Be Careful What You Measure

To see what Databox can do for you, including how it helps you track and visualize your performance data in real-time, check out our home page. Click here.

One thing I’ve found to be true in both business and in life – if you want to improve performance, measure it.

This simple idea applies to any number of things. If you want to run faster, record your time running a 50 yard dash. If you want to increase sales from your website, track how many orders you receive and the size of the average order.

As notable Silicon Valley investor Paul Graham writes:

“Merely measuring something has an uncanny tendency to improve it. If you want to make your user numbers go up, put a big piece of paper on your wall and every day plot the number of users. You’ll be delighted when it goes up and disappointed when it goes down. Pretty soon you’ll start noticing what makes the number go up, and you’ll start to do more of that. Corollary: be careful what you measure.”

He’s right. When you track these results repeatedly and watch them over time, you’ll notice trends and be able to figure out the actions that made the numbers improve. Given this insight, measurement becomes a powerful tool.

But it’s equally important to note his corollary – “be careful what you measure.”

Measurement is a powerful tool, and like a chainsaw, it can be either tremendously helpful or massively destructive. Cutting down trees fast is nice, but cutting the wrong tree can flatten your house. You must be careful you’re measuring the right things from the start because measuring the wrong ones can easily lead you astray.

Tracking the Wrong Things

This is an important point because people do sometimes track the wrong things. Instead of tracking how many sales someone closes, sales managers might track how many calls a salesperson made that day. Instead of how many conversions the website makes, a marketing manager will fixate on how many people visited the website. That may simply be because those things are easy to track — a common pitfall.

When people do this they’re focused more on the process or activity than the results. Using this faulty standard, effectiveness can take a big hit – the salesperson who closes no sales but makes 50 calls a day is deemed more valuable than the salesperson who closes one sale but makes fewer phone calls. Likewise, a website with millions of visitors and no conversions could be perceived as more valuable than a website with hundreds of conversions and a fraction of daily visitors.

Another mistake in measurement is so common there is a saying for it in statistics, “Correlation does not imply causality.” That means just because there is a correlation between two variables does not mean one causes the other.

A good example of this would be to tie gym locker usage with weight loss. Yes, there is likely a correlation between gym locker usage and weight loss but it is doubtful gym locker usage causes the weight loss. The more likely suspect is spending time on the treadmill at the same gym!

Another example of this faulty thinking occurs annually at the NFL scouting combine, a weeklong showcase in which college players perform physical and mental tests before coaches, general managers, and scouts. The event is invitation-only and can affect the player’s draft status and salary. It’s a big deal.

Measurement performed at the combine include recording how fast players run a 40 yard dash and the number of times a player can bench press 225 lbs. There is nothing wrong with these things – football is a sport where speed, strength, and agility are valuable assets.

The problem is there is, again, only a weak correlation between the tasks and the results the NFL wants. Lifting a heavy bar off your chest while lying down (the measured task) is only loosely related to sacking a quarterback (a coach’s desired outcome). Because of this, some reporters have called the NFL combine “a complete waste of time.”

No one would have predicted that Tom Brady would go on to win four Super Bowls based on how fast he ran the 40 yard dash. I mean, watch the video, it’s hilarious. Brady is many things. But a gazelle he is not! But that doesn’t mean he can’t throw 50 touchdowns in a single season.

Avoiding Mistakes in Measurement

In all fairness, I can understand why people make these mistakes. Seeing a player run a 40 yard dash in 4.26 seconds or a website that gets millions of visitors every day triggers a sense of awe. That awe can cloud people’s judgment. The player who ran 40 yards in 4.26s played a season and a half in the NFL before being deemed a “failed experiment” and Yahoo, with its millions of daily visitors, is finally, mercifully, being sold off for scrap.

Don’t be awed by the stats. Instead focus on the ones that matter above all else. If you’re interested in football player performance, watch prospects play football. If you are interested in making money, measure how many sales your website generates, not just how many visitors drop by each day.

Never before has so much data been available to us from online sources, mobile devices, and wearable technology. It can be a challenge to sort through all the data to determine what really matters. So it’s always worth asking “Why are we tracking this?” and “What is this telling me?” Doing so will help you make sure your effort will drive the results you want.