The Limits of Data
The Limits of Data
Data can only help so much, you need to know when you’re on your own.
Data driven decision making is one of the most important shifts in business of the past few decades. As business has become more competitive, data is one of the few guaranteed ways to improve your execution as a leader and efficiency as an organization. Today, not using data puts you at a competitive disadvantage.
However, I see a lot of companies use data to justify decisions already made. Misleading data is used as the universal trump card in debates and arguments, where winning is more important than being correct. Data requires context to be useful and that context can be abused to mislead.
That isn’t data driven decision making, it’s just theater.
If you really want to use data effectively, it’s important to understand where and when data is useful and the limits to using it. Data is a form of history, it describes things that have already happened in the past. As a result, data is great at:
Explaining what has happened in the past and why.
Understanding where you are today, and what your options might be.
These are easy to understand, and reflect how most companies use data today. Your company dashboards, KPIs and status reports are all using data to understand the past. Unfortunately, you cannot change the past! These applications of data are informative but the only action you can take based on them is learning lessons to apply in the future.
Data is less effective, but still pretty good, at:
Predicting what is likely to happen tomorrow.
Predicting possible futures are likely in the next few weeks & months.
The future is where data driven decision making lives, since you can change the future. If you’re considering whether to raise your prices, your data can tell you what are the likely impacts to revenue and customers. If you are considering hiring a new person, data can tell you what impact you can expect based on the performance of existing employees.
However, there is a VERY BIG catch. Data can be used to predict the future, but its ability to do so depends on how similar the future is to the past. If the future is radically different from the past, data won’t help you.
This happens a lot more than you’d think! Sometimes there are radical events like the COVID-19 pandemic that change the world and make the future very different. Most of the time, the changes are smaller such as new competitors entering your market. These changes mean that the world in which you collected data doesn’t exist anymore, and as a result it might not be helpful in understanding the future.
As a result, in a data driven decision the data is only one ingredient. Any good data driven decision has a number of critical ingredients:
Vision. Your vision is your belief in how the future will evolve over the coming weeks, months or years. It might include some big events that will occur, some shifts in the market and other things that can’t be found anywhere in the data.
Options. There are many possible paths to your vision of the future, and you need to lay out all of those paths in order to choose one. These options might be similar or different, but you need more than one or else you aren’t really making a decision.
Data. Data informs your vision, the options available and other considerations that are important. Critically, data also allows you to rank your options based on whatever criteria is important. Some options might have higher risk but higher reward, and your data helps you estimate what that risk and reward might be.
With those three pieces in hand, you can choose an option that you believe brings you closer to your vision. The data didn’t make the decision, it just informed your choice of options. That is data driven decision making, the use of data to augment your judgment and improve your decisions.
Looking to the past might help inform the options you lay out, or the options you choose, which is why using data to explain the past is important. If you tried something similar in the past and have the data to show the results, it can help you decide if a specific option is worth it. For example, if you spent $100 on ads in the past and it generated $1000 in revenue then you might be confident in spending another $100. Those learnings are true experiences and as long as the future resembles the past they are valuable.
Note that the data needs to be unbiased, you can’t just choose data that backs up the option you would prefer to choose. This is the trap that leads companies into data theater, they allow their bias to affect the data they consider when making decisions. Or, worse, you decide which option you like and then create some false alternative options with bad data to make it seem like your choice was correct.
It’s not always easy to make data driven decisions. Sometimes the data contradicts your intuition and you will struggle between the option you feel is right and the option the data shows is more likely to be successful. At the end of the day, you are responsible for the decision and you need to trust yourself and take ownership of those decisions.
But whatever you do, don’t blame the data.