Your Data is Lying To You
Never blindly trust your data.
If you liked reading this, please click the ❤️ button on this post so more people can discover it on Substack. Thanks!
Pop quiz: first thing on Monday morning you check your metrics dashboard and see a drop in revenue. What do you do?
Well, sure, you panic. That’s a natural reaction! But what about after you panic?
You could run around trying to figure out what happened, why customers stopped buying or why your product wasn’t working. That might take days or weeks. Or, you could first make sure there is not a problem with your data. If you see a big, sudden and unexpected shift in your data it’s more likely to be a data problem than not. If it’s a data problem, we don’t want to waste weeks of time chasing a ghost.
Here’s a recent chart published by the analytics company Statcounter about search engine market share worldwide:
Source: Search Engine Land
A 9% drop in worldwide market share in just one month is a big deal! Many pundits jumped on this to talk about how Google has lost to AI tools, and its future is in danger. Many people said this was the beginning of the end of Google. But what happened in April to lead to such a drop? Was there a significant event that triggered this? No, it was a data quality issue and there was no such drop. The chart was later corrected.
We’ve reached the point of analytics adoption where every company is run with metrics, and dashboards are the golden source of truth. Unfortunately, that also has led to charts and graphs being seen as facts even when they don’t always tell us the truth. The lower the analytical expertise on a team, the more likely they are to trust whatever a chart tells them. That is a recipe for many bad decisions.
The most important ingredient in data driven decision making is critical thinking. Does this chart make sense? Do you understand why it is changing? If not, you need to learn before using it to make decisions. You need to know enough to be suspicious and ask the right questions, and know where to ask those questions.
No matter how much you spend on your data systems, you have data quality problems. It’s not a question of if, but of where. Modern data is so vast and complex, that these systems cannot operate flawlessly. The hope is that your data quality issues are so small they don’t matter, or in areas you don’t care about.
While data quality issues might be a common cause of sudden, unexpected changes they aren’t always the cause! Sometimes, the data is telling you that something sudden and unexpected has happened. That’s great, because that’s exactly why we need data in the first place.
We just want to be sure we can trust it first.
For more on Analytics, see:
Pretty good post Sean. Right on point.