The Meaning of Metrics
The way you define metrics matters more than the value of those metrics.
Every company has metrics they use to run their business. If you were to choose a specific metric and ask a random person on your team how it is calculated, would they know the answer? Would they even know where to start? If you're in most companies the answer is no.
Metrics are important, but they are also HARD. There are many different ways to calculate any given metric, and different calculations can give the metric completely different meanings. As a result, the definition of your metrics is more important than the metric itself!
For example, let’s take a metric that we might use for this newsletter: referrals. You are more likely to subscribe if your friends recommend it, so referrals are a major source of new subscribers. Referrals are also a great way to measure satisfaction since you’re unlikely to recommend something you dislike. If referrals are high then we can be confident that the newsletter is doing well and will continue to grow in the future.
Actual referrals for this newsletter, which about 20% of subscribers! Please refer more people to make this metric go up.
But, what is a referral? Is it a referral if someone follows your recommendation but then unsubscribes before reading a post? If someone clicks on a link you share, reads the article but doesn’t subscribe, are they a referral? What if they come back later and subscribe? What if a referral refers another person, does that count as two referrals? We can choose a lot of different ways to define a referral.
Even worse, we can choose lots of ways to count the number of referrals! Is the total number of referrals the total number of referred people who are subscribed right now? Or is it everyone who ever subscribed from a referral, even if they later unsubscribed? Or is it an estimate since we can’t actually track either of those very well? Each of those approaches produces a vastly different number which is useful for different reasons.
Whew, that was confusing.
Referrals is a very basic metric, so you can imagine how complex it can be to define most metrics. Many metrics that we consider “standard” can be calculated in many different ways, including basics like revenue. There is so much variation that fraud is frequent, as evidenced by the massive number of accounting scandals.
The Benefits of Metrics Education
If you want to use your metrics effectively, you need to ensure everyone at your company knows how they are calculated and what they mean. Not everyone will be responsible for calculating them, or even looking at them, but without comprehensive understanding people can make small decisions that have a big impact.
For example, let’s say one of your metrics is customer retention, measured as how many customers are still active one year after signing a contract, tracked in your Salesforce. A sales operation person, working on a small project, changes the start date for a customer contract to be the date the software implementation is completed instead of the signing date of the contract. That change, a small and technical change, completely changes the calculation of customer retention and changes one of your main metrics! The person who made it might not have any idea what they did or why it matters.
Now, you can argue that these mistakes would happen anyway and you’d be right. You need to have someone responsible for metrics calculations who can double-check all of them on a regular basis to catch problems like this one. However, no supervision can protect against the myriad ways metrics can break and as your team gets bigger the chances of mistakes increases. You need everyone understanding metrics and their importance.
Understanding is also important for making decisions! If the team doesn’t understand what a metric means, it’s hard to make decisions. For example, let’s say your main metric is revenue and everyone on your team wants to increase it. Does revenue include both subscription and professional services? For many SaaS businesses, revenue is ARR so professional services would not be included. If your team doesn’t know that they might accidentally work to add more professional services revenue, not knowing it won’t help the metric!
Okay, How Do We Handle Metric Definitions?
So, how do you educate everyone about your metric definitions?
The best way is to lead by example, by using your metrics to make decisions and showing your work. If your team sees how you use metrics (both their value and definition) they will follow your example and do the same. Seeing the metric definition matter in a decision reinforces how important it is for everyone.
Here are a few other ways to educate your team about your metrics:
Central Definitions. Have a central place where the metric definitions and their explanations are stored. This should be a place that anyone on your team, at any time, can go and read about them. There are entire categories of software that will do this for you, but personally I have found that a Notion page or Google Doc is usually sufficient.
Show & Tell. On a regular basis (quarterly?) you should walk your team through your metrics. Not just what the values are, but how you calculate them. Constant reminders are the best form of education, especially if you are constantly adding new people.
Double check your metrics. At any time, something in your metrics calculations might break. As a result, you should constantly check the historical metrics values and make sure they are the same! Everyone focuses on the current metric values, but the past is a useful check to make sure nothing changed like in some of our examples above.
Depending on the size of your team you might do a few, or all, of these. What is important is that at any time, if you pick someone on your team and ask them the definition of one of your metrics they can give you the correct answer.
Because if your team doesn’t know what your metrics mean, how valuable can they really be?
For more on metrics, see these posts:
The best kind of metrics are Second Order Metrics
There are two kinds of metrics: Success & Velocity
You Have Too Many Metrics, so here’s a way to choose fewer.
So true! The "Central Definitions" part really resonated with me. Have you seen those codified in different systems/code or are they usually just words in a doc?