Finding the Right Balance: Control Metrics in Product Analytics
This article first appeared in Medium
With over a decade of experience in product, I’ve come to appreciate analytics as a powerful tool for gaining insights. One of the most enlightening aspects of analytics is the concept of a counter or “control” metric. When focusing on driving a specific KPI or North Star Metric, including a control metric can be crucial. It ensures that a particular product or feature is not only doing good but is also doing no harm. In effect, you are creating a more comprehensive view that helps avoid unintended consequences caused by your product or feature release.
North Star metrics, One Metric that Matters, KPIs, and other versions of what I’ll refer to as “primary metrics” are all the rage right now — and rightly so. There are tons of case studies of companies using these to help get laser-focused (i.e., prioritized) roadmaps and drive meaningful results for their customers and their businesses. However, too much of a good thing can… well… be a bad thing. Like that 2nd (or if you’re me — that 3rd!) piece of homemade apple pie after a relatively “robust” family dinner, you eventually cross a threshold and risk doing more harm than good before you realize it (i.e. before you feel full).
Many of these “metrics that matter” are usually leading indicators — a potent tool for product analytics. Leading indicators can also be called input metrics and are powerful tools because they act as a predictor of a critical lagging indicator (sometimes called an output metric). Output metrics are frequently tied to revenue and a company’s bottom line.
Here’s a simple example: A leading indicator for an e-commerce site might be average items added to the cart. The lagging indicator would be something like total Average Order Value (AOV). In this case, the input metric of average items added to the cart should be a solid predictor of the output metric — AOV. If you can create a UX that increases the number of items added to the cart, you will likely increase AOV.
Focusing on leading indicators is good because you want to learn — and adjust — as early as possible from the insight the leading indicator provides. You want to have confidence that you are on the right track for the lagging indicator, which will indicate your product’s or business’s health. If all we did was focus on the lagging indicator, we would have increased our risk that we were not on the right path and potentially wasted a lot of time not responding to the data. It’s the proverbial “being caught flat-footed,” and we will react too late by the time we see what’s happening with the output metric.
Semi-fictional Case Study: E-commerce Platform
Here is a semi-fictional use case that illustrates that optimizing for one metric can sometimes have unintended consequences on another, negatively impacting other essential aspects of the business or product.
An e-commerce platform focused on increasing the average number of items in a cart (leading indicator), hypothesizing that this would drive order value (AOV — a lagging indicator). They introduced features that would nudge customers to add more items to their carts throughout their shopping journey.
The actual outcome was that while AOV did increase, the checkout abandonment rate also spiked. Customers were deterred by the constant prompts to add more items, leading to frustration and an increased rate of abandoned carts. The cart abandonment rate increase also hurt the company’s bottom line. The problem was several weeks before the company course-corrected because the product team was not as focused on the abandoned cart rate. If the company had also monitored the checkout abandonment rates as a counter metric, it could have proactively balanced its “add-more-items” UX strategy to boost AOV without negatively impacting the customer experience.
Putting Counter Metrics to Work
So, how do you actually go about implementing counter metrics? There is no one right way to do this. However, here’s a basic, high-level approach I’ve taken that may help you.
1. Get comfortable playing the devil’s advocate: For each of your primary KPIs, ask yourself and your team, “What could go wrong if we push this too far?”
2. Based on the answer to the above question, you likely have a few obvious candidates to track as a control metric. Choose the metrics that would sound the alarm — i.e., prompt the executives to message you on Slack — if those worst-case scenarios were to unfold.
3. Set your thresholds: Decide acceptable ranges for your counter metrics beforehand. This could be that the control metric should stay flat, or there could be a slight increase/decrease in the metric that is an acceptable tradeoff. Your decision doesn’t need to be perfect. You just need to have given this some thought ahead of time when you can rationally think through the consequences and not put yourself in a position where you are forced to react emotionally in the moment.
4. Make the counter metrics visible: Place the control metric next to your North Star metric and other KPIs in your analytics dashboards. You want to be transparent and accountable to the targets for the primary and counter metrics you are establishing.
5. Stay flexible: As your product evolves, so should your metrics. If the situation warrants it, don’t be afraid to revise your approach and what you’re tracking as your product matures.
One word of caution: don’t go overboard. I’ve seen teams go down the rabbit hole of endless metrics and come back never the same person. In other words, don’t get so bogged down in metrics that you forget to take some chances and build, learn, and iterate. Pick a few metrics that matter — both primary and counter metrics — and keep your focus on those.
Keeping it Balanced
A balanced measurement strategy isn’t just about your product team avoiding disasters and embarrassment (though that’s a nice perk). It’s about building products that deliver short-term and long-term value to your customers and your business. You’ll see more sustainable growth, happier users, and fewer “oh $%@!” moments in the middle of a business scorecard meeting with your executive leadership team.
Here’s my challenge to you: take a hard look at your current metrics. Are there blind spots you’re not accounting for? Are there unintended consequences that you are causing? What counter metrics could help round out your view?
Remember, excellent product management isn’t just about hitting targets — it’s about building something that makes a difference. The real lesson here is that you want to measure both the good that you’re doing AND the harm you may unintentionally be causing to have a holistic and balanced view of your product’s performance.
