Jul 18, 2022
Nobody needs metrics — what we need are better decisions
Metrics serve decisions, not the other way around.
Metrics are essential, but they are also misunderstood and misused. They are too easily treated as the goal in itself when in fact nobody needs metrics, what we need are better decisions.
How can we design for decisions — not metrics and dashboards?
1. Are we over-marketing our metrics?
“While all metrics can be gamed, this often stems from metrics being disconnected from their context or being used to describe a phenomenon that they simply cannot.” — Emily F. Gorcenski (1)
One thing I’ve learned from the best data scientist I’ve met is that they are ruthlessly precise when it comes to what is being measured and not measured. In marketing we’ve become a bit more sloppy. We want to measure something complex, but only have the time, resources and/or imagination to measure something simple. So we pretend the measure is for something it isn’t — and we market it as such to our colleagues.
2. If your car breaks down what broke?
It’s usually not the whole car that fell apart, but something somewhere inside it. If our goal is to improve things we need to know where to make improvements.
Measuring something in a complex system in the nth degree of separation from the goal will disregard every other force of influence on the outcome. We need to measure what the event can likely influence and then the chain of reactions from the event to the goal — to find what is being improved and what can be improved.
Measuring the ability of an event to have the desired goal should be done through a chain of measures, not a single proxy.
To paraphrase Cathy O Neil: Proxies are statistical correlations that make assumptions about what has a causal effect on something else (if A happens B will happen), but often we use proxies that are spurious at best — we don’t know that A has an effect on B. “A model’s blind spots reflects the judgements and priorities of it’s creators (2)
3. It’s about decisions
A discussion about metrics should not start with what can be measured. It should start with: what are the most important decisions we could make? And then figure out what combination of metrics can lead to those decisions.
An outcome (decisions) driven approach to measurement is likely to deliver a more precise and effective way for the organizartion to focus their efforts, avoid waste, see the signals from the noise and make the right decisions. While an output (metrics) approach could potentially smother your organization under the weight of unused dashboards.
Sources:
- Emily F. Gorcenski, Book Report: The Tyranny of Metrics, https://www.emilygorcenski.com/post/book-report-the-tyranny-of-metrics/
- Cathy O’Neil, Weapons of Math Destruction, https://www.penguinrandomhouse.com/books/241363/weapons-of-math-destruction-by-cathy-oneil/
