Think about the first time you were weighed. Within a few minutes of being born, you’re plopped on a scale and your weight (the first data point) is placed on what will become a graph indicating a measure of health. That measure continues with you throughout your life as you check your weight to continue to gauge your health. Weight helps you to understand where you are on the spectrum of health as compared to all the other humans. Metrics and data points are based on empirical measures, which we use every day. Why does all this data matter? Metrics and data help us make good decisions.
Think about the last time you hopped on a scale. After working up the courage you got the data point for the measure which is the numeric value of your weight. This measurement is objective. After weighing yourself you probably considered if a change was desirable. If you decided on a change that was akin to setting a goal, either how many pounds you want to gain or lose (incremental goal) or what your goal weight is (absolute goal). Both approaches are expressions of a target.
We intuitively understand the difference between objective data, metrics, and targets in everyday life because we use them to make decisions or seek more clarity. If the number on the scale is higher than what you want it to be, you might have done some more exercise or modified your diet. If you wanted to understand whether the number is appropriate you may have calculated your Body Mass Index (BMI) to understand more clearly your ideal weight. While data points, metrics, analysis and targets help us to make sense of the many decisions we make in a day, this understanding doesn’t always carry over from personal to professional settings, especially when exploring agility.
Agile has created a kaleidoscope of measures and metrics which are always a hot topic of debate within agile coaching communities. Asking a group of agile coaches about metrics is like asking which superhero is best at a comic convention. The reality is metrics themselves are not the problem. The problem is the way that metrics are arbitrarily selected, evaluated, and used within a business setting. Countless companies use a high volume of metrics but almost always with a bias. An underlying expectation that the data points must always improve. The move toward agility isn’t immune to this bias, and as a result, metrics are often chosen on what’s expected to improve and not on the ability to measure what change is actually taking place.
Metrics are also chosen at higher organizational levels which have little, if any impact on the organization as a system. This antipattern leads to the selection of metrics that are too close to the teams or individuals doing the work. Combine that with the bias for improvement and it is no longer about decisions, but about telling stories, gaming the system, or bragging about being slightly better than your coworker. In contrast, companies that view data points as objective and empirical, instead of good or bad, begin to focus their efforts on true improvement and value generation. They aren’t afraid to see a number that isn’t going in the right direction and ask what should be done about it, if anything. They measure the messy things to get answers to hard questions. They’re transparent about calculations and ruthless in their evaluation of whether a metric serves its actual purpose.
So how do you know which approach your organization takes? It’s not always easy to recognize but a strong indicator is the level of objectivity and transparency your organization has around its data points. Consider some common antipattern examples:
- A team changes the way they estimate to hit a velocity target set by leadership without considering the value add.
- A team publishes their velocity metrics at the sprint review with more fanfare than the working software.
- A project manager tries to develop a conversion of story points to project costs.
- A Service Delivery Manager removes “exception cases” from the lead time distribution in favour of maintaining the lead time target.
- A leadership team pressures Product Owners to use a specific burndown rate to keep the release in scope.
All of these examples are unhealthy, and usually lead to an unsettled and toxic workplace. In contrast, decision-based objective usage looks like:
- A team uses capacity to plan and execute a sprint without over or under delivering.
- A Project Manager assesses the cost of decision delays and tracks blockers and uncoordinated meetings.
- A Product Owner uses the latest velocity to prioritize a backlog and select value.
- A leadership team calculates the cost of constantly switching people between teams or splitting team members amongst teams.
These examples focus on inspecting and adapting instead of existing in the realm of a well crafted guise. The aim is to make the best decision without blame, bias, or political agenda. If better decisions really are the art of management, then what’s measured must make valuable decisions possible.
The Moral: Data objectivity enables decisions while bias metrics incite politics.