Sometimes, when discussing a product, metrics only get in the way.
A couple of weeks ago I met a fellow coach, we got to talking and discussed several aspects of our profession. At some point, he asked me a question that turned out to be difficult to answer.
More precisely, I answered immediately, but then this question haunted me for several weeks, I thought it over again and again. As a result, having thoroughly weighed everything, I came to a more reasonable answer.
So I was asked, "If there were only two metrics to track for an agile team, which would you choose?"I was immediately embarrassed by the word "metrics", because my experience with them was not very positive, to put it mildly: they are too often used as a weapon, and not as a means of improving performance. When using them, the collective is more inclined to feel like something like laboratory rats.
However, I still answered the question - I named the first one, it came to my mind. And I cannot say that I was proud of this answer. Ask me this question today, I would answer much more thoughtfully ...
So my original answer was, "It's obvious: you need to track metrics for user experience and story run time."Now I understand that I was in a hurry. I have fallen into the trap that falls into everyone who gets into metrics: In my rush to answer, I forgot something important - more important than any indicator of time to market or bottom line.
But we will talk about what passed by my mind later. For now, let's take a quick look at the metrics and the dangers of using them. And then I will tell you how I would answer this question today.
The dangers of using metrics
When working with metrics, you can quickly get off the right track - even if we start using them with good intentions. When we focus on the numbers, we often forget about the people - those who develop the product and those who use it. Working with numbers is easier and tempting.
Consider three pitfalls that you can fall into using metrics.
# 1. Excess metrics
Let's keep track of everything! Why limit yourself to one performance indicator when you can use more? The more the better, right? Of course not - when it comes to product teams.
Let's say the following metrics are tracked:
- Mastering the user by function.
- Loss of users over time.
- New users by region.
- Average speed over time.
- Cycle time.
- Time to complete the task.
- Total flow.
- Work in progress.
- The average lifetime of backlog items.
- The amount of tasks remaining in the sprint.
- The amount of tasks left until the release.
- Average time to recover from failure.
- Percentage of missed defects.
- The percentage of code coverage by unit tests.
When and on what metric should the team focus? The answer to such a question is not easy - it depends on the ever-changing context. Improvement in one metric can negatively affect another. Usually it all comes down to which metric is the most worried about management at the moment.
"The notion that if something cannot be measured, then it cannot be controlled is a myth that is costly."The more metrics, the more effort and people it takes to collect relevant data. At the same time, we do not even try to assess whether it is worth tracking each specific indicator. Despite all this, we are trying to get the most out of data tracking.
- William Edwards Deming
It often goes so far as to assign dedicated teams to monitor, analyze and report on the data. Unfortunately, these analytic teams are often isolated from both the actual product developer and the end users. The interpretation of data becomes akin to laboratory analysis of samples and samples: there is no human contact, there is no context, no understanding of the situation.
Therefore, it is better to reduce the number of indicators, despite the fact that intuition tells us otherwise.
# 2. Comparison between teams
As soon as you start comparing teams on a metric, it immediately becomes useless - a competition on that metric begins, as a result of which the focus shifts to it in an unhealthy way.
According to Goodheart's Principle , when you compare teams for a target, they adjust to perform better on that metric — and begin to find workarounds for that. And the original meaning of tracking a metric disappears.
"Tell me how you will evaluate me, and I will tell you how I will behave."If we think we are being watched, we begin to behave differently - this is called the Hawthorne effect .
- Eli Goldratt
Take a metric that is often overused in this way: speed. For the team, this metric is the sum of the story points that the team has brought to the "complete" state in the sprint. Speed is just a planning tool for teams, nothing more. Typically, the average speed is calculated for at least the last three sprints, and based on this value, it is predicted how many and what stories can be completed in future sprints.
Unfortunately, those who are interested in the productivity of the team use the speed metric in a different way: often "at the top" it is used to compare the effectiveness of teams. As a result, a great forecasting tool turns into a metric tied to productivity, and unhealthy competition begins between teams.
As a result, the speed metric begins to be manipulated in such a way that it becomes meaningless. The most common manipulations are:
- The story is not divided into smaller ones - because the big stories have more points.
- The difficulty of the story is overestimated to get more points.
- The team sacrifices quality in an effort to quickly translate the story into a sprint.
- The team reduces the transparency of work and tries to find as few problems as possible - in order to "pass" the story as soon as possible.
This behavior results in reduced team productivity. In addition, the original meaning of the “speed” indicator as a planning tool is lost.
And this can happen with any metric, if it becomes an external goal.
# 3. Excessive emphasis on objective data
From a young age in schools we are taught to be objective. We know that bias is unacceptable in decision making, and this approach is welcome in the business world. As a result, we find ourselves obsessed with objective quantitative data - and often ignore or underestimate qualitative data.
If the team does not have a sense of psychological security, transparency in the processes decreases, which leads to less confidence in the completeness of quality data, and they try to use it less.
"If there is fear, the numbers will lie."Gradually delving into numbers, graphs, charts and other performance indicators, we communicate with the team less and less. And this does not in any way increase the sense of security of the team - it only further reduces it.
- William Edwards Deming
I remember once the company where I worked as a coach decided to automate tracking the team's productivity. They analyzed logs from Jira, data from the continuous delivery pipeline, and data from support. Much effort has been made to automatically track this data without human intervention.
And all this was done because the management did not trust the information received from the employees. They tried to eliminate the human factor, turned to numbers and ended up communicating less with teams. But the main problem of psychological security was not solved - its level decreased even more, as the management began to consult less with employees.
The only metric you need
Now you probably think that I don't like data. Let me be clear: I don’t mean to say that data is bad; data is required. In making a decision without facts - nowhere.
"Without data, your words are just another opinion."But these three issues - overwhelming metrics, cross-team comparisons, and a passion for objectivity - prevent data from being used effectively. So what should you do?
- William Edwards Deming
This is the question I've been pondering over the past few weeks. And I came to the conclusion that the answer lies in the concept of team involvement.
Team engagement metric
The best indicator of product success is team engagement. It is assessed by the level of autonomy, professional skill and team focus.
The more you are involved, the more fun you get. Teams with high levels of engagement determine for themselves what internal organization is needed to achieve the goal. Its employees improve their professional skills, learn to deliver and manage the product more efficiently, they have a goal to which they strive, which leads them forward and keeps them in the team.
The rest of the metrics are still valuable, but they must be strictly internal: the autonomous team will track the metrics according to their context of use. By raising the professional level, the team adapts itself and guides the product so as to achieve better results. In pursuit of the goal, the team will track the desired metrics depending on the context.
Does this mean that you need to launch a team engagement tracking system, or, for example, start work on a dashboard for a team satisfaction indicator? No. If we do this, we will return to the three problems described above. And then what to do?
Communicate with the team.
Get out of your office and talk where the work is going on: go to the developers' office, go to sprint reviews, and participate in the backlog discussion. Take a close look at where they need help. Ask them if they feel involved in their work.
It's not just that we communicate in words, and not in numbers and graphs: communication is a much more important factor than any metrics and graphs. You can learn a lot from a conversation, it makes it easier to understand. Metrics and charts can help with conversation, but they will not replace it.
Let teams track and respond to metrics on their own. The involved team, with autonomy, skill and commitment, will take care of the value itself and know which metrics are best to use.
Go to teams and regularly communicate with them, support if you need help - and you will see how they begin to feel more secure, and at the same time, engagement will grow. And no metrics are needed.
So my final answer is
This article began with a conversation I had with another coach who asked me what two metrics I would track.
Now, having thought it over carefully, I would give a more balanced answer:
“Let the teams decide themselves. We need to visit them more often and communicate - so that they feel participation and support. "
The Agile methodology is focused on keeping the team working together and improving. The main thing here is not numbers, charts or graphs.
It's time to focus on increasing team engagement: fostering autonomy, commitment, and the development of professional excellence. Including the team needs to give work with metrics. Then you can start to create a working environment and provide the support you need to work.
Special thanks to Maciej Jaros and Maarten Dalmijn for their contributions to this article.
About the translator
The article was translated by Alconost.
Alconost localizes games , apps and websitesin 70 languages. Native native translators, linguistic testing, cloud platform with API, continuous localization, 24/7 project managers, any string resource formats.
We also make advertising and training videos - for websites, selling, image, advertising, training, teasers, explainers, trailers for Google Play and the App Store.
→ More