The whole subject of measuring performance of agile teams is a divisive one.
In an ideal world we should be looking to measure the value delivered by a team, rather than collecting performance statistics. After all, our primary goal in agile is to deliver some value to the product or customer. But measuring value isn’t easy, and it’s not quantifiable. It often can’t be expressed in numbers. And so often organisations measure other things.
All too often the most visible thing to count — story points — is the one thing that should never be counted!
A cautionary tale…
In my recent experience, I worked with a Release Train Engineer (RTE) — that seemed to have little previous experience in agile — who decided (against all advice) to measure and report upon story points.
They counted the number of story points committed and completed within each sprint, and in a status report compared teams working on the same project against each other. They even tried to increase productivity by getting teams to compete against each other, to see who could finish the highest number of story points in a sprint.
Obviously, it didn’t take long for the teams to pick up on this, and for them to start ‘gaming’ the system. In refinement every 5 story pointer suddenly became an 8, and every 8 a 13. And in the space of a month the number of story points being completed by all the teams combined had doubled. The amount of work being done hadn’t changed, but the count of points was higher.
And the really sad thing was, the motivation within the teams went down, but the RTE was delighted because they really thought they had managed to increase team productivity!
So what can we measure?
In my opinion looking at individual sprints is far too granular, and doesn’t give a good picture of the delivery of value. There is a natural variance of work that people complete in each sprint, as from sprint to sprint the nature of the work changes, people take holidays or are sick, or as we advance through product maturity.
To get a better idea of how we’re doing in delivering against our customer’s requirements, we need to take it up a level and only measure at the portfolio level.
Instead of measuring stories or teams, we will have a greater understanding of value by looking at features, epics, or releases. We should step away from obsessing about measuring an individual scrum team’s performance, and instead look towards the wider product team’s goals.
One useful measure I like is the epic burndown chart. It gives a good overview of our collective progress towards completing an epic over time.
An Epic Burndown chart clearly shows if we’re falling behind or likely to finish early, and also calls out if the scope of work is increasing during development — which can be useful feedback into the next epic estimation process.
We can also want look at a feature or release burndown — to view our progress towards fulfilling all the goals for the release. Or if you don’t like burn-downs, then take a look at burn-ups!
What about measuring an individual team?
I think the first question I would ask about trying to measure an individual team is “why”. Why are you looking at an individual team? What do you want to compare one team against another?
Is it because you think one team is under-performing? If so, then there’s probably bigger issues afoot than can be measured or identified in a status report.
If there are concerns about a team’s individual productivity then that’s something that their line manager will need to sort out. Because a well-formed autonomous self-organising team doesn’t tend to have productivity problems.