I attended a presentation by one of the top agile metric gurus for a leading Agile tool vendor. Their tool tracks user stories, tasks, points, hours and generates charts and reports. Because over 35,000 “projects” have been captured with the tool the company did a great deal of analysis across their big data look for trends and making observations. This presentation was on “Agile Metrics” supposedly providing us insights on performance and productivity.
After observing for an hour I reached the conclusion that every metric they shared is something the lean processes would call a “vanity metric.” Direct marketers would have laughed the entire presentation out of the room. The big agile tool company did not have one single valid observation to share.
To understand why I came to this conclusion I will describe a different industry, digital direct marketing and advertising—my second profession. Imagine I had two teams (A & B) writing ads for an a Facebook based direct marketing campaign. I take all sorts of measures on how they work and how many ads they produce, capturing the following data for the first week:
Team A: 400 ads produced and launched
Team B: 200 ads produced and launched
Can I determine that team A was more productive than team B?
You do not measure productivity in ad campaigns using ads produced and launched. You measure productivity based on the business value the ads deliver—do they deliver new profitable customers or not.
But team A produced twice as many ads so they must be twice as productive, right?
We have no knowledge about how the number of ads they wrote impacted the goals of the organization. We can make no meaningful productivity statements whatsoever from this data. Any organization’s goals always include adding value to the business, and we have no evidence that producing twice as many ads produced twice as many paid customers or twice as much revenue. We have no evidence that it produced any value at all!
What if I told you team B worked in pairs and team A worked as individuals. Could we conclude that working in pairs decreases productivity?
Because we have not established a reasonable measure of productivity we cannot use this data to evaluate any of the process steps. A reasonable measure of productivity must include how we impacted the bottom line of the business. The team writing 400 ads may have written ads that anger and lose our customers—they may actually be twice as good at destroying our business. We don’t know because we did not capture the key data required to evaluate our process.
What if I told you team A worked in pairs in a common work area and team B worked alone in private offices. Could we conclude that working in pairs doubles our productivity?
I hope by now the reason is obvious.
Now imagine a company sells tools for judging the productivity of ad writing teams. Say this company gathered data not just for one week but for 500 weeks, not just for one team but for 35,000 teams. Would that make the observations more accurate?
At least I didn’t detect one other person in the audience who considered the entire Agile metrics presentation to be entirely invalid—and it was entirely invalid.
Perhaps I am lucky, I not only founded one of the worlds most renown agile software houses I also work and teach direct marketing techniques with Perry Marshall and Associates. Direct marketers cannot tolerate any BS around how their campaigns actually perform. If they do, they go out of business, fast.
The First Metric in Agile software development must be how each story delivers business value. Given we are writing software, this calculation frequently can be automated and reported in much the same manner as digital direct marketers automate and report the success of their ad campaigns—tying individual ads all the way to revenue generated.
It is possible for many user stories to automatically calculate and display the value the story brings to the business once the software is deployed. These can even be done for stories that are simply meeting a government regulation. It is essential to calculate this value in one way or another for every user story/epic if we are every going to understand real productivity.
I suggest anyone doing Agile metrics begin by taking a good course in direct advertising. It is time for agile metrics to grow up and join the adults. It is time for agile people to laugh a presentation like the one I saw out of the room.
The First Metric is essential for you to understand, however, it doesn’t stand alone. There is also The Second Metric.