1 Comment

Be Careful with Metrics


There are a lot of pitfalls with metrics, they can seem like a magic bullet to control the output of a team. Measurement is certainly useful but how you use it is critical to ensuring your projects succeed.

Let’s take an example. I tell my teams that I’m measuring their success on defects raised. For the testers their success is the more defects they find the better, for the developers the less defects found the better. I have now introduced an extrinsic reward associated with a metric. It is highly likely that the developers don’t want defects in their system and the testers were perfectly happy when they didn’t find any but that’s about to change….

Now comes the unintended consequences. Testers start to raise defects for almost anything such as a text box not quite large enough (in their opinion) etc. This annoys the developers who could just fix the defect really quickly and don’t need it raised, slowing the team down. The developers figure that they’ll not get the software in front of the testers quickly, instead they’ll sit on it for days trying things out, increasing the cycle time. Then the developers then figure that writing less software is the best way for less defects to be found and slow down. The testers and developers and now effectively at war and will fight about whether something is a change or a defect.

This as we can see can be disastrous. So how do we fix this issue. Firstly let’s trust the team to do the best they can. Secondly let’s use the metrics to help them improve and don’t reward them externally. For example we can give the team the metrics and ask them to work out where they can improve. This gives them the information they need without attaching a reward to it, people usually want to do a good job. If you have someone monitoring metrics in this way consider moving them into the team so that they are part of the delivery. They are now as accountable for quality as the rest of the team and inform them that they are there to help.

Did you already mess this up? Here’s a way of understanding that, let’s evaluate our current metrics by asking our teams:

– What was the metric originally meant to be for?

– What is it being used for now?

One comment on “Be Careful with Metrics

  1. “Expect what you inspect” – Deming

    Indeed it’s important to realise that creating an incentive on a process output rather than focusing on the process and the inputs is just attempting to solve the symptom of the problem.

    Metrics like defect count, code coverage and so on are better used as a ‘mineshaft canary’ for problem areas rather than an explicit measure of quality, as they can be gamed when people are aware that they’re being measured on it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: