CAQA Digital / Knowing Whether It Worked

Measure the work, not the software

Usage figures describe activity, while the useful question is whether the task got better.

It is easy to count things about a system: how many people signed in, how many records were created, how many items moved through. These numbers are readily available and they answer a question nobody was really asking. High usage can mean the change is working, or it can mean people are doing more steps than before to achieve the same result. Neither interpretation is available from the number itself, and the number is comforting enough that it often ends the enquiry.

The questions worth asking are about the work. How long does a common task take now compared with before, measured from the point where the person starts to the point where it is genuinely finished, including the parts that happen outside the system. How many times does something have to come back for correction. How many exceptions require someone to be telephoned. How much duplicate handling remains. These are harder to gather and they occasionally produce answers nobody wants, which is precisely why they are worth gathering.

The uncomfortable version of this is that a baseline has to be captured before the change, and almost nobody does it. Once the old arrangement is gone, comparison depends on memory, and memory about how long things used to take is unreliable in both directions. A modest amount of measurement beforehand, even rough, turns a later argument about impressions into a conversation about evidence. It costs very little and it is nearly always skipped, because at the start of a project the ending feels a long way off.