Measure a small improvement honestly

Define an outcome, keep a comparable baseline, preserve the denominator, and leave room for uncertainty when a change looks promising.

A small change can feel better before you know whether it helped. Clearer instructions, a shorter form, or a tidier workflow may deserve a trial. A useful measurement connects that trial to a specific outcome and preserves enough context to interpret what happens.

Imagine changing the instructions at a fictional book-swap table. No observations or results are supplied here. The example shows how to prepare an honest comparison.

Choose the outcome before the number

Write the improvement in ordinary language: visitors can complete the exchange with less help. Then decide what observation would count. For this example, completion could mean placing a book in the intended area and correctly explaining the exchange rule.

Choose one main outcome and one possible cost to watch. Fewer questions could be encouraging, but it would matter if visitors misunderstood the rule or gave up. GOV.UK’s performance guidance recommends combining measurements with research so that the numbers help explain the experience.

Sources: Using performance data to improve your service: an introduction

Define the measurement boundary

Specify when an attempt starts, when it ends, and what counts as completion. Decide how to record assistance, abandoned attempts, and interruptions. Write these definitions before looking at results, and apply them consistently.

For a completion rate, retain both the completed attempts and all eligible started attempts. GOV.UK calculates completion using completed transactions divided by started transactions, including unsuccessful ones in the denominator. Your project needs definitions suited to its own task, but the principle transfers: keep the population behind the percentage visible.

Sources: Measuring completion rate

Record the starting point in comparable conditions

Observe the existing arrangement using the same task and recording method you intend to use after the change. Keep brief notes about circumstances that could affect behavior, such as where the table stands and whether a volunteer explains the rules.

Define the observation window in advance so you are less tempted to select a particularly flattering period afterward. Keep incomplete and inconvenient observations in the record. A baseline is useful because it gives the later measurement context; it should not be reconstructed from your most memorable day.

Sources: Using performance data to improve your service: an introduction

Give the comparison a fair chance

Change the part you intend to investigate and record anything else that changes. A new sign introduced alongside a different layout and extra help makes the result harder to interpret. Returning visitors may also have learned the task from an earlier attempt.

The Magenta Book’s evaluation guidance explains why causal claims need a credible account of what would have happened without the intervention. A simple before-and-after comparison leaves other explanations open. For a small reversible project, that may still inform the next test. A larger commitment warrants stronger evaluation.

Sources: Test and Learn

Keep the raw counts and the inconvenient notes

Show the number of attempts beside any rate, and keep the observation conditions with them. A rate based on a small group can shift sharply when one more attempt is added. Preserve distinctions between completion, assistance, and an attempt you could not observe fully.

Look at the notes before naming a cause. Fewer requests for help could reflect clearer instructions, a different group of visitors, or someone quietly providing explanations. An observation can guide a useful question even when it cannot settle that question.

Sources: Measuring completion rate, Using performance data to improve your service: an introduction

Choose an action that fits the evidence

Write a conclusion with three parts: what you observed, the conditions covered, and what remains uncertain. Describe the recorded difference between observation periods, then explain which causes remain plausible. If the comparison cannot distinguish between those explanations, leave the causal question open.

Decide whether to retain the change provisionally, revise it, or gather more comparable observations. Proportional evaluation means matching the strength of the evidence to the consequences of the decision.

In the Small Steps Planner, record the next observation you need and the action it could change. Careful measurement makes progress easier to build on because the record remains useful when the next question arrives.