Experiments, incrementality and low-volume learning
Did the change help, and what should we do next?
How I use it
State the decision and hypothesis
Say who the change is for, why it should affect behaviour and which outcome matters. Name the primary measure, guardrails and smallest effect worth acting on. A change from 20% to 25% is five percentage points, or a 25% relative increase.
Choose a valid comparison
Where feasible, randomise a suitable unit into comparable groups. For collaboration products, account-level randomisation may be more appropriate than individual users. Check sample needs, traffic, duration, spillovers and implementation integrity.
Plan analysis and stopping rules before starting; ordinary fixed-horizon tests do not support repeatedly peeking and stopping at the first positive result.
Use limited traffic honestly
Interviews, usability work, prototypes, cohort comparisons and before-and-after trends can reduce uncertainty. Combine sources, examine alternative explanations and agree a threshold for the next commitment.
These methods can support a practical decision without establishing the same causal claim as a valid randomised test.
Decide, then follow through
Consider uncertainty, practical effect, cost and harm together. Statistical significance does not make a small effect valuable; an inconclusive result is not proof of no effect. Choose to ship, revise, gather more evidence or stop. Follow up on durability when it can change an important decision.
Two evidence plans for the same problem
- YesPlan a credible testCheck the unit, sample, duration, spillovers and stopping rule.
- Not yetReduce the uncertaintyUse observation, interviews, prototypes or a carefully described pilot.
Both routes can help a team choose what to do next. They support different claims about what caused the result.
A busy self-serve product might test a setup change using randomised accounts, a pre-planned sample and useful activation as the primary measure, with errors and support demand as guardrails.
A product with a few new enterprise accounts each month may instead observe setup, interview customers, pilot assistance and compare carefully described cohorts. Sales mix and seasonality remain competing explanations. The decision might be whether to fund a larger pilot, rather than claiming a precise retention uplift.
What to watch for
- Crediting a launch for all growth that happened afterwards.
- Testing a bundle of changes and claiming to know which component caused the result.
- Treating test volume, p-values or an inconclusive result as a substitute for judgement.
A brief for the next discussion.
A hypothesis, evidence plan and decision rule agreed before the result.
Know whether it worked
Where is your growth work getting stuck?
Use the growth diagnostic to find where to focus, or talk it through with me.