Product growth · Ted Truscott · 15 October 2026
Why more experiments do not always mean more growth
How product teams can improve experimentation through better opportunity selection, credible measurement and clear decisions, rather than simply increasing A/B test volume.
Test volume is an activity measure
Running twice as many experiments is useful only if the work creates better outcomes or resolves important uncertainty. A busy programme can keep optimising a minor part of the journey while the main growth constraint goes untouched. Begin by asking which business decision each experiment could change.
Separate three questions
Is this an important customer problem? Is the proposed solution likely to help? Can we measure the effect credibly? These questions require different evidence. Customer conversations and prototypes can expose a misunderstood need. A well-designed A/B test can estimate the causal impact of a change. Neither automatically answers the other question.
Decide what would count as evidence in advance
Define the primary outcome, guardrails, unit of randomisation and meaningful effect before starting. Check whether the available traffic can support the decision. Watch for dependencies between users or accounts, delayed outcomes and changes in who enters the experiment. A precise estimate of a proxy metric can still be a poor basis for a commercial decision.
Treat failed experiments as a process question too
When experiments repeatedly fail to improve outcomes, examine how opportunities are selected, how solutions are designed and how work reaches users. Changes to the process may be more useful than testing further variants of the same idea.
Make the result change something
Before launch, agree who will decide whether to roll out, iterate or stop, and what evidence that decision needs. Afterwards, connect the result to the original opportunity and the wider roadmap. Record what remains uncertain. A statistically credible result is valuable when it changes the work, the investment decision or the team's understanding.
I work with product and leadership teams in London and the Midlands on growth diagnosis, experimentation and how teams operate.
Explore consulting and workshops →Discuss your team's challenge ↗