Definition
A/B Testing
A/B testing is a controlled experiment that randomly splits users into a control group and one or more variants, then compares a target metric to learn which version performs better. By isolating a single change and relying on randomization, it establishes a causal link between that change and the outcome rather than mere correlation.
Key takeaways
- A/B testing randomly splits users into control and variant, then compares a metric to establish causation, not just correlation.
- Define one primary metric plus guardrails in advance to avoid fishing for a result that looks good by chance.
- Set sample size, run time, and significance threshold up front; calling tests early or peeking repeatedly inflates false positives.
- It refines within the current design space well but is poor at discovering radically new designs or handling low-traffic, networked effects.
The discipline of A/B testing is what separates it from guessing. Random assignment ensures the groups differ only by the change under test, so any statistically significant difference in the metric can be attributed to that change. Defining one primary metric in advance, plus guardrail metrics that must not regress, prevents the common trap of fishing through dozens of outcomes until one looks good by chance.
Statistical rigor governs whether a result is real. A test needs enough sample size and run time to detect a meaningful effect; calling it early because the numbers 'look good' invites false positives, and peeking repeatedly inflates the error rate unless corrected. Practitioners decide significance threshold and minimum detectable effect before launching, not after seeing the data.
A/B testing's limits matter as much as its power. It optimizes within the current design space — excellent for refining a known flow, poor at discovering a radically different one. It also struggles with low-traffic surfaces, long feedback loops, and network effects that violate the assumption that one user's treatment doesn't affect another's.
Planoda's experimentation-friendly metrics let teams wire a test's target metric to real product events, so the result reflects genuine behavior rather than a proxy.
Related terms
- Conversion RateConversion rate is the percentage of people who complete a desired action out of those who had the opportunity — visitors who sign up, trials that become paid, or leads that close. Calculated as conversions divided by the eligible population, it is the fundamental efficiency measure of any funnel step, isolating how well one transition performs.
- Funnel AnalysisFunnel analysis tracks how users move through a sequence of steps toward a goal — such as visit, signup, activate, purchase — measuring the conversion rate and drop-off at each stage. By revealing exactly where the most users leak out, it directs improvement effort to the single step where fixing it yields the greatest gain.
- Cohort AnalysisCohort analysis groups users by a shared starting characteristic — most often their signup or first-purchase date — and tracks how each group behaves over time. By comparing cohorts side by side, it separates the effect of when someone joined from the effect of how long they have been around, revealing trends that blended averages hide.
- Leading IndicatorA leading indicator is a metric that predicts a future outcome — it moves before the result it foreshadows, giving teams time to act. Activation rate, trial signups, and pipeline coverage are leading indicators of revenue. Because they shift early, they are levers teams can influence now, unlike outcomes that are already settled by the time they appear.
- North Star MetricA North Star metric is the single measure that best captures the core value a product delivers to customers — and that, when it grows, reliably pulls revenue and retention up with it. It aligns an entire company on one number, cutting through competing departmental metrics so every team can see how its work moves the thing that matters most.