When is an A/B test worthwhile?

Which decision should the test support?

An A/B test compares variants under controlled conditions. Eligible visitors are randomly assigned, then the predefined outcome is compared between groups. Start with a concrete decision: should a different offer presentation become permanent? “We want to test something” is not yet a useful question.

The hypothesis names the problem, change and expected relationship. A clearer explanation of an introductory call might reduce uncertainty before booking. This is a testable assumption, not a promised effect. Also define potential downsides to monitor, such as poorer enquiry quality.

What data is needed?

First verify assignment and event collection. A purchase must have the same definition in both variants. Consent differences, device switching or delivery errors can complicate interpretation. An unexpected departure from the planned group ratio calls for investigation. Microsoft Research describes this check as Sample Ratio Mismatch.

Plan effect size and duration.

There is no universal minimum visitor count. Baseline rate, smallest commercially relevant effect, desired power and error risk determine data needs. Available eligible visitors per day affect how long collection is likely to take. Seasonal changes and long decision journeys also matter.

Interpretation example: moving from 2% to 2.2% is 0.2 percentage points, or a 10% relative change. Do not confuse the two. This example establishes neither the required traffic nor the likelihood of that improvement. Sample size is planned using an appropriate statistical method.

Which false conclusions should you avoid?

Do not simply stop at the first encouraging interim result. Analysis and stopping rules must fit the chosen method. Searching many metrics or subgroups after the fact increases the risk of chance findings. Document the primary metric, limitations and deviations before declaring a winner.

An inconclusive finding does not automatically mean the variants are equivalent; data may be insufficient. Conversely, a statistically detectable difference may not matter commercially. Consider uncertainty, effect size, implementation costs and quality criteria together.

What if traffic is too low?

Use suitable qualitative methods to examine usability, technical faults and comprehension. Observed problems can justify improvements even when their revenue effect cannot be isolated. A broken form needs repair, not a competition against a working version.

Before starting, record the decision, reliable measurement, feasible data collection and who will implement the outcome. If one of these foundations is missing, preparatory analysis is often the more useful investment.

Sources and further documentation

Microsoft Research: Diagnosing Sample Ratio Mismatch in A/B Testing

Microsoft Research: Patterns of Trustworthy Experimentation – Pre-Experiment Stage

Explore the topic

Conversion optimisation

Tracking & analytics

What is conversion optimisation?