A/B Testing

Overview

Which version of a design actually performs better with real users, rather than by internal opinion?

Which version of a design actually performs better with real users, rather than by internal opinion?

A/B testing (also called split testing) is a controlled experiment comparing two versions of a page or interface element that are identical except for one variable, run simultaneously on real users. The two versions (A and B) are served randomly to different segments of the live user population; behavioral outcome metrics (click-through rate, conversion rate, task completion, time on page) are measured and compared, and whichever version produces the better outcome on the target metric is the winner.

Two side-by-side cards, "Version A" and "Version B," each with an otherwise identical page showing only a "Buy Now" button in a different color — blue for A, orange for B — with a conversion-rate bar and percentage below each: 3.1% for A, 4.7% for B, the latter marked as the winner Live traffic is randomly split 50/50 between the two versions; the button color is the only thing that differs.

Variants

The technique is sometimes called virtual testing in the context of full-page redesigns — a new site design is shown to a small subset of customers while the rest continue to see the current version, letting the team compare behavior across the two groups before committing to a complete switch. This is the form credited in Web Site Development Process to Amazon.com and Google, who used it to validate site redesigns rather than launching blindly to all users. Multivariate testing (or bucket testing) extends the approach to more than two versions simultaneously, allowing multiple variables to be tested in a single experiment.

What A/B testing answers

A/B testing answers which version performs better on a specific metric — it does not answer why. It is a quantitative complement to qualitative methods. Where an informal think-aloud usability test reveals why users struggle (what confused them, what they expected instead), an A/B test measures the magnitude of a behavioral difference under real-world conditions with a statistically representative sample.

A/B testing answers which version performs better on a specific metric — it does not answer why.

The appropriate question for A/B testing is narrow and measurable: “Does version B produce more clicks on the primary action?” rather than “Is version B better overall?” Broader questions require qualitative evaluation and design judgment, not an A/B test.

Meet these requirements

A/B tests need enough traffic to reach statistical significance — a difference that’s real rather than noise. Low-traffic sites and early-stage designs rarely generate enough events to detect small differences meaningfully. Running an A/B test on a low-traffic page and reading a trend as conclusive before significance is reached is a common misuse of the method.

A/B tests need enough traffic to reach statistical significance — a difference that’s real rather than noise.

Tests should change only one variable at a time, so results are attributable to that variable. A test that simultaneously changes button color, copy, and page layout cannot tell which change drove the outcome. Multivariate tests can handle multiple variables, but require proportionally more traffic to maintain significance per variant.

Welch’s t-test is the significance test most commonly used to compare variants, since it assumes the least about the underlying data distribution; other tests (Fisher’s exact test, chi-squared, Mann–Whitney U) suit outcome data that is binomial, multinomial, or of unknown distribution respectively. Where traffic is too limited to reach significance quickly, variance-reduction techniques such as Microsoft‘s CUPED (Controlled-experiment Using Pre-Experiment Data) can shrink the sample size needed.

Segmentation can reverse the aggregate result

A single sitewide winner can hide opposite effects in different customer segments. In one illustrative case, Variant B outperforms Variant A among men while A outperforms B among women, even though A wins on the combined sample — serving each segment its own better-performing variant, instead of picking one winner for everyone, produced a materially higher blended response rate than either variant alone. This only holds if assignment is random and evenly distributed across the segmenting attribute; uneven or biased assignment confounds the segment-level comparison.

A grouped bar chart with three segments — Men, Women, Combined — each showing a Variant A bar and a Variant B bar: Men 38% vs 52% (B ahead), Women 55% vs 41% (A ahead), Combined 47% vs 44% (A ahead) Illustrative numbers matching the case above — reading only the combined bars would hide that B is actually the better choice for one whole segment.

Place it later in the design process

A/B testing fits later in the design lifecycle than informal usability testing — it is most useful once a design is live or nearly live and the goal is to optimize an already-functional flow, not to discover fundamental usability problems. Per Iterative Design, finding problems early (through expert review and informal testing) is always cheaper than measuring them after launch. A/B testing is appropriate for fine-tuning decisions that depend on behavioral scale and cannot be predicted from observation alone: headline copy, call-to-action (CTA) placement, pricing display, form field ordering.

At industry scale this runs continuously rather than as an occasional check: Google moved from its first, inconclusive A/B test in 2000 to running over 7,000 a year by 2011, and Google and Microsoft each now run more than 10,000 annually. The same infrastructure extends past marketing copy — e-commerce funnels, pricing pages, social-platform feature rollouts, and even canary-style API deployment, where a reverse proxy routes a small percentage of live traffic to a new backend version to limit exposure before a full rollout.

A/B testing at this scale still has open problems: a 2018 industry paper co-authored by researchers from Airbnb, Amazon, Google, Microsoft, Netflix, and others catalogued unresolved challenges in analysis, engineering practice, and data quality even at organizations running thousands of tests a year.

See Usability Evaluation Methods for the full treatment of formal usability studies, of which A/B testing is one specialized form.

Processes

Sources

A/B Testing (Wikipedia) is the source for the statistical-testing detail (Welch’s t-test, and the alternative tests suited to non-Gaussian outcome data, plus Microsoft’s CUPED variance-reduction technique), the segmentation-reversal worked example, and the industry-scale figures (Google’s climb from its first 2000 test to 7,000+ a year by 2011, and the 2018 multi-company challenges paper).

Usability Testing (Wikipedia) supplies the contrast this page draws between A/B testing’s quantitative, behavioral-outcome measurement and think-aloud usability testing’s qualitative, why-focused evaluation.

The Design of Sites: Ch. 5 — Processes for Developing Customer-Centered Sites is the source for placing A/B testing within the seven-phase site-development process, specifically at the Evaluation/Production and Launch/Monitor stages named in this page’s Design Stages line.

The Design of Sites: Ch. 4 — Involving Customers with Iterative Design supplies the “errors caught late cost far more to fix” argument behind running A/B testing after cheaper, earlier evaluation methods, and the three-way split (expert review, informal qualitative testing, formal quantitative studies) that frames A/B testing as the formal quantitative case.