A/B Testing (Wikipedia)
Wikipedia’s dedicated article on A/B testing: a user-experience research method that compares two (or more) variants of a page or feature through a randomized experiment and a statistical hypothesis test.
License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/
Key points
- Definition and terminology. Also called bucket testing, split-run testing, or split testing; a randomized experiment comparing variant A against variant B (or more, in multivariate testing) against a single defined, measurable outcome — sales conversions, click-through rate, registration rate.
- Worked example. A two-version email sent to 2,000 customers, differing only in call-to-action wording, produced a 5% response rate for one variant and 3% for the other — illustrating that a valid test isolates one variable and measures a defined outcome.
- Statistical testing. Welch’s t-test is the most commonly used significance test since it assumes the least about the underlying data distribution; other tests (Fisher’s exact test, chi-squared, Mann–Whitney U) apply depending on whether the outcome data is Gaussian, binomial, multinomial, or of unknown distribution. Microsoft’s CUPED technique reduces the sample size needed to reach significance by controlling for pre-experiment data.
- Segmentation can reverse the aggregate result. A worked example shows Variant B outperforming among men and Variant A outperforming among women, even though A wins overall — routing each segment to its better-performing variant instead of picking one sitewide winner produced a 30% higher blended response rate than either variant alone would. This requires random, even assignment across the key attributes being segmented on, to avoid confounding.
- History. Predecessors trace to an 1835 randomized double-blind trial and Claude Hopkins’s coupon-tracked ad campaigns (described in his 1923 book Scientific Advertising), predating the statistical-significance concept. Google ran its first A/B test in 2000 (search-results-per-page, inconclusive due to a technical glitch) and by 2011 was running over 7,000 tests a year; Microsoft Bing’s 2012 test of ad-headline display raised revenue 12% within hours with no UX cost. Google and Microsoft each now run over 10,000 A/B tests annually.
- Applications beyond marketing copy. Social platforms (LinkedIn, Facebook, Instagram) test engagement and network effects; e-commerce funnels test copy, layout, and imagery for conversion; pricing tests find the revenue-maximizing price point; the 2007 Obama presidential campaign tested four button variants and six accompanying images for newsletter signup. A/B-testing infrastructure is also used for canary-style API rollouts, where an HTTP layer-7 reverse proxy routes a small percentage of traffic to a new backend version to limit exposure to bugs before a full rollout.
- Known challenges. Requires large sample sizes to reach statistical significance; a December 2018 paper co-authored by researchers from Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, and Stanford University catalogued open challenges in analysis, engineering/culture, deviations from the traditional test design, and data quality.