Quick Answer: How do you estimate A/B test sample size?
Estimate A/B test sample size from the control rate, effect you want to detect, error rate, power and allocation. Use a method that matches the outcome and the way you assign participants.
- Define one outcome and count eligible assignment units.
- Estimate the control rate using comparable historical data.
- Specify the target effect in percentage points and relative terms.
- Calculate per-group enrollment and add the outcome follow-up window.
If the required time exceeds the decision window, reconsider the test design or research method.

The useful question is whether a test can distinguish a meaningful difference before the decision must be made. A calculator can estimate the required number of assignments, but its answer depends on its assumptions and the data you enter.
Define the Unit, Baseline and Effect
Choose the unit that will remain in the same variant, such as a visitor or account. Define a binary outcome for that unit, for example one verified order within seven days after assignment. Estimate the baseline using a similar eligible group and the same window. Do not substitute sitewide traffic for the people who can enter the test.
Specify a target effect in percentage points. From 3.0% to 3.6% is a gain of 0.6 percentage points or 20% relative to the baseline. Choose a difference large enough to matter after implementation cost, and distinguish it from what the chosen test can detect.
Choose a Method and Record Its Assumptions
For a fixed-horizon test of two independent proportions, power is the probability of rejecting equality at the specified alternative if the model is correct. The significance level governs the probability of rejecting equality when it is true. Set the direction, error rate, desired power and allocation before launch.
The statsmodels documentation describes a two-proportion z-test power calculation with pooled variance under equality and nonpooled variance under the alternative. [1] Check the assumptions of your own calculator. Unequal allocations, clustered accounts, purchases with very low counts and revenue outcomes can need a different calculation. Use the stopping method the design supports.
Work Through the Enrollment Estimate
In an illustrative two-arm test, assume a 3.0% control rate, a 3.6% alternative, two-sided 5% significance, 80% power, independent visitor assignment and a 50/50 split. A standard normal approximation gives about 13,914 eligible visitors per arm, or 27,828 total. At 1,200 new eligible visitors assigned daily across both groups, enrollment takes about 23.2 days. Budget 24 whole enrollment days plus seven days for the final visitors to finish the outcome window, subject to the minimum calendar coverage in your plan.
This example plans to detect a difference from zero at a true effect of 0.6 percentage points. It does not give 80% power to establish that the true effect exceeds 0.6 points. If the business needs a lower bound above a practical threshold, design the calculation for that decision.
Decide Whether the Test Is Feasible
Recalculate under a lower and higher baseline or traffic estimate. Consider weekly demand cycles, seasonality and scheduled changes that could disrupt enrollment. Include the business threshold and any guardrail that needs enough observations in the sample plan.
If traffic or time falls short, use an approach suited to low-traffic conversion work, or ask a more specific question. Record the sample assumptions and decision date in the experiment plan before starting.
Measure Comparable Language Versions
If you will test a multilingual WordPress journey, evaluate StagLingo on a small set of pages. Confirm the same buying action and outcome definitions across the versions you plan to include.
FAQs About A/B Test Sample Size
Does a 20% lift mean 20 percentage points?
No. A 20% relative lift from 3.0% is 3.6%. The difference is 0.6 percentage points.
Can you end the test as soon as it reaches its target sample?
Follow the stopping rule and minimum calendar coverage in the plan. Wait for the last included unit’s full outcome window before a final comparison.
Why do two sample-size calculators give different answers?
Calculators can assume different effect units, allocation ratios, test statistics, power or stopping methods. Compare their inputs and method before comparing results.
Sources
[1] statsmodels. Power for two independent proportions