Quick Answer: How do you diagnose a sample ratio mismatch in an A/B test?
A sample ratio mismatch occurs when the counts by variant depart from the planned allocation by more than the randomization would reasonably explain. Check counts at each stage before using the conversion comparison.
- Confirm the expected split and count each randomized unit once.
- Trace units through assignment, exposure and inclusion in the analysis.
- Verify the cause and determine whether the planned comparison can be recovered.
This check can flag a data problem. A passing result alone cannot establish that the experiment is valid.

Your dashboard shows a lift for the checkout variant, but that variant also has more users than expected. Before discussing the lift, find out which users each row contains, how they got there and whether the intended comparison still holds.
Check the Allocation the Test Actually Promised
A sample ratio mismatch, or SRM, is a warning that the observed split differs from the expected split beyond routine variation. Fabijan and colleagues document several causes in online experiments. [1] Compare the data to the allocation recorded in the test plan, including any approved ramps. A test designed to assign 70 percent of eligible units to one group should be checked against 70 percent. If the schedule changed mid-test, use the expected split for each applicable period or cohort.
Suppose 10,000 visitors are randomly assigned once with equal probability, and the logs show 5,250 in control and 4,750 in the variant. The expected count is 5,000 per group. The Pearson chi-square statistic is 25, with one degree of freedom and a p-value of about 0.00000057 under independent assignment. Investigate the gap. That calculation says nothing about the size or direction of a checkout effect.
Compare Counts at Each Data Stage
Count distinct randomized units in the assignment logs. Then count those recorded as seeing the experience and those included in the report. Use the same enrollment dates and identity key where possible, and separate records that have missing outcomes from records that show no conversion. A filter based on something the treatment itself changes can create an unbalanced subset even when the original assignment is sound. The SRM study recommends comparing stages and examining filters when the mismatch appears only in a subsample. [2] Apply the planned allocation to a later subset only when its inclusion rule is independent of treatment. Save the counts before adjusting any reporting rules.
Find the First Point Where the Groups Diverge
If the gap is visible at assignment, confirm the randomization unit, bucket configuration and effective rollout dates. Look for multiple assignments or a policy change that made one group eligible for a different audience. When assignment matches the plan but exposure differs, check whether one page or redirect dropped exposure events. When the gap appears at analysis, test whether a join, deduplication rule or filter removed records unevenly.
For example, a new checkout could write its exposure event only after a client-side script finishes. Compare the assignment logs with server-side order and page records. If more shoppers in the new checkout leave before that event fires, an exposure-only report may disproportionately keep shoppers who stick around.
Decide Whether the Results Can Be Used
Keep the alert in the experiment monitoring record. Do not make the rows look even by deleting users. If independent logs support rebuilding the original assigned cohort and its planned outcome records, document the correction and repeat the integrity checks before estimating the effect. If visitors received the wrong experience, restoring the counts alone will not restore the intended comparison. If the bias cannot be resolved, flag the comparison as invalid and rerun after fixing the cause. Keep that status distinct from a clean experiment with an uncertain effect estimate.
Keep Localized Pages Stable During the Test
If a WordPress test includes multiple languages, evaluate StagLingo on a few sample pages before enrollment and check that each assigned experience stays available in every included language.
FAQs About Sample Ratio Mismatch FAQs
Does a 60/40 split mean there is an SRM?
A 60/40 split can be intentional or can arise by chance, especially in a small test. Use the effective allocation and sample size to judge the gap.
Should you remove users from the larger group?
No. Removing users to match the smaller group can hide the symptom while preserving the underlying bias. Find where records diverged.
Can a test pass the SRM check and still be flawed?
Yes. Balanced counts can coexist with broken outcome logging, cross-variant exposure or measurement errors shared by both groups.
Sources
[1] Microsoft Research. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments
[2] Fabijan et al., KDD. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments