Quick Answer: When can you stop an A/B test without invalidating the analysis?
A stopping rule specifies which data can decide a test and when. Repeatedly applying an ordinary fixed-horizon significance threshold and stopping at the first positive result can increase false positives.
| Decision | Fixed horizon | Sequential design |
|---|---|---|
| When to evaluate efficacy | At the agreed final analysis | At looks allowed by the method |
| How to handle interim data | Check system health and safety | Use the method’s planned boundaries or valid intervals |
| What ends enrollment | The planned sample and schedule | A supported stop boundary or the agreed cap |
Keep operational safety and the decision about effectiveness separate.

A dashboard turns green after two days, then the apparent gain shrinks after the weekend. The fact that a result appears early can be useful for monitoring. Acting on that result requires a method that accounts for the way the team decides to stop.
Separate Monitoring From Declaring a Winner
Looking at a number does not by itself change the test. The problem comes when a favorable estimate changes the sample size, stopping time or reported analysis. Johari, Pekelis and Walsh describe this problem for continuously monitored ordinary p-values and intervals and develop sequential alternatives. [1] Decide what the dashboard can and cannot authorize before launching.
Continue checking data capture, crashes, payment errors and assignment balance. Microsoft Research recommends monitoring trustworthy metrics during experiments so teams can intervene when needed. [2] Treat a verified safety problem as a reason to protect users, independently of whether you can claim the variant won or lost.
Choose a Stopping Rule Before Launch
For a fixed-horizon test, write down the target enrollment, minimum calendar coverage and outcome window. Specify when a test is ready for final analysis. If the method controls error only for the planned final analysis, avoid declaring efficacy from repeated interim comparisons.
For group sequential testing, choose planned looks and boundaries that control error across those looks. Some continuously valid methods support more flexible monitoring under their assumptions. Verify your tool’s analysis, the metrics it supports and any limits on stopping. Scheduling several ordinary comparisons does not create a group sequential test.
Let the Outcome Window Finish
In an illustrative fixed-horizon test, enroll shoppers for a planned 21 days and measure purchases within seven days of assignment. If enrollment ends on day 21, allow the last assignments to complete their seven-day window before final analysis. Where continued exposure is part of the design, preserve each enrolled shopper’s assigned experience through that window. The final analysis would be around day 28. An interim view on day 21 may contain incomplete purchase outcomes for recent visitors.
Write the distinct points for enrollment, follow-up and final analysis into the experiment plan. Monitor gaps, but avoid changing the sample or outcome window because a result almost reached the cutoff.
Report Any Early Stop and Its Consequences
If a checkout variant stops accepting valid orders, disable it under your safety rules. Log the evidence, time and affected traffic. Record whether the study was halted for harm, a data fault or a valid statistical stopping boundary. An unplanned interruption may invalidate the planned efficacy comparison even when the operational decision is clear.
If the interruption invalidates the comparison, report which assumption or rule failed. Reserve inconclusive for a valid analysis whose results leave the decision unresolved. A corrected experiment needs a clean enrollment plan and a verified fix.
Keep Localized Experiences Stable During the Test
If your WordPress experiment spans languages, evaluate StagLingo on a small set before launch. Verify the copy and offer, then record any language changes that occur while the experiment runs.
FAQs About A/B Test Stopping Rules
Can you look at an A/B dashboard during a fixed test?
Yes. Monitor exposure, quality and safety. Avoid adapting a fixed-horizon efficacy analysis to favorable or unfavorable interim results unless the method supports that decision.
Does sequential testing mean you can stop any test whenever you like?
No. Use the stopping procedure, analysis and assumptions specified by the sequential design or testing tool.
What if a serious checkout issue appears during the experiment?
Protect users, preserve the record and explain why you stopped. Resume or restart only after confirming the fix and deciding how the interrupted evidence will be treated.
Sources
[1] Ramesh Johari, Leo Pekelis and David J. Walsh. Always Valid Inference: Bringing Sequential Analysis to A/B Testing
[2] Microsoft Research. Patterns of Trustworthy Experimentation: During-Experiment Stage