Loading article…

Table of Contents

Quick Answer: What should you do after a valid but inconclusive A/B test?

An A/B test is inconclusive for a decision when a valid analysis leaves effects on both sides of a threshold that matters. Compare the uncertainty interval with the smallest worthwhile gain and the largest acceptable loss.

  1. Check that assignment, delivery and measurement worked.
  2. Ask whether the results rule out meaningful gain or harm.
  3. Rerun only if a fresh test can change a decision worth making.

Record invalid tests separately. Evidence of practical equivalence requires a method and margins chosen for that question.

Decision steps for inconclusive A B test results
Decision steps for inconclusive A B test results

A checkout test ends with a small estimated gain and a wide interval. One person reads that as permission to ship. Another treats it as proof that the two versions perform alike. The next step depends on the test’s integrity, how much uncertainty remains and what the business can risk.

Check Whether the Comparison Is Valid

Check that users reached the intended experiences, conversion events were captured and the analysis followed the planned stopping rule. Investigate a sample ratio mismatch before trusting the outcome. If a material error undermined the comparison and the original data cannot be recovered, report the problem and design a clean rerun if the decision still matters. An invalid estimate should not drive a decision about efficacy.

Read the Interval Against a Decision Threshold

Record the estimated effect, the interval and the thresholds in the experiment plan. The American Statistical Association cautions against reducing a decision to whether a p-value passes a cutoff. [1] An interval helps show which effect sizes the test can reasonably distinguish under its analysis assumptions.

Suppose a checkout change needs a gain of 0.5 percentage points in seven-day purchase rates to justify its upkeep. The largest acceptable loss is 0.2 points. An estimated gain of 0.2 points with a 95 percent interval from −0.4 to +0.8 points leaves both an unacceptable loss and a worthwhile gain plausible under the model. That result is inconclusive for launch. With a separate estimate of +0.1 points, an interval from −0.1 to +0.3 points would rule out the required gain at the stated confidence level and support declining the rollout on business grounds.

If you need evidence that outcomes are practically equivalent, set lower and upper bounds in advance and use an analysis designed for equivalence. Lakens explains how two one-sided tests assess those specified bounds. [2] Both tests must exclude effects at or beyond their respective bounds. Reporting an ordinary nonsignificant test is insufficient. The same principle applies when business value depends on the size of the change.

Decide Whether a Follow-Up Is Worth Running

Estimate whether a fresh test could reach useful precision for the decision. If the expected gain is small and the audience is limited, evaluate a larger change with a stronger rationale, or use research suited to low traffic to locate a better hypothesis. If uncertainty covers a costly loss and the decision matters, a fresh, powered test may be worth the wait. Avoid adding enrollment to a completed fixed-horizon test and reapplying its original cutoff.

Keep a Decision Record

Log the estimate and interval, the thresholds, and the action taken. In the wide-interval example, keeping the existing checkout can be a reasonable decision while the team examines why the variant might help. If a separate operational need requires a change, document that reason and the remaining conversion risk.

Prepare Localized Pages Before a Retest

If a follow-up test includes multilingual WordPress pages, evaluate StagLingo on a few sample pages and verify the layout, wording and assigned experience before reopening enrollment.

Explore StagLingo and trial options

FAQs About Inconclusive A/B Test FAQs

Does an inconclusive result mean there was no effect?

No. A wide interval may be compatible with meaningful gain, loss or little effect. Compare the interval with the thresholds that matter.

Should you extend the test until the result is significant?

Only extend enrollment under a procedure that accounts for the decision to continue. For a completed fixed-horizon test, plan a new study if further evidence is needed.

Can you ship a variant after an inconclusive test?

Yes, a separate reason such as maintenance can justify the change. Record the unresolved conversion risk and any rollout safeguards.

Sources

[1] American Statistical Association. The ASA’s Statement on p-Values: Context, Process, and Purpose

[2] Daniël Lakens. Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses