Quick Answer: What should an A/B test plan specify before launch?
An A/B test plan records what you will change, who enters the test, what success means and what can stop the experiment. Fill it in with the people who can measure the outcome and roll back the change.
- State the decision, eligible audience and comparison.
- Define the primary outcome, sample size and final analysis.
- Set guardrails and assign a person to act on each alert.
- Write down when to ship, hold, investigate or retest.
Sign off before the first assignment. Log exceptions so the final result can be interpreted correctly.

Copy the fields below into one shared record. Approve it before exposure starts, and keep the plan alongside the assignment and analysis records. Replace the checkout examples with thresholds and definitions appropriate to your site.
Write the Decision and Hypothesis
Decision and owner. We will decide whether to [ship, keep or revise the proposed change]. The decision owner is [name], and the analysis owner is [name]. Describe the current and proposed experience in one sentence each.
Hypothesis. Changing [one defined part of the experience] should move [outcome] because [customer behavior it should change]. State the evidence behind that expectation. Microsoft’s pre-experiment guidance covers defining the hypothesis and preparing instrumentation. [1] In an illustrative checkout test, show the full delivery cost earlier to reduce abandonments caused by an unexpected total.
Set the Design, Metrics and Analysis Timing
Population and assignment. Include [eligibility condition], exclude [predefined reasons], and randomly assign [visitor, account or other unit] to [variant split] on [first eligible event]. Keep assignment stable. Log intended assignment and actual delivery separately. Describe any planned ramp and how overlap with other tests is handled.
Primary outcome. Record [event], [denominator], [deduplication] and [follow-up window]. For the example, measure the share of assigned checkout starters who complete at least one order within seven days of assignment. Verify order tracking in both variants.
Enrollment and analysis. Record [baseline], [effect used to plan sample size], [power], [error rate], [target per group] and [minimum calendar coverage]. Use a sample size calculation for the planned analysis. Write [enrollment stop condition], [last possible outcome date] and [whether assigned visitors keep their version during follow-up]. Specify how to estimate the effect and report uncertainty, and how you will handle multiple decision metrics if any. Under fixed-horizon stopping rules, complete follow-up for the last enrollment before using the final efficacy comparison.
Give Guardrails Owners and Actions
Safety check. For each guardrail, write [metric and denominator], [alert rule], [verification method], [monitor], [rollback owner] and [roll-back action]. For the example, the on-call engineer disables the variant when a test purchase reproduces a new payment-blocking error, then logs the time and affected traffic. Microsoft describes monitoring signals during a running experiment. [2] A safety response can be necessary before a performance estimate is ready.
Integrity check. Write the expected allocation and the steps for checking sample ratio mismatch, variant delivery and missing events. If a test fails an integrity check, document the cause before using the effect estimate.
Agree on How to Handle Each Result
Decision rule. Fill in [minimum worthwhile benefit], [which estimate or confidence bound must clear it], [acceptable downside for each guardrail] and [evidence required to rule out that downside]. Ship when the planned evidence meets those rules. If a decision metric or guardrail remains uncertain, use [hold, limit rollout or collect fresh evidence] as agreed before launch. Retain the current version when the evidence rules out enough benefit or shows an unacceptable trade-off. For an inconclusive result, record the interval and decide whether another test would justify the time. If the comparison is invalid, record the reason and fix it before rerunning. Enter the final action, rationale, owner and review date in the same record.
Run a Language Check Before Enrollment
If you include multiple languages on WordPress, evaluate StagLingo on a few sample pages and record which translated experiences passed the same preflight checks as the source pages.
FAQs About A/B Test Plan FAQs
Who signs off on the test plan?
Name the person who owns the launch, the person who owns the analysis and the person with rollback access. Have them review the parts they are responsible for before enrollment.
What is the difference between a guardrail and a primary metric?
The primary metric measures the main benefit you seek. A guardrail tracks behavior or system health you cannot accept worsening beyond the agreed limit.
What if an assumption changes after launch?
Record the change, the time and affected audience. Decide whether the original design still supports the planned decision before reading the result.
Sources
[1] Microsoft Research. Patterns of Trustworthy Experimentation: Pre-Experiment Stage
[2] Microsoft Research. Patterns of Trustworthy Experimentation: During-Experiment Stage