Quick Answer: How do you run an A/B test you can act on?
An A/B test randomly assigns eligible units to an existing experience and a changed experience. Define the outcome and decision before launch, then verify that the experiment delivered and measured both versions as intended.
- State the proposed change, audience and expected effect.
- Choose the assignment unit and keep each unit in its original group.
- Set the sample, follow-up window, stopping rule and guardrails.
- Check data validity and interpret the size and uncertainty of the result.
Ship only when the result meets the agreed rule and the guardrails are acceptable.

Suppose a store wants to show an earlier shipping-cost estimate. An A/B test can compare shoppers assigned to the current page with shoppers assigned to the changed page at the same time. The test is useful when the business can say which behavior should change and what evidence would support rollout.
Write a Hypothesis and a Decision Rule
Start with a specific observation and proposed mechanism. In an illustrative store, shoppers ask about shipping before entering checkout. The hypothesis is that showing the expected delivery charge next to the buying action will increase verified purchases within seven days of assignment among visitors eligible for the offer. Use support contacts about price surprises and order contribution as guardrails.
Record who is eligible, the metric denominator, what counts as an outcome and what improvement makes the change worth shipping. Estimate the required sample size from a plausible baseline and meaningful effect. Use a test plan to record what you will do for a clear improvement, harm or a result that cannot settle the choice.
Assign the Right Unit at Random
Choose what stays in one version, such as a visitor, account or store. The authors of Trustworthy Online Controlled Experiments describe persistent assignment and choosing a randomization unit suited to the design. [1] For a visitor test, keep repeat visits in the same version where possible. For shared-account settings, assign at the account level when visitors could see or influence each other’s treatment.
Run both versions concurrently so changes in demand or traffic can affect them both. Analyze using the assigned unit and original group. Include all visitors eligible under the same preplanned rule, even if they were not exposed. Stop enrollment on schedule and let the final visitor complete the full seven-day outcome window. If you change several page elements together, you can learn about the combined experience. Use the comparison of A/B and multivariate tests when the goal is to distinguish individual elements or their interactions.
Verify the Experiment Before Interpreting It
Check variant rendering and log assignment and exposure separately. Microsoft Research’s experience with online experiments emphasizes validating telemetry and using guardrails beyond a change’s direct measure. [2] Confirm that outcomes are captured for both groups and investigate any unexpected allocation mismatch. Use the stopping rule you selected before results were visible.
Make the Decision With Effects and Guardrails
Compare the estimated effect and its uncertainty with the improvement required to justify the change. Check that decision against guardrails and data quality. In the shipping example, the store might require a purchase-rate gain of at least 0.3 percentage points, tolerate no more than one extra shipping-related contact per 100 assigned visitors and accept no more than a $0.05 drop in contribution per assigned visitor. One conservative rule would require the relevant uncertainty bounds to clear all three thresholds before full rollout. Set the rule and how the bounds are calculated before starting. If the interval includes both meaningful harm and a worthwhile gain, the result is inconclusive for the business decision. Identify the uncertainty that another test could resolve or keep the current design if the follow-up would cost more than the decision is worth.
Check Language Versions Before an Experiment
If you are testing copy on a multilingual WordPress site, evaluate StagLingo on a small comparable set of pages. Check that both versions offer the same terms in each language before running the test.
FAQs About A/B Testing
Can an A/B test compare several design changes at once?
Yes. Treat the changes as one combined experience. A two-variant test of that package does not isolate the contribution of each element.
Can you compare last week’s page with this week’s page?
That comparison can reveal a pattern, but changes in customer mix, promotions or timing can explain it. Concurrent random assignment gives a stronger basis for evaluating the change.
What if a test does not give a clear answer?
Check validity, then compare the plausible effects with the business threshold. Use a new test when a focused revision or a different design can reduce the uncertainty.
Sources
[1] Ron Kohavi, Diane Tang and Ya Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
[2] Microsoft Research. A/B Testing Infrastructure Changes at Microsoft ExP