Loading article…

Table of Contents

Quick Answer: When should a website use an A/B test or a multivariate test?

Use an A/B test when you need to choose between two experiences. Use a factorial multivariate test when you need to separate the effects of several changes and learn whether they depend on one another.

Design questionA/B testFull factorial multivariate test
What do you assign?Two complete experiencesEach planned combination of settings
What can you estimate?The difference between those experiencesEffects of individual factors and their interactions
What drives traffic needs?Target effect and outcome variabilityTarget contrasts, outcome variability and allocation across combinations

With either design, define the effect that would change the decision before estimating how long the test will take.

Choosing A B or multivariate testing for a website
Choosing A B or multivariate testing for a website

A team wants to change a pricing-page headline and the main product image. If the shipping choice is between today’s page and a finished replacement, there are two experiences to compare. If the team needs to know whether the headline works with either image, that question requires a different design.

Start With the Decision the Test Needs to Support

A randomized A/B test can compare the current page with a bundled redesign. Its result tells you about the combined change in that population. For a decision about the headline alone, keep the picture fixed and randomize the headline. The finding then applies to the tested picture and surrounding experience.

A full factorial multivariate experiment assigns each combination of the settings. For two choices of headline and two choices of image, that produces four cells. NIST’s design handbook describes this full cross of factors. [1] Run the four combinations concurrently and keep each visitor in their assigned combination. Check that each resulting page makes sense to the customer and that measurement works in every cell.

Use Four Combinations to Estimate an Interaction

In a hypothetical trial, the outcome is purchase within seven days of first assignment. With the current photo, the rate is 3.0 percent for the current headline and 3.2 percent for the new one, a 0.2-point difference. With the new photo, the rates are 3.1 and 4.1 percent, a 1.0-point difference. The difference between those effects is 0.8 percentage points on the purchase-rate scale. That difference is the interaction estimate. A four-cell estimate still needs an uncertainty interval before it guides an action.

A comparison between today’s page and the page with both changes would leave out the two mixed combinations. You could learn which bundle performed better, while the separate effects would remain unresolved. NIST illustrates how factorial models estimate main effects and interaction terms. [2] Specify the scale on which you will estimate the interaction.

Calculate Traffic for the Effects That Matter

Write down which contrast would change your decision. In a balanced factorial test, an average headline effect uses both cells with the new headline and both cells with the current headline. Under equal weighting, the example’s average headline effect is 0.6 percentage points. An interaction compares differences and can be much harder to estimate precisely. Four cells do not by themselves imply four times the total traffic.

Estimate sample size for the primary contrast, using the baseline outcome, smallest effect worth detecting and supported analysis method. If you plan to declare success on any of several effects or pairwise comparisons, specify how the analysis will control false positives across those decisions. Compare precision with the business threshold as well.

Choose a Design the Team Can Run Cleanly

Use the experiment plan to specify assignment, exposure, tracking, acceptable risks and how to interpret a meaningful interaction. If the interaction question requires more traffic than you can collect in a useful window, narrow the decision to a bundle comparison or one factor under a fixed condition. Record which questions that narrower test will leave open.

Check Every Language Included in the Test

If the experiment includes localized WordPress pages, evaluate StagLingo on a few sample pages and check the layout and language of each experience before enrollment.

Explore StagLingo and trial options

FAQs About A/B and Multivariate Testing FAQs

Can an A/B test change two elements at once?

Yes. Assign one group the current experience and another group the package of changes. The estimate applies to the package.

Can independent A/B tests run at the same time?

Yes, where the experiment platform and analysis handle overlap correctly. If the changes affect one another and that interaction matters, record each unit’s joint assignment and analyze the four combinations together under a plan designed for the interaction.

Is four times the traffic always required for four combinations?

No. Required traffic depends on the planned effect, variability, allocation and analysis. More cells can make some effects harder to estimate.

Sources

[1] NIST and SEMATECH. Full Factorial Designs

[2] NIST and SEMATECH. Estimate Main and Interaction Effects