Navigation

Case StudiesExperimentation

Appearance

Pre-test Planning

Determine how many visitors you need and how long your test should run to detect a material lift with statistical confidence.

Plan Inputs

%
%

The smallest relative lift you want to be able to detect.

Adding variants increases required sample size (Bonferroni adjustment).

Prescriptive Advice

Test runtime is long

At your current traffic, this test would take 63 days. That's enough time for the surface itself to change underneath you (other releases, seasonality, traffic mix shifts). Consider: (a) targeting a larger MDE, (b) increasing traffic allocation to the test, or (c) skipping the A/B test and using qualitative research instead.

Requirement
63 Days

To detect a 10% lift on a 5.00% baseline, you need 62,462 total visitors.

Visitors / Variant
31,231
Total Sample Size
62,462
Conversions Required
3,123
Expected Winning CR
5.50%

Pro Tip

If the required sample size is too high, consider increasing your MDE. Smaller effects are harder to see and require exponentially more traffic. Aim for a 14-day minimum duration regardless of what the math says to account for full weekly cycles.