Experimentation
Sample size
How many users or sessions a test needs before its result can be trusted. It depends on your baseline rate and the smallest effect you care about.
Small samples are noisy: a few lucky conversions can make a bad variant look good. Calculating the sample size in advance tells you how much traffic the test needs and, from that, how long it has to run.
Three inputs drive it: the current conversion rate, the minimum detectable effect, and the confidence and power you want. Smaller effects need far more traffic to detect.
In practice
With a 3 per cent baseline conversion rate, detecting a 10 per cent relative lift needs roughly 53,000 visitors per variation, at 95 per cent confidence and 80 per cent power.