Experimentation
Statistical significance
Also calledSignificance, Significant result
A result is statistically significant when it would be unlikely if there were no real difference, typically less than a 5 per cent chance.
Significance is a guard against being fooled by noise. Random variation alone produces differences between groups, so a test asks whether the difference you saw is bigger than chance would usually produce.
The usual threshold is a p-value below 0.05, often described as 95 per cent confidence. It is a convention, not a law of nature, and higher-stakes decisions can justify a stricter bar.
Common mistake
Treating significant as important. A tiny, meaningless lift can be significant with enough traffic, and a valuable one can miss significance in a small test.