Experimentation
P-value
The probability of seeing a difference at least as large as the one you observed if there were actually no difference between the versions.
A p-value of 0.03 means that if A and B truly performed the same, a gap this big or bigger would turn up about 3 per cent of the time by chance alone. Smaller p-values make 'no difference' harder to believe.
It does not tell you the probability that B is better, how much better it is, or whether the change is worth shipping. Those need the effect size and its confidence interval.
Common mistake
Reading p = 0.03 as 'a 97 per cent chance the variant wins'. That is the most common misreading of a p-value.