Statistical significance
Conversion and UX
Conversion and UX
Statistical significance indicates how likely it is that the difference between two variants isn't chance. In marketing we usually work at 95% confidence.
Significance says something about the existence of a difference, not its size or value. A significant 0.1% gain on a low-traffic page is statistically true and commercially irrelevant.
Calculate up front how many visitors and conversions you need for the difference you want to detect. Otherwise you'll test for weeks on a result that never firms up.