MECLABS > Research > Statistical Validity in A/B Testing: When to use a lower level of confidence

Statistical Validity in A/B Testing: When to use a lower level of confidence

Are you naturally a risk taker? In my world, that would be synonymous with bungee jumpers, skydivers and Russian roulette players.

We all know that in the world of marketing you have to take risks to be successful. But if you want the most results in customer conversion, you can’t rely on risky hunches or best practices. You must test those hunches. You will never be 100% sure that the treatment is better than the control. But 95/5 are awfully good odds.

MECLABS Institute’s director of data sciences, Derrick Jackson, may like skydiving, but when it comes to testing marketing ideas, he prefers to play it safe and use 95% as the validation benchmark (It’s also the industry benchmark.)

There are situations, however, when using a lower benchmark is your best option. In this excerpt from a live session of the University of Florida and MECLABS Institute graduate certificate program, Flint McGlaughlin and Jackson explain why you should reach for 95% most of the time — but not always.

“Sometimes the question is, ‘Would I rather guess or take the 80 percent risk?’ It is really the lesser of two evils.”

— Flint McGlaughlin, Managing Director and CEO, MECLABS

This excerpt is part of Session 11 of the MMC 5422 Customer Research and the Fundamentals of Online Testing course, part of our graduate certificate program offered through UF. Click here to learn more about the program.

Related Resources

classroom image

4 Keys to Creating a Compelling Value Proposition

Listen in on a step-by-step explanation of the Value Proposition Framework.

Watch Now

   Testing Toolkit

Increase Marketing Testing ROI with These Free Tools

The free Testing Calculation & Strategy Toolkit provides fundamental tools to ensure your tests are driving the highest ROI and achieving your objectives. 

Learn More