Two Factors that Affect the Validity of Your Test Estimation


To be a successful marketer, you must test your ideas. And you need to be confident in the validity of your test results.

In this excerpt from a live session of the University of Florida and MECLABS Institute graduate certificate program, Derrick Jackson, Director of Data Sciences, MECLABS Institute, explains that your test estimation is important because you are trying to foresee what change might happen based on past results. If something radically different happens, it might indicate a validity threat. You should take a close look at the data to determine why there is such a variation from past performance and what you expected.

When engaged in test planning and estimation, you can only be accurate if you carefully consider two factors: (1) How large your test audience is and (2) how large a change you expect between the control and treatment(s) in your test results.

 

With a bigger change, it would be easier to detect quickly, but if we feel like it's going to be a minute impact, then it would take longer to detect a smaller change.

 Derrick Jackson, Director of Data Sciences, MECLABS Institute


These two factors go together. If your expected test results only show a small change, then you should increase your sample size. But if you foresee a large difference in your test results, then you can get away with a smaller sample size because it is less likely that the difference is due to random chance.

Just don't make the mistake of oversimplifying the process by assuming a large test group ensures accuracy. You also need to consider the size of the change.

And if test estimation is difficult for you, Jackson also gives some tips for those of us who feel that we are just bad at predicting how big of a relative difference we'll get from the test.

This excerpt is part of Session 11 of the MMC 5422 Customer Research and the Fundamentals of Online Testing course, part of our graduate certificate program offered through UF. Click here to learn more about the program.

 

Related Resources

classroom image

The world’s only graduate education program focused specifically on how to increase conversion

Learn more about the MECLABS Institute and University of Florida graduate certificate: Communicating Value and Web Conversion

Learn more

  Testing Toolkit

Increase Marketing Testing ROI with These Free Tools

The free Testing Calculation & Strategy Toolkit provides fundamental tools to ensure your tests are driving the highest ROI and achieving your objectives. 

Learn More