Event Agenda

Session 1

Introduction

Testing is the only way to know for sure that your marketing collateral will have an impact on the bottom line. In the introduction to our course on the fundamentals of online testing, you'll learn why testing is important and the most effective method for testing your pages.

Here are a few things you'll take away from this session:

• How the MECLABS team failed in designing an effective treatment for a test and the (counterintuitive) treatment that finally generated a 78% increase in conversion
• The MECLABS testing heuristic and how it helps to bring a repeatable methodology to your testing efforts
• A downloadable tool that will help you bring clarity and discipline to your testing efforts so you can be sure your tests are valid
• How the rest of the course will help you achieve better test results

The Research Question

A good test begins with a research question. This session is about how to craft an effective research question and why it's important.

In this session, you'll learn:

• How the right research question led to a small company generating an extra $500,000 in revenue per year
• What a research question is and how to craft one
• Real world examples of how not to craft a research question and common pitfalls you can avoid
• How we were able to predict their behavior and increase leads by 548% by properly researching the customer
• The six steps to selecting the best research question for your first test

Session 2

Treatment Design

Once you have your research question, you need to develop the treatment(s) to test it. This session covers the essentials of treatment design in useful tests.

In this session you’ll learn:

• The strategy MECLABS uses to find the most effective balance between a lesson and a result in a test
• The four essential definitions you need to know to design effective treatments
• How a basic testing cycle works so you know what to expect when you are testing a page
• The difference between rule-based and science-based optimization
• How to use the conversion heuristic to get "in the zone" when testing your treatments
• Four quick, real-world case studies where the treatments generated significant lifts for the marketers who tested them

Metrics

Before you can get a true win in your marketing efforts, you need to know what it actually means to "win," and you won’t know that unless you’re effectively using your metrics program to track and measure your tests.

In this session, you’ll quickly learn everything you need to know to properly measure your tests:

• How one company thought that a treatment was ineffective when it had actually won by 60%
• The four philosophical categories you can use to cut through all of the confusion regarding metrics
• How to quickly determine what you should be measuring in a given test
• How to properly use secondary metrics to add insight to your test
• How a losing treatment in a test actually yielded the most customer insight

Session 3

In a recent marketing survey, 40% of respondents claimed they did not consider even the most basic validity factor (sample size) in their tests. After the next two sessions, you’ll be able to know for sure that the results you see in your tests are accurate and reliable.

Validity — Part 1

In Part 1 of this session, you’ll learn:

• How one company examined the data in a (seemingly) invalid test and found that their treatment had actually won by 31%
• The four most common validity threats that must be addressed in every test you run and how you can examine your data for clues they exist
• How an unrelated event in the outside world rendered one of our seemingly successful tests invalid
• How a testing tool backfired and ruined one of our valid-looking tests

Validity — Part 2

In Part 2 of the Validity session, we’ll cover the most common validity threat in online experimentation in-depth: sample distortion effect.

You’ll learn:

• How a treatment at a major newspaper almost validated at a significant level of confidence, but suddenly dropped and became uncertain
• The statistical level of confidence you need to make sure your treatment is actually better than the control
• The three factors that make up the statistical certainty of a test
• The four basic steps you need to establish sample-size validity
• When you should end a test

Session 4

Test Interpretation

It is not enough to run a valid test; you must also understand why a particular treatment won over another. To do this, you need to be able to effectively interpret your results.

In this session on test interpretation, you’ll learn:

• How a cycle of test interpretation led to significant lifts across an entire website for one mailing list provider
• The three levels of test analysis you must use to effectively interpret test results
• How even an inconclusive test (when interpreted properly) can provide significant insight into the mind of the customer
• How to use the testing cycle to your advantage when you are constrained by company regulations or budget