Tips from the 2012 X Change Testing Huddle

This remains one of my favorite huddles at Semphonic's X Change conference and last year was no exception. Here is a summary of the most interesting points.

1. Losses are Good. 
This is how learnings happen. A common ratio is 1 test out of 10 will result in an actual change, though Dell's rate is higher at 1 in 3 which may reflect their ability to follow the guidelines below. Otherwise the test change is not significant. I have heard from Tim Ash (who was not at this huddle but has written a great book on Landing Page Optimization) that making bigger changes can help drive this ratio down.

2. How do you Decide what to Test?
In most cases teams upstream decide, but consider giving the following guidance. What is the level of revenue impact? What is the level of learning possible (is it an isolated variable)? What is the level of effort to run the test? How deep is the customer insight (will it cross-correlate to other areas)? And finally, the golden rule of testing: Don't test what you cannot implement. 

3. Have a Slate of Tests 
This is a previous learning from Dylan's huddle back in 2010. Always have a slate of tests in the cue. This can be helpful in declaring closure because "you need the traffic for the next test" or if you need to discard a test (see #6 below). The new insight is to hold out spots for re-tests. Ideally, every test would be conclusive and unquestioned, but this is simply not reality.

4. Testing Wiki
Once tests get going it's easy to forget exactly what was tested and the outcome. To avoid repetitive tests, document the results. This includes failed tests (where the change was not significant) as well as successful tests. Include elements such as pathing, navigation, page function, customer segment, targeting, banners. Also list the top 10 most impactful at the top and the latest learnings.


5. A/B then MVT
Start with A/B testing then move to Multivariate. Multivariate testing in Test and Target is not good because it uses the Taguchi method which is to say it's not truly random by hour, it's random by day. Test & Target also has difficulty with mass segmentation. Consider running A A B tests initially to control for stability.

6. Throw out the Test
When the data is between SiteCatalyst (web tracking tool) and Test & Target (testing tool) is off by more than 15% throw out the test. Identify data issues like this within the first 3 days. Tests often need 10 days of stability before you call it (95% confidence). Dell runs tests for about 21 days. Intuit runs for 14 days. It may take two weeks to get a test into the launch cue so identify data problems early.

7. Initial Test Analysis
Run a quick initial analysis within 4 days and a deep dive analysis within 2 weeks once the test completes. Front load the analysis by asking "What do you need to make a decision?" Do not roll out preliminary numbers before the test is complete.

Think you might like to attend a huddle? Then come to X Change! Contact me for a discount code.