A/B testing is a solid way to test creative—subject line, graphic placement or call to action. But what if you want to test e-mail frequency and timing? Stream testing might be the way to go, said Jim Wehmann, senior VP-marketing at e-commerce and marketing company Digital River.
“With stream testing, you divide your list into separate "streams,' ” he said. “Everyone in a group will stay in that stream or treatment throughout the life of the test.”
One of the best uses for stream testing is in the product or service renewal process. Will a customer renew if he or she is contacted two weeks before the license runs out? How about two weeks after? How many e-mails should you send before the actual expiration date? Wehmann suggested the following steps to figure it out.
? Plan your test. Are you testing timing, frequency, or both? “Decide if you're testing the entire list or if it's just going to be people who are expiring in the next 60 days,” Wehmann said. “Understand your criteria and randomly assign those people into two or more streams.”
? Remove test subjects from your general list. If you're testing frequency, it's very important that no one else in the company contacts anyone in a stream for the duration of your test cycle. Reach out to salespeople, customer service teams and telemarketers to let them know they need to keep their hands off those in your test streams, Wehmann said. “Make sure you flag them in your CRM system, too, so someone can't accidentally contact them,” he added.
? Make sure there's enough differentiation in the streams. If you're testing duration, it's not enough of a statistical difference to send one stream e-mails that are two days apart and another stream e-mails that are three days apart. “You need to really vary the timing by at least every few days. Sometimes people don't open e-mails right away,” Wehmann said. “You need to account for that.”
? See the test through to the end. It might be tempting to stop a test after noticing that one stream is obviously doing better than the others, but don't, Wehmann said. “We'll generally learn additional information by taking a test to completion,” he said. “That's not to say that on day 45 of a 60-day test you can't scale the test to the rest of the list—the addresses that aren't in any streams. They can start benefiting right away from what you've learned.”
? Examine your results. Marketers looking for additional conversions will be checking sales metrics, of course, but it's also worth looking at other e-mail criteria to see if your tests served other purposes as well. Were people more likely to open messages that came more frequently? Did your spam complaints increase at any point?
“Obviously, when you're testing e-mail frequency, you're going to be carefully watching spam complaints and pulling the plug on any tests that are increasing both spam complains and opt-outs,” Wehmann said. You should also consider whether customers clicked through and spent time on your website or reached out via e-mail or telephone to your call center or salespeople. “You'll want to use your data to create a metric that shows sales per e-mail address to see if your messaging is really paying off.”
Originally published June 3, 2010