E-mail secrets & lies: A/B testing

By Published on .

Most Popular
Are you testing your e-mail programs? If not, you're not alone. “We don't have a lot of customers doing it,” said Melanie Spilbeler, product marketing manager for e-mail and services at ExactTarget, a provider of e-mail marketing software and services. Even sophisticated e-mail marketers with plenty of experience are skipping testing. The reason: “There are some misconceptions about what [marketers] should be testing,” she said. “They might think it's more complicated than it actually is.”

Overall, the easiest way to get started with testing is the A/B test, a method of assessing a single variable to glean information about which strategy or option is better. BtoB asked Spilbeler to identify one “secret” and expose one “lie” to help marketers get started with their own A/B testing program.

Secret: A/B testing helps marketers build relevance over time. Boosting your open rate is wonderful, of course, but then what? If you get a customer to open your e-mail but he or she instantly abandons it, your testing was worth nothing. You should be keeping track of your A/B tests over time, and correlating those tests to measurable actions, such as downloading a white paper or making a purchase. In fact, as you get more comfortable with the process, you can start testing to see which copy spurs a download or which “From” line gets people to go to your website. “Always keep in mind [what] the overall goal of your campaign is,” Spilbeler said. “You can look at trends over time to see what the optimal [action] is.”

Lie: A/B best practices require an inordinate amount of thought and effort. A/B testing is not rocket science, and it's easy to put into place, Spilbeler said. In fact, there are some very simple rules that marketers can follow to make sure their A/B tests provide statistically significant results.

  • Choose one element of a campaign test. Subject lines are still among the easiest things to test that give marketers great results, Spilbeler said. “Test the length of the subject line—short versus long. And use key words,” she said. Try swapping descriptive calls to action with key industry words, for example.
  • Wait an appropriate amount of time before declaring a winner—about 24 hours is best, Spilbeler said. “This allows enough time for the subscribers to interact with the communication,” she said. If your message is time sensitive you can move a little quicker, but at minimum you'll need to wait four to six hours to make sure you have enough responses.
  • Consider test distribution. Marketers who have databases of more than 50,000 addresses can send out to 10% of their list—5% to each segment. Those marketers who have smaller audiences will need to test with 20% of their list, or 10% to each segment, Spilbeler said.
  • Keep testing. Continual testing is extremely important so you can see trends over time, Spilbeler said. “Testing is going to help you manage expectations of your e-mail marketing program,” she said.
In this article: