A/B testing—creating a control group, making changes to the testing group, and comparing how well-received the two groups are—can be used with almost every e-mail element, including the copy, the list, the links and the design. Before marketers get started, though, Trivunovic suggested setting a few ground rules. “Make sure your testing sample is statistically viable,” she said. “And you should also make sure you’re giving the testing enough time so you can gather enough information. A one-time success could be a fluke.”
Trivunovic suggests running A/B tests for at least 30 days—and stretching it to 60 or 90 days if you’re only mailing once a month. Another tip: You want to change only a single thing to conduct a true A/B test. “Make sure that the control stays constant,” she said. “One of the things marketers often get wrong is not holding that control.”
In addition, Trivunovic identified one little-known fact and one big misconception related to A/B testing:
Secret: Testing your newsletter’s template can reap big rewards when it comes to conversion and click-through rates.
When testing newsletter copy, most people worry about just that—the copy. But how the copy is laid out and where elements are placed can have just as much importance as what you’re writing, Trivunovic said. “It’s something that most people don’t think about,” she said. “It’s really important to perform testing to understand how creative changes can impact a program.”
Your call-to-action button is one of the most important elements, she said. It should be visible and easy to find. The newsletter’s layout should also undergo testing. Does your newsletter get more clicks if items are arranged horizontally or vertically? How about the use of images? Do ads—house or otherwise—affect conversion rates?
Lie: A/B testing is a good tool to use from a strategic standpoint when assessing subject lines.
While testing subject lines in real time is a good idea—sending out two versions of a similar headline—you can get into trouble if you create a long-term A/B testing cycle for the variable, Trivunovic said, especially if you test the newsletter’s name.
“People might see a momentary lift in open rates, but what they will find over time is that they will see a decline because people stop recognizing your newsletter,” she said. Recipients grow accustomed to looking for a specific title, and some even create rules based on a subject line so your messages are delivered directly into their inboxes. When they don’t see something familiar, they may pass over it. A better option is to find and stick with a newsletter name and to test its headlines instead. In addition, headlines should be current and related to what’s happening in your industry or in the business world at large, Trivunovic said. “You can’t say a subject line is going to drive conversion, but it can mean the difference between getting your e-mail read or having it go into the spam folder,” she said.