When asked by a client about testing creative, we advise against it for a very simple reason: The science just isn't there yet. A few years back, Ad Age ran a story about the vagaries of ad testing. They cited Volkswagen's popular 1997 "Da, da, da" Golf commercial, which some in VW management didn't want to air. The article explained how GM had tested a pair of commercials for Volkswagen of America using a custom-designed pre-testing system from Design Research International, one of the nation's leading market research firms. According to the report, the successful VW commercials flunked under GM's process.
GM might be able to afford the most sophisticated ad-testing system in the world, but even stats can't predict real-world success. Replicating the reaction of a few guys swilling beers during a Saturday afternoon ball game into a research setting remains a near-impossible feat.
When you invite someone into a research study, whether it's an online survey, a focus group or even an in-home observation, his or her behavior will change simply as a function of "being researched." They know they're being watched; they may even believe their job is one of critical analysis.
Think about how a focus group works. People are invited and paid an incentive to offer insights and opinions useful to the sponsoring marketer. The pressure to contribute something of value is immense. For a participant to admit he or she enjoys an ad at face value, or to admit it might influence a future purchase, is rare. Instead, participants tend to overstate how underwhelmed or emotionally charged the ads are, as if this serves to assist the screeners.
A desire to contribute isn't the only problem. Even an honest opinion may be difficult to communicate. According to experts on the leading edge of this stuff, people just aren't able to articulate or possess full knowledge of all of the ways advertising affects them.
There are times an agency isn't given a choice to test creative. This doesn't mean we should expect testing to yield bankable results. Anyone worth their salt in this business knows there is only one way to accurately test an ad's effectiveness: Run it.