Online research is more than a decade old, and its relative speed and lack of expense has made it a growing force, but its spotty validity increasingly has become an industry embarrassment.
Diametrically opposed results
Last September, for example, at a roundtable hosted by the Institute for International Research, Kim Dedeker, VP-consumer and market knowledge for Procter & Gamble Co., said two online surveys from the same provider conducted a week apart delivered diametrically opposed results. P&G is the world's biggest buyer of survey research, spending $200 million annually with 600 vendors. (Some at that conference criticized marketers for inviting the decline in research quality as their purchasing departments squeezed costs out of contracts.)
ComScore Networks, meanwhile, produced research last year showing that a relatively tiny number of professional respondents determine the results of online studies. ComScore's research, conducted as part of an effort to sell a panel of nonprofessional online respondents, showed that 0.25% of the online population accounts for 32% of responses in other online surveys, while less than 5% account for more than half of the responses.
"Reports of the failure of online studies to replicate when repeated are becoming more common," the ARF said in a statement. "The influence on results of the 'heavy online survey responder' is worrisome, but has not yet been studied in a disciplined and objective manner."
Online Research Quality Council
The newly formed Online Research Quality Council will hold its first meeting Sept. 10 and aim to establish a set of industry standards for evaluation by early April, when the ARF holds its annual convention in New York.
A steering committee of research-industry executives, including a mix from companies using online and offline surveys and panels, will head the council and take turns chairing it for six-month periods, starting with Robert Tomei, exec VP-global director of access panels for TNS.
Other members of the steering committee will include Efrain Ribiero, chief operating officer, Ipsos Interactive Services; Steve Coffey, chief research officer, NPD Group; Josh Chasin, chief research officer, ComScore; Jonathan Jephcott, exec VP, Aegis Group's Synovate; and Renee Smith, VP-panel quality, Harris Interactive.
Marketer advisory board
The council also will work closely with an advisory board of marketers, but the ARF named only one in its statement, Coca-Cola Co.'s Stan Sthananathan, VP-knowledge and insights. "The rigor and discipline that characterized sampling plans of the 1970s and '80s is missing in the online era," Mr. Sthananathan said in the ARF statement.
"The 'good, fast, cheap ... pick any two' syndrome has jeopardized the client credibility of online access panels," ARF President-CEO Robert Barocci said in the statement. "This trend, combined with the rise of online communities and blogs, has increased the rate of individuals conducting research online without applying statistical methodologies or research techniques essential to quality."