BATAVIA, Ohio (AdAge.com) -- Two years after decrying growing doubts about online research quality, the Advertising Research Foundation thinks it may have a solution.
Eight marketers representing a combined half billion dollars or more of research spending have agreed to pilot a process the ARF's Online Research Quality Council has developed to ensure online research panels are reliable, sources of population samples are consistent from project to project and that respondents aren't fraudulent or duplicated. Among factors to be tracked are how long respondents are on panels and how -- and how much -- they're compensated.
The process, set to be released tomorrow at a meeting of the ARF council in New York, will be tried in separate pilots starting later this year by Unilever, Coca-Cola Co., General Motors, Kraft Foods, General Mills, Bayer, Capital One and Microsoft.A checklist
The ARF's solution amounts to a checklist of items for online research buyers and sellers to cover in developing contracts and project parameters rather than a set of minimum guidelines.
"It will be a very precise process where buyers and sellers will be trained on the exact same templates and definitions," said ARF Chief Research Officer Joel Rubinson. The ORQC based the process on what the ARF's million-dollar "Foundations of Quality" study identified as the key issues that affect research quality.
Former Procter & Gamble Co. research VP Kim Dedeker, now chairman-Americas for WPP 's Kantar Group, focused attention on the issue in a September 2006 industry roundtable when she noted P&G had found two online surveys from the same provider a week apart delivered diametrically opposed results and that flawed research was affecting the company's ability to make decisions.
ComScore Networks had also produced research showing a small group of professional respondents account for much of the sample in many surveys -- with 0.25% of the online population accounting for 32% of respondents. Further research by the Burke Institute found cheating, i.e. respondents answering the same question different ways or straight-lining the same response to a series of questions, was common across numerous online research providers.
The eight research buyers each will use the ARF's quality-enhancement process to choose a supplier for a project over the next 30 days, then report how it worked by early next year.Industry standard
Individual companies could develop similar processes separately, but it's more effective to develop one as an industry, said Stan Sthanunathan, VP-marketing strategy and insights for Coca-Cola, who chairs the ARF council behind the plan. "Quality is an industry-wide issue," he said. "If you take a myopic view and just solve this problem for my company, you're not necessarily solving the big problem we're currently having."
Imposing single-company solutions also could backfire on marketers if it doesn't solve the problems or is perceived as unfair by research suppliers, Mr. Sthanunathan said. "I should not confuse my buying power with my bullying power," he said.
A set of guidelines wouldn't work easily for a wide variety of buyers, Mr. Sthanunathan said. The ARF process won't solve all the industry's problems, he said, but it should address the biggest ones that account for the majority of potential errors.