Marketing-mix models, since rising from relative obscurity a few decades ago, have become the de facto standard of evaluating return on investment, driving billions of dollars in marketing spending across industries.

There's just one problem: Some critics believe the models have been wrong all along, and work even worse after three decades of change in the media landscape. They say the models underestimate the impact of advertising, particularly of broad-reach network TV; overstate the value of price promotion, mislead marketers into buying thinly rated programming; wrongly downplay risks of going dark for weeks on end; and fail to account for how online search has made all advertising more effective.

These critics up to now have been lonely voices on a wonky topic in the wilderness of a fast-growing marketing-analytics industry. But they're about to get their day in court as the Council for Research Excellence preps a report on the pros and cons of marketing-mix models. That report will in turn form the basis of an Advertising Research Foundation inquiry into the quality of the models, led by CBS Chief Research Officer and ARF Chairman David Poltrack.

That could force leading companies in research to submit their models for independent review -- or explain to clients why they won't. Mr. Poltrack said during a talk at the ARF's Re:Think conference last month that he intends to present research about flaws in marketing-mix models at the ARF Audience Measurement 8.0 conference in June.

He's been an avid user of data from Nielsen and its Nielsen Catalina Solutions joint venture that attribute sales to purchase-based consumer groups and track how ads affect them. Such data wasn't available when models were first developed. Now that it is, the models haven't adapted, he said.

For years, Effective Marketing Management has been the most vocal critic against marketing-mix modeling. The firm uses panel-based studies to measure marketing programs rather than the econometric regression approach of marketing-mix models, and its partners, Mike Von Gonten and David Hoo, say models are flawed because they treat all sales lifts the same (see sidebar, below.)

Mr. Hoo blames reliance on marketing-mix models for fueling the CPG industry's decades-long shift of funds from advertising into promotion and for poorly optimized media plans, which he in turn blames for slow growth by such marketing-mix enthusiasts as Procter & Gamble Co.

"We believe it's important to bring a combination of modeling, information and expertise to decisions," a P&G spokesman said in a statement. "We have clear evidence that marketing-mix modeling, combined with other information and expertise, has helped to improve return on investment of our marketing spending and media buying."

Sunil Garga, founder of Sapient's marketing-mix analytics firm mPhasize, said much of the criticism of models is off, particularly EMM's contentions based on outdated models or research techniques that are themselves flawed. He said the ARF could help by setting standards for data sources used in models, but doesn't believe a sweeping inquiry into model quality is warranted.

"To some extent, the [EMM] critique is reasonable," said one veteran CPG research executive, who wasn't authorized to speak by his company. He's used EMM, but said its approach is more expensive and thus less widely used than marketing-mix models. Experienced analysts understand and account for shortcomings of models, he said, but it's also easy for executives to use unadjusted marketing-mix analyses to suit their ends.

Realistically, he said, the growing power of retailers, the internal power of sales forces and the need to meet quarterly numbers factor more into the CPG industry's shift from advertising to promotion than flaws in the models.

Critics: Marketing-Mix Models Get Things Wrong

They treat all sales lifts the same -- whether it's from an ad that attracted new customers who pay full price or a promotion that caused existing consumers to buy at lower prices. The former brings more long-term sales and profits.

Models estimate initial impact of advertising using a theory based on how long it takes recall to fade -- weeks -- rather than based on actual sales impact, which happens within days for frequently purchased products. All this underestimates the impact of ads, smoothes out differences between plans with higher-rated TV programming vs. weaker plans based on lower-rated shows and misleads marketers into "going dark" for extended periods, according to researcher Effective Marketing Management.

CBS Chief Research Officer David Poltrack says models don't account for search, which boosts effectiveness by allowing people to respond immediately.

And when models were developed, it wasn't possible to attribute sales to specific segments or groups of consumers. Tools from Nielsen, Catalina, Dunnhumby and TRA now make that possible, yet models haven't adapted. Sunil Garga, founder of Sapient's marketing-analytics firm mPhasize, said the criticism is off the mark. The EMM work, in particular, is based on a system best for measuring new-product introductions but not other marketing efforts, and its executives are basing some of their criticism of modeling on outdated modeling practices and the false belief that price promotion doesn't bring in new consumers, he said.

Some analytics firms, such as his own, take cross-media effects and the digital landscape into account, he said. More-sophisticated clients figure copy-testing scores and creative quality into their models and account for other limitations of models, he added. -- JACK NEFF