Tests of alternatives to Nielsen's TV measurement have proliferated like wildflowers after a spring rain in advance of this year’s upfronts. The idea is that tests will build confidence and eventually lead to using these currencies to actually write deals over the next year.

There’s just one problem: As marketers and agencies test, they are learning that audience numbers are all over the place among different vendors and sometimes vary significantly from week to week at the same vendors.

The wide range of results threatens to impede an industry movement toward applying multiple currencies to write TV and video deals. And it possibly plays into the hands of Nielsen, buying time for it to build its own next-generation Nielsen One cross-platform measurement system before similar “big data” competitors get a bigger foothold in a market it once monopolized.

Sign up here for our special upfronts newsletter

Disparity in the data

When comparing metrics from some of the most prominent new suppliers such as iSpot.tv, Samba TV and Comscore, among others, variances in some cases are large. 

One alternative service reported numbers for a program 10 times what Nielsen reported, said a network executive, who spoke on the condition of anonymity because he was uncertain his contracts with service providers allowed him to publicly disclose their measurements. “Number two, we were 1.5 times what Nielsen says we are,” he said. “And number three, plus or minus 20% to 40%, depending on the daypart.”

Cara Lewis, chief investment officer of Dentsu International, said that “There’s definitely a disparity in the data. “Some of it has to do with the fact that they’re all using different data sets to come to the numbers they’re using.”

Ultimately, measurement firms will overcome doubts by being transparent about how they’re gathering and modeling data, Lewis said. Even the biggest data sets of tens of millions of households still have to be adjusted to reflect a representative sample based on income, ethnicity, region and other factors—with households assigned those attributes based on third-party data sets, which also have questionable validity.

“We have seen significant variations in terms of the size of the audience for each one of the data deliveries,” said Roberto Ruiz, executive VP of research, insights and analytics for TelevisaUnivision. “It’s not to be completely unexpected, because these things need to be fine-tuned. But the needle swings are just so wild.”

Much of the issue for TelevisaUnivision, he said, is owed to its audience being disproportionately from over-the-air -only households, whose data isn’t collected from cable or satellite boxes and is only partially captured via smart TV data, meaning that modeling to estimate the audience plays a bigger role.

Even measuring the Super Bowl, which drives the biggest audience of the year, generated a fairly wide range among data vendors. Nielsen initially reported 167 million people watched the Super Bowl on NBC for at least one minute, an audience that’s more than 10% bigger than the number of people rival iSpot.tv said watched the game at any point. Then Nielsen collaborated with the NFL on a survey to estimate total viewing by capturing information about people watching the game in groups. That revised the audience to 208 million, almost 20% bigger than the original Nielsen number, and 38% larger than iSpot’s.

(An iSpot.tv spokesman noted that its 150 million number was for in-home viewing and that counting its out-of-home viewing measurement of 12.5 million viewers, its difference with Nielsen's 167 million number is only 2.7%.)

The range of Super Bowl audience measures isn’t surprising based on what Jon Watts, managing director of the Coalition for Innovative Media Measurement is seeing. CIMM is running side-by-side tests among measurement firms as it prepares its “Guide to Convergent TV Measurement Providers”, with results expected to be reported later this month. These tests have shown numbers from various providers vary by as much as a factor of 10 for the same program and time slot.

Read more: A brand's guide to Nielsen's issues 

Public comparison of their measurement, along with more transparency into their processes, will ultimately help measurement firms build confidence, Watts said. Currently, CIMM is collecting data from Nielsen, Comscore, VideoAmp, iSpot.tv, Xandr and Liveramp, and it hopes to publish side-by-side comparisons regularly in the future. Since there is no ultimate arbiter of which numbers are right, collecting and comparing different providers becomes even more important, Watt said.

“The main indication of veracity we have at the moment I think is when we see numbers from different providers converging,” Watts said, “because you assume there has to be something for them to converge around.”

Wild swings aren’t the norm for all measures and dayparts, Watts said. “In some areas of the market, you’re seeing a lot of convergence, where most of the providers converge around relatively similar numbers, plus or minus 5% either way.” Where numbers diverge the most are in areas such as ethnicity, or the presence of children in the household because the data sets behind assigning those traits to households vary considerably.

Time to retrench?

But that’s a fundamental problem, said Scott McKinley, CEO of Truthset, a firm that qualifies third-party data providers for the accuracy of their data based on matching hashed email addresses of their record against databases of known quality. 

Intuitively, the law of large numbers suggests device databases with tens of millions of households will do a better job of measuring a more fragmented media market, say, the Nielsen ratings panel. But the reality is that matching device data with household information is often difficult, McKinley said. That’s part of what’s generating divergence in data from different measurement providers, he said. And he’s concerned, particularly based on feedback from some marketers he’s worked with, that the divergence will undermine confidence in using alternative currencies. 

McKinley believes measurement firms trying to impress the market with their access to tens of millions of devices might do better to pull back to, say, 5 million households where they have the highest degree of confidence in the accuracy of the data.

TV Upfronts and Newfronts 2022 calendar

But Howard Shimmel, head of strategy for predictive analytics firm DatafuelX and former Turner Broadcasting head of research, believes the market will forgive the growing pains of big data players, and that, given the fragmentation in the media market and the likelihood that measurement firms will refine their methodologies, they should stick with the biggest data sets possible.

“The industry is about to realize that there is not a lot of consistency, alternative provider to alternative provider, and then how they relate to Nielsen,” Shimmel said. But that mainly means buyers are going to have to do away with leaning on CPMs (or the cost to reach 1,000 viewers) for Nielsen gross ratings points as its benchmark, “and basically negotiate a new unit rate and create a set of selling estimates that reflect whether you’re using VideoAmp or iSpot or Comscore.”

Buyers and sellers alike are sophisticated enough that they’ll do deals that minimize the risk of unexpected measurement outcomes, Shimmel said. 

“A year from now, we’ll be in a better place, with the alternative players having seen the data and made whatever methodological adjustments they need to,” he said. “And I think people will be smart enough to protect themselves.” 

On the sell side, he said, that might include guaranteeing only 90% or 95% of the audience in an alternative currency deal.

Related: Why TV upfront deals still use Nielsen measurement

Dug in, not scared off

“I don’t think it’s scaring anybody off,” Dentu’s Lewis said of the data disparities. “It’s really making us want to dig in more. I was just talking to a vendor and my team last week and said if we’re not comfortable with the numbers we’re getting back we need to get in behind and understand it.”

Each service provides a varying degree of transparency into its processes, she said, and those that provide more transparency are more likely to keep getting chances to do business. 

“If we can’t have a partner be transparent with us now, as we’re beginning to look at the data and potentially use it, then that would scare us away,” she said. She acknowledged there tends to be a tradeoff between how much transparency a service provides and how much they’re being paid, which is one reason the market will gravitate toward a limited number of measurement providers.

Certainly, Nielsen has no problem with an industry narrative where widely disparate numbers from alternative providers bring buyers scurrying back to the company and currency they’ve traded on for years. 

Speaking at Nielsen’s NewFronts presentation earlier this month, Ameneh Atai, general manager of digital and advanced TV for Nielsen, said: “A lot of false and inflated metrics can come in when we are measuring machines. It’s important to take machine measurement and correlate it with actual people.” She outlined the problems device sets have with bias against people of color, lower-income households and people who get their TV over the air. And she said that’s why Nielsen starts with a universe estimate and will rely heavily on its panel of more than 41,000 households to calibrate big device data sets as it rolls out its next generation Nielsen One cross-platform measurement.

But alternative providers have a different outlook, of course. “It is still very early in the ‘alternative currency’ space,” said Josh Chasin, chief measurability officer of VideoAmp. “The fact that users are beginning to scrutinize the data and ask questions is encouraging. We’ve been sharing both content and campaign data with buy side and sell side clients for well over a year. Ultimately, buyers and sellers will align around a small handful of trusted currency providers that have earned their trust, through methodological rigor and expertise, and via the kind of transparency the [Media Rating Council] process provides. I have every confidence that VideoAmp will be one of that handful.”

“The challenge with TV has always been what do you query a new service against,” said Tania Yuki, chief marketing officer of Comscore. She recalls industry events years ago where Comscore set-top box data that diverged from Nielsen raised eyebrows, but said the whole point is that the numbers may be different because they come from a much larger universe. “As much as people have complained about Nielsen over the years,” she said, “they’ve been so used to it that it’s going to feel very uncomfortable to see data that looks different.”

Sign up for Ad Age’s Media newsletter

From eyeballs to outcomes?

Another possibility, should divergent audience counts undermine credibility in alternative currencies, is shifting the focus to measuring outcomes of TV buys rather than enumerating who saw them. Certainly, that’s a case that TVSquared, which focuses much of its work on helping marketers link TV buys to sales or other outcome results, makes, and one that helped Innovid, which agreed to acquire TVSquared earlier this year, win CTV business from General Motors, said TVSquared president Jo Kinsella.

“TVSquared has 10 years of clients using our platform to optimize their media buy, and they’re comfortable that it works,” Kinsella said. “They’re comfortable that it works because they sold more.”

But Truthset’s McKinley said linking outcomes databases to household devices—such as loyalty or payment card sales data – presents some of the same problems that matching demographic data with devices does. Payment databases also can under-count cash payers, which can create bias against lower-income and minority households.

For those reasons, focusing on outcomes generally doesn’t solve the Hispanic representation issue, said TelevisaUnivision’s Ruiz. But he said the methodology of EDO—which tracks ad occurrences across the U.S. and correlates them with impact on online search activity—largely overcomes biases in other outcomes data, which is one reason the company has been using the service. In some cases, the relative simplicity of the approach can seem off-putting to buyers, he said.

EDO co-founder (and actor) Edward Norton made the case for search over big device data sets in an Ad Age Remotely appearance on May 11, noting that George Gallup proved in 1936 that “the size of your data set really doesn’t matter” when his survey sample (based on a mailing to 15,000) out-predicted a Literary Digest survey sent to 10 million readers in a presidential election.

“All signals are not the same in value,” Norton said, pointing to social sentiment measures as having little correlation with sales impact. “High-value search query is one of the most predictive data sets in the world,” he said, “indicating actual purchase intent and lining up with financial outcomes.”

He wasn’t making the case for being the only measurement firm. Putting TV measurement in a movie context, he said the industry is moving from “Highlander,” with a single ruler of the realm, to a “Justice League” of multiple players.