Concerns around the accuracy of live sports ratings are heating up, and heading into the Super Bowl this weekend, the discrepancies between Nielsen data and other measurement providers are being watched with an especially close eye. 

Measurement of live sports is critically important to media companies, with sports making up the bulk of the most watched broadcasts last year. In 2022, 64 of the 100 most watched telecasts and 49 of the 50 highest-rated shows among people ages 18-49 were sports, according to Nielsen. The prospect of a Writer’s Guild of America TV and film writers strike could make live sports an even bigger piece of the mix later this year. 

Earlier this month Disney Advertising, which operates ESPN among other properties, helped facilitate an unusual three-hour, closed-door and Nielsen-hosted meeting with key players from across industry segments, according to people familiar with the matter. The meeting included executives from Disney rivals NBCUniversal, Paramount, Fox, Amazon, Google and Apple, plus the NFL, NBA and NHL sports leagues, and agencies including Omnicom Media Group, Publicis Media and WPP’s GroupM. The topic of conversation: live sports measurement.  

The meeting followed audience data out of the NFL conference championship games in January that sheds light on the audience disparities in Nielsen and iSpot.tv (one of the alternative measurement providers being tested by networks), particularly when it comes to out-of-home viewership.  

There have also been questions around Nielsen’s measurement of “Thursday Night Football” on Amazon Prime. Amazon’s data often showed a larger audience than Nielsen figures. Details of Amazon’s methodology for its own data suggest the difference lies between what Amazon’s servers and Nielsen’s panel show to be the number of viewing households. And they raise new long-festering questions from networks about whether Nielsen panels routinely undercount viewing households.

Industry sports talk

Issues discussed at the meeting earlier this month included ensuring consistency and transparency across media companies, and maximizing scale of measurement coverage and reliability, according to one meeting participant, who discussed the meeting on a condition of anonymity.  

Nielsen One, the measurement giant’s new cross-platform offering, was also a topic of discussion. There was a desire by some in attendance to receive two years of historical data prior to using Nielsen One as currency for dealmaking in 2024, an issue network executives have raised previously, this person said. 

Nielsen agreed to provide more details in follow-up meetings on how it will handle co-viewing and out-of-home measurement—which is of particular concern with sports programming that’s often watched in bars and restaurants—according to executives attending the meeting, who spoke on the condition of anonymity. 

“We are focused on helping clients understand impact metrics before any methodology change and continue to communicate that we will deliver a full year of [historical] data to ensure a smooth transition for the industry ahead of the 2024 upfronts,” a Nielsen spokeswoman said in an e-mail. “With the launch of Nielsen One Ads this year, we have begun the evolution of our products aimed at giving customers a consistent and comparable view of measurement data.”

Disney was the first of the TV media companies to try the alpha version of Nielsen One in 2021. And while it has struck deals with alternative measurement providers like SambaTV, it has been less aggressive than some other networks in the push to adopt alternative currencies. Disney also has yet to join in other media industry measurement and data collaborations, including OpenAP, and the recently formed Joint Industry Council. The meeting did get Disney, Nielsen, the big tech platforms, and others that have yet to be involved with the JIC, deeper into an industry conversation about the future of sports measurement.

Nielsen One is expected to be part of that future, Geoffrey Calabrese, global chief investment officer, Omnicom Media Group, said after an appearance with other JIC members at NBCU’s One23 developer conference, which took place earlier this week. While Calabrese did not attend the meeting, he said having meetings where Nielsen is engaged in the conversation is welcome.  

“There’s been a lot of Nielsen bashing” at other industry events where Nielsen has less of a voice, he said.

Measurement divergence

Not surprisingly, that bashing comes largely from networks, and a look at how recent NFL playoff games were measured by Nielsen and rival iSpot.tv helps illustrate why. It’s common for Nielsen and alternative measurements to differ, and the audience numbers are almost always higher from rivals such as iSpot and VideoAmp than from Nielsen, said another media agency executive. While that might make agencies more reluctant to trade on non-Nielsen numbers, it also raises questions about why.

It would seem likely that for events with massive audiences, such as the Super Bowl and NFL conference championship games, differences in audience data would not be meaningful. But there can be substantial divergence even in these big audiences. 

Nielsen reported a total audience of 53.1 million for the evening AFC Championship game on Jan. 29 on CBS, and 47.5 million for the afternoon NFC Championship game on Fox. For the evening AFC game, iSpot reported 53.3 million average viewers, and for the NFC game a somewhat higher 53.9 million.

The difference in the AFC game was negligible, less than 0.4%. The difference in the NFC game was much bigger—13.5%. 

A big part of the difference may be in measuring out-of-home viewers. Though a Nielsen spokesman declined to provide a breakdown of out-of-home viewing in its championship game numbers, iSpot did. It reported 9 million OOH viewers for the Fox game and 5.8 million for the AFC game. A lower Nielsen number on OOH viewers for Fox could explain some—though not likely all—the difference between the two in the afternoon game.

Network executives and their trade group the Video Advertising Bureau have complained about accuracy of Nielsen OOH viewership numbers for years. A disclosure late in 2021 that Nielsen had been under-counting OOH viewership since September 2020 only fueled the complaints.

“We stand by our numbers,” a Nielsen spokesman said in an email, declining to get into details on the out-of-home component.

Nielsen gathers out-of-home estimates from panelists who wear Portable People Meters to help report their viewing. ISpot gathers its out-of-home numbers from a panel made up of people who use Tinuity, an app the company acquired last year that lets users get sound on their phones from telecasts they watch out of home.

It’s unclear if any company actually measures out-of-home audiences well, said an executive at a measurement company, adding that panelists may not always push buttons needed to activate PPMs, and there are questions about how representative a panel of Tinuity app users can be. But it’s also a problem that’s hard and expensive to solve fully, since not all cable providers have data from commercial venues, and licensing rules by streamers may prevent automated collection of data even if it were otherwise possible.

Beyond that is the question of how much trouble it’s worth to measure people watching in bars or restaurants anyway. They’re unlikely to hear the commercials and may not even be paying attention to the screens. Advertisers will only pay a discounted rate for out-of-home audiences, for those reasons, even if they’re counted fully, said Calabrese.

Amazon-Nielsen differences

The persistent and bigger differences between measurement by Nielsen’s and Amazon’s self-reported audience numbers for “Thursday Night Football” this season, raise other questions.  

Nielsen measured the average audience for 11 games this season at 9.7 million, compared to 11.5 million reported by Amazon using its own data—an 18.6% difference.

What makes the difference more striking, is that Amazon was using Nielsen for much of its own calculation. According to a spokeswoman for Amazon Prime, its own numbers are based on what its servers show to be the number of member households tuned in to “TNF.” Amazon relied on Nielsen for counting such things as out-of-home audiences and over-the-air broadcast viewership in the home markets of the teams playing. That means the basic difference is between the households Amazon’s servers say were watching and what Nielsen’s household panel indicated was watching.

While advertisers have for years expressed wariness about media companies “grading their own homework,” one agency executive said given the choice between the Nielsen panel and Amazon servers, That said, Amazon Prime has been using Nielsen as currency, so that’s what it’s getting paid on. 

People familiar with the matter said Amazon has been unhappy with Nielsen, with which it signed a three-year contract this year, and has been in talks about adding other measurement and currency providers. Amazon didn’t respond to a request for comment on that.

“Our agreement with Amazon represents a first-of-its-kind deal where a streaming service is included in, and has a live program measured as part of Nielsen’s National TV measurement service,” said the Nielsen spokeswoman. “Nielsen’s methodologies provide the industry with independent audience measurement data that is comparable, comprehensive and consistent across platforms and distribution channels.”

~ ~ ~
CLARIFICATION: A previous version of the story characterized the meeting as hosted by Disney, which instead facilitated the meeting hosted by Nielsen.