9 companies and agencies doing multicultural measurement right

Accurately and adequately representing multicultural audiences in measurement has been an industry challenge for years, and new technology hasn’t necessarily helped. Big databases built on cable and satellite TV boxes and smart TVs have innate built-in biases against minority audiences and lower-income households that often can’t afford them and disproportionately still watch TV from over-the-air signals or their phones. Abundant third-party consumer data commonly mis-identifies people ethnically. And artificial intelligence—developed disproportionately by white male technologists and analysts—wrestles with its own bias issues. Even the U.S. Census Bureau, whose data forms the standard by which fair representation of ethnic minorities and other population segments is measured, has acknowledged it seriously undercounted ethnic minority populations in the 2020 census. But there are media and measurement companies, analytics firms, industry organizations and marketers that stand out for doing things right—or at least better—by developing and applying fixes. Here’s a look at what some of these companies are doing and how their approaches can be applied more broadly in the industry. To be certain, this is not an exhaustive list, but rather an overview of some of the more prominent efforts to make progress. Alliance for Inclusive and Multicultural Marketing (AIMM) This arm of the Association of National Advertisers—founded in 2016 and backed by more than 80 marketers, agencies, media and research companies—has done more than just monitor multicultural inequities in marketing. It’s done something about them. Among other things, AIMM has been tracking how well third-party data providers, who provide the data backbone of ensuring accurate representation of ethnic and other minority segments in targeting and media measurement, actually report ethnicity. The basic answer is, badly. As of the fourth quarter of 2021, third-party databases as measured by Truthset, which vets them against other databases of known quality, were accurate only 83% of the time when identifying someone as Hispanic, and 77% of the time when identifying someone as Black or white. Accuracy has improved some since AIMM started tracking it in the third quarter of 2020, but still has a way to go. The study found about 80% of the difference between above-average and below-average performing ads for Hispanics was driven by the cultural relevance and persuasion effect, according to CIIM’s white paper. For Black and Asian-American consumers, the CIIM score effect accounted for only about half the difference, and AIMM is looking into the reasons behind that difference in future work. It's a big deal because data quality is the foundation of measuring multicultural marketing, said Carlos Santiago, president of consultancy Santiago Solutions Group and co-founder of AIMM. “If we cannot see multicultural segments with accuracy, then obviously the relevant messages infused with culture are going to fall short of the very careful creative process they were intended to create.” AIMM in 2020 launched the Cultural Insights Impact Measure. It’s a survey-based tool that grades ads based on such factors as representation and perceived (by consumers) cultural relevance. CIIM now has been used by more than 500 brands and agencies to evaluate their work. Big marketers including Procter & Gamble Co., Johnson & Johnson, L’Oreal, Walmart, Visa, Hyundai and T-Mobile have been among those working with AIMM and CIIM. CIIM recently released results of an independent evaluation by Sequent Partners of how its cultural relevance scores affect sales based on marketing mix and attribution models. The study found cultural relevance and the persuasion effect it powers account for 67% of the variation in ad sales effectiveness in a study covering 34 cases across six brands in five categories involving four multicultural segments. Truthset Truthset is a business that scores databases based on how accurately they report population segments, including gender, age, income and ethnicity. This works by data providers submitting their data for review and Truthset scoring them against databases of known quality (they haven’t said exactly which), matching consumer records using anonymized hashed email addresses. Participating providers receive “Truthscores” and feedback on how to improve accuracy. Data providers can share their “Truthscores” with clients or the market if they wish. Not all do, and for good reason, since Truthset has found some databases on some measures are little better than flipping a coin to guess someone’s identity. Increasingly, Truthset is becoming a sort of public utility—albeit very much a private company—for measuring data quality across a variety of factors, particularly ethnicity. Marketers and agencies pay Truthset for helping them find the most reliable databases to target marketing. They can also use it to score the accuracy of their own first-party data, including through a project recently launched by the Advertising Research Foundation, letting members get a sample of their first-party databases scores. Media company TelevisaUnivision (below) also recently used Truthset to help it develop a data graph that was rated as being more accurate than third-party databases in finding and accurately identifying Hispanic people in the U.S. And AIMM uses Truthset for its ongoing monitoring of third-party data quality. Not every third-party database, however, has submitted its data for evaluation by Truthset. For example, Experian, which provides data support for Comscore and VideoAmp, and possibly others in the industry, does not, according to a spokesperson for Experian. “We are not a Truthset participant,” the spokesperson said in an email. “However, we do perform extensive testing on all of our data sources and capabilities.” Experian’s accuracy on ethnicity is particularly important because it provides data support for Comscore and VideoAmp (and possibly others in the future) in eliminating multicultural bias in their numbers. Big data sets from set-top cable and satellite boxes or smart TVs are biased against ethnic minority households that are less likely to have the devices and more likely to watch TV using over-the-air signals or streaming via mobile carriers. TelevisaUnivision Media companies focused on Black and Hispanic audiences have complained for decades about being undercounted by Nielsen panel ratings and under-served by other measurement providers. TelevisaUnivision took the rare step this year of developing a product to do something about it. The media company released its Hispanic data graph during the upfronts in May. It worked with Truthset to develop and validate a database that exceeds those of third-party providers both in terms of the percentage of Hispanic households it includes and the accuracy by which it identifies them. Truthset’s analysis for AIMM late last year of 15 third-party databases found on average they covered only 59% of the Hispanic market and accurately identified people as Hispanic only 83% of the time. By comparison, TelevisaUnivision’s data graph covered 85% of U.S. Hispanics with an accuracy rate of 86%. That combines for a visibility score (from multiplying those two numbers) of 73%. It’s far from perfect, but it’s way better than the 49% average visibility score for other databases evaluated by Truthset. Under-representation of Hispanics in other databases is “why we took a really big stance a year ago to say that we need to create this household graph,” said Donna Speciale, president of sales and marketing for TelevisaUnivision. The graph will let agencies and advertisers better engage with TelevisaUnivision’s audience, which should help build business, said Dan Aversano, senior VP of data, analytics and advanced advertising for TelevisaUnivision. The company will also create partnerships with other media companies to help them better reach Hispanic audiences in their inventory portfolios, he said. VideoAmp Certainly every measurement provider tries to accurately represent demographic diversity. But the challenges are real. Nielsen, despite efforts to maintain a representative panel within its 41,000-plus TV ratings households, has missed the mark at times. Undercounting of Black and Hispanic households during the pandemic played a key role in problems that led to the suspension of its Media Rating Council accreditation last year. “VideoAmp takes diversity, equity and inclusion seriously in how we live and how we measure,” said Chief Measurability Officer Josh Chasin. Getting ethnic measurement, particularly Hispanic measurement, right is particularly hard, because they’re over-represented in over-the-air-only households that don’t have set-top box data, he said, and tend to have lower match-rates with third-party data sets. To overcome those issues, VideoAmp has looked to ensure strong representation of over-the-air households using smart TV data and ensure it gives enough weight to Spanish-speaking households. One way it does this is by focusing not just on representative numbers of households but on how many people are in the households, Chasin said. And the company’s board of advisors includes Roberto Ruiz, executive VP of research, insights and analytics for TelevisaUnivision. The effort has paid off. VideoAmp won backing in June from TelevisaUnivision as a national currency alternative to Nielsen. Aversano said VideoAmp stood out from other alternative providers. "We were really pleased just how leaned in they are,” he said. “They acknowledged the challenges, and they were extremely willing to roll up their sleeves, invest and put resources against ensuring that their data really is best in class in terms of representation and inclusivity.” Procter & Gamble Co. While many marketers have talked about the need for better multicultural representation in research and measurement, P&G appears to have gone further than other big marketers in doing something about it. The company has strived to go beyond measuring representation of ethnic minorities in ads toward measuring relevance of those ads to various groups and resonance of the marketing based on the context of their media placement. A key example brought up repeatedly by Chief Brand Officer Marc Pritchard is Old Spice. Its advertising with prominent pitchmen Isaiah Mustafa and Terry Crews last decade certainly represented Black men. But it didn’t actually resonate with them until a more recent makeover by the brand and its agency Wieden+Kennedy, Pritchard said. P&G has been an early user of CIIM, but also has its own proprietary artificial intelligence tools for gauging how Black, Hispanic and other multicultural audiences, are responding to ads and other content. Though P&G usually keeps its analytics and measurement strategies relatively close to the vest, it published its “4Rs Framework” (for reach, representation, relevance and resonance) developed in an effort led by Lela Coffey, VP of brands for P&G North America hair care. “Multicultural brand building is the single biggest opportunity we have in places like North America, and actually extends to other countries,” Pritchard said in an interview. Making that work means focusing on fundamentals, including research. “We need to change the habit of research. Research now needs to be among Black, Hispanic, Asian, indigenous, people with disabilities,” he said. “It takes more time. But it’s critically important to lead to products and packaging and insights that allow you to connect. That’s meant a lot of work with our research partners and our teams.” Pritchard still sees plenty of room for progress, and some of that will come from better measurement of campaigns developed to reach multicultural audiences. Pritchard said he hopes P&G’s efforts to buy and develop more programming and media inventory from diverse owners and creators will in turn create more opportunities to measure the business outcomes. And he’s hoping progress in measurement, such as TelevisaUnivision’s Hispanic identify graph, can help P&G ensure that it’s reaching multicultural audiences at the levels it wants. IBM Artificial intelligence has long been billed as the next generation way of using automated measurement and machine learning to optimize advertising creative, media placement and other marketing budget decisions. But its ugly secret is that it often has biases built in. AI is, after all, the collective product of a technology industry with well-publicized diversity issues. IBM has set out to counter bias in the AI systems driving marketing decisions with its Advertising Toolkit for AI Fairness 360, an open-source toolkit with 75 “fairness metrics” and 13 algorithms that can help identify and mitigate biases in data sets. It includes sample code that can be incorporated into AI driving media planning, programmatic buying or creative optimization. WPP, Delta and Wieden+Kennedy Of course, a bias-mitigating AI toolkit doesn’t mean much if no one uses it. WPP and its GroupM media arm, in particular, have been enthusiastic early adopters, with Mindshare Global CEO Adam Gerhart working on coaxing clients to give it a try. Gerhart has been approaching marketers about using IBM’s tools for a series of media and creative applications that can lead to published studies in time for CES in January. Delta Air Lines has been working with its agency, Wieden+Kennedy, to apply the IBM toolkit to its advertising, said Emmakate Young, managing director of marketing. And this would not be Wieden’s only contribution on fostering better use of multicultural measurement. To its credit, when some of its prominent work for P&G’s Old Spice from last decade turned out not to resonate well with Black consumers, the company listened to the research and made changes in current work despite the agency’s legendary resistance to creative pre-testing. Subscribe to Ad Age Sign up now for the latest industry news and analysis Click here ~ ~ ~CORRECTION: A previous version of this story included an incorrect title for Adam Gerhart's role.

9 companies and agencies doing multicultural measurement right

Featured Stories