The lawsuit alleged that Arbitron's launch of Portable People Meters (PPM) to measure the listenership of radio stations in California violated state law by dramatically undercounting minority audiences. According to court documents, stations that served minority audiences in Los Angeles experienced substantial ratings decreases after the introduction of the PPM system.
Two practices were at issue. First, to recruit blacks and Latinos to its audience panels, Arbitron relied primarily on land-line telephones, which excluded many young black and Latino listeners who rely on cellphones, a large segment of the audience. Secondly, the exclusion of country of origin as a category in the Hispanic sampling strategy overlooked the fact that the over- or under-inclusion of foreign-born Hispanics can substantially influence audience ratings .
As a result of this settlement, Arbitron will be required to pay $400,000 to the plaintiffs, $100,000 each to the state of California and the City of Los Angeles, and $100,000 to the city and county of San Francisco. It also must take reasonable steps to increase minority participation in the audience panels and incorporate country of origin as a standard demographic characteristic from Hispanic households.
This is not the first time that Arbitron has been required to clean up its dirty laundry. In 2009, it settled lawsuits brought by the states of New York and New Jersey alleging that the company's marketing and commercialization of the PPM service violated consumer fraud and civil-rights laws. Arbitron agreed to pay $490,000 as part of a settlement to ensure that its ratings panels were racially diverse.
Should you be concerned if you are not using Arbitron's rating services? The answer is "absolutely yes," for two reasons. First, unlike Arbitron, the vast majority of research suppliers are not held accountable by a federal agency for the accuracy of their media ratings . Secondly, the absence of a definable set of standards for multicultural research means that the media buyer is likely to encounter inaccurate ratings that result from questionable industry practices.
A few examples of these questionable practices will illustrate my point.
Monolingual interviews. Hispanic and Asian respondents are not always provided an interviewing option other than English, despite substantial evidence that providing a native-language option in addition to English improves response rates and results in more valid responses.
Outsourcing. The practice of outsourcing data collection to research shops in foreign countries may reduce costs, but creates more problems when coupled with interviewers that have heavy accents, lack familiarity with U.S. culture, and exhibit social skills that clash with U.S. respondents.
Subjective Adjustments of Survey Responses. Purporting to take "cultural factors" into consideration, some researchers are making adjustments to the ratings made by Latino respondents because they are considered inflated when compared to non-Latinos. Such adjustments appear to have little scientific justification, and reflect a lack of understanding about cultural differences. Indeed, is it inconceivable that higher ratings are reflective of a more passionate engagement with a product or service?
Single vs. Mixed Modes of Data Collection. Survey respondents differ in terms of how comfortable they feel in completing surveys by mail, online or telephone -- preferences that are influenced by reading ability, security issues, access to the internet and the desire for a personal conversation. Multicultural audiences, in particular, reveal distinct preferences for more than one mode of data collection -- a finding that is instrumental in improving survey response rates.
Sample Imbalances. Imbalances in a survey sample (i.e., too many females, not enough immigrants) can greatly distort the conclusions if left uncorrected. Although Census data is updated annually with demographic information that can be used to correct such imbalances, research practitioners are not consistent in making the needed adjustments.
Careless Use of Ethnic or Race Labels. Qualified multicultural respondents are sometimes dropped from participation in a survey when interviewers are given strict instructions to include only respondents who identify with specific ethnic labels. Ethnic labels such as black, African American, Hispanic, Latino or Asian can have a different meaning to respondents, especially if they are immigrants. Rather than drop respondents, a better approach is to describe the meaning of these labels or define the type of persons that they include.
Although this is not an exhaustive list of questionable practices, it underscores one important point: Accurate measurement of multicultural audiences is a complex business that should not be left to general practitioners in the research industry. Arbitron is learning its lessons the hard way, but more proactive educational efforts are needed to encourage the adoption of sound practices when measuring multicultural audiences.
The academic community, which has been lax in addressing such issues, is a good place to start. Survey methods and marketing-research courses should devote attention to sound practices in the measurement of multicultural audiences, while textbook publishers should require writers to update their content.
Do we need the courts to get the research industry up to speed? No, but to remain relevant, our models of research cannot remain in a time capsule of the past, and must adapt to the changing diversity of the U.S. population.