Mr. Hoffman went into his pitch about how Deja News' Web site traffic is undercounted because most of its users log on from work, and measurement companies don't have representative business panels. He got out his own log files, showing the site's traffic at 4.2 million unique visitors in February, compared with RelevantKnowledge's figures of 820,000 unique visitors. In the end, he won some business, but not all he wanted.
In the competitive world of Internet advertising, the monthly ratings lists can be deal-makers, or in many cases, deal-breakers. If you're not on the top 25 list, it's often an uphill battle.
"I'm being held hostage by the measurement companies," said Mr. Hoffman.
These lists are so important that sites work diligently to make sure their traffic is being counted correctly by the measurement companies.
Search engine company AltaVista has even appointed a "rankings relationship expert" to figure out exactly how its traffic is being counted.
And that's no small task, considering all of the measurement companies have different methodologies and often come out with divergent rankings. For example, AOL.com ranked No. 1 on the Media Metrix Top 25 list for February, but was ranked No. 5 by RelevantKnowledge for February and No. 5 by newcomer NetRatings for the week of March 30.
(RelevantKnowledge and Media Metrix both note in their reports that AOL users are underrepresented on their panels, and traffic on AOL.com is undercounted).
NIELSEN TO LAUNCH SERVICE SOON
Media buyers have some complaints, too.
"I don't think any of them are any good," said John Nardone, director of media and research at Modem Media. "They're all systematically deficient in one way or another."
For a while, Media Metrix (formerly PC Meter) was the sole ratings service for the Web, using a panel of PC users whose online activity is tracked by software loaded on their computers and sent back to Media Metrix for monthly reporting.
But within the past year, newcomers have entered the field, including RelevantKnowledge, founded by former Turner Entertainment executives, and NetRatings, which published its first Web ratings report last week, tracking banner activity as well as Web site visits.
And the competition is only going to intensify in July, when Nielsen Media Research launches its Web ratings service, which will probably be called NielsenWeb or NielsenNet (it's leaning toward NielsenWeb), according to Manish Bhatia, VP-interactive services, Nielsen Media Research.
As the field becomes more crowded, accusations are flying about who's got the biggest panel, how panelists are recruited, if panels are representative and what services they're measuring.
Both Web sites and media buyers are frustrated that there's no single measurement model that solves all their problems.
"Our biggest hope is that we, as an industry, get some measurement that everyone can use," said Rich LeFurgy, chairman of the Internet Advertising Bureau and senior VP-advertising at ABC News/ESPN Internet Ventures.
IAB REVIEWING SITUATION
"Until we have something everyone can agree upon, we're stuck in the he said-she said lines of competing claims," said Mr. LeFurgy. The IAB is now reviewing the situation very closely, he said.
"It's important to raise the level of urgency on this matter," he said.
One of the issues is recruitment methodology. Media Metrix uses a combination of direct mail and random digit dial (RDD) recruiting; RelevantKnowledge uses RDD; NetRatings uses RDD and online recruiting; and Nielsen will use RDD.
For Mr. Nardone of Modem Media, recruiting methodology is important because it can influence the makeup of panels and their size.
"A technically pure methodology would be total random digit dial, where everyone has an equal chance of being picked," he said.
However, RelevantKnowledge co-founder and CEO Jeff Levy admits that RDD is expensive, and results in a smaller panel.
HARD TO RECRUIT BUSINESS USERS
Another key issue is representation of business users. Media Metrix is the only service with a separate business panel, although it has only 1,000 users on the panel, with a monthly in-tab (users actually reporting data) of between 650 and 700.
RelevantKnowledge compensates for underrepresentation of business users on its panel by giving them a higher weight when reporting.
Nielsen said it won't have a business panel.
"There are too many issues with a business panel," said Mr. Bhatia of Nielsen.
"A lot of companies would resist installing software," he said. "Tracking activity in the workplace could be perceived as being too intrusive."
Other measurement companies agree.
"The largest single issue is the issue of empowerment," said Doug McFarland, senior-VP and general manager at Media Metrix, and business users asked to be on panels often need to get approval.
But for sites aimed at business users and other under-represented segments, such as international and educational audiences, these issues need to be resolved soon.
BE UPFRONT WITH DATA
Industry executives say that until measurement standards are defined, the answer is to make information widely available.
"All these companies should be upfront and tell everyone how they recruit, what the incentives are, how long users stay in the sample and what services they measure," said Marshall Cohen, president of research consulting company Marshall Cohen Associates, whose clients include America Online, MTV and other media services.
"Then those in the industry can decide who is good and who's not good."
Contributing: Patricia Riedman.