Jon Steinberg, CEO of North America at Daily Mail, couldn't believe what he was hearing Wednesday morning: Google's DoubleClick, which serves ads on publishers' websites, was down -- across every site that used it, worldwide.
"The idea that it was everyone was inconceivable," Mr. Steinberg said. "It's like saying it's raining everywhere."
Support staff at MailOnline then pulled up other websites -- The Wall Street Journal, the New York Post, Forbes, Time -- to show him the extent of the DoubleClick failure. Not only had banner ads disappeared, but so had many native ads that should've been mixed in among editorial content. That's because DoubleClick serves them all.
"I've never seen anything like this," said Mr. Steinberg, the former president of BuzzFeed and a one-time Google employee.
The outage provided a rare glimpse into Google's power in the digital advertising industry. With DoubleClick out of the picture, prominent websites began displaying blank spaces instead of ads, costing publishers thousands of dollars an hour. Meanwhile, automated exchanges selling ads on sites that use DoubleClick saw their volume plunge.
Publishers that don't rely on DoubleClick, meanwhile, raked in cash as demand funneled to them.
"This is a stark reminder of DoubleClick's dominance," said Andrew Casale, VP-strategy at ad exchange Casale Media.
Another publishing executive wondered if the outage would bring attention to what he called "Google's complete monopoly on ad serving,"
"They have a glitch in their systems and quite literally a large majority of the internet businesses grind to a halt," the executive said in an email. "Speaks to their concentrated power and associated risk."
For consumers, the outage, which lasted from roughly 9 a.m. to 10:45 a.m. Eastern Time, meant a relatively ad-free experience online. But publishers experienced nearly two hours of anxiety as they watched digital-ad revenue vanish by the second. It will run The Daily Beast about $25,000 because under-delivery of ad and native impressions would require make-goods on future days, according to a person familiar with the matter. The outage will cost the Daily Mail roughly $70,000 to $100,000 in partial make-goods for advertisers, a spokesman for the company said.
Capitalizing on chaos
"Having the core revenue delivery engine be completely knocked out for two-plus hours was deeply troubling," said Julie Hansen, president and COO of Business Insider. "It's of grave concern to us any time we can't provide spot-on delivery to our clients."
While many publishers struggled, some sites not reliant on DoubleClick began raking in the cash. That's because money on advertising exchanges -- which are used to buy ads in an automated manner -- went to ads served by companies other than DoubleClick. Data from ad exchange Casale Media shows these publishers made 26% more money during the outage than the same time period the previous day.
"Some publishers literally doubled," he said.
During the outage, Casale Media saw a 28% drop in impressions across its exchange compared to the same period the previous day. The company sees several hundred million impressions an hour. "For something to cascade globally, that must have been something remarkable," Mr. Casale said.
Google declined to say much about the incident. "DoubleClick for Publishers experienced an outage this morning impacting publishers globally, across their video, display, native and mobile formats," a Google spokeswoman said in a statement. "Our team has worked quickly to fix the software bug and DFP is now back up and running, so our publisher partners can return to funding their content."
Publishers said it could've been much worse. The amount of lost revenue was relatively small given the scale of the problem. "The DoubleClick outage was only about an hour, so impact wasn't material -- just annoying," an executive at a large traditional publisher said.
Mr. Steinberg echoed this notion, saying it could have been worse if the outage occurred on a Thursday or Friday, when movie studios roll out targeted campaigns for weekend openings.
Quartz, a business news site owned by Atlantic Media, wasn't affected at all by the outage, according to Jay Lauf, the site's publisher. "We're on OpenX," he said, referring to another platform that serves ads. "Chalk it up to luck as much as anything."
Contributing: Tim Peterson