When Nate Silver correctly predicted the 2008 election, it ushered in a new era of politics. No more were we concerned with every twist and turn of every poll; no more would talking heads waste airtime on individual polls. Instead, we could all look to one model to tell us what was happening. Each side of the political spectrum could collectively agree on when to freak out and when to relax.
Confidence in modeling only grew after the Senate was accurately predicted in 2010, the presidency in 2012 and the Senate again in 2014. There was so much confidence in this model that for the past year and a half the collective blood pressures of millions of Americans rose and fell with little red and blue lines.
Not surprisingly, success bred competitors. The New York Times "Upshot" section quoted no fewer than eight other models, including Silver's. But none of them gained the popular appeal of Silver's own FiveThirtyEIght model -- and they all seemed to roughly point in the same direction anyway.
And they were all wrong.
Now, "wrong" is a strong word. Each of these models work in terms of probabilities, but none of them gave a high, or even a moderate, probability to the eventual outcome. In Silver's defense, his model was the most bullish on the prospect of a President Trump. Other models gave that eventuality as little as a 1% chance.
More than enough ink will be spilled on what all this means for politics, but let me talk for a second about what this all means for advertising. This is important because the problems that political modelers face are, to a large extent, the problems faced by people who measure ad effectiveness. That's the bad news. The good news is that the ad world isn't identical to the political world. It's important we take advantage of those differences.
Picking a model
The biggest challenge for a marketer or pollster is picking a model to begin with. In politics, even among the nine models posted by The New York Times, there was a fair amount of disagreement. There was no expert consensus on which model was best, and if you weren't an expert -- well, good luck figuring out which one to follow. This is a familiar state of affairs for many CMOs, who are presented with an array of reasonable-looking models, and then bombarded with terms like "game-theory" or "LaGrangian multiplier." Picking the right model can seem nearly impossible.
But let's say you pick the perfect model. Things are still tricky. For instance, how do you assess whether marketing caused a boost in sales, or whether it would have happened anyway? How can you tell what would have happened had you not shown that ad to that person? Similarly, political modelers will now debate for years whether the presidential debates, or video tapes or FBI letters actually changed the race, or whether they came at moments when the race was already changing.
One reason Nate Silver's model was so bullish on Trump is that he didn't think his estimates of the vote tallies were very precise. He figured the polls might be off by quite a bit in either direction. He was right. When races are close or margins are tight, precision can mean the difference between winning and losing, or between profit and loss.
Precision is expensive, though. In politics, this means more polls to more voters to get a better read on the race. In marketing, this means running bigger campaigns, tracking more data. In 2016, we actually had fewer high-quality polls than in 2012 -- calling voters had gotten more expensive just as newspaper and pollster budgets were contracting. Rising costs and shrinking budgets are things about which everyone can commiserate.
The good news is that we can do better. We don't just have one election; we have many campaigns. Therefore, we can use those campaigns to try different models against each other, seeing how well they predict reality. Marketers have another advantage: we don't just observe events, we control them. We can change our media strategy and see if a model reacts the way expect. Increasingly, we can even run truly scientific experiments across both digital media and television. Experiments give us access to truths that aren't generally available to the team at FiveThirtyEight.com.
Nate Silver's problems may be our problems, but at least we have a chance to fix them.