Randomized Testing Is Fast and Cheap, but Few Seem Interested

Skip the Gurus and Focus Groups and Take Your Ideas to the People

By Published on .

"The End of Intuition" is a terrible name. So boring. But Ian Ayres didn't believe it. That's what he wanted to call his new book about how much better it is to test ideas through random trials rather than just trusting some marketing guru or focus group -- or intuition.
Random people have spoken: 'Super Crunchers' is a better title.
Random people have spoken: 'Super Crunchers' is a better title.

Anyway, his editor thought he was nuts and insisted that "Super Crunchers" was a much zippier name. So the two of them decided to do some random testing of his book on random testing. They took out a Google ad and half the time someone was doing a search on "data mining" or "number crunching," a little ad on the right would appear for a new book called "The End of Intuition." Half the time the same ad appeared for a new book called "Super Crunchers." "In just a few days we had real-world reactions from more than a quarter of a million page views," Ayres reports proudly.

A little less proudly he adds that, ahem, "Super Crunchers" got way more traffic -- 63% -- and thus became the title of his book, brimming with just such stories about how quick, easy and cheap it has become to test pretty much any marketing idea that pops into your head.

And how you'd be crazy not to.

And yet, how few companies do.

First, a look at some of the Great Randomized Tests in History, as culled by Ayres, a Yale law professor and numbers geek:

In Mexico in 1999, the wonky president Ernesto Zedillo decided to take 506 impoverished villages and flip a coin. Heads, the village got enrolled in Progresa, a program that gave cash incentives to moms who came in for prenatal care and nutrition counseling. They also got money if their kids stayed in school and got checkups. Tails was the control group. It got nada.

At the end of just two and a half years, 10% more of the boys were attending school in the Progresa towns, as were 20% more of the girls. Serious illnesses were down 12%. Most stunningly, the kids in Progresa villages were a full centimeter taller. In 2001, the program went nationwide -- and has since expanded to another 30 countries -- for one reason: The results of a giant, randomized test are simple to see and impossible to refute.

Now, consider a more mercenary application: Capital One was wondering which offer would bring in more business: a 4.9% teaser rate for six months or a 7.9% rate for a year?

"Theory isn't going to tell you the answer," Ayres says.

Randomized testing did. The company sent out a huge direct mailing, half with one offer, half with the other. The 4.9% proved overwhelmingly more popular and is probably sitting in your mailbox right now.

"This is something I wish we would do a lot more of," says Edward Manzitti, head of research at the Direct Marketing Association. And yet, he admits, most companies don't. Why not?

"They don't have the time."

So here's what's new: With the internet, it doesn't take any time.
"You could take the front page of The New York Times and do randomized tests of 50 different ideas for changing the layout, and I bet you one or two of those ideas would be more successful than the status quo," Ayres says. And you'd know it by the end of the day.

Incredibly, though, most websites -- even the ones selling stuff, sites that could be trying out all sorts of different offers -- are still not using randomized trials. Maybe that's because the brass there don't want to see their great ideas lose in a simple, unbiased test, as judged by the consumers themselves.

Maybe they'd rather be wrong and not have anyone know it.

Scary thought.
In this article:
Most Popular