Metrics Mess: Five Sad Truths About Measurement Right Now

Turns Out the Most Measurable Medium Is the Most Difficult to Measure Accurately. Here's Why

By Published on .

A competitor's mobile app, you find out, has been downloaded more than 100,000 times in just a few days. Impressive, right? Maybe, maybe not.

Boston-based mobile analytics firm Localytics just released a study that shows a discouraging trend toward instant app abandonment among users who are clearly being bombarded with more apps than they have the time or inclination to actually use. Localytics tracked the number of one-time-only users of thousands of apps and grouped them by quarter throughout 2010, graphing them as a percentage of new app downloads during the same quarter. The results are pretty grim: In the first quarter of the year, 22% of the time apps were downloaded, used once, then abandoned. By the fourth quarter, the number had risen to 26%. To put that more bluntly: Roughly a quarter of the time, mobile apps are effectively dead on arrival in terms of user engagement.

Credit: Source: Localytics

To arrive at this analysis, Localytics massed data from its proprietary metrics system that tracks hourly and daily usage across multiple platforms (primarily Android, BlackBerry, iPhone, iPad and Windows Phone 7), plus session length, number of previous sessions of any given user (a measure of app loyalty), usage by country and city, etc. (Click on the "Demo" tab at for sample charts.)

The pre-Web 1.0 paradigm -- when dial-up portals such as AOL and CompuServe ruled the world -- worked a lot like the new app era: Walled gardens within proprietary systems kept competitors largely in the dark about one another. The late-stage Web 1.0 paradigm offered nominal transparency, with public-facing analytics options including Compete, Quantcast and Amazon's Alexa offering estimated traffic data on major sites to all comers -- rounding out the data available from enterprise-level players including ComScore (which surged to prominence when it bought former market-leader Media Metrix in 2002), Omniture (acquired by Adobe in 2009), Nielsen Online (the successor to Nielsen/NetRatings and Nielsen BuzzMetrics) and Experian Hitwise, not to mention Google with its free Google Analytics offerings. (Quantcast also lets customers who sign up to use a special Quantcast tracking tag on their website identify themselves as "Quantified," with "directly measured data" that is shared openly.)

In contrast, the app paradigm returns us, in many ways, to the Portal Age, with few public-facing resources for competitive customer-engagement data.

On Facebook, anyone can glean public-facing information on fans and "likes" -- so those figures are widely touted and cited. But consider the current Top 10 list of Facebook fans for brands, as monitored by Facebook-analytics company Socialbakers of Pilsen, in the Czech Republic. (The company offers enterprise-level tracking to clients around the world including BMW and HP; see for details.)

Top 10 Brands on Facebook chart
Source: Socialbakers

Coca-Cola, Starbucks and Disney top the list of numbers of fans -- not particularly surprising. But it doesn't take long for the chart to devolve into Pavlovian fanboy territory. Picture, if you will, the demographic that washes down Skittles and Oreos with Red Bull while checking out Victoria's Secret model galleries. Converse appears twice on the list (thanks to the Converse All Star shoe getting its own fan page) -- and, actually, Victoria's Secret also makes a second appearance, with its Pink sub-brand showing up just beyond the Top 10, in the No. 17 spot. The top 25, in case you're curious, is rounded out by the likes of Pringles, PlayStation, Monster Energy, Starburst, Nutella and Xbox, which should tell you something.

The truth is, many of the most-fanned (i.e., most-liked) brands on Facebook are already beloved by the "like"-prone nerd set -- a teen/collegiate demographic that can be both easily engaged and instantaneously distracted. Another way of saying "easily engaged": easily induced. Take, for instance, Skittles, which has been growing its Facebook fan base with goofy promotions including, recently, a give-away of a full-size Skittles vending machine (perfect for your dorm room!). Skittles may well be iconic (and yummy) enough to deserve every one of its 15 million fans, but from a consumer perspective, going public about your Skittles love (or jones for Nutella or addiction to Monster Energy) is most likely incentive-induced behavior. Quite simply, the affection ("like" or fandom) being expressed is most often just a brief manifestation of desire for a prize, a discount or a fleeting bit of amusement. (Too bad there's no "Like for the next five minutes only" button.)

Of course, there's nothing wrong with that, but it's worth remembering that much of the heat in the fan/"like" space is driven by very old-fashioned marketing concepts.

Credit: Source: Twittercounter

Speaking of Skittles, a few years ago the candy brand got a lot of attention (from Ad Age's DigitalNext blog, among others) for putting up raw social-media chatter on its website,, including an undifferentiated feed of Twitter search results on the term "Skittles." Today, the sixth-most-fanned brand on Facebook has -- drum roll, please -- fewer than 9,000 followers on Twitter. According to, it would take Skittles 3,864 days -- 10 years! -- to increase its follower base 10 times (to a still modest 90,000 Twitter followers), extrapolating from its current growth rate.

For whatever reason, Skittles is putting its social-media efforts into Facebook, not Twitter, and it shows. (Both efforts are similarly goofy in tone and affect, with the Twitter feed offering sweet-nothing tweets like "People who live in glass houses should hang up rainbow curtains.") The fact that Skittles has 15,154,430 fans (and counting) on Facebook but so few equivalents (followers) on Twitter just shows how synthetic -- non-organic -- fandom and followings typically are.

In the U.S., with a population of 310-plus million, more than 2 million people die each year. It stands to reason then, that Facebook, with more than 600 million members worldwide with an average age of around 40 (one recent third-party estimate says 38; another says 44) is losing millions of members annually to death. (Facebook's friend-connecting system has been known to spookily suggest that members "reconnect" with recently deceased friends and relatives.) In fact, mortality rates may well be elevated among those with diets high in Oreos, Skittles, Pringles, PlayStation, Monster Energy, Starburst, Nutella and Xbox.

But never mind death. Simple disinterest in branded chatter among the living is probably the biggest problem any marketer active in the social-media sphere faces. Not to mention zero chance of engagement with social-media drop-outs and opt-outs who disappear without really telling anyone.

Here's a simple test: Log into with your Twitter account. It's a service that lets you quickly figure out which of your followers haven't been active on Twitter. You can select a cut-off -- say, all of those followers who haven't tweeted in more than 30 days, or 60 days, or more -- and then decide if you want to unfollow them for being, well, deadbeats. The UnTweeps UI shows the last time each of your idle followers has tweeted, and if you've got a large following, you may be surprised to find dozens or even hundreds of folks who ceased tweeting, full-stop, months, or even years, ago, without actually formally quitting Twitter. And yet Twitter still calls them your followers.

Over the past few years, companies including Lithium (which acquired and absorbed Scout Labs last summer), Radian6, Sysomos, Trendrr, Viralheat and Visible Technologies have sprung up to help marketers track what's being said about them across the social web. Though they vary in their approaches and comprehensiveness, they all face similar problems when it comes to parsing the principal currency of the real-time economy: thoughts expressed in often surprisingly elusive human language.

At Ad Age, we've had a content partnership with Trendrr for more than a year, resulting in the Trendrr Chart of the Week at that tracks conversation about all manner of mass-market topics and brands -- from Lady Gaga to Apple. Masterminding the chart each week increasingly involves thinking outside the algorithm -- because as the social web has exploded, so have the number of ways that people choose to talk about the subjects they care about. Lady Gaga fans, for instance, may not formally mention "Lady Gaga" or "Gaga" at all in a tweet or an update, instead opting for a variation on her name (e.g., "Lady Dada," a recent trending topic on Twitter) or a sentiment (e.g., "#thankgod4gaga," another recent trending topic) or a related topic coded in language specific to insiders (e.g., "#BornThisWayFriday," which fans tweeted in anticipation of the release of Lady Gaga's latest single, "Born This Way," on Friday, Feb. 11).

There is no algorithm or machine logic than can decipher those conversational mutations among fans. Parsing and deciding to track them involves human beings paying close attention -- seeing the trees in the forest, if you will, while also trying to make sense of the forest overall.

Credit: Source: Trendrr

And then there's what we at Ad Age call The Salt Lake City Effect, aka The Bieber Distortion. Last summer, in the course of tracking chatter about various Hollywood blockbusters via Trendrr, we saw, over the course of just a few hours, a seemingly inexplicable surge in tweets including the word "Salt," the name of the then-latest Angelina Jolie vehicle.

Beyond the core problem that "Salt" has an unfortunately generic name for a blockbuster (unlike, say, "Inception"), we wondered: Did a new "Salt" trailer get released? Did Jolie experience a nip-slip or other wardrobe malfunction on a red carpet somewhere while promoting her movie? No and nope. It took awhile for us to figure it out, but by parsing individual tweets, we ultimately discovered that pop star Justin Bieber, who was then on tour, had just arrived at his latest destination, where he tweeted, "Salt Lake City is super chill. Air just feels clean. Feels like it's gonna be a good day." His fans retweeted that bland observation endlessly, giving the word "Salt" a boost out of nowhere.

In other words, seemingly meaningful facts -- absolute numbers about the usage of a certain term in the social-media sphere -- can actually be meaningless because they've suddenly become about something you don't care about at all. (Thankfully, Trendrr has technology, including so-called natural language processing, that can filter out off-topic tweets.) Social-media conversations are constantly being hijacked, both accidentally and on purpose.

One lesson we have to relearn every time we're seduced by stark new social-media data sets: Human beings are gloriously erratic, unpredictable and hard to read; the data they throw off is, too.

Most Popular
In this article: