Unemployed Gamblers From Oregon? Report Profiles Web Bots
Reid Tatoris, co-founder of digital security company Are You a Human, doesn't like the term "ad fraud" when it comes to describing bots. "I think 'ad waste' is better," he said.
According to AYAH, about 58% of all internet traffic recorded in its report came from bots.
But, he said, "the majority of bot traffic is doing something else on the Internet other than ad fraud. Bots are out there for reasons unrelated to advertising and some of them are good."
One of the primary goals of the company is to separate non-human traffic from human traffic. AYAH recently released its third quarter State of the Internet Report, which showcases data gathered about online human activity.
The company gathered data from 3.2 billion impressions collected from more than 600 million devices across 3 million websites for its report.
So what websites are bots visiting? You might be forgiven for thinking they were unemployed gambling addicts. Job-search sites and gambling sites get over 93% of their traffic from bots, according to the report.
And a lot of them prefer Oregon. Oregon has the worst level of bot traffic out of any state, with 76% of all traffic coming from bots, the report said. Mr. Tatoris said that is mainly because Amazon has its servers based in the northwestern state and hackers like to use them to spread malware.
Meanwhile, about 64% of all activity on education sites is being done by actual humans. That's rather high when compared to dating sites (30%).
Bots and humans also have different preferences in top-level domains.
More than half (53%) of all activity on .com sites came from verified human users. That's a far cry from the 71% found on .gov sites, the report said.
"What's interesting is that 13 to 19 percent of the bots imitated human behavior," said Ben Trenda, CEO of AYAH. "Bots are becoming more sophisticated. They are not just hitting a page and mimicking human behavior. Some of them are trying to trick detection."
While bots have a stigma -- especially in the ad world -- not all of them are bad. Retailers, for example, have bots that scrape other sites for product and pricing information. Search-engine crawlers makes everyone's life easier by finding what you need on the internet. Those bots don't try to hide themselves, so there's less of a negative impact, Mr. Trenda said.
"Good bots are not there to waste ads," Mr. Tatoris said. "A big category of good bots is aggregating data that presents consumers with where to find the best prices for a category, like hotels."
Still, bots have a negative impact on digital ads as websites still count many of them as human visitors.
And because bots are created every day, creating an industry standard for blacklisting them can prove challenging.
A list updated daily by the Interactive Advertising Bureau, for example, only accounts for .14% of the unverified traffic websites see, according to the report.
"The average lifespan of a bot is six days," Mr. Trenda said. "Once it's found, it's blocked. The problem is it's essentially trivial to create a newer, smarter bot that is better at avoiding detection."
AYAH uses machine-learning technology to look for patterns on how both humans and bots behave. It then feeds that data into a sophisticated algorithm that spots clusters of unusual behavior. Data scientists then analyze data such as mouse movement, time spent on a page and what IP address is being used to determine whether the activity is human or non-human.
"We are solving the very real problems that more than half of all internet traffic is generated by non humans," Mr. Tatoris said. "More than half of all Internet traffic is generated by non humans. Those problems range from social media fraud, spam, bank account hacking, content piracy and advertising waste."