Why marketing's next battleground is your face

FaceValue Marketers are using facial recognition technology to track your behavior—and personalize your shopping—but privacy advocates are worried By Adrianne PasquarelliIllustrations by hitandrun Published on November 19, 2019 Marketers are using facial recognition technology to track your behavior—and personalize your shopping—but privacy advocates are worried By Adrianne PasquarelliIllustrations by hitandrun Published on November 19, 2019 At Ruti, a new store in New York’s Nolita neighborhood, consumers can find apparel in trendy silhouettes, like draped pants and voluminous blouses, along with the Israeli brand’s best-selling wrap cardigan. They can try on such ready-to-wear clothing in one of the shop’s velvet-draped dressing rooms and peruse shelves of accessories including shoes and handbags. Like other stores in the popular shopping district, Ruti has a sandwich board on its sidewalk with cute messages for shoppers. A recent missive urged them to “Put on your happy pants.” Marketers are using facial recognition technology to track your behavior—and personalize your shopping—but privacy advocates are worried By Adrianne PasquarelliIllustrations by hitandrun Published on November 19, 2019 At Ruti, a new store in New York’s Nolita neighborhood, consumers can find apparel in trendy silhouettes, like draped pants and voluminous blouses, along with the Israeli brand’s best-selling wrap cardigan. They can try on such ready-to-wear clothing in one of the shop’s velvet-draped dressing rooms and peruse shelves of accessories including shoes and handbags. Like other stores in the popular shopping district, Ruti has a sandwich board on its sidewalk with cute messages for shoppers. A recent missive urged them to “Put on your happy pants.” Perhaps “face” should replace “pants,” because the 10-year-old womenswear brand is using facial recognition technology to identify customers to deliver, Ruti says, a more personalized shopping experience. As soon as a customer comes to the store, “we have her name on the screen with all of the info that’s relevant to this customer that’ll make her life better,” says Ruti Zisser, the designer who founded the company. “We just show her what she likes. If the system knows she already tried these white pants and she doesn’t like white pants, we won’t show her white pants.” Zisser says the technology empowers the ultimate level of customer service, and that it’s been very good for business, too. As marketers scramble to better connect with consumers, the use of facial recognition is spreading. More than 120 patents were filed by app and tech companies between 2015 and 2017, according to market research firm CB Insights. Some restaurants and hotels, including Marriott, are testing the software to offer personalized, speedy and friction-free experiences for harried consumers. Marketers are also using the technology to gauge consumer reactions to campaigns. Stores including Target and Lowe’s, on the other hand, have piloted facial recognition to curb shoplifting. But such technology could threaten the balance between the need for information and an individual’s right to privacy, and consumers—and lawmakers—are noticing. Some cities, including New York and San Francisco, are paying more attention to businesses and law enforcement’s use of facial recognition as well as biometric scanning. Yet experts warn the technology is growing faster than laws attempting to regulate it. “Over the last few years, there’s been a real explosion in face recognition technology,” says Farhang Heydari, executive director of the Policing Project at the New York University School of Law. The group works with police and communities to promote public safety and transparency. “Part of that is because the algorithms are getting better and better … that’s allowed there to be a huge number of people who have gotten into the facial recognition algorithm game. With more people, that brings down costs and makes it more widely available.” Consent is implied Consumers might welcome facial technology when using it to unlock their smartphones—the percentage of enabled smartphones shipped worldwide will increase from 40 percent in 2019 to 64 percent in 2020, reports eMarketer—but they might be surprised to know that, by just walking into a store, they’re giving their consent to be filmed. “For the most part, the word ‘consent’ is used very broadly here,” says Heydari. “When you walk into a store or a hotel, we assume you’ve consented to pretty much anything that can include surveillance in their public areas—there’s nothing limiting.” Some stores have signs announcing video surveillance but, with the exception of dressing rooms, private businesses are within their rights to photograph customers and deploy facial recognition technology, according to experts. Consumers who want to opt out are out of luck. At Ruti, where cameras are discreetly located in ceiling corners, customers who provide their email addresses will be notified of the practice. When a repeat consumer visits the store, employees are able to use the facial records to identify the person by face (not name) and pull up shopper history and preferences. Larger retailers have primarily used facial mapping for security purposes—but only briefly. Four years ago, Lowe’s conducted a three-month test at three of its stores, according to a spokeswoman, who declined to comment on why the test was not expanded. Similarly, Target ran a facial recognition software test in a small number of stores within its 1,800-unit fleet. Prior to the test, which was conducted last year, the Minneapolis-based retailer updated its privacy policy and posted signs near the entrances of stores. “As we always do, we’ll continue to test and learn from new technologies that have the potential to keep our guests and team members safe,” a Target spokeswoman said via email. Walmart has reportedly also used the software for theft prevention, but the retailer did not return a call requesting comment. In addition to security, facial tech is also being used for social media sharing and product recommendations. H&M last year installed a smart mirror in its Times Square store that uses facial recognition to dole out fashion advice and take selfies consumers can share online. Other retailers, particularly cosmetics companies like Sephora, are using facial technology for augmented reality to help customers sample makeup. Mapping facial expressions In recent years, large companies including Kellogg and Mars have used facial coding technology to analyze how people view their ads. Expedia and the Hawaii Tourism Authority teamed up two years ago to offer a customized vacation for travelers willing to watch a two-minute video and allow a camera to map their facial expressions. “Marketers are very careful in the types of applications they’re promoting with facial recognition,” says Thomas Sineau, a senior analyst at CB Insights. In China, Sineau notes, it’s common to be recognized by face as an individual, but most marketers in the U.S. use the technology only to determine age, gender and ethnicity. And when it comes to biometric advancement, marketers aren’t limiting themselves to facial recognition. Walmart has filed a patent for a shopping cart handle that can track a person’s heart rate and grip strength so that store associates will know if the customer needs assistance, according to reports. Amazon has a patent for its Alexa device that would enable it to know when a customer is sick and make medical recommendations. Many of these products are in their infancy, but all will be marketed in a way that makes them seem helpful and indispensable to consumers. Of course, they’ll also be providing valuable data that can ultimately help increase sales. Value for consumers? Even in a climate of intense privacy awareness, experts say consumers will be more comfortable with such technology if it is accompanied by value and convenience. Victoria Petrock, principal analyst at eMarketer, says consumers might feel it’s invasive to get blatant personalized promotions from marketers that have used facial recognition to identify them. Yet those same customers might be more amenable to being scanned if it saves them time or money. “Nothing is going to work from a marketing side that doesn’t include an opt-in,” says Petrock. “Just the same way they’re willing to trade their faces for security and safety [at airports], people may be willing to trade their faces for convenience if they feel the value proposition is there.” Yet some marketers are shying away from the technology altogether, believing the benefits fail to outweigh the privacy concerns. Greg Revelle, chief marketing officer of Kohl’s, says his company is not participating in facial recognition testing because it isn’t “right” for the Kohl’s brand or its customer. “We always look at what benefit it has for the consumer, and we don’t feel that this is something our consumer would view as beneficial to them,” says Revelle. However, Petrock notes, consumers are moving toward “privacy nihilism”—a defeatist attitude about technology in which they feel they cannot stop the onslaught of digital advancements so they might as well take advantage of them. Marriott has been dabbling in the facial recognition space in recent years. Late last year, the hotel giant ran a test at two of its hotels in China that allowed guests to use facial mapping to check in, promoting the service as a speedier way of providing customers access to their rooms. At a self-service machine, guests confirm terms of agreement, scan their ID cards, get a photo and put in their contact info. The entire process takes less than a minute, says Peggy Fang Roe, chief sales and marketing officer for Asia Pacific at Marriott International. Once identification is confirmed, the machine will produce a room key card. “It’s getting good traction,” says Fang Roe, who notes that the innovation frees up employees to work on other services for customers. “When you walk into a store or a hotel, we assume you’ve consented to pretty much anything that can include surveillance in their public areas—there’s nothing limiting.” Farhang Heydari, executive director of the Policing Project, NYU School of Law Delivering a personal touch While technology has typically been viewed as cold and impersonal, brands now tout it as fostering more personalized service. “People really miss this kind of personal touch,” says Zisser. “The world has become so not-personalized with everything that people are really appreciative; more than anything, they appreciate that we care about what they want.” Zisser says Ruti’s technology allows her to offer personalization commonly seen in Old World shopping districts, where store owners know customers on a very intimate level, even though she can’t be in all her stores at once. Facial recognition software helped generate a 40 percent increase in year-over-year sales in the Ruti stores where it is installed—and without a single customer complaint, says Zisser. Ruti began using the technology—which Ruti itself created—about nine months ago and has since rolled it out to its 10 stores. Cooler Screens, a Chicago-based startup that operates digitized beverage and freezer doors in stores, uses technology similar to facial recognition—but to identify customer behavior, not identity. Monitors described by founder Arsen Avakian as “IoT sensors” track a person’s movements, like standing in front of a door or opening it. Such information allows brands, including Tyson Foods and Gatorade, to see what items resonate with shoppers and lead to sales. After a pilot with Walgreens, Cooler Screens is rolling out to all of the drugstore chain’s 2,500 locations. Avakian says the company does not capture personal data because it doesn’t need it. “You can bring all of this power of technology, all of this benefit, but without compromising privacy,” he says, adding that Cooler Screens is “identity-blind.” Still, some lawmakers are concerned with the direction of facial recognition. San Francisco banned facial recognition by local law enforcement earlier this year. And last month, a New York City Council member introduced a bill that would require landlords and business owners to disclose the use of facial recognition technology, how long they are storing the information gathered and whether it’s being sold to third parties. The Policing Project’s Heydari says a few federal bills have also been proposed, though most of the regulation is at the city level. Yet consumers should be concerned, he warns. “Once you blanket an area with cameras, it’s just a matter of what kind of algorithm, software or analytics you’re going to apply to that video,” Heydari says. “When it comes to facial recognition, more than anything, you can’t change your face—once a company has information about you that’s tied to your face, that’s never going away.” Web production by Corey Holmes.

Why marketing's next battleground is your face

Featured Stories