Three years ago, on the way to my apartment in Brooklyn to cook dinner for myself and a friend, we stopped into a grocery store that offered build-your-own six-packs of beer. The store had dozens of options of craft beers and ciders in weird flavors, from obscure breweries we had never heard of.
We idly showed a few of them to each other, put them back, and got a six-pack of Stella Artois. Then, in my kitchen, scrolling through (Facebook-owned) Instagram, my friend screamed and pointed to an ad for a beer she had never tried, never read about, never heard of, never purchased, barely even held in her hand long enough to read its label out loud just one time, 15 minutes before.
Obviously, her phone was listening to her. Yes?
I had mostly forgotten this happened, until seeing my former boss, Verge editor Nilay Patel, tweet about how many of these stories he had listened to over the recent holiday weekend.
“It’s striking that everyone assumes clandestine listening instead of coincidence — and that the actual targeting data used is likely creepier than simply listening,” he noted, before conceding to a reader that he (or any other reporter) can’t explain exactly how all this targeting works.
We know that Facebook and Google and Apple have a lot to lose if it ever came out that they are recording us through our phones, and even though we try not to think about it, most of us also know that they have already gained more data about us than can possibly ever be parsed or used without accessing our microphones.
Still, there are perennial personal “investigations” into the question, and dozens of Reddit threads, and hundreds of news hits: Do our phones listen to our conversations on behalf of advertisers? And does Facebook help them?
Not all these coincidences can be explained — at least not in a way that will make you feel better
Facebook published an official denial of this practice on its company blog in June 2016, which reads in part: “Facebook does not use your phone’s microphone to inform ads or to change what you see in News Feed.” More recently, Michigan Sen. Gary Peters posed the question to Mark Zuckerberg point-blank, and he said, “No,” in front of all of Congress. (Both relevant only if Mark Zuckerberg’s word means anything to you.)
Last November, reporters Alex Goldman and P.J. Vogt attempted to debunk a whole batch of anecdotes about Facebook and Instagram “listening” through users’ phones — submitted by listeners to their popular internet culture podcast Reply All.
They spoke to former Facebook engineer Antonio García Martínez, who more or less launched the ad-targeting division of the company. Most notably, he created Facebook Pixel, which is essentially a piece of code that can be put on virtually any website, allowing Facebook to collect information about what you do there — which items you look at and click, what kinds of things you spend time reading, what you almost buy but don’t quite buy, etc. Martínez had no qualms about explaining the murkier capabilities of Facebook ad targeting, yet he denied that the company could listen to you.
Then they spoke to ProPublica investigative journalist Julia Angwin, who explained that Facebook can quite easily supplement its own data about your online behavior with purchasable data about your offline behavior. Consumer credit reporting agencies — like, for example, the now-notorious Equifax — have files that include things like your income, your marital status, your legal history, “the square footage of your house within 25 square feet,” and so on. These data brokers also manage the majority of loyalty programs in grocery stores and major pharmacy chains, which means they don’t need you to pay for things with a credit card in order to know what you’re buying.
Facebook had more than 52,000 ad-targeting categories that users could belong in, Angwin explained, including typical “interest” categories for Adele fans and Nascar enthusiasts, but also sorting people by which decade their homes were built in, whether their families drank a lot of iced tea, and if they thought they might have dissociative identity disorder. “There was one that was just my favorite called, ‘a person who likes to pretend to text in awkward situations.’” Angwin did not know how the company figured this out.
But in call after call, Goldman and Vogt reasoned their way through listeners’ stories, explaining how location tracking and social connections and other predictable elements of their personality and shopping history could explain the bizarre ads. Hardly anyone believed them.
Facebook is almost definitely not listening, but what it’s doing is arguably worse
Antonio García Martínez, the same engineer who spoke about the Facebook Pixel on Reply All, wrote an article last year for Wired that made the technical argument as to why Facebook would not be listening to us through our phones.
Essentially, collecting and storing that data would be too huge a task: “Constant audio surveillance would produce about 33 times more data daily than Facebook currently consumes,” he wrote. And “such snooping would be eminently detectable, ringing up noticeable amounts of data on your smartphone as Facebook maintained your always-on call to Zuckerberg.”
The obvious rebuttal here is that voice assistants like Alexa and Siri are always ready to talk to us, which must mean that our devices are always ready to listen. But those assistants have to be woken up by a couple of trigger words (“Hey, Siri”), he argues, and there would be no way for Facebook to listen for “the millions or perhaps billions of words and phrases [that] could land you in a Facebook targeting segment.”
We think Facebook needs more than it actually does in order to figure us out
More importantly, advertisers don’t care about the vast majority of even your most personal data — it’s just not that useful to know what you’re talking about. Particularly because knowing the words that you’re saying doesn’t really give much insight into what you mean: “Human language is overrun with sarcasm, innuendo, double-entendre, and pure obfuscation,” Martínez wrote. “To assume that at-Facebook-scale AI will be able to figure out, even at the fluky level of internet advertising, just what you crave based on any given statement gives these technologies more credence (or paranoia) than they deserve.”
We’re narcissists, he argues, basically: We think Facebook needs more than it actually does in order to figure us out. And we think marketers actually care about our highly specific preferences, rather than the mostly broad, socially generated ones that really determine whether you’ll purchase any given soda or shirt.
Still, take Martínez’ defense with a grain of salt. It makes almost no sense for Facebook to surveil us through our microphones, but the reason the company will never thoroughly engage with the topic is because it would mean explaining the way its ad targeting actually works — which is way worse.
Related
Advertisers can easily track you, your kid, your doctor, and the president
While we don’t know everything, we do know that Facebook has patented a method to use your previous location data, in conjunction with the previous location data of people you know, to predict your future location. We also know that it keeps “shadow profiles” made up of information that users (or non-users) have not actively supplied but can be easily tied to them (your work email address, your loyalty discount cards, your likely acquaintances that you haven’t yet “friend requested”). It can also figure out if two people know each other by looking at the metadata of photos uploaded in a small time frame in a small geographic area and then comparing the scratches and dust on the lens of the camera that took them.
The most valuable information Facebook has about you isn’t what you like or where you go or what you click on, but who you know. It always has been.
There are apps that are listening through your phone’s microphone, but they’re not the big ones
Just because Facebook isn’t listening doesn’t mean your phone’s microphone isn’t betraying you a little bit.
Last December, the New York Times reported that hundreds of mobile game apps were using software from a startup called Alphonso, which can “detail what people watch by identifying audio signals in TV ads and shows, sometimes even matching that information with the places people visit and the movies they see.” The information is used for ad targeting, and — somewhat quaintly — to gauge the effectiveness of various local TV ads, for things like car dealerships.
(If you’re over your cellphone gaming phase, it might still concern you to know that Alphonso has a working relationship and data-sharing agreement with the popular music identification app Shazam.)
(Also, Shazam is owned by Apple.)
This is creepy, obviously, but not quite the same as recording what you’re saying. Christopher Wylie, the former Cambridge Analytica employee who leaked information about how the British firm had manipulated US elections via Facebook user data, explained to Parliament in March, “It’s not to say that all audio has to be somebody speaking. There’s audio that can be useful in terms of, are you in an office environment? Are you outside? Are you watching TV? What are you doing right now?”
In a study conducted last year, researchers at Northeastern University found no evidence of the apps they studied triggering the microphone without obvious cause, or of audio files being transmitted from the phones. They did, however, find photo and video transmissions, which were screen recordings. Another report, published in Wired last year, explained how hundreds of apps use “ultrasonic tones” to track where you’re going in physical space. These apps need access to your microphone, but not to actually listen to you, just to “beacon” noises emitted in stores and by advertisements. So … rest easy.
In short: Facebook isn’t listening, but it doesn’t matter
I know, of course, that my friend and I are more predictable than we believed, and that whatever beer we touched must have been a subject of curiosity for other people like us. I know that 20-something-year-old women in my neighborhood with my type of social circle and my alcohol budget and my Instagram-honed susceptibility to clever branding were buying that beer, or at least looking it up. And I know that I had given more than enough information — by signing up for a club card in that grocery store six months prior, by growing up in a region of the state known for craft beer production, by living with a college friend who was in the “artistic” frat — to allow Facebook to serve my friend that ad without wiretapping her. But do I really know it?
What’s frustrating about this whole conversation is that none of it matters. If Facebook can serve you an ad for a beer you’ve held in your hand only once in your life, and never encountered in any other way, that’s scary no matter what’s behind it. (To add pointlessness to injury: Advertisers probably don’t even need this info, and Facebook’s ad view metrics are probably fake.)
It stings, too, that Mark Zuckerberg — the professed global altruist, the unmovable monopolist CEO — feels it appropriate to dismiss anyone’s questions about Facebook’s surveillance tactics as “conspiracy theories,” considering how many of his users’ fears have already turned out to be founded. All we can really do about any of it feels like almost nothing: We can turn off microphone permissions, try not to download weird apps without combing their privacy policies carefully, consider some browser extensions that block Facebook’s Pixel or other tracking cookies, and maybe figure out ways to live a little less predictably.
None of it will really help, though, unless we also cut ties with everyone we know and love — our most valuable data treasure trove, and the only one making Facebook billions of dollars a year.
Sourse: vox.com
