[liberationtech] Fwd: <nettime> Algorithmic / Biometric Governmentality

[Cecilia Tanaka]

---------- Forwarded message ----------
From: Ian Alan Paul <ianalanpaul at gmail.com>
Date: Sun, Oct 29, 2017 at 7:21 PM
Subject: <nettime> Algorithmic / Biometric Governmentality
To: nettime-l at mail.kein.org


This very digestible short talk (22:00) on the emerging threat of
algorithmic/biometric governmentality from Zeynep Tufekci may be of
interest to those who research control societies, etc..:
https://www.ted.com/talks/zeynep_tufekci_we_re_building_a_dystopia_just_to_make_people_click_on_ads

The transcript is below:

So when people voice fears of artificial intelligence, very often,
they invoke images of humanoid robots run amok. You know? Terminator?
You know, that might be something to consider, but that's a distant
threat. Or, we fret about digital surveillance with metaphors from the
past. "1984," George Orwell's "1984," it's hitting the bestseller
lists again. It's a great book, but it's not the correct dystopia for
the 21st century. What we need to fear most is not what artificial
intelligence will do to us on its own, but how the people in power
will use artificial intelligence to control us and to manipulate us in
novel, sometimes hidden, subtle and unexpected ways. Much of the
technology that threatens our freedom and our dignity in the near-term
future is being developed by companies in the business of capturing
and selling our data and our attention to advertisers and others:
Facebook, Google, Amazon, Alibaba, Tencent.

Now, artificial intelligence has started bolstering their business as
well. And it may seem like artificial intelligence is just the next
thing after online ads. It's not. It's a jump in category. It's a
whole different world, and it has great potential. It could accelerate
our understanding of many areas of study and research. But to
paraphrase a famous Hollywood philosopher, "With prodigious potential
comes prodigious risk."

Now let's look at a basic fact of our digital lives, online ads.
Right? We kind of dismiss them. They seem crude, ineffective. We've
all had the experience of being followed on the web by an ad based on
something we searched or read. You know, you look up a pair of boots
and for a week, those boots are following you around everywhere you
go. Even after you succumb and buy them, they're still following you
around. We're kind of inured to that kind of basic, cheap
manipulation. We roll our eyes and we think, "You know what? These
things don't work." Except, online, the digital technologies are not
just ads. Now, to understand that, let's think of a physical world
example. You know how, at the checkout counters at supermarkets, near
the cashier, there's candy and gum at the eye level of kids? That's
designed to make them whine at their parents just as the parents are
about to sort of check out. Now, that's a persuasion architecture.
It's not nice, but it kind of works. That's why you see it in every
supermarket. Now, in the physical world, such persuasion architectures
are kind of limited, because you can only put so many things by the
cashier. Right? And the candy and gum, it's the same for everyone,
even though it mostly works only for people who have whiny little
humans beside them. In the physical world, we live with those
limitations.

In the digital world, though, persuasion architectures can be built at
the scale of billions and they can target, infer, understand and be
deployed at individuals one by one by figuring out your weaknesses,
and they can be sent to everyone's phone private screen, so it's not
visible to us. And that's different. And that's just one of the basic
things that artificial intelligence can do.

Now, let's take an example. Let's say you want to sell plane tickets
to Vegas. Right? So in the old world, you could think of some
demographics to target based on experience and what you can guess. You
might try to advertise to, oh, men between the ages of 25 and 35, or
people who have a high limit on their credit card, or retired couples.
Right? That's what you would do in the past.

With big data and machine learning, that's not how it works anymore.
So to imagine that, think of all the data that Facebook has on you:
every status update you ever typed, every Messenger conversation,
every place you logged in from, all your photographs that you uploaded
there. If you start typing something and change your mind and delete
it, Facebook keeps those and analyzes them, too. Increasingly, it
tries to match you with your offline data. It also purchases a lot of
data from data brokers. It could be everything from your financial
records to a good chunk of your browsing history. Right? In the US,
such data is routinely collected, collated and sold. In Europe, they
have tougher rules.

So what happens then is, by churning through all that data, these
machine-learning algorithms -- that's why they're called learning
algorithms -- they learn to understand the characteristics of people
who purchased tickets to Vegas before. When they learn this from
existing data, they also learn how to apply this to new people. So if
they're presented with a new person, they can classify whether that
person is likely to buy a ticket to Vegas or not. Fine. You're
thinking, an offer to buy tickets to Vegas. I can ignore that. But the
problem isn't that. The problem is, we no longer really understand how
these complex algorithms work. We don't understand how they're doing
this categorization. It's giant matrices, thousands of rows and
columns, maybe millions of rows and columns, and not the programmers
and not anybody who looks at it, even if you have all the data,
understands anymore how exactly it's operating any more than you'd
know what I was thinking right now if you were shown a cross section
of my brain. It's like we're not programming anymore, we're growing
intelligence that we don't truly understand.

And these things only work if there's an enormous amount of data, so
they also encourage deep surveillance on all of us so that the machine
learning algorithms can work. That's why Facebook wants to collect all
the data it can about you. The algorithms work better.

So let's push that Vegas example a bit. What if the system that we do
not understand was picking up that it's easier to sell Vegas tickets
to people who are bipolar and about to enter the manic phase. Such
people tend to become overspenders, compulsive gamblers. They could do
this, and you'd have no clue that's what they were picking up on. I
gave this example to a bunch of computer scientists once and
afterwards, one of them came up to me. He was troubled and he said,
"That's why I couldn't publish it." I was like, "Couldn't publish
what?" He had tried to see whether you can indeed figure out the onset
of mania from social media posts before clinical symptoms, and it had
worked, and it had worked very well, and he had no idea how it worked
or what it was picking up on.

Now, the problem isn't solved if he doesn't publish it, because there
are already companies that are developing this kind of technology, and
a lot of the stuff is just off the shelf. This is not very difficult
anymore.

Do you ever go on YouTube meaning to watch one video and an hour later
you've watched 27? You know how YouTube has this column on the right
that says, "Up next" and it autoplays something? It's an algorithm
picking what it thinks that you might be interested in and maybe not
find on your own. It's not a human editor. It's what algorithms do. It
picks up on what you have watched and what people like you have
watched, and infers that that must be what you're interested in, what
you want more of, and just shows you more. It sounds like a benign and
useful feature, except when it isn't.

So in 2016, I attended rallies of then-candidate Donald Trump to study
as a scholar the movement supporting him. I study social movements, so
I was studying it, too. And then I wanted to write something about one
of his rallies, so I watched it a few times on YouTube. YouTube
started recommending to me and autoplaying to me white supremacist
videos in increasing order of extremism. If I watched one, it served
up one even more extreme and autoplayed that one, too. If you watch
Hillary Clinton or Bernie Sanders content, YouTube recommends and
autoplays conspiracy left, and it goes downhill from there.

Well, you might be thinking, this is politics, but it's not. This
isn't about politics. This is just the algorithm figuring out human
behavior. I once watched a video about vegetarianism on YouTube and
YouTube recommended and autoplayed a video about being vegan. It's
like you're never hardcore enough for YouTube.

(Laughter)

So what's going on? Now, YouTube's algorithm is proprietary, but
here's what I think is going on. The algorithm has figured out that if
you can entice people into thinking that you can show them something
more hardcore, they're more likely to stay on the site watching video
after video going down that rabbit hole while Google serves them ads.
Now, with nobody minding the ethics of the store, these sites can
profile people who are Jew haters, who think that Jews are parasites
and who have such explicit anti-Semitic content, and let you target
them with ads. They can also mobilize algorithms to find for you
look-alike audiences, people who do not have such explicit
anti-Semitic content on their profile but who the algorithm detects
may be susceptible to such messages, and lets you target them with
ads, too. Now, this may sound like an implausible example, but this is
real. ProPublica investigated this and found that you can indeed do
this on Facebook, and Facebook helpfully offered up suggestions on how
to broaden that audience. BuzzFeed tried it for Google, and very
quickly they found, yep, you can do it on Google, too. And it wasn't
even expensive. The ProPublica reporter spent about 30 dollars to
target this category.

So last year, Donald Trump's social media manager disclosed that they
were using Facebook dark posts to demobilize people, not to persuade
them, but to convince them not to vote at all. And to do that, they
targeted specifically, for example, African-American men in key cities
like Philadelphia, and I'm going to read exactly what he said. I'm
quoting.

They were using "nonpublic posts whose viewership the campaign
controls so that only the people we want to see it see it. We modeled
this. It will dramatically affect her ability to turn these people
out."

What's in those dark posts? We have no idea. Facebook won't tell us.

So Facebook also algorithmically arranges the posts that your friends
put on Facebook, or the pages you follow. It doesn't show you
everything chronologically. It puts the order in the way that the
algorithm thinks will entice you to stay on the site longer.

Now, so this has a lot of consequences. You may be thinking somebody
is snubbing you on Facebook. The algorithm may never be showing your
post to them. The algorithm is prioritizing some of them and burying
the others.

Experiments show that what the algorithm picks to show you can affect
your emotions. But that's not all. It also affects political behavior.
So in 2010, in the midterm elections, Facebook did an experiment on 61
million people in the US that was disclosed after the fact. So some
people were shown, "Today is election day," the simpler one, and some
people were shown the one with that tiny tweak with those little
thumbnails of your friends who clicked on "I voted." This simple
tweak. OK? So the pictures were the only change, and that post shown
just once turned out an additional 340,000 voters in that election,
according to this research as confirmed by the voter rolls. A fluke?
No. Because in 2012, they repeated the same experiment. And that time,
that civic message shown just once turned out an additional 270,000
voters. For reference, the 2016 US presidential election was decided
by about 100,000 votes. Now, Facebook can also very easily infer what
your politics are, even if you've never disclosed them on the site.
Right? These algorithms can do that quite easily. What if a platform
with that kind of power decides to turn out supporters of one
candidate over the other? How would we even know about it?

Now, we started from someplace seemingly innocuous -- online adds
following us around -- and we've landed someplace else. As a public
and as citizens, we no longer know if we're seeing the same
information or what anybody else is seeing, and without a common basis
of information, little by little, public debate is becoming
impossible, and we're just at the beginning stages of this. These
algorithms can quite easily infer things like your people's ethnicity,
religious and political views, personality traits, intelligence,
happiness, use of addictive substances, parental separation, age and
genders, just from Facebook likes. These algorithms can identify
protesters even if their faces are partially concealed. These
algorithms may be able to detect people's sexual orientation just from
their dating profile pictures.

Now, these are probabilistic guesses, so they're not going to be 100
percent right, but I don't see the powerful resisting the temptation
to use these technologies just because there are some false positives,
which will of course create a whole other layer of problems. Imagine
what a state can do with the immense amount of data it has on its
citizens. China is already using face detection technology to identify
and arrest people. And here's the tragedy: we're building this
infrastructure of surveillance authoritarianism merely to get people
to click on ads. And this won't be Orwell's authoritarianism. This
isn't "1984." Now, if authoritarianism is using overt fear to
terrorize us, we'll all be scared, but we'll know it, we'll hate it
and we'll resist it. But if the people in power are using these
algorithms to quietly watch us, to judge us and to nudge us, to
predict and identify the troublemakers and the rebels, to deploy
persuasion architectures at scale and to manipulate individuals one by
one using their personal, individual weaknesses and vulnerabilities,
and if they're doing it at scale through our private screens so that
we don't even know what our fellow citizens and neighbors are seeing,
that authoritarianism will envelop us like a spider's web and we may
not even know we're in it.

So Facebook's market capitalization is approaching half a trillion
dollars. It's because it works great as a persuasion architecture. But
the structure of that architecture is the same whether you're selling
shoes or whether you're selling politics. The algorithms do not know
the difference. The same algorithms set loose upon us to make us more
pliable for ads are also organizing our political, personal and social
information flows, and that's what's got to change.

Now, don't get me wrong, we use digital platforms because they provide
us with great value. I use Facebook to keep in touch with friends and
family around the world. I've written about how crucial social media
is for social movements. I have studied how these technologies can be
used to circumvent censorship around the world. But it's not that the
people who run, you know, Facebook or Google are maliciously and
deliberately trying to make the country or the world more polarized
and encourage extremism. I read the many well-intentioned statements
that these people put out. But it's not the intent or the statements
people in technology make that matter, it's the structures and
business models they're building. And that's the core of the problem.
Either Facebook is a giant con of half a trillion dollars and ads
don't work on the site, it doesn't work as a persuasion architecture,
or its power of influence is of great concern. It's either one or the
other. It's similar for Google, too.

So what can we do? This needs to change. Now, I can't offer a simple
recipe, because we need to restructure the whole way our digital
technology operates. Everything from the way technology is developed
to the way the incentives, economic and otherwise, are built into the
system. We have to face and try to deal with the lack of transparency
created by the proprietary algorithms, the structural challenge of
machine learning's opacity, all this indiscriminate data that's being
collected about us. We have a big task in front of us. We have to
mobilize our technology, our creativity and yes, our politics so that
we can build artificial intelligence that supports us in our human
goals but that is also constrained by our human values. And I
understand this won't be easy. We might not even easily agree on what
those terms mean. But if we take seriously how these systems that we
depend on for so much operate, I don't see how we can postpone this
conversation anymore. These structures are organizing how we function
and they're controlling what we can and we cannot do. And many of
these ad-financed platforms, they boast that they're free. In this
context, it means that we are the product that's being sold. We need a
digital economy where our data and our attention is not for sale to
the highest-bidding authoritarian or demagogue.

(Applause)

So to go back to that Hollywood paraphrase, we do want the prodigious
potential of artificial intelligence and digital technology to
blossom, but for that, we must face this prodigious menace, open-eyed
and now.

Thank you.


_____________________________________

Dr. Ian Alan Paul
www.ianalanpaul.com
Assistant Professor of Emerging Media
Art Department, Stony Brook University

“What can I do?
One must begin somewhere.
Begin what?
The only thing in the world worth beginning:
The End of the world of course.”

           -Aimé Césaire


#  distributed via <nettime>: no commercial use without permission
#  <nettime>  is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: http://mx.kein.org/mailman/listinfo/nettime-l
#  archive: http://www.nettime.org contact: nettime at kein.org
#  @nettime_bot tweets mail w/ sender unless #ANON is in Subject: