[liberationtech] Crowdsourcing a Democracy Index

Thu May 5 09:51:24 PDT 2011

I think that this is particularly challenging because I doubt that there is
agreement about what democracy means (by people, by governments, by funding
agencies, by international orgs...)

For example, in Eurasia (AKA former USSR), in public opinion polls, people
overwhelming say that democracy means economic prosperity and rate liberty,
voting, free speech, etc. way down on the list...

-----Original Message-----
From: liberationtech-bounces at lists.stanford.edu
[mailto:liberationtech-bounces at lists.stanford.edu] On Behalf Of Mark
Belinsky
Sent: Thursday, May 05, 2011 11:48 AM
To: liberationtech at lists.stanford.edu
Subject: [liberationtech] Crowdsourcing a Democracy Index

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

Here's an interesting piece that I thought you'd enjoy on Crowdsourcing
Freedom House's #democracy Index:
http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-index
.html

It utilizes a tool that we've built into a "democracy game" where people can
vote on their definitions of what democracy means. You can see the initial
Arabic version here - http://bit.ly/democracy_ar or in English -
http://bit.ly/democracy_en . We quickly it up overnight with some Egyptian
and Saudi friends and are now building it into "Idea Booths" around Egypt
where anyone - from the elderly to the youth - can come to it and vote for
what tangible ideas that they'd like to see, based on ideas seeded from the
community and built with open hardware.

Curious to see the power of crowdsourcing and communitysourcing to bubble up
perspective from the ground.

Best,
Mark

- -- 

. . . . . . . . . . . . .
Mark Belinsky
President & Co-Founder
Digital Democracy
109 W 27 St, 6 Fl
New York, NY 10001 USA
w. +1-347-688-DDEM [3336]
m. +1-347-466-9327
mbelinsky at digital-democracy.org
www.digital-democracy.org
. . . . . . . . . . . . .

      Crowdsourcing a Democracy Index

<http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-inde
x.html>

(Sorry for the recent neglect of the blog. I just started teaching again,
and that tends to absorb all my energy. So here?s a teaching-related post on
something I?ve been doing in one of my classes).

One of the things my students are doing in my ?Dictatorships and
Revolutions? class this term is constructing a democracy index/regime
classification like those produced by Freedom House
<http://www.freedomhouse.org/>, the Polity project
<http://www.systemicpeace.org/polity/polity4.htm>, or the DD dataset
<https://netfiles.uiuc.edu/cheibub/www/DD_page.html> of political regimes
I?ve used in this blog in the past (see, e.g., here
<http://abandonedfootnotes.blogspot.com/2011/02/inequality-injustice-and.htm
l>

and here
<http://abandonedfootnotes.blogspot.com/2011/01/visualizing-political-change
.html>).[1]
We
are looking at examples of how different regime classifications can be
constructed, discussing some of their problems, and then collectively
constructing a set of criteria for classification, which we will ultimately
use to actually code all 192 or so countries in the world at intervals of
about five years for a couple of decades. (If you are interested in the
actual details of how the exercise is organized, e-mail me; this whole thing
is still quite experimental, so I would not mind some feedback. It?s turning
out to be a bit complex). Since there are over 100 students in the class
(around 120, in fact), this means that we can achieve full coverage (and
even some overlap) if each student codes just 2 countries (at various points
in time), and I am planning to assign 4-5 countries to each student (so each
country gets at least 2 coders).  We will then examine how our crowdsourced
index or regime classification compares to some of the other indexes and
regime classifications.

As a warm-up exercise, I set up a democracy ranking website
<http://www.allourideas.org/democracyrankings> usingallourideas.org
<http://www.allourideas.org/>, which I learned about some time ago via the
goodorgtheory <http://orgtheory.wordpress.com/> people. Basically, this is a
webpage where you are presented with a comparison between two countries, and
asked which one is more democratic (you can answer ?I don?t know,? and give
a reason). The results of the pairwise comparisons can be used to generate a
ranking, which represents something like the probability that a given
country would be more democratic than a randomly selected country. (But
rather than read this explanation, why not go play with it
<http://www.allourideas.org/democracyrankings>? It can be addictive, and
it?s basically self-explanatory once you see it). I asked the students to go
to this website in the first class of the term, and to vote; a lot of them
voted (an average of about 14 times, i.e., 14 comparisons). I didn?t know
exactly what to expect, but I was sort of hoping for a ?wisdom of crowds?
effect. And there is, indeed, something like that, but the effect is small.
Here?s a graph (link for full screen
<http://public.tableausoftware.com/views/ExploringtheAllourideasranking/Shee
t1?:embed=yes&:toolbar=yes&:tabs=no>):

Powered by Tableau
<http://www.tableausoftware.com/public?ref=http://public.tableausoftware.com
/views/ExploringtheAllourideasranking/Sheet1>

The y axis represents the sum of Freedom House?s political rights and civil
liberties scores: 2 is most free, 14 least free. The x axis represents the
?ranking? of the countries as calculated by the Allourideas software,
ranging from 4 (North Korea has only a 4% chance of prevailing in a ?more
democratic? comparison against a randomly selected country) to 93
(Australia; New Zealand scored 92, and was for a time in first position,
which is to be expected from a group from New Zealand; see the complete
ranking here <http://www.allourideas.org/democracyrankings/results>). Note
that these numbers do not reflect the judgments of ?individual? students,
but the calculated probability of prevailing in a comparison against a
randomly selected country, /given /the information available from previous
pairwise comparisons. (No student or set of students actually ?ranked? North
Korea last or Australia first). The size of the bubbles is proportional to
the class? subjective ?uncertainty?: basically, the number of times a
country was involved in an ?I don?t know? answer divided by the total number
of times the country appeared in any comparisons. There were 1250 votes
submitted, but since there are 192 countries, the number of possible
comparisons is 36,672, which means that a relatively large number of
potential comparisons never appeared. (Which is part of the reason I am
posting this here ? I want to see what happens if lots of people engage in
this informal ranking exercise).

There?s clearly a correlation between the rating by Freedom House and the
informal rankings generated by the pairwise comparisons produced by the
students ? about -0.62, which is pretty respectable. (Some of the
correlations between Freedom House and other measures of democracy are not
much higher than this). A simple regression of the Freedom House ratings on
the rankings generated by the students gives a coefficient of -0.11 (highly
significant, not that that matters much in this context), which means that
an increase of 10 points in the student-generated ranking is associated with
a decrease of about 1 point in the combined Freedom House PR+CL score. (A
more thorough analysis could be undertaken, but I don?t feel qualified to do
it; I?ve put up the data here
<http://www.google.com/fusiontables/DataSource?dsrcid=556776> for anyone who
is interested in doing some more exploration, and will update it later if
enough other people participate in the ranking exercise).

Most of the ?obvious? cases appear at the extremes ? developed, well-known
democracies get a high ranking, while obvious dictatorships mostly get a low
ranking. Many of the countries that seem to be misplaced, however, appear to
be either small and little talked about in the news or not especially
well-known to students; see, for example, Ghana (which is ranked lower than
it should be, if Freedom House is right) and Armenia (which is ranked higher
than it should be, if Freedom House is right). Would this change if more
people contributed to the ranking, especially people from a variety of
countries around the world (I know this blog gets a small readership from a
number of unlikely countries ?could my kind readers send this link
<http://www.allourideas.org/democracyrankings> around to people who might be
interested, e.g., students?). Here's a heatmap of the student-generated
rankings <http://www.google.com/fusiontables/DataSource?snapid=160209>
(darker is more democratic):

<http://fusion.google.com/ig/add?synd=open&source=ggyp&moduleurl=http://www.
google.com/ig/modules/geomap.xml>

The map seems reasonable enough to the naked eye. It seems that even a
simple informal ranking exercise can be a reasonable approximation to a
professional ranking (like that generated by Freedom House) if the people
doing the ranking have /some/knowledge of the countries being compared, so I
would expect that more people participating would probably move the informal
ranking closer to Freedom House?s measure.
(Maybe this is a most cost-effective method of generating a democracy index
? ?the people?s democracy index,? as it were). But it could also be the case
that the ranking would diverge more from the Freedom House ranking as people
from diverse countries participated, with different understandings of
democracy. Perhaps /global /opinion about which countries count as most
democratic would diverge sharply from the opinions of Freedom House?s expert
coders. Or perhaps it would be affected by national biases ? people from
particular countries would have a tendency to rank it higher/lower than a
more ?objective?
ranking would. It would be interesting to know ? so it would be great if you
could spread the word by sending  this link
<http://www.allourideas.org/democracyrankings> around!

(I have also wondered whether this method would work for generating
?historic? data on democracy. But the obvious way of doing this would
introduce many very unlikely or difficult comparisons? e.g., could we
meaningfully compare democracy levels in 1964 Gambia vs. 1980 Angola using
this method? ? and the less obvious way would require one to set up a
website for each distinct year).

- ----------------------------------------------------------------------
[1] Technically, an index of democracy and a regime classification are two
different things. The Economist and Freedom House produce indexes of
democracy/freedom ? an aggregated measure of the degree of democracy in a
given country at any particular point in time, ranging from 0 to 100. A
regime classification instead takes regimes as types, and attempts to
determine whether a given country should be categorized as one kind or
another.
Posted by Xavier Marquez at 6:34 PM
<http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-inde
x.html>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3Cxq8ACgkQzUoxO63J1acDtACfR2VuDwpZlFtDxX9aYnZSipHt
Bl8AoJuoEcWH0PlLxztmbe5AOYIx9sgG
=UEY5
-----END PGP SIGNATURE-----

_______________________________________________
liberationtech mailing list
liberationtech at lists.stanford.edu

Should you need to change your subscription options, please go to:

https://mailman.stanford.edu/mailman/listinfo/liberationtech

If you would like to receive a daily digest, click "yes" (once you click
above) next to "would you like to receive list mail batched in a daily
digest?"

You will need the user name and password you receive from the list moderator
in monthly reminders.

Should you need immediate assistance, please contact the list moderator.

Please don't forget to follow us on http://twitter.com/#!/Liberationtech