[liberationtech] Crowdsourcing a Democracy Index

Thu May 5 08:48:04 PDT 2011

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

Here's an interesting piece that I thought you'd enjoy on
Crowdsourcing Freedom House's #democracy Index:
http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-index.html

It utilizes a tool that we've built into a "democracy game" where
people can vote on their definitions of what democracy means. You can
see the initial Arabic version here - http://bit.ly/democracy_ar or in
English - http://bit.ly/democracy_en . We quickly it up overnight with
some Egyptian and Saudi friends and are now building it into "Idea
Booths" around Egypt where anyone - from the elderly to the youth -
can come to it and vote for what tangible ideas that they'd like to
see, based on ideas seeded from the community and built with open
hardware.

Curious to see the power of crowdsourcing and communitysourcing to
bubble up perspective from the ground.

Best,
Mark

- -- 

. . . . . . . . . . . . .
Mark Belinsky
President & Co-Founder
Digital Democracy
109 W 27 St, 6 Fl
New York, NY 10001 USA
w. +1-347-688-DDEM [3336]
m. +1-347-466-9327
mbelinsky at digital-democracy.org
www.digital-democracy.org
. . . . . . . . . . . . .

      Crowdsourcing a Democracy Index

<http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-index.html>

(Sorry for the recent neglect of the blog. I just started teaching
again, and that tends to absorb all my energy. So here?s a
teaching-related post on something I?ve been doing in one of my classes).

One of the things my students are doing in my ?Dictatorships and
Revolutions? class this term is constructing a democracy index/regime
classification like those produced by Freedom House
<http://www.freedomhouse.org/>, the Polity project
<http://www.systemicpeace.org/polity/polity4.htm>, or the DD dataset
<https://netfiles.uiuc.edu/cheibub/www/DD_page.html> of political
regimes I?ve used in this blog in the past (see, e.g., here
<http://abandonedfootnotes.blogspot.com/2011/02/inequality-injustice-and.html>
and here
<http://abandonedfootnotes.blogspot.com/2011/01/visualizing-political-change.html>).[1]
We
are looking at examples of how different regime classifications can be
constructed, discussing some of their problems, and then collectively
constructing a set of criteria for classification, which we will
ultimately use to actually code all 192 or so countries in the world
at intervals of about five years for a couple of decades. (If you are
interested in the actual details of how the exercise is organized,
e-mail me; this whole thing is still quite experimental, so I would
not mind some feedback. It?s turning out to be a bit complex). Since
there are over 100 students in the class (around 120, in fact), this
means that we can achieve full coverage (and even some overlap) if
each student codes just 2 countries (at various points in time), and I
am planning to assign 4-5 countries to each student (so each country
gets at least 2 coders).  We will then examine how our crowdsourced
index or regime classification compares to some of the other indexes
and regime classifications.

As a warm-up exercise, I set up a democracy ranking website
<http://www.allourideas.org/democracyrankings> usingallourideas.org
<http://www.allourideas.org/>, which I learned about some time ago via
the goodorgtheory <http://orgtheory.wordpress.com/> people. Basically,
this is a webpage where you are presented with a comparison between
two countries, and asked which one is more democratic (you can answer
?I don?t know,? and give a reason). The results of the pairwise
comparisons can be used to generate a ranking, which represents
something like the probability that a given country would be more
democratic than a randomly selected country. (But rather than read
this explanation, why not go play with it
<http://www.allourideas.org/democracyrankings>? It can be addictive,
and it?s basically self-explanatory once you see it). I asked the
students to go to this website in the first class of the term, and to
vote; a lot of them voted (an average of about 14 times, i.e., 14
comparisons). I didn?t know exactly what to expect, but I was sort of
hoping for a ?wisdom of crowds? effect. And there is, indeed,
something like that, but the effect is small. Here?s a graph (link for
full screen
<http://public.tableausoftware.com/views/ExploringtheAllourideasranking/Sheet1?:embed=yes&:toolbar=yes&:tabs=no>):

Powered by Tableau
<http://www.tableausoftware.com/public?ref=http://public.tableausoftware.com/views/ExploringtheAllourideasranking/Sheet1>

The y axis represents the sum of Freedom House?s political rights and
civil liberties scores: 2 is most free, 14 least free. The x axis
represents the ?ranking? of the countries as calculated by the
Allourideas software, ranging from 4 (North Korea has only a 4% chance
of prevailing in a ?more democratic? comparison against a randomly
selected country) to 93 (Australia; New Zealand scored 92, and was for
a time in first position, which is to be expected from a group from
New Zealand; see the complete ranking here
<http://www.allourideas.org/democracyrankings/results>). Note that
these numbers do not reflect the judgments of ?individual? students,
but the calculated probability of prevailing in a comparison against a
randomly selected country, /given /the information available from
previous pairwise comparisons. (No student or set of students actually
?ranked? North Korea last or Australia first). The size of the bubbles
is proportional to the class? subjective ?uncertainty?: basically, the
number of times a country was involved in an ?I don?t know? answer
divided by the total number of times the country appeared in any
comparisons. There were 1250 votes submitted, but since there are 192
countries, the number of possible comparisons is 36,672, which means
that a relatively large number of potential comparisons never
appeared. (Which is part of the reason I am posting this here ? I want
to see what happens if lots of people engage in this informal ranking
exercise).

There?s clearly a correlation between the rating by Freedom House and
the informal rankings generated by the pairwise comparisons produced
by the students ? about -0.62, which is pretty respectable. (Some of
the correlations between Freedom House and other measures of democracy
are not much higher than this). A simple regression of the Freedom
House ratings on the rankings generated by the students gives a
coefficient of -0.11 (highly significant, not that that matters much
in this context), which means that an increase of 10 points in the
student-generated ranking is associated with a decrease of about 1
point in the combined Freedom House PR+CL score. (A more thorough
analysis could be undertaken, but I don?t feel qualified to do it;
I?ve put up the data here
<http://www.google.com/fusiontables/DataSource?dsrcid=556776> for
anyone who is interested in doing some more exploration, and will
update it later if enough other people participate in the ranking
exercise).

Most of the ?obvious? cases appear at the extremes ? developed,
well-known democracies get a high ranking, while obvious dictatorships
mostly get a low ranking. Many of the countries that seem to be
misplaced, however, appear to be either small and little talked about
in the news or not especially well-known to students; see, for
example, Ghana (which is ranked lower than it should be, if Freedom
House is right) and Armenia (which is ranked higher than it should be,
if Freedom House is right). Would this change if more people
contributed to the ranking, especially people from a variety of
countries around the world (I know this blog gets a small readership
from a number of unlikely countries ?could my kind readers send this
link <http://www.allourideas.org/democracyrankings> around to people
who might be interested, e.g., students?). Here's a heatmap of the
student-generated rankings
<http://www.google.com/fusiontables/DataSource?snapid=160209> (darker
is more democratic):

<http://fusion.google.com/ig/add?synd=open&source=ggyp&moduleurl=http://www.google.com/ig/modules/geomap.xml>

The map seems reasonable enough to the naked eye. It seems that even a
simple informal ranking exercise can be a reasonable approximation to
a professional ranking (like that generated by Freedom House) if the
people doing the ranking have /some/knowledge of the countries being
compared, so I would expect that more people participating would
probably move the informal ranking closer to Freedom House?s measure.
(Maybe this is a most cost-effective method of generating a democracy
index ? ?the people?s democracy index,? as it were). But it could also
be the case that the ranking would diverge more from the Freedom House
ranking as people from diverse countries participated, with different
understandings of democracy. Perhaps /global /opinion about which
countries count as most democratic would diverge sharply from the
opinions of Freedom House?s expert coders. Or perhaps it would be
affected by national biases ? people from particular countries would
have a tendency to rank it higher/lower than a more ?objective?
ranking would. It would be interesting to know ? so it would be great
if you could spread the word by sending  this link
<http://www.allourideas.org/democracyrankings> around!

(I have also wondered whether this method would work for generating
?historic? data on democracy. But the obvious way of doing this would
introduce many very unlikely or difficult comparisons? e.g., could we
meaningfully compare democracy levels in 1964 Gambia vs. 1980 Angola
using this method? ? and the less obvious way would require one to set
up a website for each distinct year).

- ----------------------------------------------------------------------
[1] Technically, an index of democracy and a regime classification are
two different things. The Economist and Freedom House produce indexes
of democracy/freedom ? an aggregated measure of the degree of
democracy in a given country at any particular point in time, ranging
from 0 to 100. A regime classification instead takes regimes as types,
and attempts to determine whether a given country should be
categorized as one kind or another.
Posted by Xavier Marquez at 6:34 PM
<http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-index.html>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3Cxq8ACgkQzUoxO63J1acDtACfR2VuDwpZlFtDxX9aYnZSipHt
Bl8AoJuoEcWH0PlLxztmbe5AOYIx9sgG
=UEY5
-----END PGP SIGNATURE-----