[liberationtech] Crowdsourcing a Democracy Index

Thu May 5 08:56:38 PDT 2011

Marginally related is our attempt to identify the top online
indicators to measure the the contributions of local governments to
democracy using technology. Obviously our indicators are built on a
base of generally functioning representative democracy. That said, it
is notable when it gets to operationalizing democracy at the local
level, how far we have to go:  http://e-democracy.org/sunshine

Steven Clift - http://stevenclift.com
  Executive Director - http://E-Democracy.Org
  Follow me - http://twitter.com/democracy
  New Tel: +1.612.234.7072

On Thu, May 5, 2011 at 10:48 AM, Mark Belinsky
<mbelinsky at digital-democracy.org> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> Here's an interesting piece that I thought you'd enjoy on
> Crowdsourcing Freedom House's #democracy Index:
> http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-index.html
>
> It utilizes a tool that we've built into a "democracy game" where
> people can vote on their definitions of what democracy means. You can
> see the initial Arabic version here - http://bit.ly/democracy_ar or in
> English - http://bit.ly/democracy_en . We quickly it up overnight with
> some Egyptian and Saudi friends and are now building it into "Idea
> Booths" around Egypt where anyone - from the elderly to the youth -
> can come to it and vote for what tangible ideas that they'd like to
> see, based on ideas seeded from the community and built with open
> hardware.
>
> Curious to see the power of crowdsourcing and communitysourcing to
> bubble up perspective from the ground.
>
> Best,
> Mark
>
> - --
>
> . . . . . . . . . . . . .
> Mark Belinsky
> President & Co-Founder
> Digital Democracy
> 109 W 27 St, 6 Fl
> New York, NY 10001 USA
> w. +1-347-688-DDEM [3336]
> m. +1-347-466-9327
> mbelinsky at digital-democracy.org
> www.digital-democracy.org
> . . . . . . . . . . . . .
>
>      Crowdsourcing a Democracy Index
>
> <http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-index.html>
>
> (Sorry for the recent neglect of the blog. I just started teaching
> again, and that tends to absorb all my energy. So here?s a
> teaching-related post on something I?ve been doing in one of my classes).
>
> One of the things my students are doing in my ?Dictatorships and
> Revolutions? class this term is constructing a democracy index/regime
> classification like those produced by Freedom House
> <http://www.freedomhouse.org/>, the Polity project
> <http://www.systemicpeace.org/polity/polity4.htm>, or the DD dataset
> <https://netfiles.uiuc.edu/cheibub/www/DD_page.html> of political
> regimes I?ve used in this blog in the past (see, e.g., here
> <http://abandonedfootnotes.blogspot.com/2011/02/inequality-injustice-and.html>
> and here
> <http://abandonedfootnotes.blogspot.com/2011/01/visualizing-political-change.html>).[1]
> We
> are looking at examples of how different regime classifications can be
> constructed, discussing some of their problems, and then collectively
> constructing a set of criteria for classification, which we will
> ultimately use to actually code all 192 or so countries in the world
> at intervals of about five years for a couple of decades. (If you are
> interested in the actual details of how the exercise is organized,
> e-mail me; this whole thing is still quite experimental, so I would
> not mind some feedback. It?s turning out to be a bit complex). Since
> there are over 100 students in the class (around 120, in fact), this
> means that we can achieve full coverage (and even some overlap) if
> each student codes just 2 countries (at various points in time), and I
> am planning to assign 4-5 countries to each student (so each country
> gets at least 2 coders).  We will then examine how our crowdsourced
> index or regime classification compares to some of the other indexes
> and regime classifications.
>
> As a warm-up exercise, I set up a democracy ranking website
> <http://www.allourideas.org/democracyrankings> usingallourideas.org
> <http://www.allourideas.org/>, which I learned about some time ago via
> the goodorgtheory <http://orgtheory.wordpress.com/> people. Basically,
> this is a webpage where you are presented with a comparison between
> two countries, and asked which one is more democratic (you can answer
> ?I don?t know,? and give a reason). The results of the pairwise
> comparisons can be used to generate a ranking, which represents
> something like the probability that a given country would be more
> democratic than a randomly selected country. (But rather than read
> this explanation, why not go play with it
> <http://www.allourideas.org/democracyrankings>? It can be addictive,
> and it?s basically self-explanatory once you see it). I asked the
> students to go to this website in the first class of the term, and to
> vote; a lot of them voted (an average of about 14 times, i.e., 14
> comparisons). I didn?t know exactly what to expect, but I was sort of
> hoping for a ?wisdom of crowds? effect. And there is, indeed,
> something like that, but the effect is small. Here?s a graph (link for
> full screen
> <http://public.tableausoftware.com/views/ExploringtheAllourideasranking/Sheet1?:embed=yes&:toolbar=yes&:tabs=no>):
>
> Powered by Tableau
> <http://www.tableausoftware.com/public?ref=http://public.tableausoftware.com/views/ExploringtheAllourideasranking/Sheet1>
>
> The y axis represents the sum of Freedom House?s political rights and
> civil liberties scores: 2 is most free, 14 least free. The x axis
> represents the ?ranking? of the countries as calculated by the
> Allourideas software, ranging from 4 (North Korea has only a 4% chance
> of prevailing in a ?more democratic? comparison against a randomly
> selected country) to 93 (Australia; New Zealand scored 92, and was for
> a time in first position, which is to be expected from a group from
> New Zealand; see the complete ranking here
> <http://www.allourideas.org/democracyrankings/results>). Note that
> these numbers do not reflect the judgments of ?individual? students,
> but the calculated probability of prevailing in a comparison against a
> randomly selected country, /given /the information available from
> previous pairwise comparisons. (No student or set of students actually
> ?ranked? North Korea last or Australia first). The size of the bubbles
> is proportional to the class? subjective ?uncertainty?: basically, the
> number of times a country was involved in an ?I don?t know? answer
> divided by the total number of times the country appeared in any
> comparisons. There were 1250 votes submitted, but since there are 192
> countries, the number of possible comparisons is 36,672, which means
> that a relatively large number of potential comparisons never
> appeared. (Which is part of the reason I am posting this here ? I want
> to see what happens if lots of people engage in this informal ranking
> exercise).
>
> There?s clearly a correlation between the rating by Freedom House and
> the informal rankings generated by the pairwise comparisons produced
> by the students ? about -0.62, which is pretty respectable. (Some of
> the correlations between Freedom House and other measures of democracy
> are not much higher than this). A simple regression of the Freedom
> House ratings on the rankings generated by the students gives a
> coefficient of -0.11 (highly significant, not that that matters much
> in this context), which means that an increase of 10 points in the
> student-generated ranking is associated with a decrease of about 1
> point in the combined Freedom House PR+CL score. (A more thorough
> analysis could be undertaken, but I don?t feel qualified to do it;
> I?ve put up the data here
> <http://www.google.com/fusiontables/DataSource?dsrcid=556776> for
> anyone who is interested in doing some more exploration, and will
> update it later if enough other people participate in the ranking
> exercise).
>
>
> Most of the ?obvious? cases appear at the extremes ? developed,
> well-known democracies get a high ranking, while obvious dictatorships
> mostly get a low ranking. Many of the countries that seem to be
> misplaced, however, appear to be either small and little talked about
> in the news or not especially well-known to students; see, for
> example, Ghana (which is ranked lower than it should be, if Freedom
> House is right) and Armenia (which is ranked higher than it should be,
> if Freedom House is right). Would this change if more people
> contributed to the ranking, especially people from a variety of
> countries around the world (I know this blog gets a small readership
> from a number of unlikely countries ?could my kind readers send this
> link <http://www.allourideas.org/democracyrankings> around to people
> who might be interested, e.g., students?). Here's a heatmap of the
> student-generated rankings
> <http://www.google.com/fusiontables/DataSource?snapid=160209> (darker
> is more democratic):
>
> <http://fusion.google.com/ig/add?synd=open&source=ggyp&moduleurl=http://www.google.com/ig/modules/geomap.xml>
>
>
> The map seems reasonable enough to the naked eye. It seems that even a
> simple informal ranking exercise can be a reasonable approximation to
> a professional ranking (like that generated by Freedom House) if the
> people doing the ranking have /some/knowledge of the countries being
> compared, so I would expect that more people participating would
> probably move the informal ranking closer to Freedom House?s measure.
> (Maybe this is a most cost-effective method of generating a democracy
> index ? ?the people?s democracy index,? as it were). But it could also
> be the case that the ranking would diverge more from the Freedom House
> ranking as people from diverse countries participated, with different
> understandings of democracy. Perhaps /global /opinion about which
> countries count as most democratic would diverge sharply from the
> opinions of Freedom House?s expert coders. Or perhaps it would be
> affected by national biases ? people from particular countries would
> have a tendency to rank it higher/lower than a more ?objective?
> ranking would. It would be interesting to know ? so it would be great
> if you could spread the word by sending  this link
> <http://www.allourideas.org/democracyrankings> around!
>
> (I have also wondered whether this method would work for generating
> ?historic? data on democracy. But the obvious way of doing this would
> introduce many very unlikely or difficult comparisons? e.g., could we
> meaningfully compare democracy levels in 1964 Gambia vs. 1980 Angola
> using this method? ? and the less obvious way would require one to set
> up a website for each distinct year).
>
> - ----------------------------------------------------------------------
> [1] Technically, an index of democracy and a regime classification are
> two different things. The Economist and Freedom House produce indexes
> of democracy/freedom ? an aggregated measure of the degree of
> democracy in a given country at any particular point in time, ranging
> from 0 to 100. A regime classification instead takes regimes as types,
> and attempts to determine whether a given country should be
> categorized as one kind or another.
> Posted by Xavier Marquez at 6:34 PM
> <http://abandonedfootnotes.blogspot.com/2011/03/crowdsourcing-democracy-index.html>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk3Cxq8ACgkQzUoxO63J1acDtACfR2VuDwpZlFtDxX9aYnZSipHt
> Bl8AoJuoEcWH0PlLxztmbe5AOYIx9sgG
> =UEY5
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> liberationtech mailing list
> liberationtech at lists.stanford.edu
>
> Should you need to change your subscription options, please go to:
>
> https://mailman.stanford.edu/mailman/listinfo/liberationtech
>
> If you would like to receive a daily digest, click "yes" (once you click above) next to "would you like to receive list mail batched in a daily digest?"
>
> You will need the user name and password you receive from the list moderator in monthly reminders.
>
> Should you need immediate assistance, please contact the list moderator.
>
> Please don't forget to follow us on http://twitter.com/#!/Liberationtech
>