1
Proposal Abstract
Measuring Democracy: A Multidimensional, Tiered, and Historical Approach
Principal Investigators
Michael Coppedge (University of Notre Dame), John Gerring (Boston University), Staffan Lindberg (University of Florida),Jan Teorell (Lund University, Sweden)
Project Managers
Michael Bernhard (University of Florida), Steven Fish (University of California, Berkeley), Allen Hicken (University of Michigan), Kelly McMann (Case Western Reserve University), Pamela Paxton (Ohio State), Carsten Schneider (Central European University, Budapest), Holli Semetko (Emory University), Svend-Erik Skaaning (Aarhus University, Denmark), Jeffrey Staton (Emory University)
Academic researchers, policy practitioners, and the general public alike have come to depend on indicators of democracy such as the Freedom House and Polity indexes. Such indicators are used to trace waves of democracy, educate students about world politics, test theories about the causes and consequences of democracy, allocate foreign aid, assess efforts to promote democracy, and withhold or grant legitimacy to regimes and governments around the world. Because democracy indicators guide such momentous decisions, it is crucial that they measure democracy accurately.
It is increasingly clear that existing indicators are not adequate for these purposes. Ideally, indicators of democracy would reflect the varied meanings of democracy in different parts of the world, measure each aspect of democracy precisely and without bias, and pass standard tests of validity and reliability. It is also essential that they cover almost all countries over a long span of time because it is only in a broad, long-term perspective that fundamental patterns and trends in democratization can be perceived and complex causal relationships can be sorted out.
Although there are now dozens of indicators of democracy, only a handful cover most countries for many years. They are a) the Polity index, which covers all sizable independent countries from 1800 (or their date of independence) to the present; b) the Przeworski-Alvarez-Cheibub-Limongi (PACL) democracy/dictatorship dichotomy, which covers 135 countries from 1950 to 1990 (and now updated to 2002 by Cheibub and Gandhi); and c) Freedom House’s ratings of Political Rights and Civil Liberties, which cover all countries and some dependent territories from 1972 to the present.
All of the indicators with extensive coverage suffer from multiple weaknesses. The PACL dichotomy, which shows only whether or not “governing parties lose elections,” is insensitive to all but the most dramatic regime changes or differences of degree. Although Freedom House’s detailed checklists seem to take a wide variety of information into account, its opaque process of translating raw information into final scores makes it difficult to determine exactly what it measures.Rigorous analysis suggests that it produces two separate indicators of what turns out to a single concept—contestation(Coppedge et al. 2008). Polity focuses on the formal institutional aspects of democracy. Both Polity and Freedom House do a good job of distinguishing the most democratic polities from the least democratic ones, but neither makes much more precise distinctions reliably. Polity, for example, cannot reliably say that the United States is more democratic than the next 50 countries ranked below it (Pemstein et al. 2008; Treier and Jackman 2008). The fact that most of these indicators are strongly intercorrelated is not necessarily reassuring because agreement at the most and least democratic extremes produces high correlations even when there is little correlation among scores in the intermediate range, and because a shared bias can also produce high correlations. All three indicators – and in fact most of the indicators with less extensive coverage as well – measure a primarily liberal electoral concept of democracy that ignores participatory and social dimensions of democracy that are often considered important outside the United States.
We propose to measure democracy better in three respects. First, we wouldradically disaggregate, i.e., create separate indicators of democracy’s dozens of components and subcomponents. Much of the measurement error found in existing indicators comes from their attempt to score countries on variables that are complex and probably multidimensional, such as “freedom of expression” (Freedom House) or “constraints on the executive” (Polity). No matter how detailed the coding criteria are, measuring such concepts with a single variable inevitably leads to imprecision or false precision because the defining attributes of the concept do not align according to a single simple rank ordering. Our approach is to measure each of the more specific attributes separately. This would enable us to measure them more precisely and reliably because we would be working with more concrete and less subjective phenomena. For example, instead of creating a single score for “constraints on the executive,” we would score separately whether the executive is selected by the legislature; whether the legislature has the power of the purse and other powers; whether the executive’s party controls a legislative majority; whether the executive by-passes the legislature with decrees and referendums; whether the executive appoints the supreme court; whether the executive accepts judicial review; the extent to which the executive manipulates the court through budget constraints, impeachments, or “packing” appointments; and so on. Each one of these attributes can be rated more reliably than vague “constraints on the executive.”
Second, we would measure democracy more comprehensively by creating indicators of neglected aspects of liberal and electoral democracy and of the participatory and egalitarian dimensions of democracy, which most existing indicators tend to ignore. Thus, we would construct indicators in all of the following thirteen categories: sovereignty, voting, elections, descriptive representation, the executive, the legislature, the judiciary, political parties, the media, other civil society organizations, local government, civil liberty, and social equality.
Third, we would measure each of these categories extensively, to cover all countries (and some dependent territories) from 1900 to the present whenever possible. Some indicators would cover this entire range while others would cover only more recent years, but there would be full coverage within each category by at least one indicator. Once data are collected, we would aggregate selected variables into indicators of various models of democracy—electoral, liberal, participatory, and egalitarian—and would make it possible for other scholars or agencies to select and combine our variables into indicators of their own preferred versions of democracy or components of democracy. Our indicators would include estimates of reliability.
The Principal Investigators (PIs) would work with a dozen Project Managers who are full-time faculty at universities in the U.S., Sweden, Denmark, and Hungary. Each Project Manager would be in charge of a category of indicators and would finalize the operational procedures for the indicators in that category in consultation with the PIs. Some of the indicators can be produced by the Project Managers themselves or by teams of coders they recruit, but most would require coding by country experts, who would be paid to gather the necessary information and submit it via the Internet. For many indicators we plan to have teams of three expert coders per country. Some of the Project Managers would also serve as area coordinators who help identify, recruit, and oversee country experts.The Helen Kellogg Institute for International Studies at the University of Notre Dame would be the central administrative office for the project. It would require a full-time staff person to work with Michael Coppedge to develop the data collection website, coordinate with the Project Managers and other PIs, standardize and distribute the questionnaires, compile and clean the submitted data, publicize the findings, and plan three international conferences. The first would be an initial organizational meeting of Project Managers with an advisory board. The second would be a midterm follow-up to head off any problems that arise after the project is in the field. The third would be a public conference presenting analyses of the newly collected data.
We anticipate that a project of this magnitude would require funding from more than one source. In addition to the usual applications to the National Science Foundation and our own universities, we plan to propose this project to USAID and several European development and democracy-promoting agencies, preferably through the OECD’s Development Assistance Committee (DAC).
A successful project would bring several major public benefits. First, it would give governments and NGOs more precise, valid, and reliable indicators of democracy, which they can use to evaluate the impact of their own democracy promotion efforts and decide where to allocate future funding. Second, it would enable researchers to do more rigorous research on the causes and consequences of democracy. Third, it would open up a new area of research on causal relationships among components of democracy, which would also be useful for democracy promotion. Fourth, it would apprentice hundreds of researchers all over the world in rigorous measurement technology. Finally, it would increase U.S. scholarly recognition of the broader criteria for democracy that are considered relevant in other parts of the world.
References
Coppedge, Michael, Angel Alvarez, and Claudia Maldonado. 2008. "Two Persistent Dimensions of Democracy: Contestation and Inclusiveness." Journal of Politics 70 (3):632-47.
Pemstein, Daniel, Stephen Meserve, and James Melton. 2008. "Democratic Compromise: A Latent Variable Analysis of Ten Measures of Regime Type." In Social Science Research Network.
Treier, Shawn, and Simon Jackman. 2008. "Democracy as a Latent Variable." American Journal of Political Science 52 (1):201-17.