Using Integrative Propositional Analysis (IPA) to Study the USAID/PERU Country Development Cooperation Strategy
By: Steven E. Wallis, PhD
Director, Foundation for the Advancement of Social Theory
Adjunct Faculty, Capella University
Fulbright Specialist Roster Candidate
BRIEF BIO
Steven E. Wallis earned his Ph.D. in 2006 at Fielding Graduate University, focusing on the rigorous analysis and integration of conceptual systems. He has a decade of experience as a facilitator and organizational development consultant in Northern California. At Capella University, Steve mentors doctoral candidates. As Director for the Foundation for the Advancement of Social Theory (FAST) he supports emerging scholars working to identify rigorous paths for the validation of theory through critical metatheory and metapolicy analysis. An interdisciplinary thinker, his academic publications cover a range of fields including ethics, management, organizational change, and policy. His recent book “Avoiding Policy Failure” shows how a systems view of policy models can be used to estimate the effectiveness of policies before implementation as well as improving policies for reducing cost and improving results. Recently, Dr. Wallis was appointed to the Fulbright Specialist Roster. The Fulbright Specialist program supports US Scholars in projects to help improve the capacity of academic institutions outside the US.
ABSTRACT
Complexity theory (CT) has been used to analyze policy situations. However, CT is difficult to use. Therefore, we need new approaches that are easier to use – without sacrificing rigor. Here, Integrative Propositional Analysis (IPA) is used to analyze the policy within the “USAID / PERU Country Development Cooperation Strategy.” That strategy supports governmental decentralization, ecological sustainability, economic prosperity, governmental services, and more. These goals are incredibly challenging and ambitious, so a new view may provide improved opportunity for success. IPA is a “conceptual systems science” approach providing new insights into the non-Toulminian logic structures in the policy. IPA helps us identify the extent to which the policy is likely to be effective, which parts of the policy are likely to be effective, opportunities for more effective administration, and opportunities for improving policy success. Importantly, this approach provides an innovative approach that scholars and practitioners may apply to critical policy discussions.
KEYWORDS
Integrative Propositional Analysis (IPA), Metapolicy analysis, Science of conceptual systems, Peru, USAID, Decentralization
INTRODUCTION
In today’s civilization we find ourselves surrounded by unprecedented wealth, power, and technology. We also face unprecedented challenges including ecological disaster, economic collapse, poverty, injustice, and governmental inefficiency. “The complex and difficult global environment has overwhelmed, exasperated and saddened many observers” (Dennard, Richardson & Morçöl, 2008, p. 17). As leaders (and, presumably, as researchers) we have an ethical responsibility to develop new understandings so that we may deal with these issues more effectively (Fuchs & Hofkirchner, 2005).
Our policies (policy models) are an important type of conceptual system because they help us to understand the complex and systemic world in which we live and work. Historically, however, our policies have not been very reliable. To mention a few examples, the failure of economic policy have led to national and global economic woes (Wroughton & Kaiser, 2008). The US anti-drug policy has created a situation that is arguably worse than before (Baum, 1996). In this research, I investigate conceptual systems – what Boulding would call a symbolic system (Hammond, 2003, p. 217).
While systems approaches are strongly suggested for understanding policy situations (Dennard et al., 2008), most methods for building policy have been non-systemic (Shaw & Allen, 2012). Indeed, accepted methods include luck, intuition, and trial and error (Friedman, 1997). Other methods include “muddling through” (Lindblom, 2010), looking for what “makes sense” (Bastedo, 2005), and Kingdon’s “policy soup” (White, 1994). Unfortunately, none of those approaches has proven to be effective for developing highly reliable policy. Clearly, we need a better approach (Sabatier, 1999) because our present approaches to creating policy have outlived their useful life (John, 2003).
Part of the difficulty of creating effective policy may be attributed to the lack of useful theory to support the policy process. Even empirical approaches to program evaluation do not seem to have proven effective (Schmidt, Scanlon & Bell, 1979). The failure of policy (and, by implication, the underlying theory) has prompted calls for new approaches to understanding policy (Sabatier, 1999) such as complexity theory and computer modeling (Johnson & Givel, 2014). And, this kind of complexity and systems approaches seem to be gaining traction, as evidenced by the increasing number of researchers who are suggesting new methods for policy analysis (e.g. deLeon, 1999; Elliott & Kiel, 1999). Although systems approaches are increasingly applied to study social systems, there has been comparatively little effort applied to study conceptual systems (Wallis, 2013b). Our lack of understanding is problematic because we use our conceptual systems to understand the problems we face in our social systems. So, our ability to deal effectively with those problems is based on the effectiveness of our conceptual systems (including theories and policies). That lack shows a critical gap in our science and highlights the need for “a systemic evaluation of our metatheoretical assumptions” (Lamborn, 1997, p. 212).
One emerging approach is to focus on the policy’s logic model. Logic models provide, “a picture of how something works as they provide a link to outcomes (both short – and long-term) from program variables and processes” (Bruder et al., 2005, p. 187, citing the W. K. Kellogg Foundation). “Evaluators have found the logic model process useful for at least twenty years” (McLaughlin & Jordan, 1999, p. 66). Logic models include sets of interrelated propositions which describe relationships. For example, one might describe how: The popular support of a government is likely to increase with increasing transparency of government functions. Such propositions guide policy actions and policy goals.
Bechtel and Abrahamsen (2005) discuss the benefit of addressing the logics within such propositions. This appears to be a valuable perspective because a good theory (like a good policy model) has, “an internal logic that interrelates the conceptual elements” (Shaw et al., 2012, p. 486). By rigorously and scientifically measuring the quality and quantity of those logics, we can gain new insights into those conceptual systems (Wallis, 2010a).
Generally, such an approach would be a “Science Two” approach to determining the validity of policy. Science Two includes pattern recognition, causality (when combined with necessary conditions), synthesis, usefulness, and the observer being part of the observed (Umpleby, 2010). Other key concepts of Science Two include nested and embedded processes and self-organizing processes (Müller & TOŠ, 2012). This distinction is very much in line with the second scientific revolution (Morin, 2005). There, the classical science principles or universal determinism, reduction, and disjunction are replaced with a “Scienza Nuova” transdiciplinary approach seeking greater understanding through greater connectedness and context. These newer systems perspectives stand in contrast to Science One, which is considered to be the traditional, linear, approach to science
The logic model has also been criticized because “it still only represents stakeholders’ best guess or theory for what will be most effective” (Hernandez & Hodges, 2001, p. 10). And, we are back to square one because it is difficult to evaluate guess-work. Indeed, traditionally, our “criteria for comparing frameworks are not well developed” (Schlager, 1999, p. 252). Of course, if we can’t objectively compare policy models, we can’t effectively decide if one is better than another. We must rely on our unreliable intuition (or worse, trust to luck). Differing guesses may lead to differing recommendations and arguments emerge between partisans. Instead of using policy to create collaboration, our process of developing policy causes conflict.
Within the field of policy studies, the logic model is a potentially useful tool. However, it is a tool that has not been applied or understood to maximize its potential, because we have not sufficiently studied logic models scientifically as conceptual systems in policy situations.
Morin (2005) has presented the benefits of understanding social and biological systems by investigating relationships and contexts. The general claim is that organizations which are more systemic, will be more effective in practice. However, he has not provided a method for measuring how systemic those organizations are. Similarly, Dubin (1978) has suggested that theories with higher levels of structure will be more effective in practical application. Like Morin, Dubin does not provide a method for measuring that structure with any great rigor. And, significantly, the past use of those approaches has not led to the creation of more effective policy.
Past efforts to understand the logic of the model have been focused on Toulmin’s approach. For example, Bozeman & Landsbergen (1989, p. 355) follow Toulmin’s six-step model including claim, data/evidence, warrants/supporting arguments, backing, qualifiers, and rebuttals. However, that approach is a reification of “Science One” empiricism that has not proved useful in developing effective policies.
Recently, studies have identified alternative structures of logic. These include atomistic (A is true), linear (More A causes more B), circular (More A causes more B causes more C causes more A), branching (More A causes more B and more C), and concatenated (More A and more B cause more C). Importantly, research has shown that some structures are better than others for predicting the successful application of conceptual systems such as theories and policies (Wallis, 2010a, 2011, 2013a). Briefly, conceptual systems that are more complex and have a higher percentage of concatenated concepts are more likely to be effective in policy application. We will delve into the more below.
The present investigation may be understood as a form of narrative analysis (e.g. Pentland, 1999) or content analysis (Hjørland, 2002; Hood & Wilson, 2002). More specifically, this can also be understood as a form of propositional analysis. However, the analysis of propositions has been generally limited to testing the correspondence between propositions and empirical data (Yin, 1984). Here, in contrast, I will be looking at the extent to which the propositions within a policy are “interrelated,” where the propositions might be seen as, “reciprocally or mutually related” (Dictionary, 1993, p. 998).
With such a view, a policy may be seen as a kind of system consisting of concepts and their relationships. Importantly, “…any part of the system can only be fully understood in terms of its relationships with the other parts of the whole system” (Harder, Robertson & Woodward, 2004, p. 83, drawing on Freeman). This emphasizes on a systems approach to understanding the validity of a policy based on the extent to which its concepts are interrelated with the other concepts (the “systemicity” of the policy) rather than a more traditional approach of understanding the validity through the veracity of the data (although, of course, both approaches are useful).
Recently, a more quantitative approach was developed to rigorously measure the complexity and systemicity of conceptual systems represented by propositions in textual format (Wallis, 2008). Since then, Propositional Analysis (PA) and its more complex sequel Integrative Propositional Analysis (IPA) have been applied to gain new insights in a variety of fields including economic and military policy (Wallis, 2011), ethics (Wallis, 2010b), management (Wallis, 2012), studies of drug use and policy (Wallis, 2010c), and others.
Although IPA was developed independently following insights gleaned from complexity theory and systems thinking (Wallis, 2008), IPA is similar to Axelrod’s process of cognitive mapping (Axelrod, 1976). Axelrod’s approach has been applied in fields as diverse as management and policy and has evolved in a variety of directions including hard, and soft approaches (Ackermann & Eden, 2004). Hard approaches include those that are tested empirically. Soft approaches are those that are used to stimulate conversation. However, neither of these approaches has been amenable to testing “on the page” before application. In contrast, IPA deepens and extends cognitive mapping approaches because IPA is used to rigorously quantify the complexity and systemicity of conceptual systems such as policies. Importantly, studies have shown that conceptual systems with more complexity and systemicity are more effective in practical application.
In one study, Wallis (2010a) used PA to investigate the evolution of electrostatic attraction theory from ancient times through the scientific revolution. That study showed how theories of ancient times exhibited only a very low level of systemicity and they were not very effective in application. During the scientific revolution, theories were more systemic and were also more useful. At the end of the scientific revolution, the final theory (Coulomb’s law) exhibited a very high level of systemicity. That theory, of course, is highly effective in practical application. Its use enables the creation of cell phones, computers, and is a pillar of today’s society and technology.
Integrative complexity (a similar approach) was used to investigate the way students in a college course understood the concepts presented over the course of a semester. Those students who understood systemic relationships between the concepts scored higher on their papers than students whose understanding was not so systemic. In short, the study found a 17% correlation between the systemicity of students’ understanding of concepts covered in a semester and their test scores (Curseu, Schalk & Schruijer, 2010).
In the policy realm, PA was used in a set of three comparative case studies to investigate the link between the systemicity, complexity and effectiveness of policy (Wallis, 2011). Areas covered were military policy, economic policy and international treaties for the formation of international organizations. Those studies quantified how policies with higher measurable complexity were more likely to succeed while simpler policies were more likely to fail. In short, IPA enables us to choose objectively between conflicting policy proposals (Wallis, 2013a) and identify opportunities to improve and integrate policies.
For the present paper, a “policy” or “policy model” is understood as a map or understanding of how the world works. Therefore, this new approach leaves tacit those discussions around what policy “goals” should be or what “actions” should be taken to reach those goals. The “collaboration” of human policy players is likewise not considered. Those aspects of correspondence, goals, actions, and collaboration are all very important to the policy process. However, they are beyond the scope of the present paper which will focus on the policy model.