Research and Analysis for Public Policy and Public Management:

Principles and Practices from Active Learning

Dahlia K. Remler

School of Public Affairs, Baruch College, City University of New York

Department of Economics, The Graduate Center, City University of New York

National Bureau of Economic Research

Draft

Do not cite without the author’s permission

November 2, 2011

Acknowledgements: Gregg G. Van Ryzin and I developed many of these ideas and approaches jointly. All mistakes are my own.

1. Research and Analysis in Public Policy and Management

Most Masters of Public Administration (MPA) and Masters of Public Policy (MPP) programs have some kind of statistics, quantitative data analysis or research requirement, although the specific courses’ content and names vary (NASPAA, n.d.). Such requirements seem essential, since public sector managers, non-profit managers and policymakers at all levels are increasingly called upon to use evidence-based practice and data for both management and accountability purposes. Yet students often find their research and analysis (R&A) courses irrelevant—as well as difficult. In this paper, I describe principles and practices for MPA[1] R&A courses that improve student learning, interest and motivation.

Most of the principles and practices I describe are relevant for a wide variety of courses: statistics, research methods, evaluation, data analysis, research in public administration and so on. Although I emphasize quantitative methods, qualitative methods are also included and many of the principles and practices apply equally to both. While the practices and principles apply widely, they are particularly focused on some general learning objectives, which I expect most MPA R&A courses share to some extent. Specifically, the objectives are that after the course(s) students will be able to:

  • Critically consume research
  • Spot weak or invalid conclusions in formal and “informal” research
  • Extract relevant and valid conclusions from research
  • Perform research in policy and practice capacities at a basic level
  • Deal effectively with the quantitative aspects of public affairs

Since most MPA students do not become researchers or analysts, I recommend that primary goals for all programs be that graduates are effective consumers of research and analysis and are quantitatively literate. (Steen (2001, 2004) defines quantitative literacy, often also referred to as quantitative reasoning, and which I discuss extensively below.)

I developed the practices and principles described in this article through two main means: Trial and error in my own teaching[2] and in collaboration with colleague Gregg Van Ryzin as we wrote a research methods textbook (Remler and Van Ryzin, 2011); The teaching and learning literatures on active learning and quantitative literacy.

The paper is organized as follows: In section 2, I describe those aspects of the diversity and context of MPA students most relevant to R&A courses. In section 3, I describe the active learning and quantitative literacy literatures, extracting the parts relevant for MPA R&A courses, providing some examples and concluding with general principles. In section 4, I provide descriptions of good practices for MPA R&A courses and many examples. In section 5, I briefly conclude.

2. Diversity and context of MPA students

MPA students are diverse in many ways: age, ethnicity, other demographics, but one form of diversity has a tremendous effect on R&A courses: prior quantitative training. I will characterize (or caricature) that variation with three made-up composite bios, characteristics of students I have taught. Alice hated math in high school, avoided quantitative courses in college, majored in Ethnic Studies and is now a community organizer with a small not-for-profit. Brenda was a math major in college who now, under the supervision of a senior researcher, does data management and some analysis for a headquarters of a national religious organization. She wishes to become an analyst in her own right. Carlos majored in history, is now a human resources manager in a Federal government agency and wishes to move up as a manager in the Federal government. Brenda is the only student with a strong quantitative background and an interest in research per se, but Alice also is motivated to learn about research, because her organization’s funders want evidence of its effectiveness. Carlos is only interested in management and sees no relevance of the MPA course to his career, beyond meeting a requirement. The ideal class would be useful, interesting, and challenging but not impossible for all three students. As the sample composite bios illustrate, a second form of MPA diversity is also important: the research and analysis requirements of students’ planned careers.

Which stages students are at in their careers are also important for designing and implementing R&A courses. Students who have substantial career experience, particularly “executive” students, have a wealth of examples and potential applications. Such students also tend to be impatient with any material whose relevance they cannot see. Their math skills may be weak since it may be many years since they last took a math or quantitative course in college or even high school. In many public affairs careers, students well along in their careers have probably seen the emphasis on data and evidence, giving them particular motivation for R&A courses. With a few careers, however, students well advanced in their careers may have not missed quantitative and analysis skills, and resent being forced to jump through a hurdle they see as irrelevant. For many MPA students, relevance to policy and practice, and particularly relevance to a job they have or would like to have, is essential for motivating and engaging them.

Other students, however, have less career experience and some even start an MPA program directly after an undergraduate degree. Yet other students may have experience in a very different career and plan to make a career change. Therefore, when crafting assignments and exercises, it is important to be aware of these students also. Nonetheless, since they choose to do an MPA, they should at least have public affairs interests to draw upon.

Some MPA students attend full-time with little work outside school, while others attend part-time and continue to work. Working while attending school can provide examples and motivation in the same way that career experience does, but it also can make students only interested in material they see as relevant. Working also takes time away from studies and reduces the flexibility of students’ time, as can family obligations or other constraints. Attending class at night after working makes it harder for students to pay attention, increasing the need for methods for keeping their attention. The long, intense sessions common in executive programs cause similar problems.

For all students, it is important to keep the learning goals relevant for MPA students. A minority of MPA students, and even of MPP students, become researchers or analysts. We turn next to principles that can help the achievement of R&A learning goals.

3. Active Learning and Quantitative Literacy: Principles for Research and Analysis Courses

Today most instructors have heard the idea that they should not just lecture but rather employ active learning. But what exactly is active learning? Bonwell and Eison (1991) say that active learning requires that “students must do more than just listen: They must read, write, discuss or be engaged in solving problems. Most importantly… [they] must engage in such higher-order thinking tasks as analysis, synthesis and evaluation.”

Of course, almost all courses require active learning tasks outside the classroom. What is the evidence about the effectiveness of active learning in the classroom? Pascarella and Terenzini (2005) summarize the experimental and quasi-experimental literature on active learning in higher education, saying that, “studies either report better mastery of course content when actively engaged in learning…or no significant learning differences or mixed learning effects when comparing active to passive lecture instructional approaches” (p. 102). Although not all studies provide enough data to determine an effect size, among those that do, Pascarella and Terenzini estimated an average effect of .25 standard deviations. Studies using observational data and control variables were consistent with the experimental and quasi-experimental findings.

A critical feature of active learning approaches, as with any teaching strategy, is to clearly define the learning objectives and then ensure that the approaches further those objectives. Bonwell (1996) suggests several active learning techniques suitable for a predominantly lecture class. Some that are suitable for R&A classes include: Short Writes, which could be short interpretations or simple problem-solving; Think-Pair-Share, in which a short problem or question is given and two students discuss it for two or three minutes, prior to a full class discussion; Formative Quizzes, with a few short questions that allow students to test their own understanding and practice applying the material. Bonwell and Eison (1991) describe many lecture-substitute activities, including case studies, debates, and peer teaching.

Many active learning approaches described in the literature do not fit well with students trying to learn essential quantitative skills that they could not learn on their own, particularly when skills build on one another. For example, a debate using empirical evidence to support positions would motivate students and is a wonderful way to synthesize and reinforce application and critical analysis of evidence. But it is not a good method for first teaching statistical significance to students who do not yet understand it. The literature on quantitative literacy is therefore particularly useful, since it is focused on the right sort of learning for R&A courses.

Steen (2001, p.8) describes several components of quantitative literacy, of which the most relevant for MPA R&A courses are:

Confidence with Mathematics…Individuals who are quantitatively confident routinely use mental estimates to quantify, interpret, and check other information…

Interpreting Data. Reasoning with data, reading graphs, drawing inferences, and recognizing sources of error…

Logical Thinking. Analyzing evidence, reasoning carefully, understanding arguments, questioning assumptions, detecting fallacies, and evaluating risks…

Making Decisions. Using mathematics to make decisions and solve problems…

Number Sense. Having accurate intuition about the meaning of numbers, confidence in estimation, and common sense in employing numbers as a measure of things.

Practical Skills. Knowing how to solve quantitative problems that a person is likely to encounter at home or at work…

I suspect that MPA programs all require some quantitative analysis or research course as much to enhance their students’ general quantitative literacy as to teach statistics or any other particular skills or knowledge. Despite the variation in MPAs’ career goals, some tasks should be doable by all MPAs, both immediately upon graduating and long after finishing their degree. Consider an example: An MPA’s organization has implemented a new policy and wants to see if it “worked” to improve an outcome. The organization has individual level outcome measures over time. (For example, an educational not-for-profit has added math majors as tutors and wants to know if their centers have more students in the community coming for help with math.) An MPA should be able to take the data in a spreadsheet or statistical package, graph averages over time and interpret the results. More importantly, any MPA graduate should be aware of the problems of statistical significance (that a change could be a fluke), even if they cannot implement the correct statistical significance test, and be aware of the problems in assuming that any change in outcomes was caused by the program change. Finally, he should consider the outcome measure’s validity and reliability. Another example, which is both qualitative and less formal, would be that when seeking feedback or thoughts on an issue, the MPA graduate will always be attuned to the issue of the representativeness of those providing feedback, even when time and resources constraints prevent obtaining a representative sample.

Steen (2001, 2004) and references therein describe a variety of teaching practices to increase quantitative literacy. I have selected the most important relevant recommendations for MPA R&A courses from Collison et al’s (2008) adaption of Steen’s recommendations. They are listed in Table 1.

Table 1: Strategies for Quantitative Literacy

  • Rule of Four: All applications and concepts presented as:
  • Words
  • Numbers
  • Graphs
  • Symbols
  • Translate from any one to the other
  • Practice
  • Interpreting and writing about numbers
  • Explaining equations in words
  • Reading, interpreting and applying technical writing
  • Assignments and tests that require students to apply skills in applications that are meaningful to the students
  • Examples involving familiar concepts are more effective than examples that require extra learning
  • Examples which motivate and interest students are valuable
  • A variety of different applications
  • Increasing student role in framing the problem and in abstracting
  • Spreadsheet and/or statistical software exercises integrated into course content throughout the curriculum

The first strategy, the “Rule of Four,” means that concepts and applications should be approached in four forms: equation, graph, numbers (data), and words, and that students should be able to translate between any two forms. Although the Rule of Four was developed for teaching calculus (Hughes-Hallett, Gleason and Flath, 2008), I have found it useful in all quantitative subjects. While all four forms are useful, not all MPA students will become comfortable with equations in some contexts, and therefore, to some extent, rationing equations can be useful.

Consider a simple regression example, with data from a representative sample providing individuals’ annual earnings and the number of years of education. The equation is

Earnings = const + B* Education.

(I recommend using the names of variables: Earnings, rather than “y”; Education, rather than “x.”) A useful introduction or application of regression would include a scatterplot of the data, with the best-fit ordinary least squares regression line, and a table (in software) showing the numerical data. Students should also interpret the coefficient and constant in words, “For each additional year of education a person has, we expect him to earn $B more annually” and “We expect that a person with zero years of education would earn $const.”[3] (They should use the actual numbers, rather than B or const, of course.) Students should also describe other aspects of the relationship in words, including tightness of fit. All four approaches together should be used for introducing the concept and when students apply it, using statistical software to estimate the constant and coefficient. It is relatively common to teach regression using the Rule of Four, but the approach can be applied to hypothesis testing and many forms of data analysis.

While all four parts of the Rule of Four are valuable, communicating in words about quantitative content is particularly important. First, it is an extremely important skill in its own right. MPA students, even those who become researchers or analysts, will work in the worlds of policy or practice. Sentences like, “the logit results were an odds ratio for X1 of 1.6 with a p-value of .007 and an odds ratio for X2 of 1.1 with a p-value of 0.26.” will not serve them well in their careers even if they understand such sentences well. MPAs will need to be able to express quantitative and technical information in the most accessible and meaningful manner possible. Miller (2004, 2005) has written two books on writing about numbers and about multivariate analysis, which include sections on writing for relatively broad audiences.

Second, describing numbers, equations and analytical concepts in words is an effective way for many to learn. Students should practice interpreting entries in tables and statistical package output in words. Being able to express the results in words both requires and aids understanding. I recommend repeated exercises in which students interpret both individual numbers within a table of results and the overall picture from the table. Sometimes non-native English speaking students resent an emphasis on words in a quantitative class, having expected that their weak English would matter little, if at all. For such students, suggest to them that they start by practicing explaining in their native language and note that being required to explain well in English will make them more effective at explaining well in their native language.

The third bullet in Table 1 refers to the importance of relevant applications. For MPA students, who may already work or be well advanced in their careers, such applications are extremely important, particularly for motivation. But applications are critical for all students and they should be rich and compelling. The fourth bullet refers to the importance of many different applications. Difficult concepts and methods take repeated efforts to learn. No matter how motivating and useful a single example, students may not learn to generalize to other situations unless they have practice generalizing. I return to this issue in later sections. Finally, it is important for students to work with data in spreadsheets and/or statistical packages. Even students who will do little data work of their own gain a far fuller understanding of the meaning of statistics and data analysis by doing it themselves. I return to this issue later also. However, it a class should not be dominated by either data cleaning or the details of a particular statistical package, because most MPA students will be primarily consumers of research and analysis, not producers.