Alternative Approaches to in North American Research on Data Use in PolicyMaking and Management Practice:Some Comments on Methods since 1990
Darryl Hunter
University of British Columbia
June 1, 2006
Abstract:
This paper provides a synopsis of the methods adopted to study data use in North American research over the past decade and a half. Research questions are analyzed in terms of authors’ purpose, whether for interrogating public administrative policy-making or managerial practice. Diverse definitions of evidence as ‘data/information/knowledge’ and their ‘use’ are analyzed as operationalized in various research designs.The instrumentation of quantitative and qualitative researchers is examined in terms of assumptions. The extent to which the manager or policy-maker is depicted as an active or passive agent in data use is also considered. Strengths and shortcomings are identified in the operational definitions and research designs chosen. In general, the paper explores the interrelationships between researchers’ views of administrative practice with data, and their practices in collecting research data from administrators. Recommendations are made for new types of questions that should be posed and pursued with alternative methods of qualitative inquiry to advance our understanding of ‘evidence-based decision-making’ in educational administration.
Outline
1.Introductionto the meta-review
2. Delineating purposes and research questions
a. Public administrative policy-making
b. Managerial practice
3. Defining‘evidence’ and ‘use’
a.Transfer ofknowledge as documents
b.Mobilization of knowledge as personified
c.Transactions with knowledge as politically-infused beliefs
d.Diffusion of knowledge as public ideas
e. Fusion of formal and tacit knowledge
f. Interpretation of knowledge as symbolic representations
4. Delimiting through instrumentation and analyses
a. Measurement scales and limitations
b. Qualitative researchers’ instruments and limitations
5.Depicting the user through sampling
6.Drawing conclusions through the unit of study
a. Inter-organizational transmission
b. Intra-organizational transactions
c. Individual use
d. Individual reading and apprehension
7. Describing new lines of inquiry
a. Recent trends
b. Future directions
c. Conclusions
Introduction to the meta-review
“Two anecdotes do not equal data” and “no ideology without metrology” are the maxims of ‘evidence-based decision-making’. However, these are less policy prescriptions than predispositions that have migrated from the western provinces to the eastern within Canada over the past decade. The recent interest in ‘evidence’ in North American education can be seen as a muted echo of transatlantic-wide controversies about the authority of scientific research in health care. Research attention has also been stimulated by the proliferation of educational testing programs across Canada as accountability has crept up the public policy agenda. The attendant public reporting of educational performances across North America in the past decade has fostered reconsideration of the role played by statistics in public administration and educational reform in general. It has also dusted off scholarly works from an older research tradition known as ‘evaluation utilization’ within assessment circles, thus drawing attention to a three-decade debate about the vagaries in ‘transfer’ of research knowledge from the academy to policy-making circles and practitioners’ work places.
The call for ‘evidence-based or -informed decision making’ is therefore a new label for a long-standing field of inquiry. Previous meta-reviews of North American research into the ‘utilization problem’ anthologize some of this field’s history, encapsulating shifts in scholarly preoccupations and documenting the increasing sophistication of its central concepts. The field can be said to have opened in the wake of American legislation mandating program evaluations under President Johnson’s Great Society initiatives of the 1960s, and animated by the ‘guns or butter’ debates in Congress over whether military or social welfare programs deserved priority funding. Researchers were disconcerted to discover that policy makers ignored the evidence assembled about social program effectiveness, thus raising the central question why ‘scientific knowledge’ did not resonate directly in the policy making process. Leviton and Hughes’ (1981) subsequent review of those evaluations conducted in the 1970s revealed increasing sophistication in the language of ‘use’, taxonomizing this central construct as having symbolic (political), behavioral (instrumental), or conceptual manifestations. A seminal debate between Carol Weiss and Michael Quinn Patton in 1981 in a California beach house and thence from the rostrums of the American Educational Research Association (Alkin,1990) lent new vibrancy to the key questions of evaluative impact, researchers’ motivation, and managerial or policy maker’s purpose. Cousins and and Leithwood’s (1986) compendium of North American research on the topic sought to identify the sundry predictor variables that impacted on ‘use’, thus beginning a series of attempts to model ‘use’ for subsequent empirical testing. By 1994, Huberman indicated it was time for those scholars plowing the field to move beyond exploratory foraystoward testing causal relations. And Shulha and and Cousins’ (1997) synthesis described the increasing orientation toward ‘process uses’ in evaluation research, a growing awareness that ‘use’ embraced organizational problems and not just individual proclivities, heightened appreciation for context in ‘use’, and emerging and vexatious issues of ‘misuse’.
Since then, lines of inquiry have diverged. Although many research questions are still associated with organizational or decision theory, now too are medical, communications and information technology issues falling within the ambit. Studies of ‘data utilization’ are therefore dispersed across journals in several disciplines, and the zones of inquiry are many. The journal Knowledge: Creation, Diffusion and Utilization, once the multidisciplinary clearinghouse for such investigations, was renamed Science Communication in 1999 under a different, less policy-specific and more natural science-oriented editorial purview.
With this background, the current review set out to answer five ‘methods-oriented’ questions about recent research on data `dissemination and utilization:
- What kinds of questions have been posed about evidential use during the past decade and a half in light of policy-making and managerial practice?
- How is the “evidence”defined and operationalized as “used” in recent studies of evidence-based decision-making?
- What instruments are scholars now employing to document or measure the use of data?
- How is the policy-maker or educational ‘reader’ depicted as sampled in recent studies?
- What conclusions about ‘the use of evaluative data’ have been recently been drawn in light of the unit of studyadopted by researchers and evaluators?
The present synthesis does not pretend to be comprehensive of all empirical research conducted over the past decade and a half into ‘data use’ questions. Rather, a purposeful sample (Patton, 1990) of published articles and insider accounts has been drawn from this literature to illustrate diversity and variation in inquisitorial approachesacross more than one discipline among educational policy makers and managers. Thus, among the interrelated ways of ‘knowing’ that Aristotle identified in pursuit of ‘knowledge’— theoria, technė and praxis—my focus is on the latter two in service of the former in scholarship. My overall aim is to highlight the range of research techniques recently adopted to investigate the phenomena of ‘data’ or ‘research use’. By isolating the questions now being posed and outlining the methods of inquiry recently adopted, my care has been to: demonstrate the span of inquiry, rather than its accumulated depth; to depict current operational assumptionsrather than to identify best practices as revealed in findings; to describe different research techniques rather than to pass judgment on (in)appropriate methods.
In part, this decision to look laterally stems from a dearth of empirical research directly addressing ‘data use’ questions by school principals during the past twenty years; in part, the choice stems from the desire to consider how and where scholars in fields adjacent to educational leadership and policy making are now pursuing their inquiries for indications of uncharted territory.This objective then led to my decision to include studies from cognate fields and to a further question:
- What recent lines of inquiry hold promise for future studies of data use in management or evidence-based leadership?
The review entailed extensive searches of relevant databases for education, public administration, evaluation utilization, and management, namely ERIC, ABI Global Forum, and Google Scholar. Hand searches were also carried out. The searches were extended from educational leadership to include a broader management literature for three reasons. First, since initial searches revealed only four recent studies directly engaging school administrators, it was deemed appropriate to consider studies on data utilization’ from a broader perspective. Second, several scholars have noted that recent studies of ‘data use’ by policy makers and managers straddle several disciplines. While there may be a hierarchy of evidence in health care, educational research sits at the confluence of different paradigms which hold contrasting views about the probative value of different kinds of ‘evidence’. And third, I posited that issues in ‘data use’ by school administrators share shares important similarities with those in other management or scientific fields.
Searches of electronic and on-line data bases using thesaurus search terms were carried out and tracked during the February 2004 to December 2005 period. The intention was to ensure that, as far as possible, the major scholarly pieces were identified, while keeping the searchlight on those studies deemed pertinent to the questions posed above. The search was limited to publications since 1990 because several literature reviews covering the two antecedent decades have been prepared. Through such a meta-review, the goals were to identify limitations and suggest new methodological approaches and to sketch prospects for future research in ‘evidence-based decision-making’ in educational administration.
The initial search yielded 134 papers (empirical, theoretical and opinion) selected for further scrutiny from among over 5000 citations. All searches were documented and requests for literature were tracked using a data base. The review considered English language reports from the United States and Canada. Although there existmanyinteresting studies of ‘research use’ in the United Kingdom, continental Europe and Israel, the supposition was that North American school systems share many structural characteristics not present across the Atlantic. Australian findings have been considered asa comparison point for American school administration. One un-refereed, government-commissioned publication and a dissertation were included because they provide the results from recent systematic inquiries not yet presented in scholarly outlets. Those studies treating higher education, nursing and medicine were generally excluded, except for one dealing with executive behavior in hospitals; my rationale here was that strategic management principles are usually deemed applicable across senior levels of administration, whereas middle or program management is customarily considered as constrained by structural characteristics unique to each sector. More controversially perhaps, I included several reports of investigations conducted within private sector firms within this purview. Private and public sector interests differ, and those organizational features of ‘knowledge transfer’ or ‘evaluation utilization’ may have different ends than those in public administration. However, I ultimately drafted three conducted in private firms or large-multinational corporations because their methods and designs appeared transposable with minimal adjustment to inquiries within a large-school setting or the education sector.
Each paperof these papers was then given three detailed readings, animated successively by the purposes of synthesis, analysis, and appraisal. In the first, a standard framework was adopted to ascertain and extract key features in each study: the research questions, operational definitions, sample instrumentation, design and analytic method, findings, conclusions and implications. In the second stage, the studies were re-read not for their extrinsic applicability or generalizability to larger utilization issues nor their alignment with broader theories, nor less to trace the sociological contours of ‘knowledge creation’. Rather, the aim was to describe and compare the intrinsic elements, primarily the internal consistency or rigor by which the researcher linked purposes with questions, operational constructs, instrumentation, data collection methods and analytic procedures. And finally, the articles were appraised from an evaluative point of view, using the quality criteria suggested by McEwen (1992) for maximizing research use, but modified for the array of methods employed: utility; propriety; accuracy. In doing so, the presumption was that scholars of research use would seek to maximize ‘use’ of their own findings, and thus would be amenable to an appraisal of their work on its own terms.
The twenty research papers eventually chosen for detailed study thus met the following criteria: 1) whether the study illustrated either a particular or peculiar definition of the constructs under investigation; 2) whether the research method deployed was unique in one or more features; and 3) whether the line of inquiry was innovative, not because of its theoretical premises, but because it posed new questions about the larger utilization problem. ‘Empirical research’ was considered to include any published article informed by systematic observation or experience, thus embracing both quantitative and qualitative inquiries.
In this meta-review, I argue that ethnomethodology, narrative case studystudies,and autobiographical or historical methodshold promise for addressing themajor unanswered issues in the field of study. Those are questiontasks relate to s of managerial practice, not of executive policy making, namely:1) what are the processes by which a datum is transmuted from information to knowledge to practical actionand 2) what is the role that research/assessmenthave has in informing or improving middle-managerial behaviourpractice. Because of notions of ‘knowledge’ and , because “usage” questions have become increasingly more complex, because the processes by which a datum is transformed in userequire description, because the structure of Canadian education systems is variegated, and because of the immense variety in position and circumstance among administrators or managers in the North American administrative hierarchy, qualitative methods appear mostamenable to exploring the microdynamics of data use. Thus and perhaps ironically, the problems of quantitative proficiency, or numeracy, may be best plumbed through methods that are essentially literary in nature.
Delineating purposes and research questions
Scholars of data use bring an underlying purpose, sometimes explicit but often implicit, to their study. This ‘animating impulse’ leads them to frame their research questions in ways that achieve those intentions. Among those studies undertaken in the past decade and a half, a fundamental distinction can be drawn between those whose purposes relate to issues in grand policy-making in public administration and those addressingmanagerial practice.In Great Britain, there is a tendency to regard management as the higher function, but in North America, “administration” is generally held to be hierarchically superordinate to and subsuming of “management”. The former deals with executive issues in policy making and general strategy;the latter is concerned with middle echelon and material matters such as program implementation. Whereas the public administrator deals with the translation of politics into policy principles, the manager addresses issues that execute policy through programmatic practice (Hodgkinson, 1991). Although the distinction may be useful for identifying whether data informs policy or practice, that dichotomy is particularly difficult in the North American education sector because of the combination of public administrative and managerial roles that a dean, a school principal (headmaster), deputy minister, ministerial assistant or policy analyst adopt in decision-making.
Figure 1: Overview of Research Questions/ Purposes in Selected Data Use Studies,
North America, 1990-2005
AuthorYear / Research Questions / Purpose / Method of Inquiry
Bedard (1999) /
- How do blue-ribbon panels construct knowledge about educational reform, how do they perceive multiple purposes, and how should research be undertaken in policy-making?
Berman & Wang (2000) /
- What is the capacity of U.S. county level administration to undertake performance measurement?
Biddle & Saha (2002) /
- What is school principals’ use of educational research knowledge?
- What are the techniques employed to acquire their knowledge?
- How do principals evaluate research?
- What do principals know about research knowledge, and how it might have been used in their schools, if at all?
Cousins & Walker (2000) /
- To what degree do experience and self-efficacy, prior course work, gender and organizational capacity predict attitudes toward systematic inquiry in schools?
Englert, Fries, Goodwin, Matin-Glenn & Michel (2004) /
- What policies and practices are principals using to meet new accountability demands?
- Are principals using policies and practices that research and literature have identified as being effective?
- Do policies and practices differ between low and high performing schools?
Goering & Wasylenki (1993) /
- How can academic evaluators assume multiple roles to bridge the gap between research and program renewal?
Graham & Neu (2004) /
- How did Alberta government use large-scale testing to help construct governable subjects?
Landry, Amara & Lamari, (2001) /
- What is the extent of social science research use in Canada?
- Are their differences among social science disciplines in use?
- What are the determinants of use?
Landry, Lamari, & Amara, (2003) /
- To what extent is university research used in government agencies?
- Are there differences between the policy domains regarding extent of use?
- What determines the use of university research in government domains?
Levin (2001) /
- How does research shape agenda-setting of government?
Mohrman, Gibson & Mohrman (2001). /
- How are the roles of joint interpretive forums, perspective taking and self-design informed by research results in determining practitioner perspectives of usefulness?
Rich & Oh (2000) /
- What is amount of use for 12 information types in political and technical decision-making about service provision and financing?
- How much do decision-makers think information is helpful in defining policy problems, designing programs/policy options, choosing specific course of action/ justifying policy programs?
Roth & Bowen (2001) /
- How do professionals read, make sense with and interpret graphic representations of natural phenomena?
Szulanski (1996) (2000) /
- What are impediments to transfer of knowledge of best practice within a firm?
Thomas, Clark & Gioia (1993) /
- How do senior managers make sense of information and how they act to influence organizational outcomes?
Torrence (2002) /
- How do American elementary school principals vary in their use of data as a tool of instructional leadership?
- What is the relationship between data use and personal and environmental variables?
Wainer, Hambleton & Meira (1999) /
- What is assessed performance of readers with alternate displays for National Assessment of Educational Progress findings for policy makers?
Willinksy (2003) /
- What is the role of online research in the workplace of policy makers?
- What are obstacles to its use?
Zellner & Fornahl (2002) /
- What are the types and nature of knowledge and how do they differ from information?
- What are the implications for maximizing both knowledge and information transfer?
Policy-making