Forthcoming:
Shortall, S. (2013) Sociologia Ruralis Vol 52 No. 3 2013
Using evidence in policy: The importance of mediating beliefs and practices
Abstract
This article argues that to understand the use of evidence in policy, we need to examine how meanings and practices in the civil service shape what is accepted as knowledge, and how differences between the beliefs and values of the academy and the polity can impede the flow and transfer of knowledge. It considers the importance of social context and shared meanings in legitimating knowledge. Who counts as legitimate knowledge providers has expanded and here the role of stakeholder groups and experiential knowledge is of particular interest. How hierarchy, anonymity, and generalist knowledge within the civil service mediate the use of evidence in policy is examined. The difference in values and ideology of the civil service and the academy has implications for how academic research is interpreted and used to formulate policy and for its position in knowledge power struggles. There are particular issues about the social science nature of evidence to inform rural policy being mediated in a government department more used to dealing with natural science knowledge. This article is based on participant observation carried out in aUK Department of Agriculture and Rural Development.
Introduction
‘A good many times I have been present at gatherings of people who, by the standards of the traditional culture, are thought highly educated and who have with considerable gusto been expressing their incredulity at the illiteracy of scientists. Once or twice I have been provoked and have asked the company how many of them could describe the Second Law of Thermodynamics. The response was cold: it was also negative. Yet I was asking something which is the scientific equivalent of: Have you read a work of Shakespeare’s?’
C.P. Snow (1959)
The study of the civil service and its role in creating social order is a well-established research topic for social scientists. It is central to political studies, social policy, the study of public administration, the study of bureaucracy (Weber, 1947), and power struggles between technocracy and ideology (Habermas, 1970). This research has largely focused on hierarchy, bureaucracy, how public policy is formulated and relationships between civil servants and politicians. With the recent emphasis on evidence-based policy, and the need for academics to demonstrate the use value of their research, there is now a research imperative to reflect on how the beliefs, values and ideology of the civil service mediate what is constructed as knowledge to inform policy.
Academics are increasingly funded by research bodies to work in government, the private sector and the third sector to provide evidence to inform particular policy questions and problems. As a result of this an increasingly sophisticated academic body of knowledge has developed reflecting on the complexities of evidence-based policy. This has developed from the various ways academics are now engaged in the policy process, for example; as policy advisers (Stevens 2011; 2007; Wilkinson, 2011; 2010), through systemic reviews of policy documents (Monaghan 2009; 2010), through reviews of Independent Commissions (McLaughlin and Neal, 2007), and by comparative analysis of the ways evidence and policy interact across Nation States (Denzin, 2009; Denzin and Giardina, 2008). While evidence-based policy sounds intuitively to be a good thing (Hammersley, 2005), this recent body of knowledge demonstrates the difficulties and complexities of the idea. Surprisingly, these recent debates make little direct reference to the Sociology of Scientific Knowledge (SSK), despite the fact that this body of research directly questions the social construction of knowledge, the importance of context, and the mediating role of values, beliefs and ideology (Collins, 1983; Yearley 2009).
The ESRC in particular has recently funded a number of Knowledge Transfer Research Fellowships, placing academics in the civil service, industry and the third sector, to foster the transfer of academic knowledge into these environments. This has contributed to the nuanced understanding of how evidence is used in policy detailed above. As a result of this some very rich ethnographic studies of the behaviour of civil servants in particular contexts, how practice of policy formation differs from the rhetoric, and how hierarchies are established through practices and behaviours have emerged (Stevens, 2011; Wilkinson, 2010; 2011). While related to but not part of the more recent tradition, the work of Rhodes (2011) and Bevir and Rhodes (2003) has followed an ethnographic approach which they term ‘interpretivism’. Rhodes (2011) specifically makes an argument for the importance of observation as an important tool for social science research (p.6). To date, the study of the different meanings, values and beliefs of the civil service and the academy in how social knowledge is constructed has not been considered. This is the subject of this article.
The article focuses on rural policy. There is some confusion across the UK government about what rural policy is, and this was also the case in the Department of Agriculture and Rural Development in Northern Ireland (DARD). Often it is unclear whether rural policy applies to all rural areas, or is targeted at disadvantaged rural areas. Nor is it clear whether the policy goal is to embed rural into all policies, or to develop specific rural policies. Part of the difficulty arises because a government department is given responsibility for an area where it does not have the policy instruments to deliver the relevant policies (for example, rural schools, rural roads, rural employment). Academic research has also debated what rural policy might be (see for example, Gray, 2000; Marsden, 1998; Shucksmith, 2010). A House of Commons Report (2008) chastised the UK Department for Environment Food and Rural Affairs (Defra) for a lack of clarity on whether its focus was rural or disadvantage in rural areas; a lack of a clear rationale for why its focus is rural; ‘woolly’ objectives for what it was trying to achieve in rural areas; a rural proofing process that was not rigorous or systematic or clearly identified policies that should be rural proofed; and that failed, through its blanket rural approach, to acknowledge the diversity between and within rural areas.
An added difficulty is the different nature of the ‘Two Cultures’ which C.P. Snow described in his Rede lecture and later published as a book (1959). While Snow was more concerned that the humanities have been over-rewarded at the expense of the natural sciences, my experience is that social science to inform rural policy is often seen as woolly, interpretative and always open to debate. Rural policy tends to be housed in departments of agriculture and here there is less of a tradition of dealing with social scientific knowledge to inform policy, and more of a tradition of dealing with the natural sciences. This has implications for the legitimacy assigned to different social science knowledge providers, and how to integrate knowledge into policy.
This article begins with an overview of the recent literature on the use of evidence in policy, and some of the key elements of the Sociology of Scientific Knowledge[i] that explore how the construction of knowledge is related to the social circumstances of its production. Next an overview of the UK civil service is presented, paying particular attention to the peculiarities of the Northern Ireland civil service, where this case study takes place. Next the methodology, participant observation, is reflected on. The case study and findings are presented.Rural policy continues to create some confusion in a government department that is more used to dealing with primary industries and the natural sciences. The rural policy making context is confused and how to use evidence within the rural policy sphere is unclear to civil servants. It is concluded that we need further reflection on how the different meanings, values and beliefs of the civil service and the academy shape how they interact with each other and determine how social knowledge is constructed in a policy process.
Evidence and knowledge: an overview of the literature
There is nothing new about the idea that policy and practice should be informed by the best available evidence. Nonetheless, the current high profile emphasis on using evidence-based policies in the UK can initially be traced back to outside the academy in the first instance to the Blair administrations of 1997 and 2001. Reforming and modernising the machinery of government was a central part of their agenda and this emphasised a commitment to evidence-based policy (Davies, 2004; Nutley et al, 2002). The Modernising Government White Paper (Cabinet Office, 1999) stated that government policy must be evidence-based, properly evaluated and based on best practice. The academy subsequently embraced the idea, with the ESRC establishing the Evidence Based Policy Unit, and Burawoy’s (2004) call for a public sociology which has generated enormous sociological debate. However, early research on evidence-based policy found the concept to be deeply problematic and to display a lack of understanding of how the policy making process occurs (Nutley, 2003; Nutley et al, 2003; Pawson, 2006). The literature on evidence and policy is enormous, with a specifically dedicated journal to the topic. Only a number of aspects of this literature are highlighted here (for a more thorough review, see Author, 2012).
Much research has highlighted that there is not the logical process of absorption of evidence into policy that textbook analyses suggest (Monaghan, 2009; Wilkinson et al, 2010). Evidence is used selectively, to support a preferred argument (Tittle, 2004; Stevens, 2011). The policy context can impact on how evidence is judged or absorbed; in ‘adversarial’ policy areas such as drug policy, the tensions between normative beliefs and evidence are magnified (Monaghan, 2010), or in times of crises, such as disease outbreak, scientific evidence is valued differently and more rapidly absorbed (Wilkinson, 2011). The power struggle between normative and empirical knowledge has also been explored, and how each establishes the legitimacy of their truth claims (Shortall, 2012). Research has considered whether it is the power of the idea (Stevens, 2007) or the power of the supporter (McLaughlin and Neal, 2007) that matters. There can be little doubt that what matters is who has the power to decide what counts as evidence.
The idea of who counts as legitimate knowledge providers has expanded enormously. Savage and Burrows (2007) speak of the coming crisis for empirical sociology brought on by the huge number of providers of sophisticated quantitative evidence that now reside outside of the academy. Similarly qualitative sociology no longer provides the only source of case study material or in-depth interviews. These are techniques also used by journalists, lobby groups and stakeholder groups and the legitimacy of ‘experiential’ evidence has developed since the public disputes between conflicting forms of scientific evidence (Collins and Evans, 2003). Science is disputed, most noticeably around global warming and genetically modified foods. Generating more evidence around these questions does not lead to a conclusive decision; it generates more debate (Oreszczyn, 2008). Including stakeholder experience in these instances, broadens the sources of knowledge and evidence relating to such policy questions, and gives greater legitimacy to policy and political decisions (Porter and Shortall, 2009; Shortall, 2012).
While many of the tenets of SSK are evident in the evidence and policy literature, they are not as overtly or as sociologically discussed as they might be. SSK shows the link between the social and what we accept as knowledge. What we ‘know’ we do so within an existing body of knowledge given to us by our society, and our organisations. Ideas are generated within our social structures and come from our collective habits (Mills, 1939; Shapin, 1995). What is viewed as knowledge depends on trust, judgement and our values (Yearley, 2009). As Shapin puts it ‘knowledge that corresponds, or coheres ...is deemed the right stuff’ (1999, p. 1).
Beliefs come from our backgrounds, but not only our social backgrounds; what we accept as knowledge depends too on the profession in which we were socialised (Bevir and Rhodes, 2005; Shapin, 2012). Sociologists try to establish the origins of beliefs and how knowledge is accepted, and part of this will involve examining the background and professional socialisation of people offering types of knowledge. Comparative research also sheds light on this question of professional socialisation (Shortall and Warner, 2012). What is accepted as truth depends on conventional rules. To paraphrase Mills (1939; p. 674) to say something is illogical is similar to saying something is immoral; they are both deviations from the norm.
Until the late 1980s, Northern Ireland, like many parts of Europe did not have a specific rural policy. In 1988, the European Commission published a document titled Future of Rural Society. This document heralded a significant policy shift to begin reforming the Common Agricultural Policy and alter the sectoral policy of only funding agriculture to instead also funding area-based rural development. Governance was devolved, and the Department of Agriculture was given responsibility for the rural development programme. This government department was not ‘socialised’ for this new responsibility. There were no conventional rules to help civil servants establish the truth about the direction of rural policy.
Reason and experience are vital parts of the story of knowledge (Bloor, 1998). It is argued that the making of knowledge is a mundane affair, established in face to face interactions, the acceptance of truths, and in the familiar (Shapin, 2009; Mills, 1939; Rhodes, 2011). Knowledge is constructed and accepted within a given social structure. This article will shortly turn to the use of participant observation to examine the way in which meanings and practices in the civil service shape what is accepted as knowledge, and how differences in beliefs and values of the academy and the polity can impede the flow and transfer of knowledge. But first, the social structure of the UK civil service is considered.
The UK civil service
Bureaucracies are historically constituted and differ from place to place (Dahlstrom et al, 2010; Barzelay and Gallego 2010; Painter and Peters 2010). The model of the UK or Whitehall civil service is also predominantly the model of most British ex-colonies (Hardiman, 2010). For almost two hundred years, the need for an English civil service that is efficient, permanent, and apolitical has been accepted (Vandenabeele et al, 2006). It provides stability to parliamentary governments that change. While governments with different ideologies and values come and go, the civil service remains intact, and ideally impartial. It is a permanent bureaucracy separate to government that provides the main policy advice to government members and it is responsible for implementing policy (Vanderabeele et al, 2006).
The UK civil service is quintessentially Weberian (Chapman and O’Toole, 2009). Many of the founding principles remain. It is very rule-based and very hierarchical in structure (Wilkinson, 2011).The civil service is a ‘command and control’ hierarchy, where those in authority control the work load of junior colleagues (Bordua and Reiss, 1966; Behn, 1995). Authority is ritually reinforced through formal and informal practices (Wilkinson, 2011; Rhodes, 2011).Those in authority appraise junior staff. Officials are esteemed because of the hierarchy, authority and power of their office (Weber, 1947). They pursue careers that bring them higher up in the hierarchy of their bureaucracies.
Throughout this article, I refer to the ‘ethos’, ‘structure’ and ‘culture’ of the civil service. When I do so, I draw on Weber (1979) and Bourdieu (1990). While Weber did not specifically refer to ethos, he described values peculiar to a specific people, culture or movement, and a collective self-representation that is characteristic of a group of people. Bourdieu’s concept of habitus refers to the values and ways of interaction of particular social groups that are acquired through the activities and experiences of everyday life. Habitus is dependent on history and human memory. The civil service imbues civil servants with a particular way of knowing and self-representation. This is acquired through every day practices as will be described later in the article.
Some government departments have civil servants who are technical advisers, scientists, specialists or ‘experts’. Policy making civil servants on the other hand are not specialists, rather their expertise is competency based and they are experts on the workings of the civil service. Wilkinson (2010) notes that the hierarchy between experts and policy makers is clearly defined and rigidly maintained; policy has higher status (p. 1). In the Northern Ireland Civil Service, technical experts can only advance up the hierarchy so far, and to progress beyond that point, they must become generalist policy makers. As Stevens (2011) notes, civil servants move between policy areas in which they not specialists.Developing competencies was a key element of New Public Management Reforms (Horton, 2010) and civil servants are promoted to and make sideways moves to different policy areas. Sideways moves are often undertaken to develop the generic competences needed to progress to the next grade in the civil service. They are promoted on the basis of their competence in understanding how the civil service functions and their competence to solve problems within it and design policy, rather than on the basis of specialist knowledge of a particular area. Hardiman (2010) has identified the process of sideway promotions as problematic, arguing that while it was intended to widen the talent pool, it has the unintended consequence of dissipating the skills base because the specialist policy understanding built up in one departmental area does not necessarily translate to another area.