'Perceptions of the UK's Research Excellence Framework 2014: a media analysis'
Tony Murphy, Department of Law and Criminology, Sheffield Hallam University
WhiteHouse 251, Collegiate Crescent, Sheffield, S10 2BP
Daniel Sage, School of Applied Social Science, University of Stirling
Stirling, FK9 4LA
Abstract
This paper explores perceptions of the UK's Research Excellence Framework (REF) and its implications for individuals, institutions and wider academia through an analysis of media coverage of the REF over a two-year period. In recent years, the importance attached to the REF has become an increasing focus of concern for academics and other commentators, particularly vis-à-vis issues such as staff morale, funding, ‘impact’, working practices and institutional inequalities. In examining media stories related to the REF, we uncover three key findings in terms of the main ways in which the REF is presented and perceived. First, media discussions of the REF are overwhelmingly negative, although this appears to have abated over time. Second, there are significant differences in the how the REF is discussed; this variation stems from characteristics such as the disciplinary background, institutional type and professional seniority of commentators. Third, although there are a wide range of themes discussed in relation to the REF, several tend to dominate media portrayals: these include ‘impact’, ‘funding’ and ‘marketization’. The implications for the role and legitimacy of research assessment processes are discussed.
Key words
Impact; intellectual freedom; pressures to publish; Research Excellence Framework
Introduction
This paper considers perceptions of the UK's Research Excellence Framework (REF) in terms of its implications for individual academics, institutions and wider academia through an analysis of media coverage of the REF over a period of two years. An earlier paper by Hare (2003) highlighted the possible effects of such a process for assessing research quality and the allocation of research funding in the earlier context of the Research Assessment Exercise (RAE). This paper draws on much of that analysis and utilizes media data to explore the various contexts in which the REF is discussed, the extent to which perceptions are positive or negative, and what this suggests about the effects of the REF in terms of shaping the HE landscape. The literature in this area, including Hare (2003), Bowring (2008), Wells (2013) and others, suggests that such assessment processes bring with them adverse outcomes for staff morale and working practices and even the very development of knowledge itself. Similarly, recruitment practices and inequality between institutions, as well as within institutions, appear to be related to such processes. We explore the possibility of these outcomes and others in the course of our research, finding that the media coverage of the REF is generally negative, with the effects on individuals, as opposed to institutions and general academic practice, resulting in the most negative score. However, reporting on the REF is most often in relation to the effect on institutions, where 'impact', 'funding' and 'marketization' are frequently cited. We also uncovered concerns for how the REF has been envisaged as affecting academics, as well as academia more broadly. Our analysis suggests for example that there is a very real sense amongst academics that the REF is adversely shaping the nature of research itself. Here, reporting on the REF signals heightened pressures to publish and a developing culture of disregard for certain types of research in favour of short term ‘REF-able’ work.
Research background
Over the past two decades, governments from around the world have increased the extent to which university research is measured and evaluated. According to Ab Iorweth (2005), this is part of a long-term, inexorable shift in the relationship between science and government/society, in which – on two counts - the former is increasingly accountable to the latter. First, in an era when public expenditure is subjected to ever greater scrutiny, researchers are under increasing pressure to justify their share of public resources. Second, and linked to this process, politicians now demand that research must show itself to have demonstrable public and policy benefits. As a consequence of this, governments in many rich countries with extensive higher education sectors have developed evaluation mechanisms to inform decisions regarding research funding. Purportedly, there are numerous advantages to such ‘performance-based’ systems, such as improved research performance, higher quality output, greater public accountability, closer links between academia and policy-making and more effective and efficient targeting of funding (Ab Iorewth, 2005). In practice, methods for evaluating research output vary across those countries where evaluation exercises have been introduced, ranging from the analysis of various indicators – such as publication output and citations – to peer review and case studies.
The UK’s Research Excellence Framework (REF) is one such mechanism for assessing the quality of research within UK universities, presently over a six-year cycle. The REF replaced the Research Assessment Exercise (RAE), which had served the same purpose up until 2008. The REF will produce assessment outcomes for submissions with the view to informing allocations of funding from funding bodies, accountability for public investment, and the development of ‘benchmarking information’ and ‘reputational yardsticks’ (REF, 2012). Thus, there are significant funding and reputational consequences. REF results are also set to inform university rankings and the trajectory of academic careers, as have results from the RAEs. Building on the RAE, the REF has brought with it a stronger emphasis on research ‘impact’, which broadly relates to the extent to which research has a bearing on wider society (HEFCE, 2010). ‘Impact case studies’ form an important part of institutions’ submissions. However, the notion of ‘impact’ itself, and how one might assess this in a meaningful sense has been the subject of debate throughout the REF 2014 cycle (for example see Bishop, 2013, Curry, 2013).
The first REF cycle has involved the assessment of research undertaken between 2008 and 2013 and this culminates in the publication of results in December 2014. The REF 2014 has 36 units of assessment, each assessed by sub-panels of subject experts. Submissions from universities are then placed in to one of four categories of overall quality (‘one-star’ to ‘four-star’, plus an ‘unclassified’ category), determined by the weighted sub-profiles of ‘output’, ‘impact’, and ‘environment’. ‘Four-star’ denotes research that is ‘world-leading’, whilst ‘one-star’ denotes research of national recognition (ref.ac.uk, 2012).
During the course of the present REF cycle, reports suggest that academics from many disciplines feel heightened demands at work, particularly in relation to pressures such as publishing and demonstrating ‘impact’ (Davey, 2013). In particular, the REF process has raised questions about how academics and higher education institutions experience and interpret these new challenges and demands (Murphy, 2013). As the REF’s objectives imply, there is a lot riding on these assessments, such as the allocation of research funding, academic reputations and public accountability (REF, 2012). This article considers the effects of such pressures on academics, institutions and academia more widely through an analysis of media reports related to the REF.
Authors have signalled adverse effects on the nature of research being undertaken, how it is reported and the kind of knowledge being developed as a result of increased pressures to publish (see Fanelli, 2010). Similarly, Miller and Sabapathy (2011) remark on how the ‘impact’ agenda associated with the REF has influenced publishing habits and the kind of work being undertaken. Similarly, Moriarty critiques the research councils’ role in shifting academic work to focus on ‘problems of direct short term socioeconomic interest’ (2011:57). Furthermore, Smith and Meer (2012) have argued that through measuring and rewarding certain outcomes the REF will inevitably shape behaviours, and worthy intentions do not guarantee desirable results. Rewarding evidence of ‘impact’ may encourage academics to focus on producing research that is relevant to short-term policy agendas and thus narrow opportunities for research focusing on longer-term yet potentially still policy-relevant issues. More worryingly perhaps, Martin (2011) has argued that measures for assessing research performance have become increasingly complicated. In fact, it is argued that it is the case that the ‘costs’ of such a system now exceed the benefits, with the prospect (and perhaps reality) of ‘game-playing’, where academics are encouraged to game the system to their best ability. This is in contrast to any real advancement of knowledge, contrary to one of the core functions of the HE system (see Holmwood, 2011).
Writing earlier in the Guardian in the context of the RAE, Bowring (2008) claimed that the pressure to publish was hurting universities, stating that it is perverse to value research publication so highly that it threatens teaching standards. Bowring (2008) argues that this is precisely what has been happening over the last 30 years. Bowring noted how he himself had been pressured to publish an article too early because it was needed for an RAE submission, and colleagues of his had resigned early for similar reasons. Purportedly then, the RAE thus narrowed choices and perspectives and penalized those who wished to open up new areas of interest. According to recent critiques, the REF is having comparable effects. For example, writing for the LSE Politics and Policy Blog in early 2013, Wells (2013), a notable commentator on the REF, discussed the extent to which the process can adversely affect staff morale, as well as the very nature of how research itself is conducted. Echoing what others have noted of the process, Wells claimed that ‘Publication becomes the primary end, not knowledge creation or intrinsic interest in the subject’.
The literature to date thus demonstrates that academics are under to pressure to publish work and respond to the demands of the REF and earlier assessments, certainly if they are to progress careers and avoid adverse performance management outcomes. Furthermore, the literature also demonstrates that there is a strong possibility that the very nature of the research being undertaken is governed as much by what the rules and processes of research assessment exercises mean for what is ‘REF-able’ and what will count, as any underlying academic freedom or curiosity. Hare (2003) explored the possible implications of the RAE in relation to individual academics, departments, institutions and the wider process of assessment and resource allocation itself. In short, Hare (2003) raised many questions concerning how such assessment exercises might shape behaviours in relation to research, and whether this would be positive for the sector and the very development of knowledge itself.
In this article we explore these perceived pressures through an empirical analysis of media stories related to the REF. In doing so, we aim to highlight some of the most common pressures and problems related to the REF process, thus signalling the perceived nature of the assessment’s effects.
Aims
The overall aim of the study is to consider if and how the REF has had, or is perceived as having, effects on individual academics, institutions, and the wider process of academic research. Through an analysis of media articles we explore what press coverage of the REF suggests in relation to
- Implications for individual academics
- Implications at the level of institutions
- Implications for the nature of academia
Within this, we also consider whether there are differences in perceptions between different kinds of commentators, different types of institutions, and different fields of study.
To achieve these ends, we sought to collect data on the following:
- Prominent themes emerging from media coverage of the REF, the commonality of these, and who is writing about them.
- The level of ‘positivity’ in the discussion of such themes and if this varies according to different types of authors, institutions and fields of study.
- Connections between themes; the extent to which one theme is cited during the discussion of another.
- Whether there has been a change in the nature of debate over time, particularly with regards to ‘positivity’.
Methodology
To achieve these aims, we undertook an analysis of media articles related to the REF. This included both editorial and commentary pieces from academic contributors and journalists writing on the subject. We wanted to consider how the REF has been represented, in which contexts the REF was discussed, and whether such representations had been positive or negative. Using the Nexus UK database we collected all articles citing the term “research excellence framework” in all UK media publications between the period 1st July 2011 and 30th June 2013. From the articles collected, we sought to explore the key issues and topics discussed in relation to the REF, how they relate to one another, and whether or not this varied according to author.
Prior to the main pilot work we explored the literature around the topic in order to acquire a sense of the issues and themes of possible relevance. In effect, this meant that we had conducted very early pilot work by reading many of the articles we would subsequently collect via the Nexus UK database. From this, we developed a list of themes and issues that seemed to be common or reoccurring. Next, our pilot work involved both researchers analysing a standardized sample of articles to identify which themes were evident within those articles, and the extent to which the coverage of the REF was positive or negative. We assigned positivity scores on a scale of 1-5, based on how the REF was reported on by the author. We then considered whether our responses matched. We initially sought to achieve a minimum of 70 per cent matching between our efforts before progressing to the data analysis proper. However, this proved to be quite difficult in practice. Indeed, we conducted 3 pilot phases, each time modifying our approach. In the end we moved from simply identifying all issues discussed within a single article, to selecting a maximum of 3 themes – just the most prominent ones in the article. This sometimes resulted in just 1 or 2 themes being highlighted in some articles. We also collapsed some themes in to others where there was sufficient overlap, and changed others where there was enough ambiguity to cause unnecessary error. After such changes, our final pilot phase resulted in an 80 per cent matching rate in our responses in relation to both the themes and positivity scores.
Eventually, we developed a framework consisting of 12 themes which informed the final pilot work. Then, following a very strong matching for the pilot sample, this informed the main data analysis process. In total, we collected 341 articles, 109 of which were disregarded because they cited the REF in a peripheral or non-relevant context.
Results
Table 1 shows descriptive statistics for all of the variables collected as part of the media analysis. Following a qualitative examination of a sample of media articles during the pilot phase of the study, twelve themes were identified as part of three broader categories of interest: implications for individuals (performance management; recruitment policy; staff morale; academic and management tensions); implicationsfor institutions (institutional inequalities; funding; impact; marketization/quantification); and implications for academia (intellectual freedom; pressure to publish; dishonest practice; disregard for certain research). Table 1 shows that ‘impact’ was the most prevalent issue related to the REF during the past two years, with 91 mentions in all 232 articles (39 per cent). Other prominent concerns included ‘funding’ (20 per cent), ‘marketization/quantification’ (18 per cent) and ‘recruitment policy’ (16 per cent). All in all, the analysis suggests that the largest concerns are associated with the implications of the REF for institutions. Out of a total 420 mentions of the 12 themes across the 232 articles, 49 per cent of these were associated with the implications of the REF for institutions. Correspondingly, 27 per cent of mentions were associated with the implications of the REF for academia and just 24 per cent on the implications forindividuals.
In addition, Table 1 also shows the different characteristics of the 232 articles. Due to the aims outlined above, we identified the following characteristics of newspaper articles: (1) newspaper type; (2) author profile; (3) author background; and (4) author institution. We also assigned a positivity score to each article, ranging from 1 (very positive) to 5 (very negative). The results from Table 1 show that the vast majority of articles related to the REF come from the Times Higher Education (82 per cent), followed by the Guardian (13 per cent). Somewhat surprisingly, the REF was barely mentioned in other British newspapers during the two-year period. There was, for example, just one reference to the REF in both the Times and the Telegraph. This strongly suggests that the REF is far from a mainstream issue for the UK media, remaining a specialist news story with little diffusion amongst the wider public.