REF 2014, HE Pedagogic Research and Impact
Final report - October 2015
[Pedagogic research] ‘was valued highly in our submission (about half higher education, half other phases) perhaps because I have worked in all phases and value it all and see much theoretical work as generic although contexts produce different environments in which to test out and experience theory in practice. But actually I know other departments could have submitted HE pedagogy but didn’t value it, both in our uni and elsewhere …’
‘Despite our hard work of our team on pedagogic research and publications over many years, the Uni's approach was pretty much to dismiss it. I worked long and hard with the Education professors building the narrative for that UoA but couldn't get our publications taken seriously. Only one pedagogic researcher from a faculty did get into the Education UoA.’
‘I just don’t know about impact I feel very ambivalent about it because I do appreciate all the arguments about why it’s important but I think once you start measuring something you start defining it in ways that the rigidity is not always helpful.’
‘And I think one of the issues with impact, I remember going back a bit now, reading through this stuff, the Guidance on impact in the REF, and it was saying if it impacts just within HE then we're not particularly interested in that, we're looking at impacts outside HE, and of course that's always tricky isn't it for pedagogic research to demonstrate?’
Professor Pauline Kneale
Professor Debby Cotton
Dr Wendy Miller
Acknowledgements
This project was funded by the HE Academy, ASC-2053, to explore the ‘Research Excellence Framework Impact’ rules on the submissions of higher education pedagogic research to Education (Unit 25).
The research was undertaken by the authors, through the Pedagogic Research Institute and Observatory (PedRIO) at Plymouth University. The research involved desk-based analysis and interviews with 15 HE colleagues at 13 HE Institutions, with various roles and perspectives in relation to the REF and pedagogic research.
The research team is very grateful to all colleagues who gave their time to explain their engagement with the REF processes.
Abbreviations
ECU / Equality Challenge UnitFTE / Full time equivalent
GoS / Guidance on Submissions
HE / Higher Education
HEFCE / Higher Education Funding Council for England
HEI / Higher Education Institution
NSS / National Student Survey
NUS / National Union of Students
OERs / Open Educational Resources
OFFA / Office for Fair Access
PedRIO / Pedagogic Research Institute and Observatory, Plymouth University
QAA / Quality Assurance Agency
QR / quality-related research
RAE / Research Assessment Exercise
RCUK / Research Councils UK
REF / Research Excellence Framework
SP / Sub Panel
SHRE / Society for Research into Higher Education
UoA / Unit of Assessment
UC / Unclassified
UUK / Universities UK
Contents
Acknowledgements 3
Abbreviations 3
Tables 5
Executive Summary 6
1 Introduction and research approach 9
1.1 Context: the REF processes 11
1.2 UoA 25 Education and Panel C 12
1.3 Assessing Research Impact 13
2 Findings 15
2.1 Main characteristics of UoA25 submissions 15
2.2 Selection Strategies 16
2.3 Submission of HE-related outputs 17
2.4 Development of Impact case studies 19
2.5 Type of impact claimed 22
2.6 Evaluating Impact 23
2.7 The future 25
3 Discussion 26
4. Recommendations 31
5 References 33
Appendix 1. Interview schedule 36
Appendix 2. Profiles for UoA25 ordered by 4* 38
Appendix 3. Profile details 40
Appendix 4. Example impact case studies from higher education 47
Durham University: Changing educational practice through ‘Threshold Concepts’ 47
University of Edinburgh: Enhancing learning, teaching and assessment at university 49
Open University: The impact of the National Student Survey: changing the behaviour of institutions, teachers and students 51
Plymouth University: Selection of doctors to specialty training on the basis of aptitude 52
University of Sheffield: Developing Higher Education in Further Education colleges 53
Tables
Table 1. Interviewee profiles...... 10
Table 1. Overall quality profile: Definitions of starred levels ...... 11
Table 3. Sub-panels within Main Panel C ...... 12
Table 4. Comparison of Education overall profile with other subject groupings...... 13
Table 5. Average of UoA25 overall profiles by HEI characteristics...... 15
Table 6. Example impact areas ...... 22
Table 7. Impact case studies which were not submitted to REF 2014...... 28
Executive Summary
This project explores issues surrounding the submission of and value given to higher education (HE) pedagogic research within the 2014 Research Excellence Framework. It explores with those involved in the submission of the Education Unit of Assessment (UoA), and other stakeholders the following research questions:
• What proportion of the impact case studies submitted through UoA 25 in the 2014 REF were targeted at each sector?
• How does the proportion of case studies in HE relate to the proportion of outputs?
• In what ways and to what extent did REF guidance about impact case studies affect submissions?
• In what ways and to what extent do stakeholders believe that the REF impacts on HE research?
The impact case study rules specified the number of case studies required according to number of Category A staff submitted (FTE), and that impacts needed to be wider in reach than the submitting higher education institution. Concerns have been voiced that this may have skewed Education UoA submissions away from HE pedagogy.
The research included an exploration of outputs submitted to the Education UoA in REF2014 and analysis of the impact case studies submitted. This was followed by fifteen interviews with UoA co-ordinators and other stakeholders with an interest in pedagogic research.
The desk-based study of the UoA 25 submission to REF2014 indicates that:
· Of the 76 HEIs submitted to the Education REF in UoA25 (Education), the most successful submissions came from Russell Group Universities, those who had a member of staff on the REF assessment panel, and pre-1992 universities;
· In relation to other UoAs in panel C, the education submission overall had a higher than average proportion of 4* outputs and impacts but also a higher than average proportion of 1* and 2* submissions;
· An estimate of the proportion of HE-related outputs in the whole UoA25 submission gives a minimum level of 9% of total submissions (far lower than other education sectors, e.g primary, secondary etc.);
· HE outputs were published in a total of 122 journals, with 50% of these published in ten journals;
· Of the 106 named research groups in the Education UoA, only five explicitly include Higher education or HE in their title (less than 5%);
· In terms of impact case studies primarily focusing on HE, the proportion of the whole sample is estimated at 8% (17 of 216 impact[1] case studies on the REF database).
The raw data indicate that HE research formed a relatively low proportion of both the submitted research and the impact case studies in the Education UoA.
There are several competing explanations about why there was a low proportion of HE pedagogic research submitted into this UoA. It may be that such research is limited in quantity, or that it rarely meets the quality threshold for REF submission. Interviews with UoA co-ordinators revealed that some felt that quality was an issue, with pedagogic research often being small-scale and localised. However, other respondents believed that pedagogic research was consistently under-valued by those putting together such submissions, and in some cases by university management. Six concerns were raised by staff engaged in pedagogic research, as well as by some of the UoA co-ordinators. These focused on the following points:
• Credibility problems for pedagogic research with university or faculty colleagues – the idea that HE pedagogic research was ‘not REFable’ remained widespread, and may have influenced submissions;
• The potential tension between research which will have impact on practice and high quality academic outputs – as in other vocational disciplines, there was a tension between the need to share innovations and evaluation with other practitioners, and producing highly academic research outputs;
• Local political issues around entering individuals from outside the School of Education. Most submissions were co-ordinated from within Schools of Education, but much pedagogic research happens outside of them, leading some respondents to feel that they were either overlooked or deliberately ignored;
• Contractual issues with some pedagogic researchers not on academic contracts – a number of respondents reported that staff on teaching-only or non-academic contracts had REFable pedagogic research outputs which were not included because the HEI was unwilling to change the contract to enable inclusion, reinforcing perceived research/teaching divides;
• The need to keep the submission ‘simple’ and ‘safe’ – the need to develop a coherent environment statement may have led to the exclusion of some pedagogic research outputs which were felt to fall outside of the traditional education research areas of schooling or teacher education;
• The inhibiting effect of HEFCE definition of eligible impacts – most respondents felt that the exclusion of impacts on HE students in the submitting institution was unfair and this rule may have unfairly disadvantaged pedagogic researchers since the target of such research is often the potentially very large number of students in their own institution.
Overall, the findings suggest that concerns about pedagogic research and REF continue to be raised, and that the addition of the ‘impact’ element has not alleviated these. With the sector moving towards the use of more teaching-only type contracts, these issues may become worse before they get better.
Recommendations arising from this research are made at four levels:
· Individual pedagogic researchers are advised to collaborate with education researchers both within their own and other UK and international institutions
· HEIs are advised to remain open to moving staff between different contract types to enable their REF submission to contain high quality pedagogic research
· The HEA can continue to support pedagogic research activity in HE through events and networks, and can feed into consultations on the Teaching Excellence Framework for some metric of resourcing of pedagogic research
· HE policy makers more broadly could strengthen efforts and support for the UoA25 panel to ensure representation of HE pedagogic research.
1 Introduction and research approach
Higher Education (HE) in the UK – and internationally – has come under increasing pressure to deliver research and teaching which provide value for money to the taxpayer. In the research realm, there is mounting interest in assessing the public benefits of research undertaken in universities, and this has become, in a period of fiscal austerity, a major source of political debate. Research funding in HE in the UK is managed through the ‘dual support’ system. Funding for research amounting to around £3bn is distributed via the Research Councils (RCUK), through grants for specific projects and programmes. In addition, funding is disbursed directly to institutions as a result of research evaluation exercises conducted at around 6 year intervals.
The Research Excellence Framework (REF 2014) was the most recent evaluation point; a process designed to assess the quality of research in UK HE which built on previous rounds of research assessment in the UK (e.g. the Research Assessment Exercise – RAE – in 2008). The results of these assessments determine the allocation of ‘Quality-Related’ (QR) funding (worth £1.6bn in 2015/16) to UK HE institutions. In total, 154 higher education institutions took part in REF 2014, with 1,911 submissions across all units of assessment (UoA), comprising 52,061 FTE academic staff, 191,150 research outputs and 6,975 impact case studies. The number of institutions submitting to each UoA ranged from 14 (Civil and Construction Engineering) to 101 (Business and Management Studies). A total of 76 institutions submitted to UoA25, Education, making this a fairly competitive UoA.
This project investigated the role of HE pedagogic research within Education Unit of Assessment (UoA25) submissions to REF2014. It explores, with those involved in the submissions to this UoA, and other stakeholders, the following research questions:
• What proportion of the impact case studies submitted through UoA 25 in the 2014 REF were targeted at each sector?
• How does the proportion of case studies in HE relate to the proportion of outputs?
• In what ways and to what extent did REF guidance about impact case studies affect submissions?
• In what ways and to what extent do stakeholders believe that the REF impacts on HE pedagogic research?
In particular, we were seeking to investigate individuals’ experiences of undertaking HE pedagogic research and of having such research assessed for REF2014, and to learn about the experience of developing impact case studies for the REF. We were especially interested in the number of HE pedagogy publications, where publication takes place, and the effect of the rules for eligible impact case studies to UoA25 which embraces education at all stages from early years to life-long learning. Notably, these rules specified (i) the number of case studies according to number of Category A staff submitted (FTE), and (ii) that impacts needed to be wider in reach than the submitting higher education institution (HEI). Concerns have been voiced that this may have skewed UoA25 submissions away from HE pedagogy.
The study spanned five months with a work programme of desk-based research, interrogating the REF submissions and impacts databases[2], and a literature search which encompassed a range of sources from peer-reviewed journals to media reports on the REF process. This was supplemented by fifteen phone and face-to-face interviews with participants from 13 HEIs (see Table 1 below). This was an exploratory study, gathering qualitative data through a series of in-depth semi-structured interviews, using an interpretivist approach. The aim was to illustrate key issues from an insider’s perspective and, for this reason, purposive sampling was utilised, focusing initially on UoA Co-ordinators for Education submissions, identified through contact with UK School of Education research leads. These were followed by further interviews with selected stakeholders having interest and expertise in pedagogic research. (See Appendix 1 for interview schedules.) The latter group were identified primarily through networks including: the Association of National Teaching Fellows; the Heads of Educational Development Group (HEDG); the Higher Education Academy (HEA) pedagogic research network; and the Staff and Educational Development Association (SEDA). This group were able to provide valuable insights into the opportunities and challenges of undertaking pedagogic research in the context of REF. We also interviewed one university impact co-ordinator, one institutional REF co-ordinator and one REF assessor to gather some wider perspectives on REF and impact. Based on a small sample, this study does not make claims to generalisability, but rather illustrates some of the key issues which may nonetheless resonate with others working in this field. Further research would be of great interest in terms of exploring with a wider group, the provisional themes identified here.