Interdisciplinary Research
and the
Research Assessment Exercise
Summary
The Interdisciplinarity Project
Context
The Project
Objective 1: Volume
Objective 2: Impacts
Perceptions
The RAE and Researcher Behaviour
Other Factors
Objective 3: Practices
Institutional and Researcher Practices
Panel Practices
Objectives 4 & 5: Strategies and Objectives
Strengthening the RAE: Consistency
Strengthening the RAE: Boundary Critical Submissions
RAE Recommendations
Objective 6: The Longer Term
The Evolving RAE
Agenda 2006
Annex 1: Funding Bodies’ Consultation
Annex 2: Tables
The Respondents
Factors that Affect Research
Taxonomy
Notes
A Report by
Evaluation Associates Ltd
March 1999
For the UK Higher Education Funding Bodies
Figures
Figure 1. Percentage of research time spent on interdisciplinary research
Figure 2. Percentage of departments’ research projects that are interdisciplinary
Figure 3. The 20–70–10 model
Figure 4. Percentage of time on interdisciplinary research by subject area
Figure 5. Percentage interdisciplinary research conducted by respondents mapped against their RAE 1996 panels
Figure 6. Percentage of research time spent on interdisciplinary research by age and sex
Figure 7. Modes of working adopted for single disciplinary research
Figure 8. Modes of working adopted for interdisciplinary research
Figure 9. Percentage lone researchers against RAE 1996 panels
Figure 10. Percentage time on interdisciplinary research for holders of Research Council grants
Figure 11. Heads of departments’ views on the impact of the RAE on interdisciplinary research in general
Figure 12. Heads of departments’ views on impact of RAE on their own department's interdisciplinary research
Figure 13. Heads of departments’ views on impact of RAE on their department's interdisciplinary research
Figure 14. Researchers' and panel members' views on the impact of the RAE on interdisciplinary research
Figure 15. Researchers that believe the RAE strongly inhibits interdisciplinary research by 1996 rating
Figure 16. Researchers that believe the RAE strongly inhibits interdisciplinary research by time spent on ID research
Figure 17. Perceptions of the impact of the RAE by area
Figure 18. Researchers' views on the impact of the RAE on interdisciplinarity by panel
Figure 19. Impact on researcher's behaviour the RAE 17
Figure 20. Factors that promote (positive scores) and inhibit research (negative scores)
Figure 21. Perceptions of the impact of the RAE by area
Figure 22. Perceptions of the current environment for interdisciplinarity by area
Figure 23. Perceptions of the future environment for interdisciplinarity from researchers who believe the
current environment is worse than that for single disciplinary research
Figure 24. Reasons for excluding researchers from submissions to RAE 1996
Figure 25. Number of panels to which single departments submitted in 1996
Figure 26. Ratings awarded to split submissions
Figure 27. Small and large teaching institutions
Figure 28. Mixed economy and specialist institutions
Figure 29. Percentage of split and cross-referred departments
Figure 30. Ratings for split and cross-referred departments
Figure 31. Rating for department submission compared to researchers’ time on interdisciplinary research
Figure 32. Rating for department submission compared to percentage of department's projects that are ID
Figure 33. Rating for department submission compared to department’s time on Interdisciplinary research
Figure 34. Difference in ratings for interdisciplinary and single disciplinary researchers by panel
Figure 35. Interdisciplinarity of panel members and researchers
Figure 36. Importance of assessment criteria: all panels
Figure 37. Assessment criteria by subject area (criteria rated important or very important)
Summary
Messages from the Interdisciplinarity Survey
Our report has six main messages:
- Interdisciplinary research is pervasive through higher education research. Around four-fifths of researchers report that they are engaged in at least some interdisciplinary work.
- Departments and researchers widely believe the RAE inhibits interdisciplinary research. Many institutions and individuals are changing the way they do research as a result of the RAE.
- Despite these beliefs, there is no evidence that the RAE systematically discriminated against interdisciplinary research in 1996.
- In 1996, there was however:
a lack of consistency in the treatment of interdisciplinary research by individual panels, and
a problem with departments that needed to split researchers between panels or seek cross referral between panels: these boundary critical submissions gained lower ratings.
- The procedural changes we propose for RAE 2001 should give researchers the confidence that interdisciplinary research will receive parity of treatment by all panels in RAE 2001.
- There remains a need for a longer-term debate — Agenda 2006 —about:
the assessment of interdisciplinary research in the RAE and elsewhere
the impact of the RAE on institution and researcher practices.
The Interdisciplinary Research Project
Our project was commissioned by the Funding Bodies[1] and had six objectives. We set out to meet these through a large-scale survey of practices and attitudes in higher education research. Below, we briefly outline our conclusions based on the 5,832 responses to our questionnaires, and on discussions with institution managers and researchers.
Objective 1: Volume
To estimate the current proportion of UK research which is interdisciplinary.
We have found that interdisciplinary and multidisciplinary research is pervasive throughout the higher education research system. The majority of researchers practise a ‘duality’ of research, switching from single disciplinary to interdisciplinary research according to the nature of the research problem or project they are engaged with. Key findings are:
researchers spend an average of 46% of their research time on interdisciplinary research
20 % of researchers are engaged only in single disciplinary research
10% of researchers are engaged only in interdisciplinary research
70% of researchers are engaged in both single and interdisciplinary research
researchers in the medical and biological sciences spend the highest proportion of their research time on interdisciplinary research (53%)
the arts and humanities are least involved (43%)
female researchers are more involved with interdisciplinary research (50%) than their male colleagues (45%).
Objective 2: Impacts
To assess whether and to what extent the RAE acts as a disincentive to the pursuit of interdisciplinary and multidisciplinary research, and if it does, whether certain areas are more affected than others.
There is strong criticism of the impact of the RAE on interdisciplinary research:
one quarter of researchers (24%), and nearly one fifth of RAE 1996 panel members (17%), believe that the RAE strongly inhibits interdisciplinary research
researchers that are more heavily involved in interdisciplinary research are the most critical, as are those that achieved lower ratings in RAE 1996
criticism is strongest in the social sciences, and in the arts and humanities.
A smaller number of researchers believe that the RAE is having a positive influence:
6% of researchers and 6% of RAE 1996 panel members believe that the RAE strongly promotes interdisciplinary research.
Heads of departments report that about one third of departments have adapted their strategies as a result of the perceived treatment of interdisciplinary research in the RAE:
17% report the RAE has discouragedtheir departments from pursuing interdisciplinary research
14% state it has encouraged interdisciplinary research in their departments
69% report that the RAE has had no effect on their departments’ interdisciplinary research.
The RAE has also encouraged researchers to concentrate on:
the quality of their research
publishing more papers, particularly in peer reviewed journals
research that produces results in the shorter rather than longer term.
Objective 3: Practices
To identify institutional, researcher and panel practices and tactics in handling interdisciplinary submissions to the 1996 RAE, including multiple submission and cross-referral.
Overall, RAE 1996 panels did not strongly discriminate for or against interdisciplinary research, although the most interdisciplinary researchers received slightly lower ratings. The average ratings awarded to researchers’ departments, based on a seven point scale,[2] were:
4.77: high single disciplinary researchers[3] (those committing <=33% time to interdisciplinary research)
4.77: interdisciplinary/single disciplinary researchers (33-66% interdisciplinary)
4.72: high interdisciplinary researchers (67%+ interdisciplinary).
There was, however, variation between panels, with some awarding higher ratings to single disciplinary research, and others rewarding interdisciplinary research.
A particular problem was the fit between departmental structures and the RAE. Nearly one quarter of departments (23%) split their researchers between panels, or sought cross-referral between panels. We call thesesubmissions boundary critical:
overall, boundary critical submissions receive a 0.5 point lower rating than non-critical submissions
in the former polytechnics and colleges, boundary critical and non-critical submissions received almost equal ratings (3.3 and 3.2 respectively)
in established universities, boundary critical submissions received an average rating of 4.6 compared with 5.1 for non-critical submissions.
Although small, these differences in ratings are of concern because they suggest that departments suffer during assessment if they are not organised in a way that approximates to the RAE panels.
Objective 4: Strategies
To identify cost-effective RAE and institutional strategies for ensuring proper assessment of interdisciplinary research.
Our conclusions under this objective are concerned with building confidence in RAE 2001 and ensuring that necessary improvements are identified. We call this Agenda 2001. Our aim is to ensure that institutions and researchers are not deterred from investment in interdisciplinary research and that they can have confidence that their work will be equitably treated in the RAE. We believe that the problems we have identified with assessment in 1996 will be addressed by changes proposed by the Funding Bodies and by our recommendations (Objective 5). We identify improvements that will strengthen the consistency of assessment in the RAE. Specifically:
the range of panel members should:
reflect all sub-areas of the units of assessment (UOAs)
represent significant cognate areas
have experience of single and interdisciplinary working.
panels should explicitly state the criteria they intend to apply to interdisciplinary research, and examine and monitor methods that might potentially discriminate against interdisciplinarity
new procedures should be introduced for documenting and actively managing boundary critical submissions
the new feedback and reporting mechanisms introduced by the Funding Bodies for RAE 2001 should explicitly address how interdisciplinary research was treated by each panel.
Objective 5: Recommendations
If necessary, to recommend revisions to the RAE and provide advice for institutions submitting interdisciplinary research.
We have made five specific recommendations:
1.The Funding Bodies and panel chairs should take steps to ensure the representativeness of panels across all forms and areas of research, including interdisciplinary research.
2.Panel methodologies should specifically address assessment of interdisciplinarity.
3.New mechanisms for boundary critical submissions should be introduced based on a new proforma (5C).
4.The Funding Bodies should introduce monitoring mechanisms to ensure the effectiveness of cross-referral.
5.Feedback and reporting should embrace interdisciplinary research.
Following our interim report in November 1998, the Funding Bodies accepted these recommendations.
Objective 6: The Longer Term
To advise on:
monitoring the volume of interdisciplinary research in the long term
making comparisons with other countries
monitoring the impact of any recommendations.
The RAE has become one of the most important features of academic life. It is likely to remain so while higher education research is dual funded. Its unpopularity with many researchers should not detract from its development as a long-term vehicle for research assessment. For this reason, we believe that the debate for RAE 2006 should commence now — even though the Funding Bodies are unlikely to confirm before 2003 whether that Exercise will proceed. Researchers, funders and policy makers need to more fully understand and reach a consensus on:
the impacts, positive and negative, of the RAE
the nature, extent and quality of interdisciplinary research
how interdisciplinary research interlocks with single disciplinary research
the relative contributions of single and interdisciplinary research to development of fundamental knowledge and to the relevance and application ('worldliness') of research.
1
Interdisciplinary Research and the RAE
The Interdisciplinarity Project
Context
1. Dual Funding
Higher education research in the UK is supported through dual funding from government:
project specific funding is provided through the Research Councils — this currently totals £1.3 billion[4]
core, flexible funding is provided through the higher education Funding Bodies — this currently totals £1.0 billion.
Dual funding rests on two very different approaches to assessment of funding priorities:
continuous project and programme level appraisal by the Research Councils, most often based on peer review
periodic subject level evaluation by the higher education Funding Bodies through the Research Assessment Exercise (RAE), also based on peer review.
2. The Research Assessment Exercise
In 1986, the former University Grants Committee (UGC) launched the first systematic assessment of research in UK universities. The purpose of its Research Selectivity Exercise was to identify high quality research to inform the allocation of funds. A second exercise followed in 1989, and from 1990/91, the UGC’s successor, the Universities Funding Council (UFC) began to allocate research and teaching funds separately to the universities and to selectively reward high quality research. The assessment took on added significance in 1992 with the abolition of the binary divide between polytechnics[5] and colleges and the universities.
The four higher education Funding Bodies established in 1992 (hereafter the Funding Bodies) brought the former polytechnics and colleges into the same funding structure as universities.[6] This allowed the ‘new universities’, as they became known, to benefit from dual funding for the first time. The 1992 and 1996 Research Assessment Exercises provided a mechanism, based on quality and volume, for selective distribution of research funds across the expanded sector.
The Research Assessment Exercise (RAE) has consistently generated debate and controversy. This was inevitable, given its unique scale and importance to researchers’ funding and status. In each successive round, efforts have been made to ensure greater fairness of assessment, openness and transparency. However, criticism persists of the treatment of interdisciplinary research.
3. Interdisciplinarity
Interdisciplinary research has long been of interest to:
researchers - who wish to break out of disciplinary constraints and develop innovative research concepts
policy makers, funders and users of research - who see interdisciplinary research as more relevant to solving the practical problems of the world.
In recent years, many models of disciplinarity and interdisciplinarity have been put forward. While recognising their value, we do not subscribe to any of these or promote any specific taxonomy of disciplines. Rather, we observe a continually evolving landscape of disciplinarity, where research areas grow and divide, merge, and fade. Interdisciplinarity permeates this landscape and is one of the essential forces in its evolution. The challenge for this project was to define and quantify this elusive concept.
4. The RAE and Interdisciplinarity
Much criticism of the RAE’s impact on interdisciplinarity stems from its organisation around units of assessment (UOAs). Some of these approximate to single disciplines but others are broader. In 1989, the UFC noted that the 1986 Selectivity Exercise did not allow proper assessment of interdisciplinary research groups.[7] Since then, the number of UOAs has been reduced: to seventy-two in 1992 and sixty-nine in 1996, assessed by sixty panels (listed on page 49). Also in 1996, institutions were asked to flag interdisciplinary submissions and could request cross-referral between panels. These changes may have helped interdisciplinarity. In 1997, Ian McNay was able to conclude:
There seems little evidence that the RAE has had a negative impact on interdisciplinary or applied research. This could be because fears about this are much overstated; or that there are counter balances, and institutions act despite perceptions about disadvantage. A different kind of study would be needed to consider whether such research is properly assessed in the RAE.[8]
However, in a detailed qualitative study of interdisciplinarity, also in 1997, the Scottish Universities Research Policy Consortium noted that:
Interdisciplinary research is increasingly important for practical, academic and social reasons, but is currently difficult to deliver. There is widespread belief that many internal pressures of ethos and structure in higher education institutions, reinforced by the external pressure of the Research Assessment Exercise, tend to favour research in traditional academic disciplines at the expense of interdisciplinary developments.[9]
And in the same year, the National Committee of Inquiry into Higher Education (the Dearing Committee) made a specific recommendation:
We recommend that the Funding Bodies and the Research Councils commission a study to evaluate the funding of interdisciplinary research, including the incentives and disincentives. The report should be ready to inform the next Research Assessment Exercise.[10]
Our project is a partial response to the Dearing recommendation.
5. A Focused Project
Our project concentrated on the assessment of interdisciplinary research by the RAE, and any disincentives that might arise from that. It was not a study of the nature of, or environment for, interdisciplinary research, or of its funding.[11] Our research was limited to higher education institutions that entered the 1996 RAE. This tight focus has allowed us to provide actionable policy recommendations for RAE 2001. Nevertheless, as we note on page 35, there is still much to be learnt about interdisciplinarity in higher education research.
The Project
6. Consultation
The Funding Bodies have already extensively consulted on the procedures for RAE 2001. Picking up concerns expressed after earlier exercises and by Dearing, they also commissioned this study of the RAE and interdisciplinarity. The contract for the project was awarded to after competitive tender. Our remit was:
To examine whether the RAE inhibits interdisciplinary and multidisciplinary research and to advise on any need for changes to the RAE.
The project was to be neutral towards interdisciplinarity — we did not have a remit to promote interdisciplinary research. A Joint Working Group drawn from funders and institutions oversaw the project.[12]