Chapter 1 Background to the review

The Research Assessment Exercise

  1. The Research Assessment Exercise (RAE) provides ratings of the quality of research conducted in universities and higher education colleges in the UK to inform the selective allocation of funds in accordance with the quality of the work undertaken.
  1. The system was designed to maintain and develop the strength and international competitiveness of the research base in UK institutions, and to promote high quality in institutions conducting the best research and receiving the largest proportion of grant. The outcomes of the exercise are published to provide public information on the quality of research in higher education throughout the UK. The results of the RAE may also be used to inform policy development.
  1. The first RAE was undertaken in 1986, introducing an explicit and formalised assessment process to standardise the information received from existing subject-based committees. Further exercises held in 1989 and 1992 were markedly more transparent and comprehensive. The fourth exercise in 1996 considered the work of over 50,000 staff designated by higher education institutions (HEIs) as research active. It determined the allocation of over £4 billion over five years. Its costs (including opportunity costs) have been variously estimated at between £27 million and £37 million (estimated as 0.8% of the total funds distributed on the basis of the exercise).
  1. The most recent RAE in 2001 was the most rigorous and thorough exercise to date. It had by then become the principal means by which institutions assured themselves of the quality of their research. It had also evolved into an intense competition in which HEIs strived not only for funding but also for prestige.

How the present system works

  1. The RAE is essentially a peer review process. Research is divided into 68 subject areas or units of assessment. Assessment panels are appointed to examine research in each of these areas.
  1. HEIs are invited to make submissions, in a standard format. There is no limit on the number of units of assessment an institution can submit to, nor is there any limit on the number of staff submitted. Each panel produces and publishes a set of assessment criteria and working methods, to which it is bound to adhere. Panels score each submission on a 7 point scale according to how much of the work is judged to reach national or international levels of excellence. A full guide to the 2001 RAE is at Annex C.
  1. The 2001 RAE was strengthened in a number of respects to address some of the concerns previously expressed about aspects of the exercise, including issues of publication behaviour, interdisciplinary research, consistency of scores, equal opportunities, and staff movement.
  1. Each funding body uses RAE ratings to allocate research funding by formula to the institutions it funds. The RAE supports the policy goal of selectivity, ensuring that scarce resources are directed towards those with the capacity to produce research of the highest quality. The RAE makes it possible for funding bodies to discriminate in their funding (this is done in a transparent way by formulae related to quality and volume). At present, the principle which underpins research funding is that of selectivity based on quality, wherever it is located, not explicit concentration of funds in a selected number of institutions.

Strength of UK research

  1. On current measures of performance, research in UK universities is in excellent health. The results of the RAE, supported by other evidence, demonstrate that research in the UK continues to improve and to do so relative to other industrialised countries, despite increasing competition.
  1. Substantial improvements are reflected in the results of the 2001 RAE itself. In 1996 32% of staff that were submitted to the RAE as research active were in departments rated 5 and 5*. In 2001 the figure was 55%. This improvement was validated by the opinion of overseas experts.
  1. This achievement is further validated by new research which shows that the UK’s share of the most cited 1% of research papers has increased from 11% to 18% over the assessment period. The average citation rate of UK papers relative to the rest of the world has improved by 12% over the period. UK researchers are now cited at a rate 38% higher than the global average.

Reviewing research assessment policy

  1. By any measure, the RAE has been extremely successful. It has evolved from a quality assurance process to a competition for funding, while successfully retaining its original function of driving up standards through reputational incentives. At the same time it has enabled funds to be concentrated in those departments best able to produce research of the highest quality. It has helped to drive up research quality, transformed the management of research within institutions, and gained the acceptance of the research community and its stakeholders. In 2000 HEFCE asked all interested parties to respond to a consultation on its research policy. Faced with the proposition that there should continue to be a research assessment process based on peer review, building on the foundations of the RAE, 98% of respondents agreed.
  1. The RAE evolved over the years to take account of changing circumstances. Following the outcome of the 2001 RAE, the funding bodies moved towards the view that change may be needed to ensure the continued fitness for purpose of the RAE. The following reasons were identified:
  1. effect of the RAE upon the financial sustainability of research
  2. an increased risk that as HEIs’ understanding of the system becomes more sophisticated, games-playing will undermine the exercise
  3. administrative burden
  4. the need to properly recognise collaborations and partnerships across institutions and with organisations outside HE
  5. the development of researchers
  6. the need fully to recognise all aspects of excellence in research (such as pure intellectual quality, value added to professional practice, applicability, and impact within and beyond the research community)
  7. the ability to recognise, or at least not discourage, enterprise activities
  8. concern over the disciplinary basis of the RAE and its effects upon interdisciplinary and multidisciplinary research
  9. lack of discrimination in the current rating system, especially at the top end with a ceiling effect.
  1. In June 2002 a review of research assessment was launched, owned by the four UK HE funding bodies and led by Sir Gareth Roberts, President of Wolfson College, Oxford. A steering group of 12 members was established, selected on the basis of personal knowledge, experience and standing in the HE and research communities. The group’s membership and terms of reference are at Annex A.

Chapter 2 The review process

The review structure

  1. The review of research assessment has been led by Sir Gareth Roberts, administered and informed by a review team resourced from HEFCE, and supported by other funding body officers and outside experts. The review team worked closely with Sir Gareth in developing alternative approaches to research assessment for consideration by the steering group.
  1. Throughout the review process the funding bodies sought to consult widely on the issues being addressed. An early consultation was published identifying issues to stimulate debate, inviting interested parties to discuss the issues and contribute to the review. Respondents were encouraged to convene focus groups or workshops and to submit the formal record of their discussions. The chair and review team held a series of public meetings across the English regions and devolved administrations, commissioned workshops within institutions, and met with individual stakeholder groups. Towards the end of the process the team also visited a selection of institutions to discuss the emerging model.
  1. Each funding body was responsible for consulting its relevant governmental stakeholders and ensuring that these views were reflected in the process. To facilitate this communication, a government stakeholder group, chaired by Sir Gareth, was established and met through the course of the review.
  1. In order to ensure the review remained fully transparent and to enable the community to participate in the discussion, a dedicated website was established ( This made available the workplan for the review, reports of key meetings, and progress on the models being developed. All other evidence considered within the course of the review will be published on completion.

Development of research assessment models

  1. The review provided distinct approaches to assessment systems for the steering group’s consideration. As these models were developed they were made available to the wider research community, the funding bodies and stakeholders for discussion. From this process the steering group identified one preferred model to be presented as an option for consultation.

Assessment of research quality: the context

  1. Research assessment may in principle be straightforward, but in order to arrive at the best possible assessment process the review has considered both the philosophical questions, such as ‘what is meant by quality in research’, and practical issues to do with designing a system that will provide a fair and accurate assessment of quality while minimising burden on all concerned.
  1. From the outset the review made certain assumptions:
  1. The dual support system for research will continue. This is the system whereby public funding for research in HEIs comes from two funding streams. The first forms part of the block grant from the UK HE funding bodies, based on past performance as measured by the RAE. The second comes through project grants allocated to particular researchers by the research councils, in response to proposals for programmes to carry out future work. The HE funding bodies have an ongoing need for a method of allocating funds selectively. Research assessment of some description will continue to be used for this purpose.
  2. Block grant funding to institutions will continue. Within the block grants system, institutions are free to allocate funding according to their own priorities.
  3. The quality of research will continue to be considered in a global context. It will therefore need to be assessed at a national and international level.
  4. The regional dimension will be considered separately both by the devolved administrations and other regional agencies, including the research funding model’s implications for regional policy.
  1. Other relevant factors were also considered:
  1. There needs to be emphasis upon the ‘people dimension’ – that is, the contribution made by institutions to the supply and development of researchers.
  2. Alongside research funding, there are now public funds available to universities and colleges to support knowledge transfer activities. Work is under way to develop measures of excellence in those activities, many of which involve research services to external partners.
  3. Competition for research funding is increasingly keen and the costs of research in many subjects are increasing. Funding bodies need to consider whether targeted help is required to enable new subjects and new fields to develop. It may (or may not) fall to the research assessment process to identify suitable candidates for any such assistance.

Chapter 3 The evidence base

Sources

  1. In the course of the review, we reviewed four major strands of evidence:
  1. 420 responses to our public invitation to contribute
  2. an operational review of RAE2001 undertaken by Universitas higher education consultants
  3. a report on international approaches to research assessment (update of a 1999 study), undertaken for the review by the Science Policy Research Unit at the University of Sussex (‘Report on responses’)
  4. a programme of nine workshops with practising researchers undertaken for the review by RAND Europe.
  1. Each of these strands produced a report[1]. These reports are available on our website Most of the responses we have received are also available on the site.
  1. In addition we held an extensive programme of meetings, which provided valuable insights into the views of the community:
  1. 44 informal consultative meetings with key stakeholders
  2. open public meetings in Sheffield, Birmingham, Edinburgh, London, Cardiff and Belfast.

Key themes

  1. A number of themes emerged strongly:
  1. the importance of expert peer review
  2. the need for a clearer link between assessment outcomes and funding
  3. the need for greater transparency, especially in panel selection
  4. the need to ensure consistency of practice across assessment panels, including the trade-off between comparability of grades across subjects and the flexibility for assessors to develop methods appropriate to their subject
  5. the need for a continuous rating scale
  6. the need for a properly resourced administration.

The importance of expert peer review

  1. Almost all those who have made their opinions known to us have argued that only expert peer review can identify the best research. It is generally accepted that proxies can properly be used by reviewers to help them arrive at judgements about the quality of research. It is clear, however, that any assessment not based ultimately upon the judgements of experts would lack credibility and legitimacy.
  1. Support for expert peer review is accompanied by a willingness to accept the burdens imposed upon researchers by the current system. As RAND Europe has noted in its report:

’The overwhelming majority of the academics and research managers who took part in the study felt that research should be assessed using a system based on peer review by subject-based panels...The participants also indicated that these panels should be informed by metrics and self-assessment, with some input from the users of research.’

The need for a clear link between assessment outcomes and funding

  1. In theory, assessment and funding are entirely separate issues: the credibility of an assessment process should rest upon the reliability of its results rather than the way in which they are translated into funding decisions by the HE funding bodies.
  1. In practice, the RAE is the mechanism by which institutions earn research funds from the funding councils. Although it is not a competition for funding in a direct sense, it needs to have many of the characteristics of such a competition – principally transparency in its own process and an explicit link between assessment and funding outcomes.
  1. We began this process insisting that assessment and funding were separate issues – and indeed it is not any part of our purpose to suggest what the funding policies of the funding councils ought to be. However, it has become clear to us that the link between assessment and funding needs to be clarified in advance of the next assessment, if the assessment process is to retain credibility and consent. As the report on the responses to the ‘Invitation to contribute’ has noted:

’Of the (unsolicited) comments recorded, a majority focus on funding problems generated by the (post RAE) financial settlement. There is strong support among HEIs in particular for the next round of assessment to make the financial outcome of attaining a particular grade explicit before the exercise is run.’

The need for greater transparency

  1. The operational review found that many institutional RAE contacts believed that the RAE had achieved its goal of transparency. However, the very high value placed upon this dimension by the community is reflected in the outcomes of the workshops and the ‘Invitation to contribute’, both of which suggest that further progress could be made. The report on responses notes that:

‘...there is strong support for the workings of the assessment panels to be made more transparent, including:

  1. the selection procedures of UK and international panel members
  2. the panels’ weighting of the various assessment criteria
  3. cross-referral processes
  4. the definition of international, national and sub-national standards of research excellence.’
  1. A similar picture emerges from the workshops, with the same emphasis upon the selection of panels and the need for greater international input:

‘There was a strong desire for a system with clear rules, and transparent procedures that were established and not modified during the assessment process. The appointment of panels and the selection of the criteria they used were thought to be critical areas for transparency. Participants in the study considered that the panels themselves should be professionalised, and that there should be increased and earlier involvement of international members. They suggested that chairs from outside the subject area with more experience of facilitation should be used.’

The need to ensure consistency of practice across panels, including the need to balance comparability and flexibility

  1. All our consultative exercises have revealed concern about the comparability of RAE grades across subjects. It is acknowledged to be very difficult to guarantee that a given grade in one subject is equivalent to the same grade in another, especially where there is little overlap between those subjects.
  1. The operational review found that a majority of institutional RAE contacts felt that RAE2001 had failed to achieve its goal of comparability (the only one of its goals that the exercise was deemed to have failed to achieve). Respondents to the ‘Invitation to contribute’ also expressed concern about comparability.
  1. The academic community recognises that comparability, although important, can conflict with other imperatives to which it attaches even greater importance. In its workshops, RAND Europe explored the priority to be given to various features of the exercise. It found that there was strong support for allowing panels more freedom to define their own criteria and information requirements, even if this led to divergent practice.
  1. A consensus position could be defined as follows:
  1. there is scope to improve the comparability of RAE scores
  2. it may not be possible to achieve absolute comparability, and the limitations of the system should be recognised by policymakers
  3. comparability should not necessarily take precedence over the freedom for panels to adopt assessment methods appropriate to the subject area
  4. every effort should be made to achieve comparability consistent with the above.

The need for a continuous rating scale

  1. The RAE awards submissions one of seven grades. The grades are wide and there is scope for submissions of a rather different quality to receive the same grade. For example, a submission rated 5 could be deemed to display international excellence in anywhere between 11% and 49%[2] of its research activity.
  1. The rating scale is seen as artificially raising the stakes of participation in the RAE by imposing a step-change in the prestige (and inevitably the funding) of university departments. There is a consensus that the rating scale needs reform. The preferred option is to adopt a system that allows for continuous grading. As RAND Europe has noted:

’Suggestions (for change) seemed to arise from the concerns regarding the step-changes in funding in the current system...Most...preferred a move to a continuous scale in which a submission’s score was the sum of the scores of the individuals included in the submission.’