Research Assessment Exercise 2001:
Assessment panels’ criteria and working methods
December 1999
RAE 5/99
This document contains details of the criteria and working methods which Research Assessment Exercise (RAE) panels will adopt in making assessments in 2001. It should be read in conjunction with ‘Research Assessment Exercise 2001: Guidance on Submissions’ (RAE 2/99).
Enquiries about this circular or the conduct of the exercise should be addressed to the RAE Team. It would be helpful if enquiries could be channelled through institutions’ own nominated RAE contacts wherever possible.
John Rogers RAE Manager Tel: 0117 931 7237
Anita Jackson RAE Officer Tel: 0117 931 7260
Greg Williams RAE Officer Tel: 0117 931 7003
Sue Wood RAE Administration
Officer Tel: 0117 931 7012
Rebecca Witts RAE Team Secretary Tel: 0117 931 7293
Richard Puttock RAE Information
Process Manager Tel: 0117 931 7472
Anne Southworth RAE Analyst Tel: 0117 931 7131
RAE Team, Northavon House, Coldharbour Lane, Bristol BS16 1QD
E-mail: (for general enquiries)
(for software and data enquiries)
This circular, together with other documents and information on the 2001 RAE, is available on the RAE web site at http://www.rae.ac.uk.
Table of Content
Page
- Overview of the Research Assessment Exercise
- Introduction to the Criteria and Working Methods
- Panels’ Criteria and Working Method Statements
No. / Unit of Assessment
1 / Clinical Laboratory Sciences
2 / Community-based Clinical Subjects
3 / Hospital-based Clinical Subjects
4 / Clinical Dentistry
Joint Panel
5 / Pre Clinical Studies
6 / Anatomy
7 / Physiology
8 / Pharmacology
9 / Pharmacy
10 / Nursing
11 / Other Studies and Professions Allied to Medicine
13 / Psychology
14 / Biological Sciences
Joint Panel
15 / Agriculture
16 / Food Science and Technology
17 / Veterinary Science
18 / Chemistry
19 / Physics
Joint Panel
20 / Earth Sciences
21 / Environmental Sciences
22 / Pure Mathematics
23 / Applied Mathematics
24 / Statistics and Operational Research
25 / Computer Science
Joint Panel
26 / General Engineering
31 / Mineral and Mining Engineering
27 / Chemical Engineering
28 / Civil Engineering
29 / Electrical and Electronic Engineering
30 / Mechanical, Aeronautical and Manufacturing Engineering
32 / Metallurgy and Materials
33 / Built Environment
34 / Town and Country Planning
35 / Geography
36 / Law
37 / Anthropology
38 / Economics and Econometrics
39 / Politics and International Studies
Joint Panel
40 / Social Policy and Administration
41 / Social Work
42 / Sociology
43 / Business and Management Studies
44 / Accounting and Finance
45 / American Studies
46 / Middle Eastern and African Studies
47 / Asian Studies
48 / European Studies
49 / Celtic Studies
50 / English Language and Literature
51 / French
52 / German, Dutch and Scandinavian Languages
53 / Italian
54 / Russian, Slavonic and East European Languages
55 / Iberian and Latin American Languages
56 / Linguistics
57 / Classics, Ancient History, Byzantine and Modern Greek Studies
58 / Archaeology
59 / History
60 / History of Art, Architecture and Design
61 / Library and Information Management
62 / Philosophy
63 / Theology, Divinity and Religious Studies
64 / Art and Design
65 / Communication, Cultural and Media Studies
66 / Drama, Dance and Performing Arts
67 / Music
68 / Education
69 / Sports-related Subjects
Section I: Overview of the Research Assessment Exercise
1.1 The arrangements and regulations for the 2001 Research Assessment Exercise (RAE), and detailed information on the data requirements, are set out in full in ‘Research Assessment Exercise 2001: Guidance on Submissions’ (RAE 2/99). This document should be read in conjunction with RAE 2/99. The criteria and working methods for assessment panels contained in Section III below should be read in conjunction with RAE 2/99 and with the information in Sections I and II of this circular.
Purpose and Conduct of the RAE
1.2 The primary purpose of the 2001 RAE is to produce ratings of research quality which will be used by the higher education funding bodies in determining the main grant for research to the institutions they fund, with effect from 2002-03. The RAE is conducted jointly by the Higher Education Funding Council for England (HEFCE), the Scottish Higher Education Funding Council (SHEFC), the Higher Education Funding Council for Wales (HEFCW), and the Department of Higher and Further Education, Training and Employment. The exercise is managed by the RAE Team, based at the HEFCE offices, on behalf of the four funding bodies.
1.3 The assessment process is based on peer review and is not mechanistic. Assessment panels will use their professional judgement to form a view about the overall quality of the research activity described in each submission, taking into account all the evidence presented. They will form their judgements in the context of the statements of criteria and working methods published in this document.
1.4 The other principles by which the RAE is governed are:
a) Clarity. A large and complex body of regulations and procedures governs the RAE. All written documents and statements about the RAE should therefore be clear and consistent, and misinformation about the RAE will be challenged wherever possible.
b) Consistency. Assessments made through the RAE should be consistent, especially across cognate areas and in the calibration of quality ratings against international standards of excellence.
c) Continuity. The RAE develops through an evolutionary process, building on and learning from previous exercises. With each successive exercise a balance has to be struck between continuity and development. In general, changes are only made where they can bring demonstrable improvements which outweigh the cost of implementing them.
d) Credibility. As was demonstrated through responses to the funding bodies’ consultation on the conduct of the RAE, the methodology, format and processes employed in the exercise are credible to those being assessed. It is essential that this credibility be maintained.
e) Efficiency. Previous RAEs have been extremely cost-effective given the value of public funds distributed according to their ratings. The cost of the exercise, including the cost to Higher Education Institutions (HEIs), should continue to be the minimum consistent with a robust and defensible process.
f) Neutrality. The RAE exists to assess the quality of research in HEIs. It should carry out that function without distorting what it is measuring. In other words, the RAE should not encourage or discourage any particular type of activity or behaviour, other than providing a general stimulus to the improvement of research quality overall.
g) Parity. The RAE is concerned only with assessing the quality of research in participating HEIs, regardless of the type, form or place of output of that research.
h) Transparency. The credibility of the RAE is reinforced by transparency about the process for making decisions. All decisions and decision-making processes will be explained openly, except where there is a need to preserve confidentiality (for example in panels’ discussions, or when dealing with the names of nominees for panel membership or with the strategic research plans of institutions).
1.5 The 2001 RAE will follow broadly the same approach as previous exercises. Eligible HEIs will be invited to submit their research activity for assessment. Submissions will be made to a number of subject-based Units of Assessment (UoAs). The information supplied by HEIs will provide the basis for peer review assessment of research quality by specialist panels. Submissions will be in a standard form which includes both quantitative and descriptive elements. As well as providing information specifically requested by the funding bodies, institutions will have the opportunity to draw panels’ attention to any matters which they consider particularly significant to judgements of research quality. All information submitted will be subject to audit by the funding bodies. A statement of the audit arrangements for the RAE will be issued during 2000.
1.6 The census date for the exercise is 31 March 2001. Institutions are invited to provide information on staff in post on that date, and on publications and other forms of assessable output which they have produced during the assessment period. The assessment period is 1 January 1994 to 31 December 2000 for arts and humanities subjects (UoAs 45 to 67 inclusive), and 1 January 1996 to 31 December 2000 for other subjects (UoAs 1 to 44, 68 and 69). All information must be submitted by 30 April 2001.
1.7 There is an overlap between the assessment period for the 1996 and 2001 RAEs. Research outputs and other information relating to the overlap period may be returned to the 2001 RAE and will be given equal weight with all other evidence.
1.8 Institutions are invited to list up to four items of research output for each individual (Category A or C) whose research is to be taken into account (up to two items for Category A*). These may be any form of available assessable output. All forms of output will be treated on an equitable basis: panels will be concerned only with assessing the quality of research presented.
1.9 Proper account will be taken of the complete range of research. Panels will give full recognition to work of direct relevance to commerce and industry, as well as to the public and voluntary sectors. All research, whether applied, basic or strategic, will be given equal weight: panels will be concerned only with the quality of the work submitted for review.
1.10 Research into the teaching and learning process within higher education (pedagogic research) is regarded by the funding bodies as a valid and valued form of research activity. It will be assessed by all subject panels on an equitable basis with other forms of research.
Definition of Research and Eligible Outputs
1.11 The definition of research which applies in the exercise is as follows.
‘Research’ for the purpose of the RAE is to be understood as original investigation undertaken in order to gain knowledge and understanding. It includes work of direct relevance to the needs of commerce and industry, as well as to the public and voluntary sectors; scholarship*; the invention and generation of ideas, images, performances and artefacts including design, where these lead to new or substantially improved insights; and the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products and processes, including design and construction. It excludes routine testing and analysis of materials, components and processes, e.g, for the maintenance of national standards, as distinct from the development of new analytical techniques. It also excludes the development of teaching materials that do not embody original research.
* Scholarship is defined for the RAE as the creation, development and maintenance of the intellectual infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to major research databases.
1.12 In order to be eligible for submission, all items of research output must embody the outcomes of research as defined for the purposes of the RAE. Teaching materials are admissible if they can be shown to embody research outputs within the RAE definition, but the preparation of teaching material in itself is not accepted as a research activity for the purposes of the RAE.
Assessment Panels
1.13 The Units of Assessment for the 2001 RAE are the same as those used in 1996 with the exception that, following consultation, the separate Biochemistry UoA has been discontinued. All UoAs are described at the beginning of the appropriate criteria statement, in a format which indicates the key subject areas they encompass.
1.14 The membership of the assessment panels, the method of their selection, and an account of their role and operation, are set out in ‘Research Assessment Exercise 2001: Membership of Assessment Panels’ (RAE 3/99.)
Umbrella Groups
1.15 Throughout the RAE the Chairs of assessment panels in cognate areas will meet in Umbrella Groups. The Umbrella Groups are:
I Medical and Biological Sciences (UoA 1 to 17)
II Physical Sciences and Engineering (UoA 18 to 32)
III Social Sciences (UoA 33 to 44, 68, 69)
IV Area Studies and Languages (UoA 45 to 56)
V Arts and Humanities (UoA 57 to 67)
1.16 The role of the Umbrella Groups is to facilitate cross-panel consistency in the assessment process and application of standards. Particular attention will be given to areas of work which span the boundaries of units of assessment, including interdisciplinary research.
Rating Scale
1.17 Ratings of research quality will be expressed in terms of a standard scale with common definitions of the points. The rating scale and definitions which will be used in the 2001 RAE are shown below. Full notes on the rating scale are given in RAE 2/99.
5* (five star)Quality that equates to attainable levels of international excellence in more than half of the research activity submitted and attainable levels of national excellence in the remainder.
5
Quality that equates to attainable levels of international excellence in up to half of the research activity submitted and to attainable levels of national excellence in virtually all of the remainder.
4
Quality that equates to attainable levels of national excellence in virtually all of the research activity submitted, showing some evidence of international excellence.
3a
Quality that equates to attainable levels of national excellence in over two-thirds of the research activity submitted, possibly showing evidence of international excellence.
3b
Quality that equates to attainable levels of national excellence in more than half of the research activity submitted.
2
Quality that equates to attainable levels of national excellence in up to half of the research activity submitted.
1
Quality that equates to attainable levels of national excellence in none, or virtually none, of the research activity submitted.
1.18 Panels will reach a judgement on the balance of quality across all activity submitted. Within their submissions, institutions will be able to describe the configuration of their research groups and structures.
1.19 Panels will be encouraged to indicate research areas which are identified as having a quality of research which is at least two points above the overall rating for a submission. These "flags" will be published alongside the ratings.
Content of Submissions