UNIVERSITY OF ULSTER
JOINT FUNDING BODIES’ REVIEW OF RESEARCH ASSESSMENT

The University of Ulster believes that the Research Assessment Exercise has had a major beneficial on the quality of UK research across all disciplines. The exercise is widely respected by UK academics and the ratings have achieved international currency to the great benefit of many institutions. It has also led to the better management of research in UK institutions and, notwithstanding the national funding shortfall, the exercise has led to areas of true international excellence receiving selective support. In particular, emerging institutions have benefited by being able to focus their limited research funds on areas of recognized strength. We believe however that improvements can be made to the system and these are summarized in our responses below.

The following views have been drawn together from across a broad spectrum of opinion among the research-active community in the University. The question numbers refer to those posed in the notes for facilitators (Annex B).

Group 1: Expert review

7. a)The current practice should be retained ie. a combination of prospective and retrospective assessment but with continued emphasis on the retrospective.

7. b)Assessors should consider primarily publications, research context, indicators of esteem and research plans, whilst also taking account of the number of research studentships and level of external research funding associated with an area of assessment. Knowledge transfer should also be assessed positively.

7. c)The current practice is to assess the combined performance of a group of individuals within a subject or thematic area. We believe this practice should be retained and that the current subjects areas are broadly appropriate. There is a widely recognized need to include appropriate consideration of interdisciplinarity.

Group 2: Algorithm

9. and 10. We do not believe that it is possible to assess research entirely on the basis of metrics. This is particularly true in the Arts and Social Sciences. The sole use of metrics to assess research performance would have a detrimental effect upon the research activity, thereby adversely affecting the reliability of the metrics.

As well as those metrics already employed, namely the number of research studentships and the level of external research income, metrics associated with technology and knowledge transfer should play a greater role, particularly (but not exclusively) for disciplines in science and engineering and the social sciences. These include numbers of patents, licences, spin-out companies, joint ventures, external investments, uptake of research by users including changes in professional practice etc. These measures on their own, however, do not adequately assess research excellence in any subject area. Panels need to be more rigorous and consistent, across cognate disciplines, in their assessment of external research income.

The major advantage of using an algorithm to assess research quality is ease of administration. The major weakness of this approach is that it fails to take account of research realities, in which there is no meaningful relationship between some important aspects of research activity and metrics. It would also be difficult when using metrics to take account of regional variations in the economic, social and cultural environment which can have profound influences on outputs.

Group 3: Self-assessment

13The results of the RAE have had a major positive influence on overseas perceptions of the quality of UK research. Self-assessment could reduce the credibility of the exercise.

Self-assessment should encompass all the existing criteria, including publications, numbers of postgraduate research students and external research income. It should be both prospective and retrospective.

Assessment criteria should be the same for all institutions and broadly the same for all subjects while permitting criteria to vary to some extent according to the specific nature of the subject.

Validation of self-assessment could be achieved by the assessors examining selected publications chosen by the subject area as a reliable measure of overall research performance in that area.

It is considered that self-assessment and expert review would be at least equally burdensome, although, in order to ensure fairness, the checking of self-assessment might even be more burdensome.

The strength of self-assessment is largely administrative in that the assessors would not receive a mass of raw data as at present, but an analysis of the researchers’ assessment of their performance.

Group 4: Historical ratings

16. a)While recognising that researchers build up a research culture and strengths, it is inaccurate to presume that, in an individual subject, there cannot be a rapid change in performance. A historical ratings approach may preclude such rapid change from being reflected within a single review period and may, therefore, be expected to inhibit serious improvement (e.g. 3b to 5 as occurred in Law at Ulster between 1996 and 2001).

16. b)The usual indicators should be used to establish an institution’s baseline ratings ie. publications; external research funding; research students; research organisation; research centres; research culture; indications of esteem; knowledge and technology transfer including spin-out companies and uptake of research by users.

16. c)Identification of failing/outstanding institutions flows from the baseline ratings across a range of subjects at the time of the review. It is difficult to see how ‘value for money’ might be easily assessed by focusing solely on research performance.

16. e)The weakness of a historical ratings approach would be a tendency to disadvantage new institutions which lack a strong research track record. It would be likely to reinforce the existing inequalities within the system and to favour only very strong instititutions. It would be difficult to design a system which compensated for the disparity between more and less established institutions. Future RAEs must find ways of stimulating research at ‘grass-roots’ level.

Group 5: Crosscutting themes

17, 18. a)An assessment of the research base should be used as a driver for funding decisions and to enable the Funding Councils to allocate funds more strategically. Excellence should be rewarded by the provision of sufficient funding.

17, 18. b)Research assessment should not be on a rolling basis. An appropriate interval between assessment exercises is five years and all subjects should be assessed at the same time and with the same frequency. This is important for research planning within institutions.

17, 18. c)Excellence in research constitutes production of work in a subject to the highest possible standards, as assessed by the peer understanding within the discipline.

17, 18. d)i) The question of the relative value of subjects is a meta-question which can not be answered by considering the excellence of the research carried out within them.

In determining ‘subject pots’ with respect to funding, it is crucial that there not be a bias against Arts subjects or emerging subjects e.g. Nursing and the allied health professions. The level of total research funding should be significantly higher in Arts than it is at present. The findings of the UK Universities Review, in relation to the extent of the shortfall, are crucial here: the conclusion is a call for remedial investment of £5bn for Arts and Humanities.

(ii) In principle, the application of international benchmarks is desirable, with the caveat that the quality threshold must be determined by the subject benchmark. However, such international benchmarking should not be to the detriment of strategic development of a new area of research.

iii) A metric based on external funding should not be applied to the Arts and Humanities in determining ‘subject pots.’ While external funding in this area increases in importance, and is becoming a valid guide to excellence it is not, nor will ever be, as reliable an indicator as it is, say, for Science. In general greater weight should be given to outputs rather than inputs (including research income).

iv) Retention of the current balance in funding is not desirable. A system based on assessment would allow change in funding distribution, as well as reinforcing areas of quality.

v) See above responses for opinion on how the relative quality of research in different subjects should be assessed. It is important that relative quality is transparently assessed.

17, 18. e)The assessment should be at the level of subject, not institution. There should be consistency in the assessment between institutions, in order to enable the promotion of regional excellence and to enable new or improving institutions to attain equally good results as those of established institutions.

17, 18. f)Each subject or group of cognate subjects should be assessed in essentially the same way, while taking account of variation between discipline types, which should be recognised by different panel criteria. Participants should feel that the system is fair and transparent.

17, 18. g)The current level of discretion afforded to institutions in putting together their submissions should be retained.

17, 18. h)The principle of equal opportunity is enshrined in the normal operation of Higher Education institutions and should also apply to research and to the review of research. The only dangers in this respect occur if a review panel is not transparent and inconsistent with its own assessment criteria.

17, 18. i)The most important characteristics of an assessment process are that it be judged to be:

-not burdensome

-rigorous

-fair to individuals and institutions

-transparent

Group 6: Have we missed anything?

i)Excellent scores should be rewarded with excellent funding.

ii)It should be incumbent upon the review panel to offer a feedback report substantially longer and more comprehensive than the current reports. This should detail the texts reviewed in detail, assessment criteria and basis of comparison with other output. Specific reference should be made to the work of named individuals.

It is noted that the current feedback is bland and avoids incurring the danger of litigation. Consequently, it is ineffectual in helping an area improve its performance. The threat of litigation could be removed by putting in place a comprehensive appeals process.

It is felt that an appeals process should be instituted which should be two tier: first to establish a prima facie case for review and secondly to engage in a full review. The process should be expensive and demanding for institutions in order to discourage unrealistic requests. However, this would be better surely than the current system which can lead to judicial review and all of the expenses and potential negative publicity involved.

iii)There is a strong conviction in the research community that there is not a consistent application of standards between panels, especially in applying the relative importance of publications, research environment and evidence of esteem. There is also a perception that the definition of international excellence varies between panels.

iv)Assessment concerns: the RAE 2001 (13): It is not agreed that the design of the current RAE fails adequately to accommodate interdisciplinary research. Staff from different Units of Assessment who collaborate are advantaged in that the same outputs can be submitted to each of their respective panels.

v)Funding concerns (16): It is felt that the current level of total funding is inadequate.

vi)Underpinning principles (21): The University concurs with all of the points made in para. 22, re. the principles that should underpin a selective research funding methodology.

vii)The University would wish to stress the following points:

  • The assessment process must be totally transparent and must result in an ‘even playing field’ for all subjects and institutions. A greater level of consistency across panels is essential.
  • The appointment of panel members must be a more transparent process than at present. Currently, there is a perception that there is an advantage associated with membership of a review panel, therefore there should be greater rigour in the selection process. Greater attention must also be paid too ensuring that the panel has the appropriate range of expertise for the discipline.
  • There should not be a predominance of members from particular institutions on a panel. There should be a revolving membership, with the aim of ensuring that, over a period of time, most institutions are represented on a panel
  • There should be the opportunity for further panel members to be drafted in to assess specialist submissions
  • The RAE has proved to be highly valuable in that it has led to:

-significantly improved quality of UK university research

-improved management of research

-recognition of centres of research excellence outwith the older established universities

-internationally recognised and respected benchmarks for UK university research

Notwithstanding the shortfalls associated with any method of assessment, the RAE has been a major force for improving UK university research and this review should seek to improve rather than dismantle a highly successful process.