Joint Funding Bodies’ Review of Research Assessment
University of Wales College of Medicine (UWCM)
1.Expert Review
1.1We agree that future assessments of research should be a combination of prospective and retrospective review and that such a review would be a combined assessment of teaching and research. For scientific/medical subject areas UWCM believes that assessment should be made at research group level. This would complement the subject/thematic approach that most universities (including UWCM) adopt for managing their research. The main strength of this approach is that it will encourage collaboration amongst researchers and avoid research being conducted in isolation or in departmental silos. This ties in with the way funding bodies now approach the funding of university research, i.e. collaborative research is being increasingly recognised.
2.Use of an Algorithm to Assess Research
2.1The College broadly accepts that research assessment should be based on the use of metrics but not on these alone. Issues such as entrepreneurship, which may be more difficult to measure, will need to be considered. Awards of prestigious fellowships and prizes and invitations to speak at highly acclaimed events/conferences are also worthy of recognition. Use of metrics as the only means of measuring research strength may provide an accurate picture of funding, research student numbers (the numbers of PhD students are not necessarily an indicator of quality – evidence of timely completions may be) and publications, but does not allow activity which may not be the traditional business of a university to be adequately assessed.
2.2 Strength in research could also be viewed by the quality of the research environment including infrastructure, culture, codes of practice in relation to research supervision, high quality research training programmes, professional development opportunities The introduction of site visits as part of the RAE may encourage improved recognition in this area. Success in attaining external funding is a reasonable indicator of quality but may be unduly affected by the influence of priority areas and political imperatives on commissioned research.
3.Use of Self-Assessment
3.1The type of data that should be included in any self-assessment exercise would be as indicated in Section 2 above. Such assessments should be both prospective and retrospective. Criteria that universities should apply to their work would need to be carefully considered since the subject area of the research often determines the type of approach to carrying out that research. Disciplines having an additional mission, such as clinical practice, should be self-assessed differently (from the more traditional academic disciplines) because of the additional responsibilities placed on staff. Sources of funding must be taken into account too since certain funding, such as NHS educational contracts, do not usually make a provision for research activity.
3.2In cases where self-assessment and algorithm disagree, UWCM is of the opinion that self-assessment should be coupled with the use of an algorithm with expert review. UWCM does not have a strong view as to whether self-assessment or expert review may be more or less burdensome. Needless to say the validation of universities’ self-assessment will certainly make for an additional burden for the RAE process.
4.Historical Ratings
4.1A system that rates universities by taking into account past performance may be used but, as the consultation paper indicates, this could only effectively be employed if used in conjunction with another system that identifies institutions whose performance was obviously improving or declining. Using such a system where distribution of research strength is likely to change slowly is preferable. A rapid change would not allow universities the time to adapt and refine its strategies and policies. Rapid change may also carry potentially detrimental effects on employment of research staff (early termination of contracts, lack of renewal of contracts).
4.2Benchmarking and target-setting exercises would help identify institutions that are performing well. There is an issue here though for the less mature disciplines. In the case of UWCM, disciplines such as Nursing and Allied Health Professionals (AHPs) have under-developed research capacity and cultures at present. The use of historical ratings in these examples would therefore be less helpful but could be taken into account at a national level.
5.Cross Cutting Themes
5.1An assessment of the research base can be useful but only as providing a context for the fair assessment of the quality of research. It should not in itself be a determinant of quality. Assessment should be conducted every 4 to 5 years. The administrative burden is currently not commensurate with the output, therefore assessment at any greater frequency would carry an increased burden. Excellence in research should be viewed in its broadest sense, as evidenced by the quality of the research output including the originality of ideas, evidence of critical engagement with the field of study, clarity of exposition, theoretical sophistication, fit between research question and chosen method, transparency of research process, scholarship in interpretation and discussion of results. Research can be of high quality whether it is descriptive (i.e. qualitative) or quantitative. Ultimately, it is the question being addressed which is important. All good research should be innovative.
5.2The research assessment should determine the proportion of funding allocated to each subject but only if the assessment takes into account the issues highlighted in this report, i.e. that different subjects should be approached in different ways. However, at the institutional level, the assessment should be the same. During RAE 2001, universities were aware that RAE submissions were to be made available on the web for the first time. This meant that a greater level of discretion was needed in compiling the submissions. For future assessments, this may not be the case but is dependent on what approach the Funding Councils take as far as publicising the RAE results. The priorities that should be given to any future assessment exercise include transparency of rules and clear definitions (what is defined as international versus national excellence?) There should be much less scope for games playing.
5.3As themes predominate and cross cut between areas marking growth and success in a research area, UWCM is concerned that there is a danger of a successful team losing touch with its ‘roots’, ie its predetermined area of assessment. One potential example of this is clinical dentistry.
6.Other Comments
6.1The disparity of size between the current RAE subject areas is a significant weakness. UoA3 – Hospital based subjects – is a classic example. It is very difficult to sustain research quality (and a fair rating) across such a large area. The use of sub-panel ratings did help in 2001 but as there had been no link to funding their worth was limited. UWCM firmly believes that Medicine should have 4 if not 5 UoAs dedicated to it in order to provide an equality with other disciplines.
1
J:\POLICY\Research Policy Group\RAE review\Responses\AWalesMed.doc