,
Joint funding bodies’ review of research assessment
The Royal Statistical Society (RSS) welcomes the opportunity to contribute to the present review of research assessment. This response was prepared on behalf of the Council of the RSS by myself, as President, following consultation with other members of Council. It is one of two separate submissions that the RSS has prepared; this letter concentrates on some general aspects of the RAE process which cause us concern, that apply to all units of assessment, and on which we as statisticians have a professional competence. In the language of the Invitation to Contribute these are Group 5, cross-cutting themes. A second submission is being made jointly with the other learned societies in the mathematical sciences, namely the London Mathematical Society and the Institute of Mathematics and its Applications, and focuses on subject-specific issues.
We take it as axiomatic that some form of research assessment is essential, in order to guide funding decisions, and to provide proper public accountability to justify the public funding of research. Any credible system of research assessment, even if it were to place greater or lesser weight than the present RAE on either historical ratings or self-assessment, will involve some process of soliciting information from institutions, and of summarising and extracting key messages from such submissions. Such a system must possess all of the characteristics listed in Annex B, para 18i, of the Invitation to Contribute (not burdensome, rigorous, fair, …, etc.).
These ingredients of soliciting information and analysing its content are intrinsic to almost all research studies in the social sciences. Good practice, which provides academic objectivity and possesses all these same listed characteristics, is well-understood, and indeed commonly taught by statisticians in research methodology courses for social science postgraduates. Unfortunately, this good practice was not appreciated by those designing the RAE, and the research assessment was therefore conducted to inferior standards. We do not understand why this should have been so.
Rather than provide an exhaustive critique of the existing practice, we prefer to cite just two examples, one concerned with soliciting information, the other with its analysis.
(a)The way in which information was sought from institutions allowed too much subjectivity in responses so making uniform interpretations very difficult – at least in some subject areas. On at least one panel, assessors were reduced on many occasions to `reading between the lines' because of the looseness of what was asked for. There is much expertise among survey statisticians about how to word questions so that objective analyses can be done on the responses. The RSS suggests that the HEFCE should utilise such expertise when designing procedures for the future. For example, as often as possible, respondents should be given a list of options to choose from rather than asked to provide verbal descriptions whose formats are very difficult to standardise.
(b)The sampling of outputs was subject to very unfair rules. At least in one panel, and probably in many, the rule was to read roughly the same percentage of outputs from each submission. In some subject areas this could have been 100%, but in large subject areas this meant that for large departments a sensible number was sampled to enable reasonable accuracy whereas for small departments this was not so. We know that assessors did protest at the technical inadequacy of this procedure but were told the rules could not be changed. (It is of course very well-known that in survey sampling the precision of estimation depends primarily on the absolute size of the sample, not its size relative to that of the population. That is why the absolute sample size is quoted when opinion polls are published.) The RSS suggests that a sensible procedure would be to sample (randomly) enough outputs that the percentage accuracy of the final (average) grade in terms of the outputs is the same for departments of any size. There are some additional technical issues, such as how one decides to aggregate gradings from the outputs. Statisticians are familiar with these issues.
We invite the review steering group to share our view that professional expertise relevant to the conduct of research assessment exercises, of any form,
(i)does exist,
(ii)is not being used at present, and
(iii)should be in the future.
The time-scale for this consultation prevents us from proposing a fully-designed scheme that would meet desired standards, but we invite the funding bodies to consult suitably qualified statisticians in designing modifications to the current procedures. The Royal Statistical Society is willing to assist in facilitating such consultations.