REF Briefing Event

RIBA, London

Thursday 31 May 2012

Introduction

On 31st May Professor Caroline Strange and Dr Tim Brooks attended a REF briefing event organised by the REF team, the fifth and final event of a roadshow which took place around the country during the second half of May.

Slides from the various sessions during the event (with the exception of the opening introduction) are attached to the accompanying email. These notes are intended to be read alongside them. As ever some of the most interesting information arose from responses to questions asked by members of the audience, rather than from the material the REF team had prepared.

Introduction to the Briefing Event(Graeme Rosenberg, REF Manager, HEFCE).

Survey on Submission Intentions

The Survey on Submission Intentions – which institutions will be invited to respond to by letter by the end of July 2012, and which will be open for business between September and December 2012, will collect information to enable the REF team and Main and Sub-Panels to estimate the likely workload and areas of expertise they will need to be able to handle, for example to inform the appointment of additional assessors. Information collected will include, for each proposed UoA submission:

  • staff volume (FTE for Category A staff; headcount for Category A and C staff).
  • 300-word statements on ‘areas of impact’, explaining the areas of impact in which case studies are to be presented. In this, for example, stating that a case study will focus on ‘policy impact’ is insufficient; the REF team is looking for more detail without repeating the case study, e.g., ‘impact on national housing policy’.
  • explanations of research specialisms within submissions. These may be the names/areas of the research groups within a submission, or may make use of the detail of UoA descriptors – different panels will have different expectations based on their disciplines. There is no difference between a ‘common’ research specialism or an ‘unusual’ one – while for the latter the panel may have to draft in additional specialist expertise, they still need to know about the former to ensure they have sufficient and appropriate panel membership to deal with it.
  • expected volume of cross-referral requests
  • languages other than English used in submitted outputs, and volume.

Collection of outputs

The REF team’s intention at present is that outputs will be collected by the same means as for the RAE, that physical outputs will be deposited with them in a warehouse facility, while electronic outputs with DOIs will be sourced directly from the publishers by the REF team. (There was some suggestion, not expanded on, that non-DOI electronic format outputs might be submitted electronically through the REF submission system rather than on CD-ROM, as was the case for RAE).

However, recent pronouncements by HEFCE on the desirability of open-access forms of publishing and the possibility of mandating this in some way for the next REF has, understandably, dismayed the publishers, and negotiations to arrange DOI-based access have stalled. However, the REF Manager is confident these issues will be overcome.

Panel Criteria Guidance (Graeme Rosenberg, REF Manager, HEFCE, and Professor Dame Ann Dowling, Chair, Main Panel B)

Codes of Practice (slide 7)

The REF’s Equality and Diversity Advisory Panel (EDAP) will produce reports on good practice in the Codes of Practice it assesses, as soon as possible after the close of the submission period for Codes of Practice (31 July 2012).

Institutions’ Codes of Practice will be published alongside submissions, after the assessment process is complete. (If the RAE 2008 can be used as a model, for REF this is likely to be some time in early 2015).

Clear Circumstances (slide 9)

The REF submission system will include a ‘clear circumstance calculator’ which will work out the correct reduction for individuals with clear circumstances. (See the Submissions System section slides).

Complex Circumstances(slide 10)

The REF submission system cannot calculate complex circumstances reductions.

REF 1b – the section where explanations of individual personal circumstances are made – should be completed without referring to the member of staff by name. This will allow EDAP to assess the claim anonymously. Similar practice should be followed in institutions.

If EDAP disagrees with an institution’s justification for a reduction on the basis of complex circumstances, the ‘missing’ output will be rated unclassified. There is no facility for including a reserve: the argument is that if reserves are included, this weakens any claim that individual staff circumstances justify a reduction in the number of outputs required. Where an individual is submitting four outputs, there is no need to declare either clearly-defined or complex circumstances where these exist. The one exception is ECRs, whose status as such must be declared.

If institutions have examples to which one of the existing case studies cannot be applied, they are invited to ask the REF team to have EDAP prepare a further case study.

Staff who have retired and, after a period away, are brought back on a fractional contract, can be submitted to REF. Their break in service counts as an ‘absence’ under the REF rules, and should be treated as a career break as in other examples provided.

Staff who have held a teaching contract for some of the REF period cannot count their period of teaching only as a ‘career break’ because REF rules consider a career break to have taken place outside the HE sector.

Staff on management contracts, but who remain research active and could be submitted to REF, must have an eligible contract – their management contracts will need amending. Management activities in themselves do not constitute a legitimate reason for reducing outputs required.

Sub-panel working methods (slide 14)

The calibration exercises to be run by each sub-panel, and within and between Main Panels, should mean that different disciplines and subject areas are assessed equitable and evenly. This should, for example, mean that within the Psychology sub-panel, applied psychology research will not be disadvantaged by comparison with ‘hard’ Psychology.

The level of detail to which outputs will be scrutinised will depend on the time available. This in turn is also dependent on the numbers and types of outputs submitted and the numbers of assessors and members on the sub-panel. It is expected that most outputs will be read twice, but some outputs, for example edited books and monographs, may only be read once.

Additional assessors (slide 15)

Additional assessors will be employed where needed to cover additional areas of expertise etc. Additional assessors will either have an academic or user background, and will assess either outputs or impact as a result. They will not, this implies, look at both.

Interdisciplinary research (slide 16)

Cross-referral may, exceptionally, take place and it can be either at the request of the submitting HEI (if the panel agrees) or initiated by the panel. Cross-referral may be of a single output, the work of a single individuals, or those of a research grouping – so there is a great deal of flexibility.

Output Co-authorship (slide 19)

If in a single submission, i.e. UoA, the same co-authored output is, exceptionally, submitted more than once (for a maximum of two authors) then a statement is required justifying the inclusion of that output, describing the scale of the research and the distinct and substantial contribution each author made. Where the same output is submitted to a different UoA, or to the same UoA but in a different HEI’s submission, this is not required.

Citation data (slide 21)

The sub-panel for Computer Science will make use, in addition to the SCOPUS citation data, information sourced from Google Scholar. Google Scholar citation data will not be part of the citation information provided by the REF team through the submission system.

Some examples of impact (slide 26)

The examples shown on this slide are drawn together from all of the Main Panel criteria.

Case Studies (slide 28)

Ideally, evidence of impact should be quantitative, where this is available.

Panel members and assessors won’t follow up, for example, links to websites, to obtain additional information. Case studies need to provide all of the information required to assess the impact claimed, though of course a website may be included as a corroborating reference. References and corroborative evidence will be sampled as a means of audit.

There was an interesting debate around how a piece of research can have both beneficial and negative impacts. For example, research might lead to a new work-flow process which brings greater efficiency, thereby having economic impact. However, the same efficiency savings lead to job losses. No conclusion was reached as to whether the impact might be ‘marked down’ because the benefits were offset by negative consequences, but it was an interesting discussion.

Of the panel, one academic member and one user member or assessor will look at each case study.

Underpinning research (slide 29)

Panels will not, as a rule, request copies of the outputs referenced as underpinning research. It may be that panel members want to look at some of these outputs, but in the first instance they will try and source the outputs themselves through their own institutional subscriptions etc. In the unlikely event that they cannot do so, submitting HEIs will be asked to provide them as an audit query, but REF recognise and will ensure the need for sufficient time to do this. However, HEIs are advised to ensure that outputs listed to corroborate underpinning research are available!

The assessment of underpinning research is only to ensure that the research meets the 2* quality threshold, and as such will be light-touch.

Data and Audit (Anna Dickinson and Vasanthi Waller, REF team)

Obtaining Citation Data (slides 5-9)

Citation data will only be available via the REF submission system for output information entered against those UoAs making use of it. REF team members talk in terms of ‘requesting’ citation data, but the process is essentially instantaneous.

When citation data is requested, the system may return a number of responses, ranging from ‘matched’ to ‘no match found’. In some cases, it may be necessary to take further steps – e.g. to manually identify the correct output from a list provided by the system. HEIs are required to have entered all outputs and attempted to match the outputs for citations at an (unspecified) deadline ahead of the actual submission deadline.

Ideally, the output will be cleanly matched. The screen will show both the metadata for the output originally submitted for matching, and the metadata identified within Scopus. It is not necessary for institutions to have a Scopus subscription to view this data.

Scopus data provided through the REF submission system will not be “live”, but will be refreshed on a weekly basis.

Contextual Citation Data (slides 13-15)

Contextual citation data – demonstrating, for example, citation ‘norms’ in a particular subject area – will be made available to panels in early 2014. This will cover the period from 1 January 2008 to 31 December 2012. Equivalent data, covering the period 1 January 2007 to 31 December 2011, will be made available to HEIs in early 2013. The REF team believes that this will be sufficiently in line with the data the panels will use, to make it valuable for institutional planning.

In the assessment itself, panels will use a ‘snapshot’ of citation data taken on a particular date (to be specified) after the submission deadline has passed, rather than citation data on the day of assessment, for consistency.

Environment data (slides 16-17)

Unlike the RAE, research environment data (doctoral degrees awarded and research grant income) relates to the whole UoA, rather than just the individual members of staff submitted within it. (See paragraphs 166 and 171 of Assessment Framework and Guidance on Submissions).

Data on doctoral degrees awarded for the 2012-13 academic year will be calculated by institutions alongside their preparations for their HESA return, rather than by using their HESA returns. One of the pieces of check documentation that will be returned by HESA as part of the 2012-13 data submission will be the doctoral degrees awarded data, which will be used by REF to check the data submitted directly to them. It is likely that this check documentation won’t be returned by HESA until the end of 2013, too late for HEIs to make use of in their REF submissions. Presumably the same check response applies to financial data returned, though this was not specified; certainly we are not expecting to be able to make use of HESA returns here either.

Research doctoral degrees awarded (slide 19)

Doctoral degrees awarded refers to awards made to home and overseas students only. Awards made to students recorded in the ‘aggregate offshore record’ (students studying (to date) wholly outside the UK who are either registered with the reporting institution or who are studying for an award of the reporting institution) do not count.

MPhil awards do not count, whatever HESA may have previously said about its definition of ‘research degrees’.

Awards are only counted in the first year they are awarded – there are some instances of institutions returning the same award to the same student in several years. Data going back to 2007-8 will be used to check for duplicate awards. (No response was given to my query about how this could be done, given we have previously been told that the reason for leaving the 2007-8 research environment data out of REF was that it was not defined in the same way, so using it for audit seemed contradictory).

For some UoAs, additional research environment data is requested in the REF5 narrative. For Business and Management Studies (UoA 19) and Education (UoA 25), submissions are required to include data disaggregating between PhD and Professional Doctorate awards.

Research income-in-kind (slides 22-23)

Research income-in-kind data originating from RCUK will be supplied in a few weeks’ time, via the REF team. Income-in-kind data will be provided using the template provided in slide 23. In addition to the RCUK data, various income-in-kind data will be provided by various health research funding bodies, e.g. NIHR, but no further information or timescale was given for the supply of this information. Other research income-in-kind data is not eligible.

Returning data and data adjustments (slides 24-26)

HEFCE does not expect to see data returned against UoAs to which we are not otherwise submitting; at the same time, we aren’t expected to return data in UoAs with no connection to that data. It was pointed out that the rules and regulations for REF don’t prevent HEIs from doing so, although the person posing the question acknowledged that tactically allocating environment data to UoAs went against the spirit of the exercise. The presenters agreed to take this point back to REF for additional guidance.

Various submission system limits have been put into place, which cannot be exceeded by institutions. These limits are the HESA figures for the institution as a whole for 2008-9 to 2011-12 with various caps e.g. £200k and 5% over the period. Limits for 2012-13 cannot be built into the system but the REF team will manually cross-check with HESA data for that year. The REF team does expect to spot what was described as systematic over-reporting that is within the system limits.

Where institutions are aware they wish to submit data exceeding the system limits, they can request permission to do so from the REF team. (This replaces the instruction in the REF guidance that HEIs should contact their funding council). HEIs should explain why the data they wish to submit was not included in the HESA return. If the request is accepted, the REF team will amend the system submission limits.Where UoA convenors identify the need to make such requests, they should raise it as soon as possible with Caroline and Tim.

Audit – staff (slides 29-30)

The REF team recognise that, despite the terminology used in Guidance on Submissions, contracts themselves do not always specify sufficient information to enable eligibility to be determined. Supporting documentation such as job descriptions can be used.

There will be a FAQ indicating the content of information that should be provided in REF 1b (individual staff circumstances) including for ECRs. Where evidence is requested through audit of, for example, ill-health, the REF team will not expect to see primary evidence (e.g. doctor’s notes). They will, however, require evidence that such evidence has been seen by submitting HEIs.

For staff who come from another institution and whose individual circumstances relate to their period of service there, we would need to obtain verification from that institution as to, for example, part-time working, maternity leave and so forth.

Evidence for an ECR might be an extract from their CV.

The REF team seems to be relatively understanding that, particularly for older evidence, some simply isn’t now available – e.g. HR records going back to 1992.