REF Event – 22 September 2011

The Oval, Kennington

See Slides

  • More briefing events Spring 2012
  • Panel criteria and working methods consultation: 5 October 2011
  • Separate response for each of the main panels
  • July 2011: published in draft form
  • Will produce FAQs from briefing events and booklet on REF for users.

Consultation responses to

Part 1 – generic – sets out how all panels will operate

Part 2 – main panel criteria and guidance

The working methods of each main panel and its sub-panels

Can ask for greater clarity

Consistency across the 4 main panels –

Impact – A and B have extensive examples, panel C only general comments

View is that more examples is helpful.

In Panel A

  • multiple submissions are not allowed.
  • Panel A trying to encourage interdisciplinary research
  • Need to demonstrate that areas are academically distinct.
  • Southampton would want multiple submissions in Panel A
  • Timing for decision on multiple submissions is too late.
  • The same impact can be claimed by different UoAs if there are different aspects to the impact.
  • Impact is more likely to cross disciplinary boundaries.
  • Do look at UoA descriptors – bits that are missing

In Panel B

  • Allows multiple submissions but general engineering does not permit multiple submissions
  • Does allow research groups in a UoA. Main Panel A is looking for research groups rather than multiple submission.
  • Outputs
  • Criteria
  • Output types
  • Co-authorship
  • Double-weighted outputs
  • Additional information
  • Use of citation data

In Panel D

  • Expectation about performance type outputs.

Co-authored outputs

  • Panel Abut need to provide description about contribution
  • Panel Bno description required except Physics more than 5 authors
  • Panel Cwill expect a description if output submitted more than once
  • Panel D as Panel A

Always give priority around the panel for single-authored outputs, particularly in the social sciences.

If cross referred, an output will be sent to another panel, but this advice is only advisory and the original panel still judges the output’s worth.

Co-authorship – is it clear what information is asked for in panels A and D?

Double-weighted outputs

  • not clear in the guidance.
  • Need double-weighted outputs across all panels.
  • Not in Panel A.
  • Monographs might be double-weighted.
  • Possibility of putting in a reserve item.

Citation Data

  • Panel A will be using citation data
  • Panel B variable using Google Scholar
  • Panel C variable
  • Panel D not using citation data
  • Panels will get the raw citation counts and info on the reliability of the data.
  • Very concerning about use of citation data – should not be used without contextual data.
  • What is normal in a given field in a given year?

Impact

  • Read carefully – what is criteria and differentiate from what is advice – this is guidance
  • Each panel has described a range of impact
  • Impact is an area where panels are really looking for feedback from HEIs
  • Need evidence of these:
  • quality of output
  • relation between research and impact of impact

Panel C helpful

Must demonstrate that quality of research is a minimum of 2*

Impact template

  • Panel B criteria possibly not consistent
  • Criteria defined for reach and significance: a holistic judgement - don’t need 4* in both to achieve 4*
  • Reach is not geographically defined – the spread of impact does not necessarily indicate qualify of impact.
  • Understanding the diversity of beneficiaries
  • Can have research und3ertaken within the HEI in say 1994, person has left, but research has been continued with other staff at the HEI.

Environment

  • Asking for a lot less than in RAE 2008
  • PGRs and research income
  • Panel B is asking for additional data

Template for Environment

  • Overview, strategy, infrastructure, collaboration and contribution to the disciplines
  • Criteria of vitality and sustainability
  • Panel B asking for FTE of students
  • Some panels are asking for FTE of support staff on submission date
  • Need to ensure that HESA data is being collected and applied consistently by the panels

Responses must be made by mid-day 5 October online.

Separate response for each main panel.

Responses will help refine the criteria and working methods.

Supporting and Promoting Equality and Diversity through REF

Lessons learnt from RAE 2008

RAE evaluation suggested need

  • More consistence across panels in criteria and processes relating to individual staff circumstances
  • Better (REF specific) training
  • Proactive proforma-based procedure for disclosing individual staff circumstances (we need to ensure we get the disclosures)
  • Better communication of Codes of Practice
  • More timely appeals process (must be soon enough for a staff member to be submitted)
  • To use next exercise to promote equality and diversity more

Examples of good Codes of Practice on Dianne Berry’s E&D website from the RAE.

New equality law since RAE 2008.

Overall approach in REF

  • Equalities embedded
  • Equality and Diversity Advisory Group/Panel (EDAP)
  • Promotion of E&D through environment template
  • HEIs required to develop, document and apply Code of Practice on staff selection to:
  • Ensure work of ALL excellent researchers, regardless of circumstances
  • Help compliance with equality legislation
  • Avoid inadvertent discrimination

There will be a review of effectiveness of measures for REF

Changes in HESA staff records in 2012 to include info and staff circumstances

Individual staff circumstances

  • Age
  • Disability
  • Marriage and civil partnership
  • Pregnancy and maternity
  • Race
  • Religion or belief
  • Sex (including breastfeeding and childcare)
  • Sexual orientation
  • Part time and fixed term employment status

Codes of Practice

  • Must be signed off by Head of HEI and submitted to REF team by July 2012
  • Will be reviewed by EDAP in advance of submission deadline
  • Will be published with submissions at end of process

The Guidance on Submissions provides detailed guidance

  • Return – code of practice that could not be applied effectively and fairly

Look at how your RAE 2008 code of practice matches the new guidelines and the new legislation.

Information will be placed on the E&D website next week.

Underlying Principles

Code should demonstrate fairness

  • Transparencyhow accessible is it to staff

Keep it succinct

Staff need to be aware of it, on maternity leave

  • Consistencyacross the HEI

clarity as to which staff situations are considered

  • Accountabilitywhat training has been conducted since Equality Act Oct 2010

Have all UoA convenors been trained

  • InclusivityWhat messages are managers sending out re submission to the REF?

What needs to be included

  • Information on staff and committees responsible for staff selection, including terms of reference, role definitions and training undertaken
  • Information on criteria to be used, how communicated, feedback to staff, appeals process and timescales
  • Don’t have to say the actual threshold level ie 2*, but must give contextual information about the level you are seeking. How are these levels being communicated to staff?
  • Look for FAQs to provide to provide clarity
  • If you have an appeals process, this will need to be detailed. Timescales that staff need to be aware of

Equality Impact Assessments – these are required

  • Required to conduct EIA on policy and procedures for selection staff to determine whether selection policy may have differential impact on particular groups
  • Should inform Code of Practice and be kept under review as submissions prepared.
  • Should be informed by analysis of range of data, including (eligible and submitted) staff data in respect of protected characteristics for which data available.
  • Should be reviewed in light of mock exercise, appeals, and final submission
  • Further information on ECU website

Impact assessment at the start of determining the policy of selection of staff. This assessment can be reviewed regularly and at final submission. Funding Councils require publication of impact assessment.

How to do this will be on ECU website for REF.

Equality and diversity team may have templates for this.

Question:Citations do have a gender bias. The Funding Council does not condone use of citations for selecting staff for submission.

Individual Staff Circumstances

  • Clearly defined circumstances (clearly defined periods of time)
  • ECRs, part-time ‘working’, maternity, paternity of adoptive leave, secondments or career breaks
  • More complex circumstances
  • Disability, ill-health or injury, mental health conditions, constraints relating to pregnancy or maternity in addition to clearly defined period of leave, caring responsibilities, gender realignment, other circumstances related to protected circumstances

(not easy to work out time lost to research)

Staff disclosure

  • REF team advise proactive approach (to ensure that all staff feel able to declare restrictions regarding reduced number of outputs)
  • Suggest giving every member of staff a form to complete
  • Use a central group to assess cases and advise of decisions
  • HEIs will need to consider confidentiality in procedures for staff disclosure (ethics issue)

Consideration of individual staff circumstances

  • Clearly define circumstances
  • More complex circumstances
  • Considered by EDAP who will advise main panel chairs on whether submitted number is appropriate

Submission of information

In REF1b (200 words for clearly defined circumstances)

(300 words for complex cases)

For complex cases:

  • Example cases on ECU website to provide guidance on how to structure information re EDAP’s likely response
  • By March 2011
  • The initial set will be supplemented if further cases are brought to their attention

Reduced outputs

Proposed tariffs allow for reduction in outputs in relation to:

  • ECRs – determined by date become independent researcher during the census period
  • Other clearly defined circumstances – reduction in contracts hours / FTE worked over census period

Need to look at consultation document for E & D

Guidance on submissions

Systems going to be using

Survey of submission intentions

Oct 2012 deadline – Dec 2012

  • Collect information to plan panel’s workload
  • Following this may need to appoint additional panel members
  • Proposal to use this submission intention to request multiple submissions – need to provide detail to support the request.
  • Response from REF team by end January.

None of the submissions intensions are binding

Only one change to ask for multiple submissions

Will be asking:

  • Total volume of submitted staff
  • Impact areas (and details of highly confidential impacts)
  • This is to help identify expertise needed to judge the impact
  • Highly confidential ie MOD, defence
  • Research specialism (and volume)
  • Outputs in welsh / other languages
  • Work where cross-referral will be requested?

Submitting outputs

  • REF team will attempt to source all submitted journal articles and conference proceedings in electronic format
  • Working with publishers (PLS) to gain agreement to access as in 2008 RAE
  • For all other outputs types, and where not available for publisher, we will require the output in either electronic form or a physical copy / appropriate evidence
  • Electronic wherever possible

Citation data

  • All in A, some in B & C, none in D
  • Using Elsevier and Scopus – raw citation counts
  • In September 2012 can load your outputs on pilot REF system and verify outputs and see citation count, system updated probably weekly to update citation counts
  • Panel will be looking at contextual information to go alongside citation counts. For REF 2014, the citation counts will be taken just after submission
  • Citation only to be used as a positive indicator.

HESA data

Want to align HEFCE data with HESA data

Data will be provided to us from the HESA data

Discrepancies in data

Proposal allow

  • 5% in total for all UoAs across all years
  • 10% in total for all UoAs, per year / income category
  • These limits may also have a number element
  • If an HEI expect this would be unachievable, they should have early discussion with the relevant funding body.

Impact case studies

Each study must

  • Describe and reference the underpinning research
  • Explain the link between the research and the impact
  • Explain and provide evidence of the nature

Impact stays with the institution, so research must have been done whilst staff is at institution, but that member of staff could have died, not being submitted. Panel may need access to the research.

No particular ‘model’ of impact generation is assumed.

The HEI need not have been involved in ‘exploiting’ the research

The contribution may be the sole research contribution, part of a collaboration or a whole body of research

  • Direct, indirect, linear or non-linear

No research = no impact

Panels are not going to follow up references. Case studies need to provide all the details to justify the impact.

Sources will be used for audit purposes only – to verify claims but not additional evidence.

Sources could include:

  • Details of factual statements by ‘users’ that the HEI has collected, and can be made available to the REF team
  • Documentation, web-content or report (whether in the public domain or available from the HEI)

In section 5 – do not put details of actual people, only titles

After REF 2014, HEIs will be able to redact info in the case studies or confidential case studies before they are published on the web.

Summary of Briefing Sessions

Equality Headlines

Option 2 for maternity – strong favour for this

1 maternity leave = 1 reduction in outputs

14 months too high

Panel criteria

Greater justification for sub-panel differences and panel differences

Multiple submissions in particular

Co-authorship requirement for text boxes in panels A & ?

Discussion re double weighting – option to submit a reserve across all the panels

How will underpinning research for impact be judged?

What is the disciplinary justification for using citation data?

How will reach and significance be combined and judged?

Guidance on submissions

Request for multiple submissions earlier

May not have full details for submissions intentions on case studies for impact

REF team will batch up all the questions and use this to form FAQs on the web.

What should the focus be for briefing next Spring?HHHighlyoino

1