Appendix QMEC 3/23 (11-12)

Quality Management and Enhancement Committee

Internal Quality Review Working Group

REVIEW OF IQR PROCESS 2011-12 – Final Report

______

Summary:
In January 2012,a working group of QMEC, the IQRWG, was charged to review the scope, methodologyand timing of IQR and its continued fitness for purpose.The group’s interim report to QMEC on 13 March 2012 contained a number of overarching recommendations concerning the continued focus and content of IQR which were approved. This, the IQRWG’s final report to QMEC,sets forththe group’s recommendations for enhancing the IQR process.
Action proposed to the Committee:
QMEC is invited to endorse (i) the recommendations for enhancing the IQR process set out belowand summarised at section 3 and (ii) the associated implementation plan at Appendix A.
Authors:
Professor David Shanks, Head of the Division of Psychology and Language Sciences/Chair of the IQRWG;
Ms Sandra Hinton, Senior Quality Assurance Officer, Academic Support/member of the IQRWG.

Abbreviations:

ACAcademic Committee

AMAnnual Monitoring

AMRAnnual Monitoring Report

AugAMAugmented Annual Monitoring

AugAMRAugmented Annual Monitoring Report

CSSCorporate Support Service

DLTSDepartmental Learning and Teaching Strategy

DocTCDoctoral Training Centre

DSSCCDepartmental Staff Student Consultative Committee

DTCDepartmental Teaching Committee

ESCILTAExecutive Sub-Committee for Innovation in Learning, Teaching and Assessment

FTCFaculty Teaching Committee

HEARHigher Education Achievement Record

HEFCEHigher Education Funding Council for England

HEIHigher Education Institution

HoDHead of Department

IQRInternal Quality Review

IQRWGIQR Working Group

JSSCJoint Staff Student Committee

KISKey Information Set

PGRPostgraduate Research (student)

PGTPostgraduate Taught (student)

PSRBProfessional, Statutory or Regulatory Body

QAAQuality Assurance Agency

QMEQuality Management and Enhancement

QMECQuality Management and Enhancement Committee

SESSelf-Evaluative Statement

StARStudent Academic Representative

UCLUUCLUnion

UGUndergraduate

1BACKGROUND

1.1The most recent QAA Institutional Audit of UCL in March 2009 identified IQR as an example of institutional good practice. This can be taken as evidence that the process is largely robust, effective and consistently applied. Since its inception in 1992, the IQR methodology and its practical implementation have been periodically subject to a number of major reviews and in addition, to continuous minor revision and enhancementby QMEC, the IQR Panel and by Academic Support[1]. It was in keeping with this ethos of continuous enhancement therefore, that QMEC agreed that it would be timely to review IQR once more to ensure that it remained fit for purpose. A working group, the IQRWG, was established to review the scope, methodology and timing of IQR and to report its findings to QMEC. The Terms of Reference and Membership of the IQRWG can be found at Appendix D to this report.

1.2The IQRWG has now met three times; on 9 February, 19 April and 29 May 2012. On 13 March 2012, the working group submitted an interim report[2] to QMEC which detailed the preliminary discussions of the working group on some overarching questions regarding the future focus and content of IQR and which made a number of recommendations in respect of these which QMEC approved. These were as follows:

(i)That the normal unit of review for IQR should continue to be the academic department, with the process being adapted where necessary.
(ii)That the focus of IQR should remain on the management and delivery of teaching and the student experience and not be expanded to incorporate periodic programme review.
(iii)That a working group of QMEC be set up specifically to review the periodic programme review process (AugAM) at UCL.

1.3In the course of the working group’s discussions regarding the optimal unit of review which would enableall the issues covered by IQR to continue to be adequately explored, the group noted the proliferation of alternative units (centres, institutes, DocTCs etc.) for the management and delivery of some or all of these elements. In view of the increasing complexity of these units, the IQRWG recommends that, although the normal unit of review for IQR should continue to be the academic department or equivalent (and as formally constituted by UCL Council), UCL should review the criteria for assessing how IQR should be conducted in respect of these alternative units. To this end, QMEC officers will develop appropriate criteria for defining whether a unit should, in broad terms, be reviewed discretely or as part of the IQR of the academic department with which it is most closely associated. Officers will then consult with individual Faculty Tutors to confirm how the individual units in each faculty are to be treated in accordance with these criteria.

RECOMMENDED TO QMEC
(iv)That QMEC officers (i) develop appropriate criteria for the application of the IQR process to those academic groupings which are not formally constituted academic units by UCL Council and (ii) consult with each Faculty Tutor to confirm the treatment of individual units within his/her faculty.

2ENHANCING THE IQR PROCESS

2.1Having received QMEC’s approval to proceed on the basis that the focus of IQR should remain, for the time being, the same and that the normal unit of review for IQR should remain the academic department or equivalent, the groupthen met to consider howthe IQR process might be further refined and enhanced and how engagement (particularly student engagement) with it might be further increased. The group’s discussionson the remaining issues set out in its terms of reference can be foundunder the relevant headings below and have given rise to a number of recommendations for action which are also summarised at section 3below.QMEC is invited to discuss and approve these recommendations and the associated implementation plan, which can be found at Appendix A to this report. Also, in the context of this implementation plan, it is suggested that the IQRWG is not stood down but retained as a useful source of advice and expertise for the resolution of any issues which may arise in implementing the recommendations.

2ALinks between IQR and other internal planning and review processes

Background

2A.1The IQRWG concentrated particularly on those review processes for strategic academic and resource planning, with a view to identifying and eliminating any perceived duplication or overlap between them.UCL has a number of internal procedures, both annual and periodic, for which staff in departments, faculties and the CSS divisions are required to complete documentation of varying kinds. These procedures are intended to serve different purposes, and to meet a number of requirements, both internal and/or external. Details of some key processescan be found atAppendix B to this report.

2A.2It can be seen that while the purposes of each of the internal procedures listed at Appendix B are fairly distinct, some of the information required, particularly factual information about a department (etc)’s provision, are similar.In particular there appeared to the IQRWG to be some duplication of information required for the DLTSwith that required for IQR and AM/AugAM. This duplication, whether real or perceived carries with it thepotential to jeopardise staff engagement with IQR and AM/AugAM and for this reason, the IQRWG considered that it should be addressed.

IQR and other internal procedures

2A.3The procedures for AM, AugAM and IQR were developed, (and are continuously reviewed, monitored and refined) with full committee and academic staff oversight (by QMEC and its IQR Panel). These processes have been explicitly developed to inform one another[3]. However, while the 2007 iteration of the DLTS proforma and guidance was developed with oversight from AC’s Executive Sub-Committee for Innovation in Learning, Teaching and Assessment[4](ESCILTA), the circumstances of the mostrecent DLTS initiativewithin the Office of the Vice-Provost (Education)were less well known and the role and purpose of the DLTS initiative seems also to have been insufficiently communicated to departments.The DLTS review group[5] charged with reporting to AC on emerging outcomes of DLTSs completed in March 2012 acknowledged in its reportto AC on 10 May 2012 that ‘many drafters had found the template unwieldy, and found the format of the guidance restrictive (and an invitation to repetition)’. The review group concludes that ‘although many strategies were comprehensive and thoughtful documents, a full review of the template is necessary to make the process of completing a strategy document a more user-friendly experience’.

2A.4On 15 May 2012,three IQRWG members[6]met with the Vice-Provost (Education) and staff from his office to (i) discuss the role, purpose and format of the DLTS proforma and (ii) consider how this and the information requirement for IQR, AM and AugAM might be more effectively integrated. A productive discussion was hadin which it was agreed that there was certainly scope for ensuring a closer integration between the DLTS and other review processes.Also, and as noted in its report to AC, the review group will consider the timing of future DLTS requests, to ‘ensure that drafters have sufficient time to plan for and complete the document, and to ensure alignment with other strategic planning and reporting activity.’Finally, the Vice-Provost (Education) will also (at his request) be included in the membership of the working group of QMEC proposed by the IQRWG to review AugAM (see 1.2 (iii) above)where the fit between AugAM and the other review processes can be more fully explored.

2A.5By these means, it is hoped that the future development of these key internal review processes can be conducted in a more joined-up way and with clearer institutional oversight of the full range of processes in which staff are required to participate, their role and purpose and their relevance to one another. Certainly any reduction in duplication of effort, whether real orperceived, would be one way to increase engagement with all processes.

IQR and resource planning

2A.6Turning to the link between IQR and strategic or academic resource planning, the IQRWG noted that sometimes, an IQR may explore issues relating to the student experience which are the consequence of planning and resource allocation decisions made by units other than the departments being reviewed (ie. Faculties and CSS divisions). Where these processes are being investigated in IQR, ithas occasionally been suggested that IQR is straying beyond its remit. However, where institutional issues or strategies are clearly impacting on the student experience (eg. in its first year of implementation, issues relating to the functioning of Portico featured in several IQRs) the Group felt that was right that IQR teams should have a view on these issues; they are integral to the student experience and delivery of teaching and review teams cannot therefore ignore them. However, it is sometimes a delicate matter for review teams to make recommendations on resourcing issues without feeling that they are straying beyond theboundaries of the IQR. Also, departments may well, in light of the new funding regimescheduled to begin in 2012-13, increasingly point to resourcing issues as a legitimate response to any criticisms raised by IQR. Teams will need to be sensitive to the financial context in which the department is operating and may need to be given more pre-visit information/data in this respect; for example, the departmental SES could give contextual information regarding the department’s overall financial positionand strategic plan.

RECOMMENDED TO QMEC
(v)That the departmental SES should give additional, contextual information regarding the department’s overall financial position.

2BStudent engagement with the IQR process

Background

2B.1Following full discussion at its meeting of 9 March 2010, QMEC approved a number of measures to increase student engagement with IQR. Interviews with students have always been key to the IQR process. Students (including UG, PGT and PGRstudents) are interviewed during the IQR visit without departmental staff present and minutesof the DTC and theDSSCC, which have student representation, form part of the documentation for the review. However, in March 2010, the IQR Panel and QMEC discussed and approved a number of measures to further increase student engagement with IQR. These were:

- the draft SES must be discussed by the DTC prior to its submission to the Review Team;

- departments must submit the IQR Report and Action Plan to the DTC for discussion (in addition to their current discussion by FTCs, which are also attended by student representatives);

- departments must make the final IQR reports and action plans accessible to students in the department, e.g. by making these public on departmental intranets;

- a student representative on the IQR Panel.

2B.2These were implemented in session 2011-12 and initial feedback received suggests that they are working well. However, the measures stopped short of inviting students to become members of IQR teams. Having considered all the issues, QMECconsidered at the time that extensive efforts to include student reviewers on IQR Teams might divert existing resource and attention away from more productive means of student involvement in IQR without delivering real benefits, but agreed to review this position when necessary. QMEC also resolved to monitor the development of pilot projects to include student members on internal review teams at Anglia Ruskin and RoehamptonUniversities and a chart showing emerging outcomes from these can be found at Appendix C to this document.

Changesto the higher education landscape

2B.3New arrangements which the QAA introduced from September 2011 emphasise the need for students to act as participants in quality assurance as well as to be recipients of information on outcomes.Each institutional review team now includes a student and there are various opportunities for QAA review teams to seek the views of students built into the new review method. The June 2011 Government White Paper – Higher Education: Students at the Heart of the System, emphasises the general intention for students to be more active participants in QME issues which affect their education, particularly in light of the introduction, in 2012, of the new funding regime. The new UK Quality Code effective from 2012-13, also emphasises a move away from the traditional ‘representational’ model of student engagement towards a more participatory approach. There is a greater focus on students being increasingly active participants in their own education and they therefore can and should be directly involved in their own institution’s approaches to QME. In the view of the working group, students can also bring a valuable perspective and experience to internal QME processes. It is therefore now the right time for UCL to include students as members of IQR teams and the group considered how this might most effectively be done.Discussions with the universities piloting use of student members of internal review teams (see Appendix C) revealsconcerns that students might be added in a ‘tokenistic’ way in order to satisfy external scrutiny, rather than to enhance the process. The IQRWG wastherefore determinedto ensure that the necessary thought be given to maximising the benefits both for the student reviewers and for the IQR process and to providing appropriate training and support.

Identification and recruitment

2B.4The inclusion of student members on IQR Teams involves a number of practical challenges, the first of which is in identifying appropriate students to be reviewers. The IQRWG first considered, given thetime commitment involved in IQR,how and on what basis should student reviewers be, identified/selected, and briefed/trained and the locus of responsibility for this.Team members participating in UCL’s IQR process make a one day visit to the department, attend a planning meeting, a follow-up meeting and read the department’s SES and supporting material. This time commitment has the potential to impact negativelyon the studies of student reviewers andHEIs have a duty of care to students to ensure that extra-curricular enabling activities do not impinge on their academic studies. UCL, in common with all HEIs, experiences difficulty in recruiting adequate student representation to fill some of its existing committees, etc. (although there are a number of initiatives underway to address this such as the Student Academic Representatives (StARS) programme). Sufficient numbers of students would need to be willing to give up significant amounts of time in order to participate on IQR Teams. If a ‘pool’ of student reviewers were used, the ‘pool’ would have to be sufficiently large so as to ensure the selection of a student for each IQR who was neither a member of the department nor the faculty being reviewed.A student’s year of study will also have a bearing on the identification of potential student reviewers and thus on the number of reviewers available. (eg. first-year students might not have sufficient experience to comment on departmental matters, third-year students might be focusing on final-year exams).

UCL Union Sabbatical Officers

2B.5UCL Union Sabbatical Officerswere considered to be well placed to be student reviewers with their institutional knowledge of UCL and experience of its committee system. However, of the six full-time officers only two (i.e. the Education and Campaigns Officer and the Welfare Officer) would not require extensive training for IQR; the other four roles do not engage with institutional QME as a matter of routine and would require more extensive training. In any event, the IQRWG feltthat it would be unreasonable to expect the Sabbatical Officers to cover all IQRs (usually around twelve per year), butagreed that they should certainly form part of any pool of student reviewers.