Working Group on Student Feedback Report to QMEC 10 October 2005

LONDON’S GLOBAL UNIVERSITY

QUALITY MANAGEMENT AND ENHANCEMENT COMMITTEE

WORKING GROUP ON STUDENT FEEDBACK

Report and Recommendations to Quality Management and Enhancement Committee on Student Feedback

Executive Summary

Academic Committee resolved at its meeting of 11 May 2004, that the Quality Management and Enhancement Committee should consider the general issue of student feedback in relation to its deliberations on UCL’s quality management and enhancement strategy. QMEC, at its 1 June 2004 meeting resolved to set up a working group to undertake a review of UCL’s current policy and practice in relation to the collection and use of student feedback and to report back in the summer term 2005.
WGSF met on four occasions to review UCL policy and practice on student feedback. In the event its work took longer than anticipated and its report to QMEC was delayed for submission to the Committee’sOctober 2005 meeting.
This report:
- includes further background on the establishment of WGSF [paragraphs 1-4]
-outlines the various issues discussed by WGSF and consideration of the evidence submitted to assist its deliberations [paragraphs 5-58] including:
- Student Evaluation Questionnaires [paragraphs 13-22]
- Student Representation on committees [paragraphs 23-39]
- Student Feedback on UCL Support Services [paragraphs 40-44]
- The proposed UCL-wide end of programme SEQ [paragraphs 45-50]
-includes 13 recommendations for the formal approval of, or further consideration by QMEC – as well as other relevant committees within UCL [paragraph 59]

Key to abbreviations:

ACAcademic Committee

CHERICentre for Higher Education Research and Information

DAPS/IADegree Awarding Powers/Institutional Audit

DSSCCDepartmental Staff Student Consultative Committee

DTCDepartmental Teaching Committee

FTCFaculty Teaching Committee

HEFCEHigher Education Funding Council for England

HEIHigher Education Institution

IQRInternal Quality Review

JSSCJoint Staff Student Committee

LTSNLearning and Teaching Support Network

NSSNational Student Survey

QAAQuality Assurance Agency

QMECQuality Management and Enhancement Committee

QSRGQuality Strategy Review Group

RECCRecommendation

SEQStudent Evaluation Questionnaire

WGSFWorking Group on Student Feedback

Background

1At its meeting on 11 May 2004, Academic Committee resolved that the Quality Management and Enhancement Committee should consider the general issue of student feedback in relation to its deliberations on UCL’s quality management and enhancement strategy. The 1 June 2004 meeting of QMEC resolved to set up a working group to undertake a review of UCL’s current policy and practice in relation to the collection and use of student feedback.

2The terms of reference of the Working Group on Student Feedback were:

  • to review UCL’s current policies and procedures relating to obtaining feedback from, and giving feedback to, students.
  • to consider, as part of its review:

(a)policy statements made by UCL on the issue of student feedback(e.g. in the Academic Manual);

(b) the effectiveness of existing student feedback mechanisms within UCL, at institutional-, faculty- and department- level;

(c) the decisions taken by AC in June 2002 in relation to the introduction of a UCL-wide end-of-programme Student Evaluation Questionnaire;

(d)the guidance set out in the CHERI/LTSN report 'Collecting and Using Student Feedback: A Guide to Good Practice' (March 2004).

The membership of WGSF was:

Dr Tom Gretton [Chair - Meeting 1]

Professor Chris Carey [Chair - Meetings 2-4]

Dr Anita Berlin

Mr Jason Clarke

Mr Marco Federighi [Meetings 2-4]

Dr Jonathan Iliffe [Meeting 1]

Mr Peter McLennan

Dr Hilary Richards

Ms Visahli Thakrar [UCL Union Medical Sites Officer, Meeting 1]

Mr Luke Yahanpath [UCL Union Medical Sites Officer, Meeting 2-4]

Ms Mary-Beth Young [UCL Union Education and Welfare Officer 2-4]

Mr Rob Traynor, Quality Assurance Officer, Academic Services [Secretary]

It was intended that WGSF would draw the various threads together regarding national and internal developments in order to inform UCL’s approach to the issue of student feedback as part of its quality management and enhancement strategy.

3It was noted that for the purposes of WGSF’s review, “feedback” refers to the gathering of student views on their experience at UCL and the reporting back to them on actions taken in response to matters raised by various student feedback mechanisms (such as student evaluation questionnaires and student representation on committees). Student feedback in this context does not refer to formative assessment of student work or of performance.

4It was noted that many of the matters considered by WGSF have been considered by a number of UCL committees over a number of years in the context of a changing national higher education landscape in relation to quality management and enhancement. For example, the (former) Academic Advisory Sub-Committee’s Working Party on Student Questionnaires had recommended to AC in 2000 suggestions for good practice in SEQs, including common questions to be included and the nature of feedback to students of the results. In 2002 Academic Committee conducted a survey of faculty practice for SEQs and AC approved in principle that a UCL-wide end of programme questionnaire be established.

Introduction

5WGSF met on four occasions, once in April and three times in September 2005. In order to facilitate its discussions, it considered three main areas of student feedback:

  • Student Evaluation Questionnaires (at meeting 1 and 2);
  • Student Representation on Committees at department-, faculty- and institutional level - e.g. Departmental Staff Student Consultative Committees, Departmental Teaching Committees, Faculty Teaching Committees and institutional level committees (at meeting 3);
  • Student Feedback on Support Services - e.g. IT, Library services etc. (at meeting 4).

6In order to inform its discussion, WGSF considered a range of evidence including:

  • key good practice and policy statements on student feedback procedures in the Academic Manual;
  • a survey of faculty practice in regard to SEQs;
  • a survey of student feedback arrangements in UCL Central Support Services;
  • UCL Union submission to the Quality Assurance Agency’s Institutional Audit and Degree Awarding Powers visit;
  • UCL Union (draft) Handbook for Student Representatives;
  • Royal Free and UniversityCollegeMedicalSchoolelectronic SEQ and related background documents; and
  • UCL Registry information on Student Feedback.

7At its first meeting in September, WGSF noted that the context to its review of student feedback had changed following discussions within UCL of the Provost’s White Paper (2004) one year on. The Vice-Provost (Academic and International) had commissioned a review of UCL’s quality management and enhancement structures and processes, which would be undertaken by QMEC’s Quality Strategy Review Group. It was intended that the QSRG would receive reports from a number of existing reviews of different aspects of UCL’s quality management and enhancement processes, with the intention to improve their efficiency, effectiveness and fitness for purpose. It was intended therefore that as well submitting its report to QMEC, the WGSF report would also be considered by the QSRG to assist in this review.

CHERI/LTSN Report

8In its terms of reference, WGSF had been asked to consider the CHERI/LTSN report 'Collecting and Using Student Feedback: A Guide to Good Practice' (March 2004) [1]in its deliberations.

9WGSF noted that the guidelines raised a number of issues to be considered when looking at student feedback mechanisms including:

  • the purposes of student feedback;
  • the level at which it should be gathered;
  • timing issues;
  • standardisation of feedback across higher education institutions, faculties and departments;
  • types of feedback mechanisms;
  • the use of questionnaires;
  • publication of results and dissemination to students and
  • suggestions for good practice on student feedback.

10WGSF also noted that the guidelines encouraged HEIs to consider carefully how the audience for student feedback was identified and who would consider its results. It observed that this could be course or programme tutors, departmental, faculty and institution level committees, external scrutiny from reviews and professional accreditations and ultimately the students themselves, in terms of dissemination of the results of the feedback to them.

11The guidelines were well constructed and gave valuable information on the types of feedback mechanisms. The report advises that HEIs consider the feedback mechanisms holistically and in combination, and not to simply rely on one method such as SEQs. There was also a section on standardisation and centralisation of feedback within HEIs, which helped to inform WGSF in its deliberations on the variation of student feedback practice at UCL. The guidelines also stressed that it was extremely important for HEIs to close the “feedback loop” and ensure that students were given information on their feedback. This was vital in ensuring that students knew that their views were valued and (where appropriate), were acted upon. The knowledge that they were taken seriously would encourage students to provide more feedback in future and participate in representative processes.

12The guidelines concluded by suggesting that HEIs review their policies and practices in regard to student feedback and consider “whether purposes are clear and continue to be appropriate, whether they are being achieved and whether they might be achieved more effectively and efficiently”. WGSF found the CHERI/LTSN report helpful in providing a framework for its discussions and kept them in mind during its deliberations.

Student Evaluation Questionnaires

Use of Student Evaluation Questionnaires within UCL

13UCL’s existing guidelines on the use of SEQs can be found in the Academic Manual as Document H2[2], Student Questionnaires -this is at Annex 1. This document is designated as “good practice” which means that departments are not required to fulfil the suggestions, although it is highly recommended that they do so. It was observed that despite this, the actual wording of the document implies that its content constitute UCL policy and that this could give rise to ambiguity as to what UCL’s policy really is and possibly cause confusion within departments. The document advises that SEQs should be distributed annually, with separate questionnaires for each course unit as well as for the end of programme. SEQs should address the teaching on courses and not the syllabus and guidance is offered on the range of questions that SEQs should ask students. It also recommends that reports should be produced summarising the quantitative data received as well as student commentary and that this should be considered by the DTC and/or the DSSCC (with the final analysis to be forwarded and kept by the Faculty Office).

14WGSF conducted a brief follow-up survey to the current use of SEQs within faculties, updating the survey results of 2002 which had helped to inform AC’s decision to introduce a UCL-wide end of programme SEQ. The results are presented as a table at Annex 2. There was wide variation in how SEQs were conducted within UCL with most departments using paper based surveys, although some had moved to electronic systems. There appeared to be inconsistencies within UCL in the provision of feedback from the SEQs to students. It was not clear to WGSF that feedback to students on the results of SEQs was routinely provided around UCL, with summaries of responses given to DTCs or DSSCCs. There also appeared to be wide variation in the monitoring of the results, with this carried out by FTCs, DTCs and programmer tutors depending on the faculty.

Use of Electronic Student Evaluation Questionnaires within UCL

15WGSF received a great deal of information and a demonstration of the Royal Free and UniversityCollegeMedicalSchool’s electronic SEQ system. This had been introduced in 2004 following frustrations with the previous paper based exercise that required a great deal of administrative resources and staff time to operate (using Optical Marker Readers). The on-line system,“Opinio”,conducted both end of course and end of programme surveys, with the latter well received by students achieving a 63% response rate in its first year of operation. The MedicalSchool gave an approximate costing of the system and calculated that it had cost £5,000 to set-up (including purchasing of the necessary soft-ware and hard-ware, licensing from “Opinio” and technical support time) and less than £1,000 per year to operate. WGSF believes that should this system be extended UCL-wide that the start-up costs would not be likely to exceed more than £30,000, with a likely operating cost of £6,000.

16It was noted that there were a number of advantages to the new system including:

  • the freeing up of administrative resources and staff time (after initial investment in setting up the system) – it was cost effective to install and operate;
  • the flexibility of the system in terms of timing and frequency of sampling:
  • core SEQ templates could be adapted to suit the requirements of individual courses;
  • data could be analysed quickly and efficiently and presented in a variety of ways;
  • it was possible to set up a series of reminder notices to automatically prompt students to complete the SEQ;
  • the process of distributing results to course organisers was more efficient;
  • the MedicalSchool is subject to professional accreditation and was able to more easily produce valuable data from the on-line SEQs in order to meet the requirements of external bodies;
  • the process for monitoring the results was much less involved than the paper system meaning that results from data was released more quickly – the nature of some Medical School courses meant that it was possible to enact changes in response to the results within the same academic year.

More information on the MedicalSchool’s on-line SEQ system, including a document outlining its principles and procedures, can be found at Annex 3.

17The positive experience of the MedicalSchool’s conversion to the electronic SEQ was, however, tempered by views WGSF heard from other departments. It was reported that the response rate to the Department of Geomatic Engineering’s SEQ had gone down by 50% since it had been run on-line. WGSF noted that the departments of Computer Science and Statistical Science identified this as an area of concern in their recent Internal Quality Reviews, specifically a reduced response rate and quality in the information they received following transfer to an electronic SEQ system. Members of the departments met with WGSF to discuss their experience and concerns in more detail and indicated that whilst electronic SEQs were more efficient (i.e. they were easier to resource and operate) they were not necessarily more effective (i.e. the response rate was considerably lower). The Department of Computer Science also reported that it had found other sources of student feedback more useful in gauging student opinion, such as its DSSCC, the tutorial system and informal meetings held between tutors and student representatives.

18It became evident to WGSF that there appeared to be clear cultural and organisational differences between disciplines and departments which might affect the success of using electronic SEQs. WGSF observed that the Medical School students were not based in individual departments so did not have a single physical “home” and did not have personal tutors;they takevarious courses throughout their programmes in different departments and medical institutes. As a result there was not the same identification with an individual department that other UCL students enjoyed and less avenues for feedback through such sources as personal tutors. Students were used to communicating with the MedicalSchool through e-mail and the team observed that this might make the use of on-line SEQs more acceptable to them than in other departments. WGSF observed that other departments regarded feedback from SEQs as of less significance and heard concerns with the administrative workload and resources that would be required to implement an electronic system.

Ownership of the feedback given to Student Evaluation Questionnaires

19The issue of ownership of the feedback given to SEQs was also considered by WGSF. It had not been clear in the discussion who owned this feedback, the department involved or the student and this raised questions of access to the data and whether individuals outside UCL could request the data under the Freedom of Information Act. The UCL Freedom of Information and Data Protection Officer informed WGSF that individuals could request information and data from the SEQs in terms of the questions asked and percentages of the students’responses. However, departments (or UCL) would not have to release individual student responses or identify students. In practice, this would mean that the summaries of the surveys (as discussed below in paragraph 20) would suffice in order to satisfy freedom of information requirements. WGSF was also informed how departments would need to deal with circumstances in which student anonymity in responding to SEQ could be broken - e.g. in cases of malicious or libellous responses or where the seriousness of the matters raised meant that urgent action and identification of the student was required (it was also noted that UCL has a duty to protect staff from malicious comments). WGSF was informed that SEQs would require a statement making it clear to students that malicious comments were not acceptable and that departments reserved the right to take appropriate action and contact students in circumstances where they were made. The statement would also need to clarify that departments reserved the right to contact students in circumstances where their comments indicated the need for urgent action to address raised problems. The statement would need to be either at the top of the form in printed SEQs or as a flagged message for electronic surveys – it would need to be clearly visible to the students and the first item the student read before completing the SEQ.

20The CHERI/LTSN’s guidelines emphasized that it was vital that the results of SEQs were fed back to students in order to “close the loop” in student feedback and to ensure that students knew that their voice was listened to and their opinions valued. It was important to consider how students are encouraged to complete SEQs and how the process is explained to them. If students did not believe that staff took SEQs seriously, they would be less likely to complete the surveys, leading to a possible further reduction in staff commitment to the SEQs, thus creating a downward spiral with ever decreasing returns. Students might develop apathy and antipathy towards future surveys if they perceived that their concerns had not been taken seriously or saw no action arising from their responses. It was clear that completing SEQs (particularly end of programme surveys) was often an altruistic act which the respondees would not benefit from, but nonetheless their feedback could still be conveyed to their student successors, showing that the Department did value and act on the feedback. WGSF considered that the mechanisms to enable students to receive feedback should include publishing summaries of the data on departmental web-sites and/or to DSSCCs.