Aspects of the Induction of External Examiners 2004-6

EXTERNAL EXAMINERS:

ASPECTS OF INDUCTION AND BRIEFING 2004-6

REPORT TO THE HIGHER EDUCATIONACADEMY

HAROLD SILVER

June 2006

External examiners: aspects of induction and briefing 2004-6

1Summary and outcomes

It will be clear from what follows that there are some potentially important conclusions from this project on induction and briefing external examiners for undergraduate programmes. There are also weaknesses in the data that might make conclusions tentative. The Academy conducted a surveyabout external examining in 2004 and some information is drawn from this, though the response to that survey was limited. The number of responses to a questionnaire at the outset of this project in 2005 was extremely disappointing, specifically with regard to information about procedures in pre-1992 institutions. The relevant data from the two questionnaires are outlined below, but the weaknesses meant that it was not feasible to pursue a question that formed part of the initial project brief, regarding external examiners’ reasons for not attending induction events to which they were invited. Instead it was agreed to focus on information that it had hoped to obtain in the 2005 survey – details of how universities that did not have induction programmes delegated responsibilities for the briefing process. It was decided to conduct interviews at what had to be a small sample of these institutions, though the ones chosen were intended to represent some important features of pre-1992 universities in England. It was not possible in the time scale to undertake interviews in other parts of the UK. These universities provided some valuable insights into common and disparate features of ways in which external examiners were briefed and with whom university staff interacted. None of these universities conducted preliminary induction events for newly appointed external examiners.

Four universities were visited:

University of Sheffield

BrunelUniversity

University of Essex

University of Bristol

At each of these a contact person on the administrative staff arranged the visit and was interviewed, and though ManchesterUniversity was not visited a telephone interview was conducted with the contact person. The contact person was in each case someone with university responsibility for conducting the arrangements concerning the external examiners and also for relating to a member of academic staff with such responsibilities on behalf of the department, school or faculty (‘sub-units’ will be used below). Most interviews were on site, three were by telephone. At Sheffield five interviews were conducted, at Brunel six and at Essex four. At Bristol a focus group discussion was held with six people (a seventh withdrew because of illness). Interviews lasted 40-45 minutes, in a few cases longer, and the focus group lasted one and a half hours. Prior to each visit information about the external examiner process (code of practice, external examiner guidance and other material was downloaded from the five universities’ websites, and additional information was provided either by the contact person or by members of the sub-units. Contact persons did their best to make the range of interviews as widely representative of the sub-units and subjects as possible.

Detailed notes were taken during all on-site and telephone interviews and during the group discussion. These notes were typed up, normally on the day following the event, and these notes form the basis of the third part of this report. Some of the following conclusions arise from the questionnaires or the interviews or a combination of the two (remembering that the interviews were only in England):

1)A majority (it is not clear how large a majority) of post-1992 institutions probably hold induction events, and fewer than half of newly appointed externals invited actually attend.

2)None of the five universities where interviews were conducted held an induction event and there was little or no support for doing so by the academic staff or the university authorities. This is probably the case in most pre-1992 universities.

3)The respondents to the two questionnaires and the interviewees were equally committed to the different forms of induction and briefing they provide. Interviewees uniformly felt that their externals did not wish for and would not attend an induction event (and as externals they were themselves not invited to induction events). The 2005 questionnaire elicited a small amount of information suggesting that increasing the number of events per year might raise the percentage of externals attending, but it seemed mainly to be accepted that a figure of 30-50 per cent attendance was inevitable.

4)All institutions, whether or not they conduct an induction programme, treat the role of, support for and contact with their externals as extremely important.

5)The briefing of externals by pre-1992 institutions is thorough, and contact with them is seen as sufficient to ensure that externals perform their roles successfully, especially their ability to help to maintain appropriate academic standards. The structures in place in the universities’ central units and formalised in sub-units seemed to support the conviction that the briefing of and support for external examiners were effective.

6)There are significant differences or ambiguities amongst these pre-1992 universities in applying criteria relating to the institutions from which it is preferred that external examiners should be drawn.

7)There are similar and related issues amongst these pre-1992 universities about the interpretation of the role of the external examiner in bringing to bear the experience of ‘other’, ‘similar’ or ‘equivalent’ institutions.

8)There are varied degrees of difficulty in recruiting external examiners, but a significant number of sub-units in these institutions find recruiting new externals a very protracted and problematic process.

9)Benchmarkstatements played little or no explicit part in the process of external examining.

Some information was forthcoming about support for first-time external examiners and practitioner externals, but not enough to include in this analysis.

Note: The questionnaire responses above have been used anonymously. Institutions and individuals interviewed are identified in the introduction to the interview section, where institutions but not individuals are named in the text were this is considered to be helpful. It was agreed with interviewees that this report should be treated as an internal document of the HigherEducationAcademy, with copies accessible to interviewees. If any wider use of the report were planned it would be necessary to consult the institutions or rewrite the section based on interviews.

2Data from questionnaires

In 2004 the HigherEducationAcademy conducted a survey of how institutions induct and support their external examiners. Information on induction has been incorporated from that survey into the present study, which is also based on a questionnaire focusing on induction to which responses were obtained early in 2006, and on interviews also conducted in 2006. At the Academy Professor Howard Colley oversaw the project and with Elizabeth Davis established the initial contact with institutions. Professor Harold Silver was engaged to prepare the latter questionnaire, conduct the interviews, analyse the data and report. One intended feature of the 2006 study was to examine reasons why a proportion of newly appointed external examiners invited to induction events do not attend, but this was overtaken by other emphases. One disappointing aspect of the questionnaire was a response rate that was too small to draw reliable conclusions. Two other related questions that emerged, however, concerned the presence or absence of induction events by institutions, and particularly how contact with and the briefing of external examiners take place in institutions that do not hold centrally organised events – and this question became the focus of the interviews conducted during 2006.

The total number of institutions responding to the 2004 survey was 33, of which 16 were pre-1992 institutions - 5 of them being constituent institutions of the University of Wales. Of the 33 institutions, 3 were in Scotland, 6 were in Wales and 1 was in Northern Ireland. Six of these institutions were non-university institutions of higher education, though 3 of these were shortly to become universities or parts of universities.

Only 11 institutions responded to the 2005-6 questionnaire, 3 of which were pre-1992 institutions – 1 in England, 1 in Scotland, and 1 constituent institution of the University of Wales. This report considers the evidence from the post-1992 respondents, but in addition to the 3 pre-1922 institutions it takes account of interviews conducted at 5 others. Four of the 2006respondents had also responded to the 2004 survey, and 3 of the institutions at which interviews were conducted had also taken part in the 2004 survey.

The poor response rate in both projects, but particularly the later one, was probably governed by timing, by questionnaire fatigue and by lack of sustainedinterest in the issues involved after a period of considerable consultation on external examiner issues with the senior academics and administrators likely to be responsible for responding. This could be the case particularly with those institutions without any centrally arranged induction programme. Any conclusions from the 2006 data collection can only be indicative, and in general it is difficult to generalise from it. The extremely positive responses to the interviews suggest that more can be securely concluded from them. It is to be hoped that the analysis below may feed into future Academy consideration of induction and briefing processes. This was one reason for suggesting that the Academy might treat this report as an internal document.

Induction events (2004)

In 2004, 4 (all in Wales, of 16) pre-1992 and 10 (of 17) post-1992 institutions held at least one centrally arranged induction event a year (a small number of larger or dispersed institutions held 2 or 3 such events). Six of the 19 that had no such event indicated that events were held ‘by the relevant school’ were ‘up to the school’, were ‘offered by faculties’ or consisted of ‘formal faculty briefings’. Such alternative structures existed in post-1992 institutions without induction programmes(one had no sort of on-site event ‘at present’, and another encouraged externals to attend an examination board ‘prior to uptake of the role’. The central authorities in the pre-1992 institutions gave schools or faculties various degrees of ‘encouragement’ or ‘guidance’ regarding arrangements. The following are examples of pre-1992 institutional responses to the question of whether they provided induction programmes:

• ‘The university would be most reluctant to institute such a programme on a wider basis’ than by individual schools.

• External examiners are ‘encouraged to contact the school and perhaps visit’.

• There was ‘no formal training’, but externals were invited for lunch and a briefing session in the department.

• Schools are advised, where resources permit, to invite new externals to the University ‘ahead of their first board of examiners’.

• ‘External examiners handbook only. The majority of them surveyed thought this was enough and did not wish to attend an institutional induction programme’.

• ‘The code or practice plus a visit are enough’.

In addition to information about induction or briefing some pre- and post-1992 institutions indicated the presence of mentoring, web-based or other developments, but this evidence was limited as no specific question about these had been asked. An important distinction between the two categories of institution was the degree of informality about arrangements that seemed to be a feature of a large proportion of pre-1992 institutional responses. This, together with the absence of significant further information in the 2006 study, was what suggested the desirability of the interviews that were conducted.

Induction events (2006)

Six of the 8 post-1992 institutions that responded to the later questionnaire indicated that they held regular induction events at least once a year. One of these responses was from an institution that did not hold an induction event but from one of its faculties that embraces a number of partner colleges, and does hold regular events. One institution has briefings in each school prior to the examination board, but ‘since this is not institutionally organised there is at present no parity across the institution’. This institution was therefore planning to develop an induction event in the near future. The existing events in these institutions had a common structure of dividing a one-day event into two parts, one concerning ‘institutional matters, such as rules, regulations, practices and procedures’, modular schemes and the assessment process, and the second consisting of meetings with the school and/or course team, meeting staff and discussing the involvement of the external examiner. The questionnaire asked respondents if they could ‘calculate or estimate’ what percentage of externals invited had attended in the last year (or current year if the event(s) had already taken place). The following responses indicated the proportion attending:

1)approx 40-50%

2)about 30%

3)27 out of 48

4)25 of 63

5)45 %

6)46%

7)‘most’

8)n/a

Some were at pains to point out, for example, that externals who had not attended might do so at a later event.

Of the 3 pre-1992 institutions responding in 2006 the information provided was as follows:

• (England) No institutional induction, ‘some departments’ hold local induction events. ‘Induction events, at the institutional level, have been held in the past but very few external examiners attended. We are considering piloting an induction event again in the future’. Externals with no, or little, experience of the role ‘are appointed a mentor’.

• (Scotland) ‘Schools are responsible for providing briefing and training… Heads of schools “should ensure” that an external examiner is appropriately briefed and trained and that responsibilities are understood’. Not possible to estimate numbers attending.

• (Wales) Centrally organised event, used to be once a year, now offered twice, so externals unable to attend the first are invited to the second. Structure of the day similar to the ones held by post-1992 institutions. In the current year 55% of new external examiners attended the first event and it is hoped most of the remainder will attend the second.

This small number of pre-1992 institutional responses is best understood in the light of the subsequent interviews.

Information and guidance

Both surveys concerned with induction also asked about handbooks and other introductory information provided for external examiners, either as a preliminary to or instead of attendance at an induction event. All institutions accept the need for newly appointed external examiners to be sent preparatory information, though there seemed to be agreement on what constitutes appropriate and necessary information. There is no sectorial difference between what is sent by pre- and post-1992 institutions, nor is there clear difference between what is provided for externals as additional to an induction event or as a substitute for one. The 2006 study was less concerned than the previous one to explore the information provided, and the responses were therefore correspondingly brief – confining themselves to references to sending a ‘handbook’, which could imply almost any level of detail about the institution, its sub-units and courses, its procedures and its expectations of the external examiner. Responses to the more specific 2004 survey produced across all types of institution references to information that could be either minimal or very detailed. All institutions sent something:

• ‘Internal Code of Practice for external examiners

Course- and Programme-specific information

Specific technical information’ (Scottish, pre-1992)

• ‘Induction procedures and regulations guide’ (Welsh, pre-1992)

• ‘A pack of information which includes the University’s guidelines for external examiners, relevant Assessment Regulations, copies of report forms’ (English, post-1992)

• ‘A pack of information including the code of practice, the Undergraduate Programme Regulations and previous externals’ reports and feedback. Individual departments are responsible for sending the Teaching and Learning Stragegy’ (English, pre-1992)

The assumption was sometimes that the external examiner needed to have detailed information only about such matters as assessment procedures, modular structure and course content, but in other cases the information was broadened to include institutional and other policy contexts:

• ‘The University’s Code of Practice on the External Examiner System

Guidance Notes for Externals and Notes for Schools of the University

University Policy and Code of Practice on Equal Opportunities for Students

University Policy on AP(E)L

University Policy on Plagiarism and Collusion’ (English, pre-1992)

• ‘Code of Practice for External Examiners

Mission and objectives of the University

Unfair Practice Procedure

Modular Degree Regulations

Rules and Procedures and Code of Practice for the Assessment of Students’ (Welsh, pre-1992)

• ‘duties of external examiners

rights and responsibilities

assessment regulations

programme handbooks

reporting procedures

liaison conventions’ (English, post-1992)

Most institutions sent their own code of practice for external examiners, a very small number sent the QAA Code of Practice or other documentation such as the National Qualifications Framework and the relevant benchmark statements. An institutional statement on students with disabilities might be a separate document or contained in a handbook. There are a few references in the responses to an existing, or currently being developed, CD-ROM, video or DVD intended for external examiners. Some signalled with whom external examiners might have contact, and one provided a ‘Staff A-Z’.

The picture overall is uncertain since it is not clear what was contained in some of the compendia sent, the detail of what was sent by schools or departments, what might be circulated at or after an induction event (absentees were sometimes sent such documentation or Power Point frames used). There is, of course, no direct impression possible from the questionnaire responses of how much information external examiners (with different levels of experience or in different subjects) expected or wanted, or how satisfied they were with the information received or the respective value ofdocumentation and insights formally or informally acquired, an induction programme, a casual visit or a meeting the day before a first examination board.

3The interviews

Interviews were held as follows, in each university with the contact person who had external examiner responsibilities and with academic staff with such responsibilities in sub-units: