Quality and Standards Sub-Committee
Subject:Institutional overview of External Examiner reports - 2011/12
Origin: Rob Pearson, Programme Quality & Teaching Partnerships
Executive Summary:An overview of external examiner reports for the 2011/12 session is provided, identifying themes and recurring recommendations that are worthy of further consideration.
Action Required: The Committee is asked to comment on the report, and in particular on the recommendations for action.
- Introduction
This report provides an overview for the University of external examiner reports for the 2011/12 session, identifying themes and recurring recommendations that are worthy of further consideration. The criterion for the selection of these themes is that they occur in more than one report and across more than one School/Department.
In most cases the themes have only been identified in a handful of reports and so should not be interpreted as evidence of major concerns about the quality and standards of our provision. It should be noted that the overall message within the external examiner reports is very positive, with considerable praise for high standards and excellent quality of provision.
All the external examiner reports have been received and considered by Schools, with a response made from the School to the external examiner. It is an expectation that the external examiner reports will be shared with students via Staff Student Consultative Committees.
- Themes identified in the external examiner reports
The themes in the reports have been separated into three broad categories:
- Assessment Practice
- Assessment Policy
- Other issues
.1Assessment Practice
i.Use of the full marking range
As in previous years, the most common issue of concern to externals is that staff are not using the full range of marks available to them. This concern has been raised across most subject areas. For example:
Comment 1
theredidseemtobeadistinctreluctancetoaward marksmuchbeyond 72-73
Comment 2
Forthedissertationsthestaffappeartomarkbetween50%forabarepassandmarks inthe70'sforaverygoodpieceofwork.AlthoughIhaveseentheodddissertationwith amarkinthelow80'sIkeeppointingoutthatstaffshouldbepreparedtousetherange of0to100%ratherthanthevastmajoritybeingbetween50to75%!!
Comment 3
Iwouldencourageexaminers tousethefullrangeofmarkstofullyrecognisethe excellentworkofthebetterstudentsonthemodule
However, there was one exception to the rule:
I found myself uneasy at some of the dissertation marking. One in particular was marked at the highest level I have ever encountered – 95% - in long experience of external examining. I felt that the dissertations in general were of a good standard but that the range of marking at the top end perhaps exaggerated the standards achieved.
Several externals note that there has been improvement, with evidence of a wider range of marks. A couple suggest that that a review of marking criteria would be beneficial. For example:
I recommendproducingmarkingcriteriafor givingamarkfortheupperdecilesof80%and90%,sothatmarkerswillbeencouragedtouse thefullrangeofmarks.
ii.Consistency of marking
Some externals noted that in very few cases, first and second markers were fairly wide apart in marks given, for example:
The discrepancies between first and second markers were extreme in a couple of cases and despite an agreed middle ground mark I wondered how a student might only get 10% from one of the markers, and whether this student was adequately advised and supported.
In some cases, the externals have recommended better use of / provision of marking criteria to aid consistency of marking. For example:
EachyearIhavepointedoutthattheacademicstaffareusingspecimensolutionsand NOTmarkingschemes.Markingschemeswouldaidconsistencyofmarkingasitwould clarifyhowmarkshavebeenallocatedforanygivenassessment.Thiswouldmakeit easierformoderation/verificationandexternalexaminers,howevertheyareusefulif everthereisaquery/appealregardinganassessment. Althoughtheyareinitiallytime consumingforacademicstodeveloptheyareinvaluablewhendemonstrating consistencyofmarking.
iii.Consistency of feedback to students
Several externals comment that some Schools would well advised to be more consistent in the provision of feedback to students. For example:
Comment 1
… it would be good to see a standardisation or at least a bringing together of feedback and assessment forms. I saw several excellent examples of structured and considered feedback presented in a professional way. If it were me, I would select any one of these and have a standard across the modules.
Comment 2
A minor comment would be that feedback on work could be more consistently delivered across the programme and a little more timely in a couple of cases. This is a common issue across HE and one that the Teaching Team might address next year.
.2Assessment policy
i. Discretion at degree class boundaries
As in previous years, several externals have questioned the discretion available at degree classification boundaries. Regulation XX specifies that the Programme Board has discretion to lower a classification threshold, in which case the revised threshold should be applicable to all students under consideration by that Programme Board. For example:
Comment 1
Whilst the use of an algorithmic approach to assigning degree classifications is now common practise in UK Chemistry Departments, the validity of this approach is compromised at Loughborough due to the University requirement that borderline candidates must be promoted in rank order. This means that a borderline candidate who is not promoted by the algorithm can still be promoted if a borderline student ranked below them is promoted by the algorithm.
Comment 2
Apparentlyitis aUniversityrequirementthatthe degreesclasses mustbedeterminedsothatonecandidatewithahigheroverallaveragethan anotherneverreceivesalowerclass. Thisselfevidentlyhasthepotentialto leadtocaseswheresomecandidatesaretreatedunjustly.Theremaybe instanceswhere,theexamboard,constrainedbythisrule,hastoawarda different classofdegreethanthecandidatemeritsbyvirtueofan objective assessment oftheir performance(asevidenced bythetheirfullmarkrecord).
Comment 3
I remain unconvinced that the mobile degree class boundary and the insistence on the priority of rank order by weighted mean, expressed to 1 decimal place, provide an effective method for delineating the boundary in degree classification. …. In my opinion a more robust mechanism is to identify a zone of discretion below the strict boundary and have clearly defined rules (e.g. preponderance of weighted credits) for raising the degree class awarded to candidates in the zone of discretion. This is straightforward to explain and operationalise.
ii. Coursework Anonymity
Several externals recommend the University reviewit’s position on anonymity in coursework. For example:
As a general rule, I consider that anonymity should be maintained whenever possible and practice should be changed without delay to address this. At present, coursework is submitted with candidate names and marks and sample work are provided to external examiners with candidate names. Both of these should be avoided.
- Impaired Performance claims
There were some interesting comments about the use of Impaired Performance claims. These related to the number of claims and the modification of marks. For example:
Comment 1
I am concerned at the number of IP claims submitted by students this year across the Part C modules – 19 claims out of 45 students of which 17 were upheld. This is a worrying trend … it may be appropriate to review the current procedure as presently a student who submits such a claim is in a “no lose” situation.
Comment 2
I haveexperienced theexamboardsofalargenumberofuniversities,bothasexternaland internalexaminer,andLoughboroughisthefirstuniversitythatIhaveencounteredthatwill modifyrawmarkstocompensateforimpairment Thisworriesmeonanumberoflevels. The mostobviousisthatthe unitofassessmenthaspresumablybeenprofessionallymarked,and theallocatedmarkreflectsthedegreetowhichastudenthasachievedthelearningoutcomes. Byadjustingthismark,thedepartmentisclaimingthatthestudenthasachievedalearning outcomeforwhichthereisnoevidence.Thisbreaksthebondof trustbetweentheuniversity andallthoseinstitutionsdownstreamof theuniversitywhichrelyonthetranscripttoindicatethe proficiencyofthestudent. Iftheuniversitywishestoprovidethestudentwithhelpovertheir impairment,surelyit shouldeitherflaganimpairedmarkonthetranscript,orreportthis impairmentinareference.Inaddition,whilst Iamhappythatacademicscanassessapieceof worktoallocate a mark,I cannotbelievethattheycanpredictalternatehistorieswherethe impairmentdidn'toccurandthestudentreceivedabettermark. Pleasereconsiderthis nonsensicalapproachtoimpairment compensation.
.3Other issues
- Equality and Diversity
The report proforma asks externals to comment on diversity issues relating to race, gender and equality, but in practice this invitation is rarely taken up. However, this year two externals have commented on the gender mix, one in regard to staff and the other in regard to students:
Comment 1
I would still be keen to see more female staff members involved in such a teaching programme, which it might take some time for the University to respond!
Comment 2
There seems little evidence of any direct activity to encourage diversity within the student body. It remains overwhelmingly white middle class girls with no obvious disabilities. I recognise the gender/class profile is relatively common in textile courses nationally and meaningful remedies take a long time to achieve any quantifiable change. However as noted by the previous examiner strenuous efforts are underway to redress this elsewhere but I was not made aware of any strategies that may be in place to redress this at Loughborough.
- Recommendations
The themes identified in this report cover both guidance / practice of external examining, and broader regulatory issues for the University to consider. The Quality & Standards Sub-Committee is asked to consider how these issues should be taken forward.
It is suggested that the issues in regard to assessment practice could be taken forward by the Teaching Centre, with the remaining issues could be taken forward by the Academic Registry.
It should be noted that most of the issues in relation to assessment policy are either being considered, or will be considered, in a review of assessment policy being undertaken by the Academic Registry.
RP
April 2013
1