Introducing computers into the assessment process: what is the impact upon academic practice?

by Dr Colleen McKenna

University College London

Department of Education and Professional Development,

1-19 Torrington Place,

London WC1E 6BT

E-mail:

Paper presented at the Higher Education Close Up Conference 2, Lancaster University, 16-18 July 2001

Introduction

The increased use of computerised assessment in higher education is an international phenomenon. The United States and Australia, for example, have seen a recent rise in the use of computer-assisted assessment (CAA), with further growth predicted in the near future (Bennett, 1998; Dalziel and Gazzard, 1999; Sly and Rennie, 1999). This trend is also apparent in UK HE where brisk expansion in student numbers since the late 1980s combined with a relative decrease in resources has motivated academics to adopt new methods of assessing large groups. Additionally, the sector-wide embrace of C & IT initiatives, encouraged in part through government directives, particularly in the shape of national funding programmes promoting learning technology, has led to the embedding of computer-aided learning materials such as CAA into teaching.

For the purposes of this paper, CAA is defined as the use of computers to deliver, mark and analyse assignments or examinations. In particular, much of the following discussion is concerned with the impact of automated marking upon academic practice. Examples of CAA include

  • computer-based assessment in which students take tests in computer labs, responding to all questions online,
  • optical mark reading (OMR) technology, in which students complete responses on machine-readable cards which are automatically marked,
  • audience response system which involves giving students handsets with radio-transmitters and projecting multiple-choice questions onto a screen via a computer. The students transmit their responses using the handsets and a histogram is generated and displayed, showing what percentage of the class has selected each response.

The introduction of computers into the assessment process can have far-reaching consequences, particularly when used for summative assessment. For example, in most CAA systems, question type (usually a variant of the multiple-choice format) and test design are pre-determined, leaving the lecturer relatively little flexibility in how the examination is constructed and delivered. The use of objective testing raises issues about the capacity to assess higher order learning, validity, suitability for discipline area and potential student disempowerment (Paxton 2000) . As CAA usage increases, it is important to gain a critical understanding of how it is viewed and used within higher education, and what its perceived impact upon the academic practice of both staff and students is. Yet, until very recently, only a limited amount of research has been conducted into the use of CAA in higher education.

This paper reports on a qualitative study into the experiences and attitudes of a range of CAA users and non-users in the humanities, social sciences and natural sciences, and it aims to articulate a set of mixed attitudes to issues such as appropriateness for subject, curricular impact, student perspectives and control of the assessment process. It also attempts to preserve some of the minority perspectives on CAA which can be lost in the analysis of quantitative data. To this end, the reporting and analysis of findings is punctuated by case histories of individual participants including two long-term users of CAA, one academic who is sceptical of the use of computers in the assessment process, and one new user who has attempted to run on-line tests in a humanities department.

Background and methodology

In 1999, a national survey into the use of CAA in higher education canvassed just under 800 staff (users and non-users of CAA) in UK universities and higher education colleges via an extended questionnaire about their views and experience of computers in assessment (Bull and McKenna, 2000). Participants were asked to comment on the appropriateness of CAA for their discipline, experience with CAA, advantages and disadvantages of using CAA, attitudes towards using objective testing, concerns about using CAA for assessments which counted towards module marks , the impact of computer anxiety upon staff and students, and details of any computerised tests delivered. Where participants had used CAA, they were asked to identify how it was used (ie for summative, formative, diagnostic, self-assessment purposes), question format, what other methods it was combined with, and where it was positioned within the curriculum. After analysing this data, a qualitative study was designed and a series of focus groups and interviews were conducted to explore the field in more depth.

This paper draws upon the findings of this qualitative research, begun in spring 2000. Participants for the study were selected because they expressed particular views (favourable and unfavourable) about the use of CAA, and in some cases had devised novel testing practices using C & IT. They comprised academics and educational researchers/technologists from pre- and post-1992 universities and a college of higher education. Participants included users and non-users of CAA (though the majority had some experience with computerised testing) working in chemistry, pharmacology, engineering, history, philosophy, biology, mathematics, geography, modern languages, computer sciences and learning support units. Data were collected from two focus groups at different institutions and 13 semi-structured interviews; in total, 25 people participated in the qualitative study. The data have been analysed and grouped into themes, some of which are reported here.

Approaches to the use of CAA

Reasons for using CAA

Amongst those who used CAA, its introduction was often intended to have a direct impact upon the practice of either students or academic staff, or both. A number of participants spoke about CAA as a means of attempting to manipulate student learning behaviour in terms of what was learned (ie the breadth of a unit as opposed to selected elements) and when it was learned (across the term rather than at the end). Additionally, several participants introduced CAA as part of an attempt to encourage a shift among students towards self-directed learning. Other reasons for adopting CAA included high student numbers, coping with cohort heterogeneity, increasing opportunities for student feedback, and preparing finalists for discipline-based professional exams.

Several participants cited the use of CAA as a means of promoting regular learning across a unit or module. For example, in one department in which lectures occurred with greater frequency than tutorials, CAA tests were introduced at the start of each tutorial to assess students' understanding of the material presented in the intervening lectures. The tests counted for 25 per cent of the overall unit mark, and the lecturer suggested that with the CAA tests, the tutorial sessions were more productive because students no longer turned up to them "cold". The tests also functioned as diagnostic tools.

Following similar reasoning, another participant spoke of establishing a departmental-wide programme of CAA assessment in an attempt to foster a more incremental pattern of learning and thus improve progression rates between the first and second years of a maths degree. (See case study 1 below.)

As suggested above, another way in which CAA is used to shape student learning patterns is to encourage wider coverage by students of the course material. The capacity of CAA exams to test the breadth of a course unit was cited frequently by respondents. For one long-term CAA user "the fact that you have short answer methods means that students have to learn more of the course". There can be no "question-spotting" or selective learning.

The issue of high marking loads and time savings also arose repeatedly, and while not the main drivers for CAA for most users, these were key motivating factors for several, who mentioned rising student numbers as the main reason for adopting computerised assessment. There were, however, mixed experiences with time savings. One interviewee, when asked why she started using CAA for summative examinations, responded "Why? Numbers. We were running very large groups... Having gone down that road, I use it with smaller groups." Her first test involved 300 students and the lecturer estimated that the extra time taken to develop new questions meant that there was no time savings, but no extra cost of time either. The time savings is now substantial. On the other hand, one participant observed that although CAA eventually saves time, all the time costs are at the beginning of the process, in contrast to the production and marking of essay questions. Another lecturer who has used CAA for over 10 years suggested that he has only recently experienced a time savings: "The class sizes of over 100 mean that there is a substantial saving in time. However, this is offset by doing more and better things such as diagnostic tests in induction week."

In a case where CAA was used solely for self-assessment, such that no savings in marking time were made, the motivating factor was still large student numbers (and the resultant range of mixed subject experience.) One method of coping with cohort size and heterogeneity was to introduce CAA to offer increased and unlimited opportunities for practice and feedback. In one instance, the CAA self-assessment exercises were viewed as supplements to traditional tutorial support. (See case study 4.)

Elsewhere, CAA was implemented as part of a larger departmental shift to promote self-directed learning. (See case study 3.) In the case of the audience response system, CAA was incorporated to give students experience with objective test formats used in their professional exams. However, the main reason for continuing with the system has been to promote interaction and discussion in teaching sessions and to help tutors modify the course in accordance with student needs: "if they all get it right, we'll move on... We can focus on areas where there is a misunderstanding or confusion."

Proponents of CAA nearly always mentioned student learning as a reason for introducing technology into the assessment process. However, as can be seen in the next section, those who were sceptical about the use of CAA cited the same reason to justify their concern, suggesting that its impact upon student learning was potentially adverse.

Case study 1: Dr Smithson

Dr Smithson, a mathematician at Central University who returned to academia after a career in industry, has used CAA for over 10 years. When he began lecturing, he found that "the one thing I really hated was marking. I absolutely hated it and still do." Not long after he joined Central, a colleague introduced him to a CAA package and he began using it with students: "you can always remember the kick when you get the first set of marks back... I was hooked."

Early on he was one of very few CAA users and was viewed within his own department as an "enthusiast", a term he dislikes because of its connotations of amateurism and eccentricity: "At that stage, the assessment was not 100 per cent CAA; it was just a component." He had a very enthusiastic external examiner and supportive head of department, so he was able to increase the use of CAA in his own courses and encourage its use amongst his colleagues "lots of things gelled quite early on... it's been remarkable to see how it's caught on. For us, it's [CAA] bread-and-butter now."

A particular programme of regular testing was introduced to improve student progression rates between the first and second year, and to improve the breadth of understanding across the year one curriculum, following complaints by second year tutors about a lack of proficiency with the key concepts of the first-year syllabus. It was felt that a more incremental approach to testing (ie regular testing after each unit of material rather than examining all units at the end of term) would encourage students to learn the material throughout the term and thus improve their scores. Early results suggest that the CAA intervention is having the desired impact, and the same approach has been adopted for a second year statistics module with which many students were experiencing difficulties.

In terms of procedure, the students register in advance for a test and there are regularly scheduled testing periods in a computer lab which are invigilated by the department's learning technologist. Students are not allowed to refer to prepared notes, but they are given blank paper to work out the problems during the test.

Dr Smithson believes that although the test questions are in objective formats (MCQs and text match), the students are "practically doing a written exam. You see [on the provided paper] what would be a traditional maths solution."

In terms of the impact on his own practice, Dr Smithson suggests that he now experiences "a noticeable time savings" but he acknowledges that for those starting out on CAA, a significant investment of time is required: "People get tremendously frustrated. There is a learning curve to climb." He does think that once a general competence level in CAA is achieved, staff usually stick with it.

Why CAA is not used

A number of the interviewees did not use CAA for reasons of pedagogy, discipline area and technical expertise. One participant who works in a department in which CAA is widely used, expressed concerns about the impact of CAA upon the academic practice of students: " I don't use it because I'm not terribly technologically au fait. I also have grave reservations about the effects of CAA on student learning." In particular, he argued that CAA reinforces in students a reductive view of education which sees learning as "data transfer" rather than the assimilation and manipulation of knowledge: '[Students believe] that higher education is "not about the facilitation of learning; it's about information transfer. CAA confirms this view and it absolves them [of making the effort to practice deep learning." [See case study 2.]

Another user of CAA for formative purposes, who was "nervous about doing anything summative" with computers, echoed this concern about the mechanistic process of automated assessment: "it's difficult to stop [CAA] becoming mechanical -that is, you just remember the actual answer rather than actually knowing what it is." Similarly, a concern about the possible mechanisation of the process is apparent in the language used by another lecturer (see case study 3) and a colleague in their accounts of the manner in which his first attempt at CAA was used by students. They described how, when faced with CAA self-assessments, the students adopted a "smash and grab" technique, "punching any key" to "strip off" the feedback and correct answers. When they realised that the students were using the CAA in a manner different to the reflective approach they had envisaged, the system was abandoned in favour of one which required students to work through complex pathways to solve problems.

Two interviewees ( non-users from humanities and social sciences departments) felt that the objective question types supported by CAA were unsuitable for their disciplines. Two other users in humanities departments faced resistance to the introduction of CAA because colleagues considered it to be unsuitable for discursive subjects . Additional issues surrounding non-use are discussed below.

Case study 2: Dr Hewitt

Dr Hewitt, a geographer at Western University, has a wide experience of assessment techniques and conducts research into student learning and assessment. A self avowed skeptic of CAA, he teaches in a department and institution where it is widely used and he is familiar with the standard computerised assessment systems.

His main concern about CAA is its impact upon the academic practice of students. He believes that there is no "personal construction of meaning" with computer-based materials in general, and specifically, with CAA. As far as he can see, learning technologies only promote a surface learning approach and do not enable students to "move to another conception of teaching and learning." In particular, he suggests that CAA reinforces a common student perception that understanding means "accurately to reproduce" rather than "seeing it my way." As with those who use CAA, Hewitt feels that the chief shortcoming with CAA is the objective test question format that most systems use: "Students have had MCQ tests in 6th form and A-level. Their perception is that these only require memory recall... It's very difficult to challenge [this idea] ... if you believe adamantly that the questions are about the facts, then that's all you prepare for."

Despite his concerns, Dr Hewitt spoke moderately approvingly of a colleague who uses CAA for formative assessment: "Objective testing does make it possible to have formative assessment enabling a certain amount of feedback."

Additionally, Dr Hewitt feels that CAA is introduced into the curriculum in order to reduce workloads, rather than for sound educational reasons: "I've got a feeling that a lot of people who have gone into CAA and CAL [computer-aided learning] have done so to reduce the amount of work they have to do." He also believes that the effect of CAA upon student learning has not been properly evaluated: "What I don't think people have done is evaluate what [CAA] does to the way students learn." Furthermore, he worries that his students are being over-assessed: "Western is grappling with assessment overload ... as a department, we've tried to draw back."

CAA and the curriculum

Much of the discussion in the earlier section on why CAA is used focused on seemingly positive reasons such as increased student feedback, the motivation of regular learning, and the breadth of scope for tests. However, shortcomings were also identified by both users and non-users, and these concerned the positioning of CAA within the curriculum, question formats and fitness for purpose vis a vis higher order learning, such as synthesis and analysis. Most commercial CAA packages rely on standard objective test formats such as multiple choice, true-false and matching questions. It was in discussions about the perceived limitations of such question styles that the most critical views of CAA were expressed, although, even here there was disagreement over the extent to which the use of objective questions was a strength or weakness.