Developing competent e-learners: the role of assessment.

JANET MACDONALD, Open University in Scotland, Edinburgh, UK

Paper presented at the Learning Communities and Assessment Cultures Conference organised by the EARLI Special Interest Group on Assessment and Evaluation, University of Northumbria, 28-30 August 2002

ABSTRACT We know assessment plays a major formative role in driving student learning appropriately, but what implications does this have for online courses? Is it more important than in a face to face context, or less so? Should we reconceptualize the ways in which we assess students, or are existing methods, tried and tested in conventional teaching and learning situations, appropriate? This paper draws on recent research of student and staff perspectives on two networked courses at the Open University, which employ constructivist pedagogies. It discusses the practical implications of implementing these online pedagogies and illustrates the powerful formative effects, both intended and unintentional, of assessment on student learning and behaviour on these courses.

Introduction

The use of electronic media for flexible course delivery provides a new context for teaching and learning. The emphasis on asynchronicity brought by email and online conferencing brings with it an increased scope for flexibility in study routines, and meets a growing demand for part time study, continuous professional development and lifelong learning.

Electronic media offer a variety of ways of presenting or structuring learning opportunities which were not previously available on distance courses. They offer new ways to access and combine information, and the possibility to keep in touch on a more regular and continuous basis, so that students need no longer work in isolation, but belong to an electronic “community of learners” (Collis, 1998).

E-learning has recently achieved prominence as the “Philosopher’s Stone” for future development in Higher and Further Education, and yet there is little common understanding as to its meaning.

Mason (2002) outlines the wide spectrum of interests and understandings attached to the term, from those with social constructivist approaches, whose aim is to make use of the communicative potential of online learning, to those who have a more behaviourist approach, and see it as a convenient medium for content delivery and testing students. There is also a wide variation in the types of student who may be involved in e-learning, in terms of age or motivation, their situation, which may be campus based, or remote, or a combination of both, and also the discipline which they are studying.

As part of this debate, a recent ESRC manifesto describes networked e-learning as:

“..those learning situations and contexts which, through the use of ICT, allow learners to be connected with other people, and with shared information rich resources. Networked e-learning also views learners as contributing to the development of these learning resources and information of various kinds and types.”

(ESRC, 2002).

This paper concentrates on those courses where a constructivist approach is adopted, by using online media to support distributed collaborative interaction and dialogue, and sometimes access to information rich resources. Such courses place importance on understanding, rather than on memorising and reproducing facts, and on the contribution of social interaction and collaboration to learning.

Constructivist philosophy accommodates a family of closely related pedagogies, which optimise the potential of networked environments. The family includes collaborative learning (McConnell, 2001), activity-based learning (Macdonald & Twining, in press), resource-based learning (Macdonald, Heap & Mason, 2001) and problem-based learning (Ronteltap & Eurelings, 2002).

These four pedagogies lay different emphases on particular facets of constructivist philosophy, as their names suggest. They all operate by providing opportunities for students to learn by engaging in activities, which might involve collaborative work, or problem solving, or open access to electronic resources. In practice, many networked courses adopt all four approaches to varying degrees, alongside more conventional modes of delivery.

The pioneers in e-learning have tended to concentrate their studies on postgraduate courses, often having small numbers of students, where e-learning has been the subject of study, as well as being the medium used for interaction. Many useful lessons have been learnt from these courses, however, e-learning is rapidly being mainstreamed at all levels of study, for a wide variety of courses.

At the OU UK there are now 275 courses involving 100,000 students in which online media are used for tuition. Over 16,000 FirstClass conferences have been set up for course related tuition, as well as student run peer support through the student union. In addition, by 2005 all students will be encouraged and expected to use web based systems for administrative services and non-academic student support; this will no doubt raise the general level of computer literacy, and also student expectations of online support.

What is a competent e-learner?

Courses using electronic networks make certain demands on the students who study them. Inevitably, using a networked computer as a study tool involves basic ICT skills which need to be mastered, and this eventually leads to a familiarity with the relevant hardware and software tools.

Beyond this, students need to learn how to use the computer as a study tool. They may need to accommodate screen based study, with its limitations on flexibility, and the implications for time management; in addition to learning how to manage files effectively. Approaches to writing and sharing information in electronic form offer enhanced possibilities for the re-drafting of scripts, as well as the use of information from a variety of sources. This presupposes some understanding of plagiarism, and the role of academic evidence in developing an argument. Arguably, all these skills and adaptations, once learnt, are fairly readily used in a variety of situations.

Courses which are networked will by definition offer students greater access to the opinions of peers and the resources of the web, and students will need to develop a self-directed approach study, and adopt a critical approach to these resources. Of course, under traditional university models of teaching and learning, encouraging students to develop any independence and self-direction in learning is a lengthy and gradual process (Perry, 1970), which develops throughout a graduate course of study. Since networked courses with constructivist approaches have been introduced at all levels of the undergraduate curriculum, e-learners may be exposed to unfamiliar demands at an early stage in their academic career.

Self-directed study using electronic networks requires competence in two major areas: information literacy and online collaborative learning, in line with the need to cope with collaborative interaction and access to information rich resources. The term information literacy has been in use particularly in the US and Australia, in connection with self-directed lifelong learning (see for example Bruce & Candy, 2000). Its characteristics include recognising the need for information, being able to identify and locate it, gaining access to it, then evaluating, organising it and using it effectively (American Association of College and Research Libraries, 1999). Students need to recognise the gaps in their knowledge, in order to establish what they need to find out, and Hill (1997) underlines the significance of subject knowledge, in providing the framework for further exploration and research, in addition to confidence and metacognition. It is clearly not enough to teach students searching techniques, and then to assume that they will be competent investigators. Indeed the support of students in this area is highly problematic, and McDowell (2002) describes a wide spectrum of approaches to the use of electronic resources within courses.

There is similar complexity in online collaborative learning. Salmon (2000) proposes a number of progressive stages of development, which include access and motivation, socialisation, information exchange, knowledge construction and development. These stages illustrate the interplay between competence, and affective factors such as growing confidence, motivation and group dynamics. And of course if students are to communicate effectively within an academic discipline, then they need to become familiar with the language of a discipline and the academic genre. Lea & Street (1998) maintain that this familiarity with the discourse is a defining factor in students' abilities to read and write appropriately within a discipline. In fact, this familiarity grows as they practice writing conference messages on course topics, and reading, or eventually responding to messages from others.

Finally, if students are required to collaborate to undertake a common task, as opposed to making optional contributions to a conference, then they need to practise team working and negotiation skills, group decision making and task management (see for example Schrage, 1990). Again, affective issues may be significant here, for example group cohesion, and the evolution of mutual trust. It follows that the whole process of competence in online collaboration certainly requires practice, and will take some time to develop.

In short, I have argued that the competent e-learner will have developed communicative and interpretive ability using electronic media. This involves the use of computers as an effective study tool, but also implies the development of critical and analytical abilities to work as a self-directed learner, together with the communicative abilities to work with, and learn from peers. In other words, competence in e-learning has many parallels with competence in learning, and is probably acquired as a developmental progression, alongside developing confidence and metacognition, and a growing understanding of a discipline. However, the use of online media for study, and the ready availability of information rich resources on the web, means that students may need to develop competence and self direction at an earlier stage in their undergraduate career than they would in more conventional distance courses. Course design and outcomes will need to reflect and support these needs.

A significant body of research supports the view that the design of assessment is critical in determining the direction of student effort, and that the formative value of assessment is vital in providing a channel of communication between students and their mentors (see for example Black & Wiliam, 1998). This role for formative assessment is increasingly important for campus based universities, as well as in a traditional distance learning context (Higgins, Hartley & Skelton, 2002). But to what extent can the formative power of assessment be deployed to develop competent e-learners, and does it differ in any way from formative assessment to develop competent learners?

In this paper I discuss the lessons learnt on assessment design for developing competent e-learners, by drawing on two qualitative studies of two networked courses at the UK Open University; both studies are reported in detail elsewhere (Macdonald, Heap & Mason, 2001; Macdonald & Twining, in press). Both courses engage students in learning overtly about e-learning, but I believe that the lessons learnt here will be applicable to any networked course which makes use of activities such as those described above, because it is likely to make similar demands on its learners.

The courses

The courses described here were both 60 point 2nd level undergraduate courses. In line with common practice at the OU UK, they were designed for part-time study, and each presentation ran for six months. Both courses adopted a constructivist approach and made use of the networked environment to engage students in a variety of activities. They were both assessed by a series of assignments spaced throughout the presentation together with a final examinable component.

The Technology course THD204 : “IT and Society” explored social and technological issues associated with IT developments. It was presented from 1995 until 2001, to approximately 1500 undergraduates each year. The course adopted a resource based and collaborative approach, delivered through a combination of printed course books, a CD-ROM indexed library of academic articles, and an internet Home Page which gave access to resources for further reading.

Empirical data is drawn from a case study of perspectives on the assessment of this course, which was undertaken over a three year period with three cohorts of students and their tutors. Data were gathered from 21 initial in-depth interviews and observations. These were followed by a series of computer conferences to which 800 students and 50 tutors were joined over two years; they ran throughout each course presentation, and provided a running commentary on the course. They were supplemented by 100 telephone interviews (Macdonald, 1999).

The Education course E211: “Learning matters: challenges of the information age”, was presented in 2001 to 200 students. The course aimed to empower students to deepen their understanding of learning, and offered a variety of perspectives on learning, with particular reference to the growing influence of ICTs. It placed an overt emphasis on experiential learning and reflective analysis, whilst being relatively low in print delivered content, in comparison with conventionally produced distance courses. Course activities represented an integration of three types of task, in addition to course readings and videos:

  • Participation in online conferencing and internet searching with fellow students;
  • Hands-on experience of ICT as a tool for structuring reflection on learning;
  • Maintenance of a portfolio in which they recorded personal learning experiences and made notes on course readings and various other activities.

A case study of both students and tutors on their perceptions of the course assessment was carried out at three points of enquiry, immediately after three of the assignments, using course conferences and targeted email questionnaires. A total of 10 out 12 tutors, and 20 students contributed feedback.

Creating e-learning opportunities at critical points

If a course is based on active learning and experiential approaches, then it is critical that students actually do the activities they are set, so that they engage with the learning process and derive benefit from the course. In a classroom situation there is some expectation that students, once in the room, will participate, at least to some extent, in the activities set them. Distance learners are not the same.

Intuitively one might expect little guarantee that students will undertake any element of a course in the prescribed manner, unless there is sufficient motivation to do so, and Lockwood (1992) describes the balance of “costs and benefits” undertaken by distance learners when considering whether to undertake in-text activities. Whether e-learners are any less likely to participate than other distance learners is difficult to say. Some students might be attracted to the "fun" element in activities, others might be motivated by online interaction, whilst others might see online activities as intrinsically "lightweight" in comparison with traditional course content. Obviously, this issue of participation is of critical importance where activities form a central part of the course. It is even more critical where activities involve online collaboration, because non-participation by one student impacts on other students.

Our own experiences illustrate this point. Throughout the course E211, students were expected to engage in online discussion in groups of 15-20, which were moderated by their tutor. The course materials provided them with discussion points to follow through in their tutor groups. Mid-way through the course, students were given an online collaborative activity, on which they were expected to work in sub-groups of four to six students. They were required to search for resources on the Internet and to reflect on their experiences of online collaborative working.

It appeared that for about half the tutor groups, participation had waned since the start of the course, and the sub-groups had achieved limited success in the collaborative task. Arguably the extent of success was related to a variety of factors, and the skills and moderation tactics of the various tutors undoubtably played a significant part in this (Mason & Bacsich, 1998). However, feedback from both students and tutors revealed that the greatest perceived hurdle to successful participation was the lack of integration of assessment with the collaborative task. At the same time, an impending assignment which was not related to the activity, materially detracted from student efforts, as these comments describe:

“Assessment tasks take priority and however valuable the learning opportunity might appear to be, it will be sacrificed in the name of survival.”

(E211 tutor)

“..timing and incentive are both important - a number of my students commented that they would have completed this activity (and taken a more active part) but stopped when they felt they had to give priority to work on assignments.”

(E211 tutor)

"..an impending assignment reduces participation significantly, and the very short gap between [assignments] 3 and 4 has more or less killed activity in my conference stone dead...Some of my students who began hopefully in the spirit of the course have been put off conferencing simply because they began to feel they were providing useful material for others who read but did not contribute."

(E211 tutor)

There will always be students who recognise the value of online participation for their learning, and appreciate the benefits of belonging to an online community. But the only time when most students will undertake activities is when they are linked to assessment. The assignment will always take priority, and may detract from non-assessed activities. Of course, there are always exceptions, and some students will ignore activities even when they are linked to assessment (Thomas & Carswell, 2000).

An assignment which requires reflection on a topic of online debate, or on the application of a particular skill in operating course software, helps to ensure participation by most students, if it is timed to coincide with the relevant activity. The following is an example, which illustrates how an assignment can cover course content while also encouraging students to undertake an online collaborative activity: