Status and Scope of Electronic Assessment in Teaching and Learning[1]
Dr. Nazir A. Sangi, Chairman, CS Department, AIOU,Pakistan
Sheeraz Ahmed, Research Associate, CS Department, AIOU.
Principal Contact:
Abstract
ICT has given growth to new applications in distance education. E-learning is now widely being used as an additional / exclusive education delivery method. Assessment in distance learning has been a topic of discussion and difference among educationists. E-assessment (assessment in electronic mode) even poses special problems and issues to individuals and organization practicing it. These issues can be identified as academic, social, ethical, technology and economic related. These issues play a very important role in making a decision about e-assessment implementation at any institute.
A recent survey conducted highlights the state of affairs in various higher education institutions in Pakistan. This paper attempts to identify; assessment practices adopted by the institutions, scope of assessment activities, current status of e-assessment and future development needs and preferences in e-assessment.
1.Introduction
Distance learning and open learning have taken new dimensions of growth with development in ICT(Henrichsen, 2001).Fast, accurate and reliable communication mechanism coupled with digital information available have given a much appreciated growth to teaching and learning (Wachholz et. al., 2005 and Baggaley & Belawati 2007). A number of universities have initiated development of digital contents for their off campus, distance students to be accessed via available (specified) technologies at learner end. Internet is mostly used as intercommunication means in e-learning (UNESCO, 2004). With such awiderange of choices, the learner often gets advantage to opt for a mechanism and technology which is commonly accessible at learner’s end. E-Learning where opened many avenues of learning, it has also posed special challenges in e-assessment. As explained later in this paper, it offers awide choice of research encompassing disciplines such as; education, information technology, communication and social sciences.
In 2004, IDRC, Canada sponsored a research synthesis meeting calling ICT planners, distance education experts and e-learning activists. A group of about 25 experts from 13 different Asian countries participated. A mega research project was launched as a follow-up of this research synthesis meeting (Malik, et. al. 2005). The research group was identified as PANdora ( PAN Asia Networking Distance and Open Resource Access). The mega project consisted of many subprojects. An “E-assessment models and methods for student evaluation in Asia” was awarded to Allama Iqbal Open University, Pakistan and University of Colombo School of Computing, Sri Lanka. Two countries were considered the representatives in Asia to focus research work on “E-Assessment” and other related issues in distance education. A Survey was envisaged as a tool to get base line information in both countries.
In this paper, survey conducted in Pakistan is presented. Although it represents institutions in Pakistan but results are representative and can be used in other Asian countries sharing similar ICT and educational situations.
2.E-Assessment Issues and Challenges
The distance between learners and teachers may be long but if the communication channel is available, learner may get a range of learning choices from a self paced, any time (asynchronous) learning to a real time live (synchronous) interaction with teachers (Sangi & Ahmed, 2007). The terms such as; e-learning, web based learning, computer based training, computer assisted learning, mobile learning, tele-conferencing and video conferencing have emerged to represent different levels of technology applications in distance education (Craven, et. al., 2004 and Henrichsen, 2001,).Many Learning Management Systems (LMS) are also available to facilitate learning and teaching process at distance (Brandt, et. al. 2003, Pearson, 2005 and EduTools, 2007). Applications of new technologies in education have resulted in use of a new term such as “ubiquitous e-learning”, where multiple technologies may be used to have a blended learning.The learner may be facilitated by interfacing physical and cyberspace (Beuschel, W, 2003).
Where most of the e-learning work has focused on improvement of teaching and learning, comparatively less work has been done on evaluation of e-learning. This is obvious due to difficulties associated with the use of ICT infrastructure for electronic assessment (e-assessment). The concept of “any time and anywhere learning” is often not applied to “any time, any where e-assessment”.In order to conduct same or equal e- assessment to a large and distributed learner population, which is usually the case in distance education, the establishment of fair and meaningful e-assessment is quite difficult. A detailed discussion on different issues is provided in a report (Sangi, 2007), a brief summary is given here.
In assessment, there are generally two components; one is formative and the other is summative. Many different assessment techniques exist and are being practiced within institutions(Suleman, 2003). Formative assessment can take place;either at learners own pace through self testsor can be implemented with background measurement and assistance from computer or it can be implemented in a restrictive and supervised mode such as laboratory based e-assessment Many of the Learning Management Systems used for e-learning also facilitate the teacher in some of these processes(Pearson, 2005). Assessment libraries or data banks are often used (Abler, et. al., 2005).An easy to use software interface and simple presentation of the questions are quite important considerations.
The summative assessment is usually performed by institution instead of the teacher.Especially with distance learning students, the major institution requirements are; secrecy and security of assessment documents, authenticity of students at distance and reliability of e-assessment system in conducting a fair summative examination.The availability infrastructure (computers, networks and software etc.) in same time frame, at all locations, are economic and technology related issues to the institutions. Technology reliability during e-assessment conduction is also a technological issue. The skill of learner to use technology besides develop know how in subject matter, is topic of discussion in academic communities. The plagiarism issue in electronic testing also needs special consideration as it is far easier to use same ICT mechanism to find, formulate and present the answer in a very short time.Focus of ICT shifts from provision of multiple electronic resources to restricting and controlling such resources and technologies to avoid plagiarism.
Withe-assessment, there are many academic issues and concerns. Such as, frequent reuse of the same electronic content (i.e. questions) is debatable in academic circles. The e-assessment faces many challenges; from designingof an e-assessment system which may evaluate students’ “knowledge” and“skills”, to thinking of mechanisms for preventing electronic plagiarism and keeping the sanctityof the examination process. MCQ based e-assessment is often applied. But a limited capacity of MCQ based e-assessment system is criticized (Quellmatz & Haertel, 2004). Some e-assessment systems deal with plagiarism issues by randomly sequencing the questions (Gaffar and Sangi, 1998) or by parameterization of questions (Doukas & Andreatos, 2007).
The complexity is further added when electronic examinations are conducted at a distanceand applied to diverse local learners.The issuesoflocalizations may also become a major obstacle in implementing and e-assessment system for mass assessment. Local learner related and ethical issuesare; the ability of the student to use the computer, e-assessment software and solve the examination paper in the same time space(Sangi, 2007) using a non native language. This puts the local students at disadvantage, if they have any deficiency, in use of computer technology, English language or examination system.
3.E-Assessment Survey
Objectives:
It was planned to conduct a baseline survey.Following were main objectives of this survey:
- Collect base line information on operational policies and practices
- Identify the scope of assessment activities
- Collect information on present status and preferences related to e-assessment.
Methodology:
The survey mainly focused on institutions of higher education in both countries and included private and public sector institutions. The survey was conducted between October 2006 to March 2007. Three survey questionnaires were used; first looked at issues related to administration of assessment and targeted at controllers, examination officers and faculty/managers. Who were officially engaged in examination planning, conduction and results preparation etc. related activities. The second questionnaire targeted faculty members who were evaluating students in their specific courses. Third questionnaire targeted students who may or may not have entered into any ICT based assessment. There were some questions repeated in all three groups to get their input on same issues and practices and balance the analysis findings if required.
The questionnaire containing multiple entry with closed questions were distributed to each group participants. In few cases, electronic copy was also e-mailed. The questionnaires were sent to about 50% degree awarding institutes (113 in Pakistan). The selection of institutes in Pakistan was mainly based on institutions with about 2000+ student enrolments. The responses were slow. Replies received are given in Table 1, as follows:
Table 1: Basic Survey Data – E-Assessment Survey: Pakistan
Category / Universities/Institutions / Faculty Members / Students
Responses / 31 / 85 / 60
Distance Education Institutions:
Two Universities in this survey provide education at a distance; these are Allama Iqbal Open University (AIOU), with about 750,000+ students and Virtual University (VU), with about 15,000+ students. AIOU is mainly practicing traditional (print based) distance learning education with audio/video and computer multimedia support in some courses. It has however, initiated formal class and laboratory facilities in many science subjects. E-learning (online) programs are also offered in CS and social science disciplines. It has about 20 million answer scripts to be checked each semester. More than 400 examination centers are managed countrywide with about 50% female only examination centers. Most examination practices are manual. The examination period takes about five months period from preparation to result declaration.
VirtualUniversity (VU) is a new university established with technology based education. It has four satellite based TV channels and broadcasts lectures to its students. VU uses e-assessment system for all its students. VU campus labs are used to conduct the electronic examinations using a customized e-assessment system. VU tutors mark both formative and summative assessments. VU e-assessment system is robust; but also facing manychallenges from students and environment as an ongoing phenomenon. A detailed profiles of these two universities are discussed by Samaranayake and Sangi, (Baggaley & Belawati (Ed.), 2007).
Other 29 universities were formal education universities. However a few had initiated pilot programs in traditional DL and online education. These will be discussed in relevant sections,
- Analysis of E-Assessment Survey
The complete survey report is being compiled and will be published. Due to shortage, of space in this paper, selected findings from this survey are presented in fourparts; 1) assessment practices (as set by the institution’s policies),2) scope of assessment activities,3) status of present e-assessment activities and 4) preferences/reservations related to e-assessment. mainly institution view (Administrators) is presented and discussed in first two parts, Whereas faculty view point is presented in part 3. All three groups (institutions, faculty and student) input is presented in part 4.
4.1Assessment Practices
The institutions have, by experience, adopted assessment policies and associated practices for their institutions. These policies and practices have a wide spectrum of operational activities. These generally relate to summative assessment performed by the controller of examinations. But may also define standard practices for the components of formative assessment. In summative assessment, it defines the processes from paper generation, conduction to certification. Some post assessment analysis may also be carried out. The questions related to who should prepare a summative question papers? When, where and how the assessment takes place in institutions is described in examination rules. The questions were asked from the controllers or faculty involved in examination administration. In order to check possibility of generalizations, a number of questions were asked and their responses are given below.
Examination System
In response to the questionsfrom controllers about general examination system, it was found that about 83.9% universities follow semester system, where as only 6.4% follow annual system, and about 9.7% follow mixed (both) systems in their general examinations. Similar responses were received from other two group where clear majority followed semester system.
Semester system was introduced in Pakistani universities in early 70’s. Before this, all the public and private sector educational universities were following annual examination system. Gradually semester system started flourishing. Due to high acceptability of this system many institutes converted their program into semester system. In a few cases, it increased the workload beyond the capacity of examination department and final results were delayed. Therefore a few universities retained back to annual system of examinations. However, a few universities are using semester system in some programs and an annual system in other programs.This has resulted in a number of variation in a semester system. The components and their relative weightage in formative assessment varies from course to course even in the same institution. Summative assessment variations range from method of paper preparation to a method of marking and preparation of final results.
Number of terminal/final papers conducted per year
The majority ( 48.4 %) of the institutions conduct the terminal/ final (summative) exams twice in a year. Where as about 22.6 % institutions perform terminal examinations three times a year. These universities add a summer semester and therefore have three semesters instead of two. Therefore 80% respondents follow a semester based pattern of examinations. Only 9.7 % universities conduct examinations once a year and represent the annual system of examinations. Other non common systems are; four times a year where the university follows a term (quarterly) system or may take multiple examinations if a qualification certification is performed.
Marking scheme adopted
Which marking scheme is used to prepare (final) student’sresult? The responsesshow that most of the institutions (67.7%) follow Grade Point Average (GPA) system as their adopted method of generatingstudent’s results. Where as 29% do not follow GPA system. If we take out about 9.7% institutions which follow annual system, we can find that about 19% institutions use students marks percentages to determine either their grades (A, B, C …. etc) or divisions (first, second or third) similar to an annual system. Interestingly, the students and institutions still encourage both method of result presentations and issue final transcript accordingly.
Method ofevaluation
As a common practice, Universities in Pakistan follow two methods, either external or internal evaluation of the students. In semester system, most institutions follow an internal evaluation in both formative and summative examinations. However, in annual examination system, external evaluation is generally performed. A few institutions have some variations to moderate either question paper generation process or answer scripts marking processthrough use of an external examiner. The summative assessment is usually conducted under a centralized (controlled) examination environment at almost all institutions.
Paper Setting
a) Internal Examinations:When a question was asked,whoprepares a question paper in internal/formative assessment? The results show that about 67.7% institutions use internal faculty members as paper setters. Interestingly about9.7% institutions use external examiners as paper setters. These exactly match with percentage of institutions who follow annual examination method. A sizable number of institutions (22.6%) use acombination where both external and internal examiners are used for paper setting.
b) Final Examinations: Almost same policy is adopted for summative assessment. The results show that about 61.3% institutions use internal faculty members as paper setters. Whereas 12.9% institutions use only external examiners. A sizable number of institutions (25.8%) use a combination where services of both external and internal examiners are for final paper setting.
Usually in the semester system, the internal faculty members prepare the formative and summative question papers. They seem more reliable source to keep acceptable standards of efficiency, secrecy and academic knowledge within the institutions. However, this method is also criticized for relying on a single person for all academic quality aspects and may result over monotonous papers over a time period. A combined group of external and internal faculty members is therefore encouraged as represented by about 25.8% institutions in the survey.
Final Paper Conduction
Final (summative) paper conduction process is usually centralized. In a response about 71% institutions used internal faculty to assist in centralized examination process. Where as in some cases where university operations are spread over to many locations or students are in large number, a decentralized examinations are conducted. About 12.9% institutions have a decentralized examination conduction system. About 16.1% respondents were undecided and have refrained from answering the question.
4.2Scope of Assessment Activities
The scope of assessment as described above,consists of two main groups; formative and summative assessments. A detailed analysis of assessment activities can reveal various components which may become candidates for electronic assessment. In this part scope of assessment activities including the nature of their components is identified.
Components of Assessment
It is necessary to establish the components of assessment, which over all define the student’s final assessed result. In general, semester system constitutes of assignments, quiz, tests, laboratory project etc. as part of formative assessment. In the summative assessment, final paper is conducted for theoretical courses. Where as laboratory test is conducted or project is examined and an oral examination is arranged. There may be some other options of assessment. Such as;oral presentation, term paper/report evaluation etc. in certain courses.
Figure 1. A sample variety of major assessment components