iask proceedings
Comparing Learning Content Management System’s and Intelligent Tutoring System’s Effectiveness
Ani Grubišić, Slavomir Stankov, Zdeslav Hrepić
Abstract — All instructional software should be evaluated before being used in educational process, because it is important to know whether it actually improves the student performance. Within the context of evaluating the educational influence of learning and teaching process, we measure educational influence by using the effect size as metric. In this paper, we presented the results of an experiment where we have compared one learning content management system, the BlackboardTM, with a representative of Web-based authoring shells for building intelligent tutoring systems, the xTEx-Sys. The experiment was coordinated remotely from distant location (another continent) by the developers of the xTEx-Sys, and directly conducted by the BlackboardTM using expert. The results gained through this experiment were not a total surprise, because this was the first time that the English speaking students, used to working with BlackboardTM, have used the xTEx-Sys, as well as, the first time that an experiment was conducted by a person who had no part in designing the xTEx-Sys.
Index Terms — E-learning, evaluation, effectiveness, intelligent tutoring systems
—————————— u ——————————
- 7 -
iask proceedings
1 Introduction
I
nformation and communication technologies (ICT) have become integral part of educational systems as a support for teachers in realization of traditional learning and teaching process. An intersection between the world of the information and communication technology and the world of the education is nowadays known as e-learning. The e-learning is enabled in the e-learning systems [1].
It is generally known that all instructional software should be evaluated before being used in educational process. Evaluation is useful for the investigation and exploration of different and innovative ways in which technologies are being used to support learning and teaching process. A well-designed evaluation should provide the evidence, if a specific approach has been successful and of potential value to the others [2].
————————————————
§ A. Grubišić is with the Department of Computer Science, Faculty of Science, Split, Croatia. E-mail: .
§ S. Stankov is with the Department of Computer Science, Faculty of Science, Split, Croatia. E-mail: .
§ Z. Hrepić is with the Department of Physics, Ford Hays State University. E-mail: .
There are different kinds of the e-learning systems. In this paper we are particularly interested in two, rather different, types of the e-learning systems: intelligent tutoring systems (ITS) and learning content management systems (LCMS).
Intelligent tutoring systems are computer systems that support and improve learning and teaching process in certain domain knowledge, respecting the individuality of learner as in traditional “one-to-one” tutoring ([3], [4], [5]). The major problems when developing ITS are their expensive and time consuming development process. In order to overcome those problems another approach has been chosen, namely to create particular ITSs from flexible shells acting as program generators.
Learning content management systems are computer systems used to create, edit, manage, and publish educational contents. There are dozens of LCMS available. Most popular LCMS, such as BlackboardTM (www. blackboard.com), is used by hundreds and thousands educational providers. It offers content management, customization and integration with student information systems and authentication protocols. The BlackboardTM allows instructors to disseminate handouts and readings to students, create tests online, maintain gradebooks, organize chat rooms, etc. [6]
In this paper, we present the results of an experiment where we have compared the students who used only the BlackboardTM in their learning and teaching process, with the ones who used only the eXtended Tutor-Expert System (xTEx-Sys) [7], which is a representative of Web-based authoring shells for building intelligent tutoring systems (ITS). This was the very first time that the xTEx-Sys has been used by English speaking students.
The paper is structured as follows. In the second chapter we review fifteen years long research and development of the Tutor-Expert System (TEx-Sys) model [8], that is a hypermedial authoring shell model for building ITS. In the third chapter we give an overview of evaluation methodology that has been used in this research. Finally, in the last chapter we present the results of our experiment deployed to see the differences in using two different approaches, the BlackboardTM system and the xTEx-Sys system, in the same learning and teaching process.
2 Background
The first implementation of an intelligent authoring shell model called the TEx-Sys is the on-site TEx-Sys (1992-2001), after that followed the Web-based intelligent authoring shell (1999-2003, Distributed Tutor-Expert System, DTEx-Sys) [9] and, finally, the system based on Web services (2003-2005, xTEx-Sys).
The xTEx-Sys is a Web-based authoring shell with an environment that can be used by the following actors: an expert who designs the domain knowledge base, a teacher who designs courseware for student learning and teaching process as well as tests for the student knowledge evaluation, a student who selects course and navigates trough the domain knowledge content using didactically prepared course content and, finally, an administrator who supervises the system.
In the past decade, there were numerous applications of the TEx-Sys model in the learning and teaching process that involved students from a primary education, all the way to an academic level. In the period from 2001 to 2007, 1302 students solved 5482 knowledge tests in one of the TEx-Sys model versions, while evaluating their understanding of different domain knowledge.
Questionnaires about the students’ impressions were given to the students after finishing the courses that were supported by the TEx-Sys model. The qualitative analysis of the questionnaires’ results revealed that most students were pleased using the model and that they were open minded for embracing that kind of the learning and teaching support. The qualitative analysis could not determine the effect of educational influence of the TEx-Sys model. That was the reason why we have conducted 11 experiments in order to evaluate the educational influence of the TEx-Sys model [10].
3 Evaluation Methodology
Different evaluation methods are suitable for different purposes and the development of evaluation is a complex process. Experimental research is common in psychology and education [11] and it is suited for evaluation of different kinds of e-learning systems, because it enables researchers to examine relationships between teaching interferences and the students’ results, and to obtain quantitative measures of the significance of such relationships. Controlled experiment is used to ensure that any differences in the measured variable have resulted from the applied treatment and not from other, uncontrolled variables [12].
The different major types of experimental designs, that is, the way treatments are assigned and applied to the available groups, are classified according to the usage of random assignment to groups. If random assignment is used, it is a true experiment. If random assignment is not used, and then if multiple groups are used, then it is a quasi-experimental design, if not, it is a non-experimental design [13].
In a variety of different experimental designs [14], we have decided use a factorial design. In a factorial design we have two or more parallel groups in which are factors introduced simultaneously, by rotation, in each cycle.
At the beginning of each cycle (Fig. 1.), initial states Si1A, Si2A, Si2B and Si1B and respectively their means Xi1A, Xi2A, Xi2B and Xi1B, should be captured using pre-test before introducing experimental factors. Pre-tests are used because we know that students have different skills and backgrounds. At a later point, students involved should take the exact comparable tests to determine the extent to which knowledge and understanding has been improved by the educational intervention. In the first cycle, group A would use the xTEx-Sys (experimental factor F1) and the group B would use the BlackboardTM (experimental factor F2), and in the second cycle the group A would use the BlackboardTM (experimental factor F2) and the group B would use the xTEx-Sys (experimental factor F1). At the end of each cycle, final states Sf1A, Sf2A, Sf2B and Sf1B and respectively their means Xf1A, Xf2A, Xf2B and Xf1B, should be captured using post-test, in order to calculate the difference in the BlackboardTM and the xTEx-Sys effectiveness. Therefore, in the first cycle experimental group is the group A, and in the second the experimental group is the group B.
In factorial design, different domain knowledge is learned in each cycle. If we find that in each cycle one and the same factor is more efficient than the other, then we can conclude that a certain factor is better regardless of a difference between taught domain knowledge.
In the same way, if the same factor is more efficient in each cycle, in spite of the groups that it has been introduced in, then we can conclude that a certain factor is better regardless of a difference between the groups.
It seems that this approach has no negative sides, but it is not true. Namely, we can only be sure in effectiveness of one factor if it was found to be better in each cycle regardless of the groups’ equivalence and domain knowledge (what is not highly possible). Therefore, it would be advisable to design statistically equivalent groups. Besides, conducting this kind of experiment is related to many organizational difficulties and that is the reason why it is not used very often.
4 Description Of The Experiment
The experiment took place at the Fort Hays State University (FHSU), a state supported University in Kansas, USA. The experimental design involved 4 sections of Physical Science Lab course (PHYS 103) for non-science majors [15]. The course meets once per week for two hours and course assessment traditionally consists of three components:
1. Pre-lab quizzes (pre-labs), which students take individually online and before the lab. Pre-labs are based on a short introductory reading related to the lab content and serve to familiarize students with the topic.
2. Lab reports, the second assessment component students complete during the class time collaboratively.
3. Tests associated with lab material which typically involves short answer and multiple-choice questions. Tests are administered as paper and pencil, in-class exams.
One of the authors taught this course each semester between Fall of 2004 and Spring of 2008, and in this entire period tests were the weakest component of the students’ performance in the course. Another three instructors taught one or more sections of the course in the same period and this entire time test results hardly depended on instructor.
In order to remedy the situation, in Fall of 2007 we deployed post-lab quizzes (post-labs) which are also taken online, but after the lab, and serve as lab material reviews and reinforcements. Post-lab quizzes, as well as pre-lab quizzes consisted of pre-defined multiple choice questions and were administered using BlackboardTM. The success with implementation of post-lab quizzes in Fall of 2007 was limited.
Semester later, in Spring of 2008, in attempt to possibly improve effect of post-lab quizzes, we administered them using the xTEx-Sys system. As described before, the xTEx-Sys system is not only a quiz-administrating tool, but also provides practice venue for students. In addition, questions that xTEx-Sys administers are not predefined but rather generated by the system based on the predefined semantic network structures related to the domain of interest.
In addition to post labs administered through the xTEx-Sys system, we used the BlackboardTM administered post-lab quizzes with pre-defined questions as control for the xTEx-Sys system effect on students’ learning. Our research question was whether or not administration of post-lab tests on the xTEx-Sys system affects i.e. improves student learning differently than the BlackboardTM administered quizzes, as measured by students’ test scores.
We have randomly assigned two of four sections to the experimental group and the other two to the control group. The experimental treatment consisted of taking post-lab quizzes related to three labs on the xTEx-Sys system and control groups was taking corresponding post-lab quizzes on the BlackboardTM system. The experimental group also had opportunity to learn the content using the xTEx-Sys learning and teaching features. All other parts of the course operated identically for students in all sections.
After first three labs involved in experiment, all sections took the same in-class test, the checkpoint test CHK1, after which control and experimental sections exchanged. After another three labs they again took identical in-class test, the checkpoint test CHK2. Each test covered only content related to three labs that preceded it.
We gauged students’ attitudes toward the xTEx-Sys and the BlackboardTM systems using an online questionnaire administered after the second test.
4.1 Subjects
The study deployed in Spring of 2008 involved four sections of PHYS 103 taught by the same instructor.
The experiment started in February 2007 and lasted until the April 2007 (in total 10 weeks). At the very beginning of that experiment there were 65 students, but eventually only 48 of them completed all parts of the experiment (74%).
Therefore, of the 65 students that agreed to participate in the experiment, 26 students were assigned to a group A (sections A and D) and 39 students to an group B (sections B and C). The group A was the experimental group in the first cycle and the group B was the experimental group in the second cycle.
4.2 Procedure
The experiment was conducted following the plan presented in Table 1. First, a short introduction during which the purpose of the experiment and general organizational issues were explained was given. Then the pre-test was conducted. Following the pre-test, a brief introduction into organizational issues related to the treatments was given.
During the experiment, there were two treatment-test cycles. The tests were used to measure the dependent variable – student knowledge. After completing first treatment, the both groups performed the first checkpoint test (CHK1) and after second treatment they performed the second checkpoint test (CHK2).
After completion of the second checkpoint test, an online survey was administered to all sections in order to gauge and compare student’s attitudes toward the xTEx-Sys and the BlackboardTM systems, as, at this point, all students used each of the systems. This survey provided data on perceived time pressure and subjective judgment of training quality (Fig. 2). Students answered each question by selecting their level of (dis)agreement on a five-point scale ranging from strongly agree to strongly disagree with neutral option included. During the whole procedure, the time slots reserved for completing a certain step of the schedule were identical for the experimental and control groups.