Considering automatic educational validation of computerized educational systems

Alexandra Cristea and Toshio Okamoto

University of Electro- Communications Graduate School of Information Systems

{alex, okamoto}@ai.is.uec.ac.jp

Abstract

Evaluation of educational systems is a difficult and often subjective process. More and more computerized educational systems are being proposed, many find their ways into the classrooms and already are forming new generations of pupils. However, it is risky to leave the actual validation of these new systems to a trial-and-error technique, especially when the involved target are humans. We propose with this paper automatic, virtual students for a more unitary and consistent testing of computerized educational systems, as an alternative to the traditional trial-and-error method.

1. Introduction

Recently, there is a trend to improve and expand education via computerized systems. Especially, the developments in the hardware industry made faster, preciser, efficient and reliable computers and software systems possible. Therefore, the usability of computers in all life sectors increased. In education, in particular, computers are mostly used for simulations, for data storage and processing, for automatically assessing students, a.s.o. The Internet especially made the new paradigm of distance learning, and with it, the life-long learning paradigm possible. The limitations and problems of the first educational applications, as well as the skepticism of some defenders of the old teaching ways could not stop this progress. Now-a-days, the stress is on collaboration via Internet, collaborative learning, as well as learner-adaptive and learner-centered systems. New systems are being built all the time, and they vary in teaching methods used, expression ways used, technical qualities, etc. Therefore, it is extremely difficult for educators who have to choose among these systems for their students, as well as for individual learners who wish to find the most appropriate system for their learning goals.

Judging by the state of the art applications and learning systems, and looking at the articles published on this subject, the best researches propose a new system, build it, test it on a class of real students or more, and then publish their results. This method, unfortunately, says little of the actual usability and efficiency of their proposed system, and the duplication of the reported results is questionable.

In this paper, we will propose an alternative method of evaluation of computerized educational systems, based on automatic students.

2. Problems with computerized educational systems: new trends

Apart from the critics warning about a "dehumanization of the learning process" [8], [5], computers in education have had some extremely positive feedback and are an evident success [2]. However, various problems appeared, some due to form (e.g., too many text-based systems, the appearance of multimedia leads to over-using of media effects, etc.), but mostly due to the lack of flexibility, adaptivity, [1] features which are natural in the human teacher. Most of these systems offer a good-for-all education delivery, which disregards the different ways of perception of the different students, their backgrounds, levels of understanding, etc.

To alleviate some of these problems, the recent trends in computerized educational system research tend towards adaptivity, multimedia, distance learning, collaboration, collaborative learning, user modeling, agent technology, intelligent tutoring systems.

Together with the learner-centered trend, there is a growing understanding that the final learning results are a combination of the used teaching strategy and the learning strategy employed by the student.

3. User modeling

Currently, many researchers are using user modeling techniques. Such research has started in the early '80s [7] from the idea that different users have different needs. In education, this is particularly true, although this process entered educational applications with a delay [4]. In educational applications, simply put, the main purpose of user modeling is to find out how the student learns best, and to offer him/her the suitable form of education. This is also called student modeling. The analyzing of the student is limited by the interaction human-computer: questionnaires (single/ multiple choice, etc.), tests, tracing the learner's steps during learning and interpreting them, etc. All the information gathered in this way is then interpreted into a simplified student model. The latest student models contain a layered evaluation of the student, from different points of view, as can be seen in figure 1 [1].

The main difficulties of student modeling are represented by the following steps:

-gathering information

-filtering the relevant information only

-interpreting this information for selecting the appropriate pedagogical strategy

Pedagogical strategy contexts can be tutor-tutee, learning companion, learning by disturbing, learning by teaching, learning with co-teacher. Within these strategy contexts, direct strategies can be learning by examples, learning by story-telling, learning by doing, learning by games, etc.

From the steps above, the last step means matching the deduced student learning strategy (resulting from his/her deduced cognitive style), with the appropriate teaching strategy. This is, of course, a difficult process, but one that has been thoroughly researched by psychologists, and that can rely on mankind's hundreds of years of teaching and learning practice. Indeed, talented teachers seem to instinctively select the best tutoring strategy for the respective student. Moreover, many educational systems (such as the US [6]) routinely lean on IQ tests, personality tests or various other cognitive and learning style tests, in order to classify students and to offer them tailor made educational programs.

4. Current evaluation methods

Currently, there are many evaluation ways of computerized educational systems. Most of these evaluations are of a technical nature (e.g., [3]). Many sites accumulate information in order to help teachers decide on one product or another, and criteria are evaluated such as transfer format, service, platforms, etc. Some of the offered educational evaluation points comprise instructor or student tools, e-mail capabilities, etc. Surely these information are useful, but they say little of the educational values of such products. As the main purpose of computerized education systems is education, moreover, the authors usually claim that their system can provide better education than the classical education systems, this lack of educational evaluations is extremely dangerous.

Of course, educational evaluations are difficult to perform and, as of mostly qualitative and not quantitative nature, they are also subjective, and results are very often not reproducible.

An important factor here is subjectivity versus objectivity. Questions in questionnaires can lead students to some specific answers. It is known that students try to please their teachers, and, for instance, only by turning a questionnaire into an anonymous questionnaire, answers can vary a lot. Moreover, a specific set of questions can only cover the area which it targets. Questionnaire designers might be able to predict some reactions, but might fail to predict all. Free comments can improve this matter, although students are often shy in expressing directly their opinions, etc. Furthermore, as has been often pointed out, students are often voluntarily or involuntarily not truthful, and they might often not know themselves well enough to give relevant answers.

The field obviously lacks benchmarks, clear effectivity measures, with which each researcher can compare his/her own application.

Moreover, most of the evaluations of educational nature that are done are usually based on the questionnaire method. A class of students is usually used to work with the new system, and at the end is questioned about the effectiveness of the new system. Sometimes comparisons are made with another set of students that work without the system, sometimes pre-tests, etc., are also involved. But these systems are basically never reported after long-term use on different types of students, with different backgrounds, etc., in other words, with different cognitive styles. Graphically represented, the experimental situation is more than often something like in the figure 2.

This situation shows clearly that a structuring and a creation of benchmarks for this field is necessary.

5. Automatic students

Among the researches that propose new computerized educational systems, a new evaluation method emerged. As testing with real students is often time-consuming, as well as resource-consuming, on one hand, and on the other hand, trial-and-error type of research is inappropriate when dealing with real students, some researchers have proposed systems based only on simulated tests. In their simulation, they have used what is called simulated, or automatic students.

Of course, many of the previous critics for the current evaluation methods, such as the subjectivity, etc., can apply in such cases as well, although out of different reasons. It is questionable if a simulated student, built entirely according to the predicted student model, will not simply always generate good results, which, translated into real life, might result in completely different real evolutions of the human students.

To validate the usage of automatic students, the specifications of these students must be more general, and not decided by the educational system designer, but should be accepted by an international community.

6. Learning Styles

The literature provides various definitions of cognitive styles (proposed initially by Allport in 1937) and learning styles (proposed initially by Herb Thelan in 1954), and often the two terms are used interchangeable. For our purpose, we are focusing on learning style, as the specific individual approach to acquiring new knowledge of each student. According to the student's learning style, the student is able to receive knowledge easier or not via a certain teaching style. The learning style is independent from the other abilities, which have direct sequels (the more, the better [6]), whereas styles are controlling mechanism and define the internal preferences and value system.

Among the different cognitive/ learning styles, we are enumerating some of the more important in the following.

6.1. Hill's cognitive style mapping

Hill has built a cognitive style coefficient as a function of symbols and meanings (i.e., the preferred form in which an individual encodes information), cultural determinants (i.e., family, colleagues, etc.), modalities of inference (reasoning style, i.e., inductive, deductive, etc.) and a memory function.

It is interesting here to note that the cultural determinants, in the form of the influence of the country and cultural background on ones information processing style (learning style) have only recently been proposed for studying in the adaptive hypermedia community.

6.2. Kolb's learning styles

Kolb (1984) defined a 2-dimensional scale to represent learning styles, which leads to 4 extreme cases:

  1. converger (abstract, active): abstract conceptualization and active experimentation; great advantage in traditional IQ tests, decision making, problem solving, practical applications of theories; knowledge organizing: hypothetical-deductive; question: "How?".
  2. diverger (concrete, reflective): concrete experience and reflective observation; great advantage in imaginative abilities, awareness of meanings and values, generating alternative hypotheses and ideas; question: "Why?"
  3. assimilator (abstract, reflective): abstract conceptualization and reflective observation; great advantage in inductive reasoning, creating theoretical models; focus more on logical soundness and preciseness of ideas; question: "What?".
  4. accomodator (concrete, active): concrete experience and active experimentation; focus on risk taking, opportunity seeking, action; solve problems in trial-and-error manner; question: "What if?".

6.3. Dunn and Dunn's Learning styles

Rita and Kenneth Dunn developed in 1974 a comprehensive learning style model on the following axes:

  1. environmental factors (sound/noise level, light level, temperature, design setting)
  2. emotional factors (motivation, persistence, responsibility, structure)
  3. sociological factors (self-orientation, colleague orientation, authority orientation, pair orientation, team orientation)
  4. physical factors (perception, intake, time, mobility)

Although this model deals very little with the cognitive factor, this model is currently used in schools for pupils of grades 3-12 and a version has been developed for adults.

6.4.Herman brain dominance model

Ned Herman classified in 1976 thinking styles according to the brain quadrants:

  1. Quadrant A (left brain, cerebral): analytical, logical, factual, critical and quantitative
  2. Quadrant B (left brain, limbic): sequential, structured, organized, planned, conservative and detailed
  3. Quadrant C (right brain, limbic): interpersonal, emotional, sensory, kinesthetic, symbolic and spiritual
  4. Quadrant D (right brain, cerebral): visual, holistic, innovative, conceptual, imaginative, artistic

His model could classify people according to their preferences, determining a dominant style (without excluding different degrees of preferences for the remaining quadrants).

6.5.Other models

There are many other models, but among all those we would like to mention also the classification into:

  1. field dependent
  2. field independent

Whereas field independence means the extent to which a person can perceive analytically, and can distinguish the study object from the surroundings. Field dependent people are dependent on external cues, and can, for instance, learn better if they have graphical support.

6.6.Considerations on learning styles

From the short presentation on learning styles above it is obvious that, although the correlation between learning styles and teaching styles has not yet been clearly defined, the existing findings already can provide guidelines. Indeed, many of the above classifications are actively used in evaluating students in the regular school system, as well as in higher education or life-long learning.

For instance, it is good to consider before introducing a new on-line course with all the new multi-media technological gimmicks, that this course might not improve learning for field independent students.

Or, before taking over collaborative learning as the latest in computer based learning developments, to consider that self-oriented students might actually be perturbed by the added interaction, and that it might actually reduce their learning results.

Moreover, it is obvious that there exists an information exchange deficit between psychological researches and the computerized educational systems community, which results often in a poor reflection of the findings of one domain into the other, to the detriment of both.

7. Proposing an automatic student pool

The evaluation of educational systems is difficult, as has been noticed by many researchers. In mathematical terms, the function:

O = f(I)

is not well defined. The input, I, which are the students, can vary a lot from case to case. The output, O, also, can be measured from many different angles, and therefore is also poorly defined. Inbetween there is the computerized educational system, which provides our function f. A good result O1 for input set I1 cannot guarantee a good result O2 with input set I2.

Looking at the same problem another way, if the students are good, the result will be more or less good, disregarding the appropriateness of function f.

Most of the current tests of educational systems presume a more or less even distribution of the cognitive styles among their students. This is more than often not the case.

To solve these problems, we propose the building of a automatic student pool for computerized educational system testing and validation. By designing specifications for simulated students representing different cognitive styles, such a pool of students can be created. A computerized educational system designer can refer to this student pool when testing his/her system, and report the results that s/he achieved by using the virtual students from the student pool (figure 3).

Although these simulated students represent only simplified versions of the real learners, the using of the same simulated students for different systems can be the common denominator, and provide a uniform measurement system and benchmark. Moreover, the pool of students can be permanently improved, as more information is gathered about the particularities of the different learning styles, and the process can be of a permanent feedback type.

Results from new researches can be of the type:

"our system performed well with the convergent students with strong field independence", etc.

In this way, concrete and clear information about the educational usage of the respective system can be expressed.

Furthermore, comparison with performances of real students and the students from the simulated student pool can validate these evaluations and suggest improvements of the virtual students in the student pool.

The virtual students representing the different cognitive styles can be exposed to the different teaching strategies provided by the system, in order to validate the system.

By showing which students performed best, the system designers can show concretely for which type of students their system is applicable.

Beside of the obvious difficulties of implementing such virtual students, real students have another characteristic that is difficult to implement: they sometimes have the tendency to change their cognitive style, i.e., their approach to new information and its processing. This change is dependent or independent on the teaching strategy used. Some teaching strategies are specifically aimed at changing the cognitive style. Such simulation as in the latter situation can also be performed, and percentile results of the students who could be "converted" by the teaching strategy can be obtained.

Here it is to be added that modifications in cognitive style are long-term processes, and it cannot be reasonably presumed that such changes could take place by, for instance, attending one single on-line course.

The final evaluation and validation results should show that a respective system is efficient for students representing some specific cognitive styles, or who are able to change their cognitive style.