Why do students using interactive learning environments game the system?
Author Information Removed
RUNNING HEAD: Why do students game the system?
Abstract
In recent years there has been increasing interest in the phenomena of “gaming the system”, where a learner attempts to succeed in an educational environment by exploiting properties of the system’s help and feedback rather than by attempting to learn the material. Developing environments that respond constructively and effectively to gaming depends upon understanding why students choose to game. In this paper, we present three studies, conducted regarding two different learning environments, which present evidence on which student behaviors, motivations, and emotions are associated with the choice to game the system. We also present a fourth study to determine how teachers’ perspectives on gaming behavior are similar to, and different from, researchers’ perspectives and the data from our studies. We discuss what motivational and attitudinal patterns are associated with gaming behavior across studies, and what the implications are for the design of interactive learning environment.
1. INTRODUCTION
In recent years, there has been increasing evidence that students choose to use interactive learning environments in a surprising variety of ways (Wood and Wood, 1999; Aleven et al , 2004; Baker et al, 2004; Mostow et al, 2002; Arroyo and Woolf, 2005; Stevens and Soller, 2005), and that some choices are associated with poorer learning (Baker et al, 2004; Beck, 2005; Aleven et al, 2006). In particular, one category of behavior, gaming the system, has been repeatedly found to be associated with poorer learning (Baker et al, 2004; Baker et al, 2005; Beck, 2005; Walonoski and Heffernan, 2006a). Baker (2005) defines gaming the system as “attempting to succeed in an educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly.” Gaming behaviors have been observed in a variety of types of learning environments, from educational games (Klawe, 1998; Magnussen & Misfeldt, 2004) to online course discussion forums (Cheng & Vassileva, 2005), and have been repeatedly documented in one type of interactive learning environment, intelligent tutoring systems (Aleven, 2001; Baker et al., 2004; Beck, 2005; Mostow et al., 2002; Murray & vanLehn, 2005; Schofield, 1995; Wood & Wood, 1999). Across the systems studied, a reasonably substantial minority of students (10-40%) appear to engage in some variety of gaming behavior, at least some of the time.
Within Cognitive Tutors and ASSISTments, the two types of intelligent tutoring system we investigate in this paper, gaming the system consists of the following behaviors:
- quickly and repeatedly asking for help until the tutor gives the student the correct answer (cf. Wood & Wood, 1999; Aleven, 2001)
- inputting a sequence of answer attempts quickly and systematically. For instance, systematically guessing numbers in order (1,2,3,4…) or clicking every checkbox within a set of multiple-choice answers, until the tutor identifies a correct answer and allows the student to advance.
Other examples of gaming the system include choosing to work on material which the student has already memorized (personal communication, Jack Mostow), re-starting problems when a tutoring system only saves at the end of each problem, and intentionally posting irrelevant material to online course discussion (Cheng and Vassileva, 2005).
Recent work to model gaming the system has resulted in systems which can effectively detect gaming behaviors in a variety of intelligent tutoring systems (Baker, Corbett, and Koedinger, 2004; Beck, 2005; Baker, Corbett, Koedinger, and Roll, 2006; Walonoski and Heffernan, 2006a; Johns and Woolf, 2006; Beal, Qu, and Lee, 2006), using a considerable variety of modeling frameworks and approaches. The existence of systems that can effectively detect gaming has spurred the development of systems that attempt to respond to gaming, when it occurs. Prior to this work, most attempts to remediate gaming consisted of attempting to make it impossible for students to game (cf. Aleven, 2001; Murray and vanLehn, 2005), but it was found that this approach both reduced the usefulness of help features for non-gaming students (cf. Baker, Corbett, Koedinger, and Wagner, 2004) and that gaming students discovered new strategies for gaming that worked despite the system re-designs (Murray and vanLehn, 2005). Newer systems have responded to gaming using models that identified exactly when and which students game the system, and by producing visualizations of student gaming that were visible to both the student and their teacher, instant feedback messages that notify the student that they are gaming, and/or supplementary exercises on exactly the material the students have bypassed through gaming. These systems have been successful in reducing the prevalence of gaming (Walonoski and Heffernan, 2006b; Baker et al, 2006), and in improving gaming students’ learning (Baker et al, 2006), over short (approximately week-long) sections of intelligent tutor curricula. However, it does not appear that these systems address the root causes of gaming; instead these systems only alleviate gaming’s symptoms. Hence, it seems quite possible that students will over time discover ways to game these interventions, reducing their effectiveness over time.
In order to have confidence that a re-design will address the root causes of gaming, rather than briefly alleviating its symptoms, our community needs to understand why students game – both in terms of context (what specific situations students choose to game in), and of student characteristics (what factors differentiate the students who game from the majority of students who do not game). There is already preliminary evidence about when students game, that was gathered in the process of developing detectors of gaming behavior, showing that students game on steps they personally find difficult. However, knowing that some students game on steps they personally find difficult does not tell us why the majority of students do not game on steps they personally find difficult.
It could be of considerable value to know what motivates students to engage in gaming. Other ITS researchers have looked at tracking students’ motivations as the students use an intelligent tutoring system. One classic example is the work by de Vincente & Pain (2002), who developed a system which tracked 9 motivational variables (including effort, satisfaction, interest and independence) – however,they did not attempt to study what motivations are associated with specific behaviors such as gaming.
In this paper, we will investigate why some students choose to game while other students choose not to game, using evidence from three studies across two different types of interactive learning environments, Cognitive Tutors and ASSISTments. By investigating this question across multiple systems with independent methods of detecting gaming, we can have some confidence that our findings will generalize beyond just a single type of learning environment.
This paper is organized as follows: first, we will discuss a set of hypotheses for why some students game, drawn from the literature on gaming, motivation, and related classroom behaviors, and from discussions and brainstorming with school teachers and other researchers. Next, we will present some evidence on teachers’ perspectives about why students game. In the following section, we will discuss three questionnaire studies where we gave a variety of items to students (relevant to the hypotheses) and correlated their responses to their frequency of gaming. Next, we consider how each of the hypotheses is confirmed or disconfirmed by the data from the students’ responses, and how the results from our studies correspond to the teachers’ predictions. Finally, we discuss the psychological and design implications of our results.
1.1. Systems Studied
Within this paper, we will consider data on the student characteristics associated with gaming the system from two different interactive learning environments: Cognitive Tutors (Anderson et al, 1995) (shown in Figure 1) and ASSISTments (Razzaq et al, 2005) (shown in Figure 2). Both environments can broadly be characterized as intelligent tutoring systems or coached practice, but differ in many ways at a finer level of detail. Within each type of learning environment, each student works individually with the computer software to complete mathematics problems.Problems in each Cognitive Tutor curriculum are designed to map to specific parts of a single state-mandated mathematics curriculum, and are organized into lessons by themes in that curriculum. Problems in the ASSISTments system are designed to map to the problems found in a state mathematics exam, the Massachusetts Comprehensive Assessment System (Razzaq et al, 2005), and are explicitly modeled on problems from exams in previous years. Problems in the ASSISTments system are also grouped mathematical topics.
Figure 1.A student using a Cognitive Tutor lesson on scatterplots has made an error associated with a misconception, so they receive a “buggy message” (top window). The student’s answer is labeled in red, because it is incorrect (bottom window).
Cognitive Tutors breaks down each mathematics problem into the steps of the process used to solve the problem, making the student’s thinking visible, whereas ASSISTments provide a complete problem, and break down the problem into the steps of the problem-solving process only if the student makes errors or has difficulties.
Figure 2.Showing an ASSISTment where a student made an error on the question, then completed the 1st scaffolding question and is in the middle of trying to answer the 2nd question related to perimeter.
In both environments, as a student works through a problem, a running cognitive model assesses whether the student’s answers map to correct understanding or to a known misconception (cf. Anderson, Corbett, Koedinger, & Pelletier, 1995). Both environments offer students instant feedback based on their errors, at each step of the problem-solving process. However, the nature of the feedback differs substantially between the two systems. Within the Cognitive Tutors, if the student’s answer is incorrect, the answer turns red; if the student’s answers are indicative of a known misconception, the student is given a “buggy message” with feedback tailored to the student’s current mistake (see Figure 1).Within the ASSISTments system, a wrong answer is responded to both with “buggy messages” and with supplementary questions which break down the problem-solving exercise into the component skills needed to get the overall problem right.
Beyond instant feedback, both systems offer multi-level on-demand hints to students. When a student requests a hint (by clicking a button), the software first gives a conceptual hint. The student can then request further hints, which become more and more specific until the student is given the answer (see Figure 3, for an example in a Cognitive Tutor). The hints are context-sensitive and tailored to the exact problem step the student is working on.
Figure 3.A student reading the last message of a multi-level hint in a Cognitive Tutor lesson on scatterplots: The student labels the graph’s axes and plots points in the left window; the tutor’s estimates of the student’s skills are shown in the right window; the hint window (superimposed on the left window) enables the tutor to give the student feedback.
Within the Cognitive Tutors, as the student works through the problems in a specific curricular area, the system uses Bayesian Knowledge Tracing (Corbett & Anderson, 1995) to determine how well the student is learning the component skills in the cognitive model, calculating the probability that the student knows each skill based on that student’s history of responses within the tutor. Using these estimates of student knowledge, the tutoring system gives each student problems which are relevant to the skills which he or she is having difficulty with.The ASSISTments system does not use Bayesian Knowledge Tracing in the same fashion, since it is based upon the entire Massachusetts Comprehensive Assessment System (for mathematics) rather than sub-sections of a single curriculum.
Both Cognitive Tutors and ASSISTments system are used in combination with regular classroom instruction, including group work; students use the Cognitive Tutor or ASSISTments system once or twice a week as part of a regular mathematics course, and have classroom lecture or group work on the other days. Both systems have been validated to result in positive student learning outcomes. Cognitive Tutor curricula have been validated to be highly effective at helping students learn mathematics, leading not only to better scores on the Math SAT standardized test than traditional curricula (Koedinger, Anderson, Hadley, & Mark, 1997), but also to a higher percentage of students choosing to take upper-level mathematics courses (Carnegie Learning, 2002). In recent years, Cognitive Tutor mathematics curricula have come into use in an increasing percentage of U.S. high schools – about 6% of U.S. high schools as of the 2005-2006 school year. The ASSISTments system is considerably newer, and has thus been less thoroughly studied, but its use has also been shown to result in significantly better learning (Razzaq et al, 2005, Razzaq & Heffernan, 2006). In 2006-07, 3,000 students are using the system as part of their normal math class.
2. HYPOTHESES
In order to determine what factors might lead students to game the system, we conducted a thorough review of the relevant literature on gaming-like behavior in both traditional classrooms and in online settings. In general, the existing literature had considerably more hypothesis than concrete data about why students engage in gaming-like behavior, though notable exceptions to this trend do exist (e.g. Arbreton, 1998). We also reviewed literature on the range of attitudes, beliefs, and motivations students show towards technology and school, and the range of attitudes and beliefs that cause people to behave in resistant and subversive ways outside of educational settings, as will be discussed in the following sections. Finally, we engaged in both structured and non-structured brainstorming (Kelley and Littman, 2001) with school teachers and colleagues in a variety of scientific areas, including educational technology researchers, educational psychologists, behavior modification researchers, and interaction designers.
It is important to note that not all of this literature review and brainstorming took place at once; instead, it took place across a span of three years. Some hypothesis formation took place before any of the three studies we conducted, but hypothesis formation continued in between each of the studies we report in this paper. Through this process of discussion, brainstorming, and literature review, we have come to thirteen hypotheses for why students game. It is important to note that this is by no means an exhaustive list of the reasons students may elect to game the system. A virtually limitless set of hypotheses may be generated; we claim only that the set that we present is reasonably well-justified by either prior research or practitioners’ and researchers’ past informal experience.
For presentation purposes we categorize our thirteen hypotheses into groups. These categories are meant only to group hypotheses together that have some relation to one another, in order to facilitate understanding. Our hypothesis groupings are not intended to make any broader theoretical claims. Within this paper, the important unit of analysis is the hypotheses themselves and their corresponding results, not the categories. That said: the five categories of hypotheses are: hypotheses relating gaming to students’ goals (H1, H2), hypotheses relating gaming to students’ attitudes (H3-H5), hypotheses relating gaming to students’ beliefs (H6-H9), hypotheses relating gaming to students’ broader responses to many educational situations (H10, H11), and hypotheses relating gaming to students’ emotions (H12, H13).
2.1. Hypothesis H1: Performance Goals.
Our first hypothesis, and the hypothesis for why students game most commonly proposed prior to the research presented here (cf. Baker et al, 2004; Martinez-Miron et al., 2004), is that students game the system because they have performance goals rather than learning goals (the distinction between performance goals and learning goals is discussed in detail in Dweck, 2000). In this case, a student using educational software wants to “succeed” in the environment, by completing all of the problems, more than he or she wants to learn the material contained in those problems. Therefore, to complete more problems, he or she engages in gaming behaviors.
Some evidence on student behavior in traditional classrooms supports this hypothesis. Specifically, some students in traditional classrooms engage in a behavior termed “executive help-seeking”. In this behavior, a student starting work on a new problem requests help from their teacher or a teacher’s aide immediately, before attempting to solve the problem on their own. Arbreton (1998) found that executive help-seeking was significantly correlated with performance goals, as measured through questionnaire items.
2.2. Hypothesis H2: Desire For More Control
Our second hypothesis is that students game the system out of a desire for more control over their learning experience. In this case, a student using a fairly constrained learning environment, such as Cognitive Tutors or ASSISTments, feels that they do not have control over their learning experience, and games the system in order to regain some measure of control over their learning experience – for example, in order to avoid problems the student does not wish to solve.
This hypothesis was formed based on informal comments made by students in previous studies, both amongst themselves, and to the authors of this paper. In addition, it is potentially congruent with prior studies that have found that giving students greater control within constrained learning environments, even over relatively minor aspects of their learning experience such as which spaceship represents them in a space-fantasy mathematics learning environment (Cordova and Lepper, 1993) or which story they get to read in a reading tutor (personal communication, Joseph Beck), can improve student motivation and even increase student learning. It is possible that one of the reasons choice improves learning is because it reduces gaming -- though some evidence suggests that choice features can also enable new ways of gaming the system (cf. Mostow et al, 2002).
2.3. Hypothesis H3: Disliking Mathematics