A Reflective Learning Framework for Programming

A Reflective Learning Framework for Programming

Lichao Li

A Thesis Submitted for the

Degree of Master of Science (Research)

School of Information Technologies

The University of Sydney

April, 2007

Abstract

Learner reflection has been found to be important and valuable in learning, especially in cognitively demanding tasks, such as learning programming. This thesis presents an online learning system, Reflect, which supports learning to solve small problems in this demanding area and emphasizes on encouraging learner reflection in the learning process.

Reflect is essentially a learning system that supports problem solving based on synthesis as well as learning from examples. It incorporates a knowledge layer to support several novel reflective elements to promote different types of learner reflection. These elements are embedded in each major part of the system. The knowledge layer enables Reflect to build learner models, which were based on the main learning objectives defined in an ontology by teachers. Student interaction with Reflect can be used to provide the evidence to build a model of the learner's knowledge of each of the learning goals. This information, when visualized provides students with informative feedback on their learning progress. The knowledge layer also enables creation of reflective elements as part of the Reflect task specification, process of submitting solutions and provides linkage between the reflective elements and automated grading of students’ solutions.

We conducted both quantitative and qualitative analysis to evaluate the system and our approach to promote learner reflection in the Reflect system. An investigation of the general system usage reveals that, within the context of an authentic learning environment, students used the system as expected and sixty percent of all students found it helpful for preparing for exams. A study of students’ attitudes to each of the key reflective elements of Reflect was also conducted. The resultsindicate that most students had favourable opinions of all these elements.Notably, both the higher achieving group and the lower achieving group saw greatest value on automatic testing aspects. However, the lower achieving group placed greater value on Reflect’s support for learning from examples.In addition, we conducted a detailed qualitative study of the ways students used several of the reflective elements of Reflect, based on a selected group of students with diverse level of achievement. This points to differential benefits to both in student attitudes and in behaviour for higher versus lower achieving groups.

1

Contents

Chapter 1 Introduction

1.1 Motivation

1.2 Context of the Research

1.3 A Definition of Terms

1.4 Thesis Aims and Research Contribution

1.5 Thesis Content

Chapter 2 Background

2.1 Student Self-assessment

2.2 User Modeling in Intelligent Tutoring Systems

2.3 Learner Reflection, Scrutable Learner Models and Promoting Learner Reflection in ITS

2.4 Diagnosis of Students’ Solutions in ITS for Programming

2.5 Learning from Examples

Chapter 3 System Overview: Learner Experience

3.1 Problem Statement and Learning Objectives

3.2 The Writing Phase

3.3 The Read Phase

3.4 Feedback and Student Profile

Chapter 4 Teacher’s Use of Reflect

4.1 Aggregate Information

4.2 Individual Information

Chapter 5 Author’s View of Reflect

5.1 Defining Learning Objectives

5.2 Creating New Tasks

5.3 Stage One: Create Task Statement and Associate Learning Concepts

5.4 Stage Two: Edit marking criteria

5.5 Stage Three: Create example solutions

5.6 Stage Four: Setup automatic student solution assessment

Chapter 6 Evaluation

6.1 Overview of the Evaluations

6.2 General System Usage

6.3 Survey

6.4 Student self-assessment behaviour analysis

6.5 Summary

Chapter 7 Conclusions

7.1 Further Directions

Bibliography

Appendix A

Appendix B......

List of Figures

Figure 1 Skill meter of the LISP tutor

Figure 2 Course progress visualization of ELM-ART II

Figure 3 Visualization of the learner model of the SQL-tutor

Figure 4 Interface of e-KERMIT displaying the student model (The screen is divided into three sections, the top section is the problem statement, the middle section is the working space where students can design entity relationship data models and the bottom section provides feedback to students including the four skillometers)

Figure 5 A screenshot of ViSMod showing a fragment of a Bayesian student model

Figure 6 A textual display of the learner model of Mr. Collins and its negotiation process

Figure 7 Student Homepage

Figure 8 Learning Objectives and Their Meaning in the Course Glossary

Figure 9 Problem Statement

Figure 10 Write First

Figure 11 Read First

Figure 12 Self-assessment

Figure 13 Example Solution

Figure 14. Comparison of Student’s (Yours) and Teacher’s (Ours) Assessments of an Example and a Measure of the Discrepancy between Them

Figure 15 Previous Solutions

Figure 16 SIV Display of the Student Model

Figure 17. More Useful Statistics

Figure 18 The HTML Version of the Learner Model Display

Figure 19 Students’ View of Reflect

Figure 20 Aggregate Class Information

Figure 21 Summary of a Student’s Work

Figure 22 A Student’s Submitted Solution

Figure 23 A Student’s Assessed Example Solution

Figure 24 The Author's Homepage

Figure 25 Edit Learning Objectives

Figure 26 New Task Creation

Figure 27 Stage One of Task Creation

Figure 28 The Problem Statement in a Popup Window

Figure 29 Edit Marking Criteria

Figure 30 Edit Example Solutions

Figure 31 Create New Example

Figure 32 Edit Additional Marking Criteria

Figure 33 Assess Example Solution

Figure 34 Upload Testing Material

Figure 35 View Entire Task

Figure 36 Task Creation Process in Reflect

Figure 37 Students’ Submissions from Week 2 to Exam

Figure 38 Students’ Submissions and Exam Marks

Figure 39 The Number of Students who Explored Their User Profiles

Figure 40 The Number of Students that Agree/Disagree with Their User Profiles

Figure 41 Students’ Opinions on the Usefulness of Self-Rating

Figure 42 Students’ Opinions of Reflect’s Self-assessment Approach

Figure 43 Students’ Opinions of the Value of Example Solutions

Figure 44 Students’ Opinions of Reflect’s Approach to Automatic Evaluation of Programming Code

Figure 45 Comparison of Stronger and Weaker Students’ Answers to Questions 3, 4 and 6

Figure 46 Stronger Students’ Answers to Questions 3, 4 and 6

Figure 47 Weaker Students’ Answers to Questions 3, 4 and 6

Table 1 Number of Tasks Published per Week

Table 2 Students’ Answer to Survey Question: Have you explored your user profile?

Table 3 Students’ Answer to Survey Question: Have you explored your user profile? (2)

Table 4 Students’ Answers to Survey Question: Do you think the user profile matches your own beliefs of your knowledge?

Table 5 Students’ Answers to Survey Question: Do you think the user profile matches your own beliefs of your knowledge? (2)

Table 6 Students’ Answers to Survey Question: Do you find self-rating of knowledge helps you realise the relevant things to focus on?

Table 7 Students’ Answers to Survey Question: Do you find self-rating of knowledge helps you realise the relevant things to focus on? (2)

Table 8 Students’ Answers to Survey Question: Do you find self-assessment of solutions helps you think about the quality of your answer?

Table 9 Students’ Answers to Survey Question: Do you find self-assessment of solutions helps you think about the quality of your answer? (2)

Table 10 Students’ Answers to Survey Question: Did you find that the sample solutions help you learn?

Table 11 Students’ Answers to Survey Question: Did you find that the sample solutions help you learn? (2)

Table 12 Students’ Answers to Survey Question: Did you find automated grading helpful?

Table 13 Students’ Answers to Survey Question: Did you find automated grading helpful?(2)

Table 14 Students’ Answers to Question 6 and Their Average Exam Marks

Table 18 Second and Third Marking Criteria of the Shortstrings Task

Table 19 Fourth, Fifth, Sixth and Eighth Marking Criteria of the Shortstrings Task

Table 20 Test Cases and the Related Marking Criteria

Table 21 Conscientious Students’ Submissions to the Shortstrings Task

1

Introduction

Chapter 1Introduction

This thesis explores methods to improve support for learner reflection in an online learning system that supports learning to solve small problems in a demanding area such as computer programming.

Learning is a process of gaining understanding through the acquisition of knowledge and skills through study and experience. It is a very complex cognitive process. There are many factors that contribute to a successful learning experience. It is common knowledge that, in both formal and informal education, there are always learners who learn more efficiently than others. This is, of course, affected by the learners’ prior knowledge in the subject. However, a good learning strategy also contributes to efficient learning. Knowing how to learn is often a valuable skill that differentiates expert learners from novice learners (Ertmer & Newby, 1996).

Here is an example to illustrate the difference between a more effective learner and a less effective learner. It illustrates some of the huge body of work on what differentiates more effective learners from others, for example (Boud, Keogh, & Walker, 1985; Ertmer & Newby, 1996; Weinstein & Stone, 1993). Two students are preparing for an end of semester history examination. One student starts revision by reading the text book from chapter one to the end and doing every exercise in the book. A few days before the exam, the student finds himself/herself running out of time. Then he/she starts to read a lot faster, skipping random content and exercises. The day before the exam, the student finds himself/herself barely remembering anything he/she has read. The other student, however, starts revision by thinking through the following questions:

What is my goal;

What do I already know about the subject;

How much time will it take me to study and;

What strategies work best for me to study for the exam (Will it be more efficient to read more example solutions or solve more problems myself)?

This student refers to these questions repeatedly as they began each study session and plan their next stage of study. They will also make a strategic, possible effective, revision plan at the beginning and they will always be aware of the learning goals, what they have learned, how much time is left and the current learning strategy. Although both students work hard for the exam and both are dedicated to their study, the second student has greater control over the process, is more strategic, with greater potential to find the learning experience much more efficient and enjoyable and, seems more likely to achieve a better result in the exam.

In summary, more efficient learners tend to take conscious control of their learning, planning and selecting strategies, monitor the progress of learning, correcting errors, analyzing effectiveness of learning strategies and changing learning strategies and behaviors when needed. On the other hand, less efficient learners often fail to stop to evaluate their comprehension of the material. They generally do not examine the quality of their work or stop to make revisions as they go along. Furthermore, better learners are more aware of when they need to check for errors, why they fail to comprehend and how they need to redirect their efforts (Ertmer & Newby, 1996).

The type of activities that expert learners do is often described as metacognitive. Metacognition (Flavell, 1987) consists of two basic simultaneous processes: monitoring the learning progress and making changes and adapting learning strategies if one is not doing well. It is about self-reflection, self-responsibility, initiative, goal setting and time management. The capacity to use such metacognitive skills is developed to different stages in different people. Not all learners apply these activities in their learning. In fact, many learners are not aware of them. The benefit of developing metacognitive activities is that, as learners become more skilled at applying metacognitive strategies in their learning, they gain confidence and become more confident learners, leading to pursuing their own intellectual needs without persuasion of a teacher. The role of the teacher is therefore to acknowledge, develop and enhance the metacognitive capabilities of all learners.

There is a wide range of metacognitive activities. In this thesis, we concentrate on helping learners to become aware of the value of reflection and develop their ability to self-reflect as a means to enhance their learning of a cognitively demanding synthesis skill, like programming. Self-reflection is an important element in metacognition. It allows a learner to be aware of his knowledge state, so he/she can make adaptations to their learning strategy. We believe that if learners are promoted using the right method to self-reflect in learning, they will become more self-reflective and thereby, enhancing their learning efficiency.

1.1Motivation

In both the literature and our experience in teaching programming, it is clear that some students are unable to comprehend topics that are considered easy by other students, even though both groups try hard and dedicate the appropriate amount of time to study these topics. This suggests that the low achieving may have used unproductive learning strategies. It seems promising to explore the possibility that learning effectiveness can be enhanced by using more efficient learning strategies. Developing the ability to self-reflect is a key factor in becoming an efficient learner.

There is a large body of research evidence suggesting that learning effectiveness can be enhanced when students pay attention to their own learning by reflecting on the state of their knowledge and the learning process (Schön 1983; Boud, Keogh et al. 1985; Schön 1987; Pirolli and Recker 1993). As described by (Boud, Keogh, & Walker, 1985), Learner reflection is a generic term for those intellectual and affective activities in which individuals engage to explore their experiences in order to lead to a new understanding and appreciation. This ability to self-reflect is developed to different degrees in different people. Some learners are able to apply their relevant previous experiences at the right moment, while others cannot. However, if a teacher can actively promote learners to correctly apply their existing knowledge in learning at the correct moment, this may be beneficial for a student who lacks the ability do so themselves. Moreover, the student will have the opportunity to learn how to self-reflect, thereby becoming more capable and proficient in learning.

Therefore, the main motivation for this research project is to help student’s enhance their learning efficiency by using a number of methods to promote learner reflection in an online learning system.

1.2Context of the Research

The research described in this thesis is conducted in the context of programming education. This is mainly because learner reflection is particularly valuable for cognitively demanding subjects. Learning programming, even at the most introductory level is cognitively challenging (Bonar & Soloway, 1983). Moreover, recent research has found that a majority of students in introductory programming courses do not learn how to program at expected skill level (Lister, et al., 2004; McCracken, et al., 2001).

For this research project, a system, called Reflect has been developed as a prototype system designed to teach students programming techniques and to promote learner reflection in the teaching/learning process. There are several earlier implementations of this system (Kay, Li, & Fekete, 2007; Li & Kay, 2005b); these versions were more advanced forms of a student self-assessment tool that enabled students to self-assess their knowledge, monitor their learning progress and judge if they have achieved required learning outcomes. These were self-assessing and self-reflective in nature.

Earlier versions of Reflect were called Assess which began as a tool for supporting reading of example answers to design tasks. By the time of this thesis, the changes in functionality and software engineering pragmatics meant that the new Reflect system needed a new design and was a re-implementation of the earlier systems.While still retaining the earlier systems’ goals andsome of the earlier ideas and features, it introduces many new functionalities and a set of new user interfaces. The most important improvement in the new system is the introduction of a consistent focus on applying reflection throughout use of the system. The system design centered on the notion of learner reflection. It employs various techniques such as user/learner modeling, student self-evaluation and computer aided assessment. These will be discussed in more detail in later chapters.

Reflect has been used in a second year C programming subject as a learning aid. We gathered valuable feedback from students and conducted several qualitative studies to evaluate our system.

1.3A Definition of Terms

Descriptions of some terms commonly used in this thesis that may be unfamiliar to the reader are provided here.

User Modeling

A user model is a representation of a set of beliefs about the user, particularly their characteristics, knowledge, beliefs and/or preferences. Early user modelling was often performed by the application system, but not by a distinct logical entity(Wahlster & Kobsa, 1989). There was often no clear distinction between system components that serveduser modelling purposes and components that performed other tasks. Information about a user was extracted from normal interaction between the user and the system. This information was used later to adapt the system to the user’s knowledge, characteristics, beliefs and/or preferences. For example, a program can ask whether a user knows about X, and at a later stage, it decides that if the user knows about X, he/she would probably want to learn about Y. Such implicit user models are not visible for inspection at any level.

Explicit user models, on the other hand, can either be a standalone application dedicated to the task of user modeling such as a user modeling shell system (Kobsa, 2000), or they can be distinct modules within applications. There are two main types of explicit user models, cognitive and pragmatic(Kay, 1999). Cognitive user modeling attempts to match the way that people actually think and know. It is of much interest to psychologist and educators. An important early example is the knowledge tracing module (Corbett & Anderson, 1995) in the LISP Tutor (Anderson & Reiser, 1985) that is based on the Adaptive Control of Thought (ACT) theory of knowledge acquisition (Anderson, 1983). This type of user modeling is a well-defined artificial construction of the psychologist-programmer. However, cognitive user modeling is computationally expensive and the model is difficult to construct. Therefore, in many cases, a more pragmatic user modeling approach is simpler and sufficient. The user model in this thesis is a pragmatic one. An explicit user model allows a system to store relevant information about a user and use this accumulated information to adapt to the user’s needs. Without it, decisions about adaptation of the system can be made only on the basis of observed learner behaviour snapshots. Another major advantage of an explicit user model over an implicit one is that it can be scrutinized, thereby offering potential educational value. This will be discussed in Chapter 2.