18
ECER2002
Constructing and evaluating an ICT-instrument to assess
students’ quality of working in groups
Paper presented at the European Conference on Educational Research, University of Lisbon, 11-14 September 2002
Alexander Minnaert, Monique Boekaerts, & Cornelis de Brabander
Centre for the Study of Education and Instruction
Leiden University
The Netherlands
MAILING ADDRESS. - Dr. Alexander Minnaert, Leiden University, Faculty of Social and Behavioural Sciences, Centre for the Study of Education and Instruction, Wassenaarseweg 52, P.O. Box 9555, 2300 RB Leiden, The Netherlands.
Electronic mail may be send to
Phone: +31 71 5273384 Fax: +31 71 5273398
Monitoring co-operative learning processes
The movement of social-constructivism that has grown in education over the past 25 years has led to the development of new arrangements of learning settings that stress self-responsible and co-operative learning (Slavin, 1996). Self-regulation was identified as the overall key to optimal learning. Boekaerts (1997) found, however, that there is no short cut to the acquisition of self-regulatory skills: it requires the commitment of both students and teachers to lengthy periods of practice with their new complementary roles. In order to become self-regulated learners in a particular subject-matter area, care should be taken that students develop a wide range of skills, including metacognitive skills as well as various control processes that experts in that field also possess. An important problem, however, in the new arrangements of learning settings is the shortage of means for monitoring the progress of learning processes in these project groups.
Students bring their own goals and needs into the classroom and these goals may be (in)compatible with teachers’ targets for learning and teaching (Vermunt & Verloop, 1999), particularly with the development of the new learner role. Teachers want their students to get good grades and they expect them to fulfil their learner role. However, in recent years this role has undergone many changes and teachers hold a new set of goals for their students, including “work autonomously”, “provide social support to peers” and “take responsibility for learning” (Deci & Ryan, 1994). Students who are well adjusted to school life may react in a totally different way to the teacher’s motivational practices than their less-adjusted peers, mainly because their values and psychological needs are in harmony with the teachers’ expectations (Boekaerts & Niemivirta, 2000). By contrast, students who discover that their psychological needs are not fulfilled at school will experience a feeling of disconnection and discongruence most of the time.
Teachers can organise classrooms in such a way that students perceive them as in conflict or in harmony with their psychological needs. Research evidence indicates that some settings facilitate the pursuit of personally valued goals, thus catering for students’ psychological needs, whereas other settings frustrate students’ psychological needs. Ames (1992) distinguished between three types of learning settings: competitive, individualised, and co-operative learning settings, and reported typical motivational practices in each of these settings. Unfortunately, there is scarce research evidence on the degree to which students perceive competitive and individualised learning settings as in balance with students’ psychological needs. There is, however, a vast body of research documenting the effect that co-operative learning settings have on the acquisition of the new learner role. Slavin (1996) as well as Maehr and Midgley (1996) illustrated that teachers, who provide plenty of opportunity for students to pursue social goals and fulfil their need for social relatedness, create a powerful environment for adopting the new learner role. We agree with Ames that the three learning settings differ in the motivational practices they elicit, and may therefore have a differential effect on the students’ goal orientation. However, these settings differ on other grounds as well. The point being made is that some learning settings facilitate the pursuit of personally valued goals, thus catering for the students’ basic psychological needs whereas other settings frustrate students’ psychological needs, particularly their need for autonomy and social relatedness.
Basic psychological needs
In self-determination theory (Deci & Ryan, 1985), the concept of intention is central for understanding the regulation of behaviour. When a behaviour is self-determined, the regulatory process is choice and the person perceives that the locus of causality is internal to his or her self. But when behaviour is controlled, the regulatory process is compliance (or even defiance in some cases) and the perceived locus of causality is external to the self. Self-determination theory has specified a set of three innate psychological needs that are relevant to intrinsic or extrinsic motivation: the need for competence, relatedness, and autonomy. Competence involves understanding how to attain various external and internal outcomes and being efficacious in performing the requisite actions; relatedness involves developing secure and satisfying connections with others in one's social milieu; and autonomy refers to being self-initiating and self-regulating one's own actions (Deci, Vallerand, Pelletier, & Ryan, 1991). The concept of needs allows one to specify the contextual conditions that will facilitate motivation, performance and development. Social contexts that provide people the opportunity to satisfy their basic psychological needs affect people's intrinsic motivation and autonomous self-regulation, and in turn the quality of their performance. Besides, opportunities to satisfy the need for autonomy are necessary for people to be self-determined rather than controlled. Overview studies support for these assertions (see Grolnick & Ryan, 1989; Deci et al., 1991; Connell & Wellborn, 1991).
Based on work with students in vocational school and in higher education, it was assumed that for any project group to be able to work productively, a balanced fulfilment of basic psychological needs of its group members is essential (Boekaerts & Minnaert, 2003). In contrast to Connell and Wellborn (1991), we presume that the context variables jointly contribute to the fulfilment of one's psychological needs and disregard the unique and mutual exclusive effect of structure, autonomy support, and relatedness on respectively competence, autonomy, and relatedness (see figure 1). Furthermore, we presumed that students' basic psychological needs might change during the various stages of the learning process. It is indeed easy to imagine that a specific classroom setting may be optimal for some students, meaning that it provides them with a focus for personal identity and for learning and achievement. The same context may be "suboptimal" for others (see, Ames & Archer, 1988; Boekaerts & Niemivirta, 2000). However, it is also true that students who are still in the initial stages of skill acquisition differ from more advanced students in their perception of autonomy, competence, and social relatedness. This implies that student basic needs for competence, autonomy and social relatedness may be met in some contexts most of the time, but not all the time. When students are working in a learning environment that they perceive as "optimal" they may be willing to invest resources to learn to self-regulate their learning, whereas they may decline such an offer when they perceive the context as "suboptimal" (e.g., not enough structure, no autonomy support). In such a context, they may feel lack of purpose (no goal-oriented behaviour), low relatedness and no inclination to engage in learning tasks set by the teacher or by the group.
Context Self Outcome
Structure skills and
abilities
Autonomy
support
adjustment
Involvement
Figure 1.
Motivational analysis of self-determined group work (see Connell & Wellborn, 1991, p.51).
In the context of co-operative projects, two aspects were distinguished in the need for (social) relatedness: namely, the need for co-operation and the need for responsibility for learning. The former refers to the social dimension of the need for relatedness, while the latter refers to the accountability to accomplish the goals being set by the group. Besides, interest was added, not intended to reflect a separate psychological need, but as a general indication of engagement (versus disaffection) in the co-operative project. In line with Krapp (2002), interest is used as a relational construct, that conceptualises the person-object conception of interest, investigated at the level of current events, states, and processes.
It is hypothesised that interest operates as a 'barometer' for the balanced fulfilment of students' basic psychological needs. Hence, autonomy, competence, and relatedness are expected to play a central role in the explanation of students' interest. A number of authors (see Schiefele, 1991, 2001) have demonstrated the relationship between interest and various indicators of academic learning (e.g. achievement, text learning, depth of learning) and have emphasised the relationship between interest and self-determination theory (see Krapp, 2002). Interest is postulated to act as a mediating variable between the psychological needs and cognitive and affective learning outcomes (see figure 1).
Rationale for an alternative form of assessment
In the course of our field work, we gradually came to realise that traditional forms of assessment do not provide a powerful framework for helping researchers to meet the needs of both students and teachers. Along with several other educational researchers (Dochy, Segers & Sluijsmans, 1999; O'Neil & Abedi, 1996), we believe that alternative forms of assessment are "a conditio sine qua non" for teaching students self-regulated learning. Talking about alternative forms of assessment has become quite common. Birenbaum (1996) pointed out that one of the most generic terms used in present-day educational psychology is "alternative assessment". It is an umbrella term for various forms of judgement.
Over the past two decades, an increasing emphasis on assessment results and on the nature of most widely used forms of student assessment is noticeable. A shift has taken place from a culture of testing to a culture of assessment. The latter puts a stronger emphasis on the integration of assessment and instruction, on assessing the process rather than just the product, and on the evaluation of students’ growth or progress relative to students’ personal base line level (Minnaert, 2002a, 2002b). The relative shift from traditional to alternative approaches in assessment is not motivated primarily by psychometric considerations (Linn & Baker, 1996). Both types of assessment need to be psychometrically sound in relation to the assessment purposes and to the uses that are to be made of the results. Many tests have been criticised for failing to provide the information we need about students and their competencies to meet specific curricular objectives. Critics contend that these tests often assess only a narrow range of the curriculum and often focus exclusively on aptitudes (Rudner & Boston, 1994). Further, these tests are lied up with general diagnostic purposes for orientation, selection, and classification, bypassing process-oriented information on how to improve the performance at individual, class, or school level. Hence, alternative assessment is developed especially for on-line assessment purposes, namely to assess students’ higher order processes in a ‘real-world’ context in which they are actually used (Linn, Baker & Dunbar, 1991; Shepard, 1991).
On-line assessment methods: advantages and shortcomings
To prepare students for democratic collaboration, the individual differences in the perception of constraints and affordances in the learning environment necessitate the construction of an instrument that can register on-line to what extent students’ psychological needs are fulfilled or frustrated under various conditions. Based on our work with students in vocational school and in higher education (Boekaerts & Minnaert, 2003), we assumed that the level of students’ basic psychological needs affects students’ perceptions of ongoing and upcoming learning processes but also their perceptions of ongoing interaction processes with peers. Large individual differences in the perception of constraints and affordances in the learning environment are, however, expected.
In order to capture the fulfilment of these basic needs, researchers must register individuals' experiential states while activities are still in progress. This calls for a recording method that can capture induced motivational states quasi on-line. When recording motivational states close in time to the event(s) that elicited them, the chance is reduced that students have forgotten their specific thoughts and feelings about the learning situation. When recording motivational states close in time to the event(s) that elicited them, the chance is reduced that students have forgotten their specific thoughts and feelings about the learning situation. Indeed, several researchers (e.g. Schwarz, 1990) reported that participants in self-recording studies about moods seemed to have completely forgotten their earlier moods, when interval-contingent recordings were used.
Larson and Csikszentmihalyi (1983) did some pioneering work with on-line measures of motivation, using the experience sampling method. This procedure consists of asking individuals to provide systematic self-reports at random occasions during the waking hours of a normal week. They asked gifted secondary school students in each class to carry an electronic pager and answer a set of questions about their cognitions, moods and activities whenever they were beeped. These students were disrupted by the beeper during class, exams, swimming, parties, or even napping. The researchers were interested in the whole spectrum of adolescent activities, and wanted to compare and contrast students' affects about schoolwork with their affects about other activities. Therefore, this procedure can be described as a signal-contingent method.
Krapp and Lewalter (2001) also used a signal-contingent method of sampling. He studied the impact of student's experiential states (i.e., the extent to which the three basic needs were fulfilled) under two different learning conditions: traditional schooling practices followed by training at the workplace, and the reverse condition. Th authors reported that, at the workplace, students reported high feelings of social relatedness, high feelings of competence and high feelings of autonomy. The experience sampling method allowed Krapp and Lewalter to conclude that students in vocational education, who begin their training at the workplace and continue later in school, have a fundamentally different learning experience than students who follow the reversed sequence.
In the area of motivation, an event-contingent sampling method is developed by Boekaerts (2002): namely the 'On-Line Motivation Questionnaire'. It was used to gain information on the way students interact with the learning activity. More specifically, students were asked to describe their cognitions and feelings, as well as their learning intention before initiating the activity, and their cognitions, feelings and attribution processes after doing the activity. Event-contingent methods of sampling have the advantage that they do not interfere with ongoing information processing, yet provide process centred information.