The organisation of assessment: the impact on study behavior in higher education

Kim Waeytens, Veerle Hulpiau & Jan Elen, University Educational Support Office, University of Leuven, Belgium

Paper presented at the European Conference on Educational Research, University of Lisbon, 11-14 September 2002

Assessment and its influence on study behavior nowadays attracts a lot of research attention. This attention focuses on variables situated at the level of specific instructional processes. Research that pertains to variables situated at the organisational level remains rare. This study aims at investigating whether the summative or formative character of assessment and the assessment period (i.e. immediately after the courses or not) affects study behavior. Freshmen indicated for each of their courses the frequency by which they attended lectures, studied course materials and performed other supplementary study activities. The expected timepattern regarding the moment of assessment got confirmed. Lectures are more attended for courses for which the assessment comes closer in time. More time is devoted to these courses both in studying and in performing supplementary study activities. This observation is made irrespective of the formative or summative character of these assessments. The summative character of an assessment influences the attendance in the lectures.

Introduction

A lot of differences exist between European countries regarding the organization of the academic year and the organization of exams in higher education (http://www.eurydice.org/). In some countries exams are organized at the end of a term, in others at the end of the academic year or at the end of the study. In Flanders, a rigid year system is still the standard. The academic year starts at the end of September and is divided into two terms (the first term ending in January). Each program consists of relative fixed curricula, which are spread out over several academic years. Success in the exams of all the courses in one year is a condition to progress to the following year of study. A student can only succeed in a given year provided all the exams are taken for the courses that are included as part of the study-program for that particular year (for more information see http://www.eurydice.org/).

As there are differences between countries regarding the exam system, there are also differences within countries in the way exams are organized. So is the case in Flanders.

In some universities exams of all courses are organized at the end of the academic year (i.e. a year system). At the end of the academic year (mostly in June) students are evaluated about the subject matter of all the courses they followed during that academic year. This strictly organized system of exams is made more flexible in universities were some exams are already organized in April (during or after the Easter holidays). In addition to these summative exams some institutions also organize formative tests for first year students at the end of the first term. The goal of these tests is to become acquainted with exams at the university. The primary goal is remedial and the results of these tests are not taken into account for the final result of the courses. In other words these tests have a formative character. In other institutions a term-based assessment system got implemented. At the end of each term students have summative exams for each of the courses they followed during that term.

The choice for a particular system is mostly based on practical organizational arguments. A strict year system has come under great pressure due to the movement towards European harmonization. It could be expected that didactical arguments also have some influence. The organization of formative tests and summative exams has a great influence on the study behavior of students. An important question is how the organization of exams could induce a more regular study pattern by students.

Nowadays assessment and its influence on study behavior attract a lot of research attention. These studies focus on variables with respect to the level of specific instructional processes, such as the influence on learning activities of the content and, more particularly, of the kind of questions (see Crooks, 1988; Entwistle, 1996). Research results can be summarized by the WYTIWYG principle: “What you test is what you get”. Students study differently for exams if they expect that they will be asked to reproduce the subject matter than for an exam in which they expect to have to solve a new problem. Also about the effect of feedback on the learning process a lot of research is performed (e.g. Bangert-Drowns, Kulik, Kulik, & Morgan, 1991).

Less research exists on the role of variables at the organizational level and their impact on study behavior. Whether the period of assessment and its formative or summative character influences study behavior remains unclear. Some authors argue that by organizing various assessments throughout the year a more regular pattern of study is obtained and the amount of self-study increases (van der Drift & Vos, 1987). The programmation of exams and intermediate tests are considered to be responsible for the studypattern of students. Students have a general tendency towards procrastinating the preparation of exams until the very last moment (van der Drift & Vos, 1987; Vos, 1998, 2000). The phenomenon of procrastination of study behavior will decline as the exams come closer (Schouwenburg, 1994). If the moment of exams approaches, participation on socio-cultural activities will also decline (Schouwenburg & Groenewoud, 2000). Research that pertains to the period of exams and the formative or summative character of exams, remains unexisting.

Nevertheless this information is important given the observation that differences between universities in this respect may hamper the growing mobility of students and the harmonization of higher education in Europe.

This study investigated the effect of (1) the summative or formative character of assessment and (2) the assessment period (i.e. immediately following on the courses or not) on study behavior.

1. Research methodology

The study has been carried out when at one specific university some faculties were organizing summative exams at the end of the year whereas other faculties were providing summative exams at the end of each term. In the case of end-year summative exams, formative tests were organized for some of the courses in the first year.

Four faculties are engaged in the research. Two faculties organized at the end of the first term (in January) only formative tests. All the summative exams took place at the end of the second term (June). In two other faculties both the first and second term ended with summative exams (see Figure 1).

January June

Faculty A Year-based assessment system Tests (formative) Exams (summative)

Faculty B Year-based assessment system Tests (formative) Exams (summative)

Faculty C Term-based assessment system Exams (summative) Exams (summative)

Faculty D Term-based assessment system Exams (summative) Exams (summative)

Figure 1: Scheme of the different systems of assessment for each faculty

1.1.  Research questions

The research deals with the differential effect of different kinds of assessments. The effect on study behavior of students at the end of the first term in their first year of study is investigated and this for summative exams at the end of the term or at the end of the academic year and for formative tests. The influence of (1) the summative versus formative character of assessment on study behavior and (2) the moment of assessment on study behavior are investigated. With respect to the first question the way students prepare for summative exams at the end of the first term is compared to their study behavior for formative tests. For the second question a comparison is made between the preparation for summative exams and formative tests programmed at the end of the first term versus the preparation for courses scheduled in the first term for which the summative exams are scheduled only in June.

1.2.  Research instruments

For each of their courses and for each month of the first term (October, November, December, Christmas holidays and January) students of the first year were asked to indicate, by means of a questionnaire, the frequency by which they attend lectures, study course materials and perform other supplementary study activities. Students were asked to rate these activities by using a Likert scale with four categories. They were asked to indicate whether they perform each of the activities never (1), rarely (2), often (3) or very often (4).

1.3.  Data collection and data analysis

The questionnaire is filled in during a course in presence of the researchers. Data are available for 937 students (out of a total of 1755) of the first year (see Table 1).

Table 1: Research population

Frequency / Percentage / Total number of students into the first year / Percentage response
Faculty A (year system) / 218 / 23 / 455 / 48
Faculty B (year system) / 159 / 17 / 338 / 47
Faculty C (term system) / 206 / 22 / 295 / 70
Faculty D (term system) / 354 / 38 / 667 / 53
Total / 937 / 100 / 1755 / Mean = 57

The mean response is 57%, with a minimum of 47% and a maximum of 70%. The questionnaire was filled out for 50 different courses.

By means of an analysis of variance, with months as within-subject variables the effect of the type of evaluation is investigated. Four types of assessment are distinguished: term or end of year exams (both summative) for a term-based assessment system and formative tests or summative exams for a year-based system. With regard to the first question (the effect of summative versus formative character of assessment) term-based exams are compared with formative tests. The second question on the moment of assessment deals with comparing the formative tests and term-based exams (both in January) with exams scheduled in June.

2.  Results

For each of the research questions only significant results are presented.

2.1. Effect of formative tests versus summative term-based exams

In Table 2 the means for each of the activities are presented.

Table 2: Mean scores for each of the activities for term-based exams and formative tests

Attending lectures / Studying course material / Performing supplementary study activities
Term-based exams / Formative tests / Term-based exams / Formative tests / Term-based exams / Formative tests
October / 3.63 / 3.74 / 2.33 / 2.24 / 1.93 / 2.54
November / 3.55 / 3.72 / 2.62 / 2.50 / 2.03 / 2.62
December / 3.53 / 3.71 / 3.01 / 2.92 / 2.04 / 2.63
Christmas holidays / / / / / 3.57 / 3.36 / 1.74 / 2.20
January / 3.20 / 3.67 / 2.73 / 2.93 / 1.90 / 2.56

2.1.1.  Attending lectures

First year students seem to attend frequently lectures during the first term both for courses with exams organized at the end of the first term and for courses with formative tests (means between 3.2 and 3.7, see Table 2). Still there are some differences with respect to the type of assessment (see Table 3[1]). An effect of time, type of assessment and an interaction effect between time and type of assessment could be observed. Univariate analyses of variance indicate significant effects for each period of time.

Table 3: Results of the multivariate analysis with regard to attending the lectures

Source df SS F p Wilks'Lambda

Multivariate Time 3 / 99.41 0.000 0.921

Time*Type assessment 3 / 50.37 0.000 0.958

Type assessment 1 45.28 82.01 0.000

Results indicate that students who have only formative tests in January attend more lectures than students with summative exams at the end of the first term.

2.1.2.  Studying course material

The extent to which students indicate to study course materials during the first term varies from rarely to frequently (Table 2). Differences could be observed in function of the type of assessment. An effect occurred both for time, interaction between time and type of assessment as for type of assessment (see Table 4).

Table 4: Results of the multivariate analysis with regard to studying course material

Source df SS F p Wilks'Lambda

Multivariate Time 4 / 1037 0.000 0.453

Time*Type assessment 4 / 29.51 0.000 0.967

Type assessment 1 2.989 6.259 0.012

Univariate analyses of variance indicate that in December (p = 0.013) and during Christmas holidays a difference exists between the two types of assessment. For October (p = 0.080) and November (p = 0.059) the differences are not significant.

In the beginning of the first term (i.e. October and November) students process course materials for both courses with the same regular pattern. At the end of the first term more attention is devoted to courses with term-based exams. In January the opposite results are found: students indicate to spend more time studying the course materials for courses with formative tests than for courses with summative exams.

2.1.3.  Performing supplementary study activities

Mean scores for performing supplementary study activities greatly differ: the means vary between never (1) to often (3). There is an effect of time and an interaction effect between time and type of assessment (see table 5). Also the between-subject effect of type of assessment is significant.

Table 5: Results of the multivariate analysis with regard to performing supplementary study activities

Source df SS F p Wilks'Lambda