The Good, the Bad Or the Insignificant?

The good, the bad or the insignificant?

Teachers’ views of Quality Assurance and Evaluation in Finnish top-ranked comprehensive school

Risto Rinne, Hannu Simola, Mirka Mäkinen & Sari Silmäri-Salo

(Gothenburg 8-9 September 2008)

Contents

1THE EVALUATION REVOLUTION

2WHAT IS “QUALITY” IN QUALITY ASSESSMENT AND EVALUATION?

3RESEARCH QUESTIONS AND DATA OF THE SURVEY

4THE STRUCTURE OF THE GROUP OF TEACHERS AND ITS REPRESENTATIVENESS

5THE SCHOOLS

6THE HIGH QUALITY OF EDUCATION

6.1High quality of education and teachers’ gender

6.2High quality of education and teachers’ age

6.3Type of school and high quality of education

7CHANGES IN THE QUALITY OF EDUCATION OVER THE PAST 20 YEARS

7.1Teachers’ sex and change

7.2The age of teachers and changes in quality

7.3Type of school and change in the quality of education

8CHANGES IN RESPECT FOR THE TEACHING PROFESSION OVER THE PAST 20 YEARS

8.1Teachers’ sex and change in status

8.2Teachers’ age and change of status

8.3Type of school and changes in status

9THE MOST IMPORTANT ASPECTS OF EVALUATION

9.1Teachers’ gender and aspects of evaluation

9.2Teacher’s age and aspects of evaluation

9.3Type of school and aspects of evaluation

10THE MOST IMPORTANT MEASURES OF SCHOOL ACTIVITY QUALITY

10.1School type and most important indicators

11VIEWS ON THE EFFECTS OF DIFFERENT FORMS OF QAE ON THE QUALITY OF EDUCATION

11.1Views by male and female teachers on the effect of evaluation on the quality of education

11.2Teachers of different ages and the effects of evaluation on the quality of education

12ATTITUDES TOWARDS QUALITY EVALUATION

12.1Atitudes are gender-specific

13EVALUATION PROCEDURES AT SCHOOL

13.1Evaluation procedures at school, the size and location of the school

14QUALITY EVALUATION AND THE WORK OF TEACHERS

14.1Evaluation forms and teaching

14.2Time used by teachers on qae

14.3The effects of quality evaluation on work

14.4Views by male and female teachers on the effects of evaluation on their work

14.5The effects of evaluation on work in relation to the size and location of the school

15CONSEQUENCES RESULTING FROM THE RANKING OF SCHOOLS AND THE PUBLISHING OF QUALITY INDICATORS

16THOUGHTS ON QUALITY EVALUATION

17CONCLUSIONS

References

Since the late 1980’s, Finnish education policy has adopted many of the features of supranational neo-liberal education policy, including the new culture of school assessment and Quality Assurance and Evaluation (QAE).[1] Following the decentralization of school administration, QAE has become main instrument for steering education in Finland. In the 1990’s the governance of the Finnish education system underwent a drastic change from a very strictly planned and governed system to one of the most decentralised in Europe. The national implementation of this policy in Finland has, however led to many controversies and collided with conflicting historical traditions, which has given the implementation of QAE a uniquely Finnish flavour. Some of these Finnish adaptations are connected with the autonomy, status and education of teachers. That is why especially in Finland it is worth asking teachers how they experience and view the new QAE policy in they everyday life; is it good, bad or insignificant.

1. The Evaluation Revolution

In the aftermath of the economic depression of the first half of the 1990’s, the political balance in Finland shifted quite dramatically to the right. It became commonplace in public discourse to blame the “bloated” state and municipal bureaucracies for the economic difficulties. It is not in the scope of this study to determine to what extent this argument was just, but in any case, it can be argued that the trimming down of the public sector was done using the economic depression as either an excuse or a cause.

In education, central planning and inspection systems were replaced by a system of quality assessment and evaluation. The state and municipalities are increasingly taking on the proverbial role of the night watch, which does not interfere in the everyday tasks, duties and worries of the citizens, but watches from afar and sounds the alarm if major problems arise.

One of the simplest definitions of evaluation has been formulated by Scriven (1967), who defines it as “judging the worth or merit of something”. Although in this sense things have been evaluated ever since the man rose to walk on two feet, or even before that, as a profession evaluation is a novelty.

Evaluation is hardly evil in itself, but it can bring about some problematic developments.

• The explicit purpose of QAE ought to be promoting quality in whatever is being evaluated. However, the rise of evaluation is historically connected with a decline in resources in education and elsewhere in the public sector. Thus, quality is being emphasised in rhetoric, but fewer and fewer resources are given for implementing this thing called “quality”.
• As every profession tends to look after its vested interests, it can be expected that a strong evaluation profession will keep the wheels of evaluation spinning, churning out evaluation reports, whether they are needed or not.
• Evaluation is connected on certain levels to the marketization of education. As in social policy, there have been pressures to introduce market mechanisms in education. Free school choice was introduced at the end of the 1990’s. It has even been suggested that school-specific evaluations be published to help parents make better choices for their children.

According to Foucault, the history of the exercise of power shows a tendency towards “softer” methods. Whereas in the Middle Ages an offender’s limbs may have been ripped off using horses, as Foucault describes in the shocking opening sequence of Discipline and Punish (1979), nowadays the offender may be put under house arrest using electronic devices. The debate concerning the effectiveness of harsh methods of punishment is one we will not dwell on in this study, but the general idea of changing methods of control is also applicable to education. If considered in the framework of Foucault’s theory of power, governance in education has softened. However, it can be argued that the apparent softness of governance by evaluation is an illusion. Evaluation can be an efficient method of control: it can lead to self-censorship, changes in working methods, surveillance of one’s colleagues, shifts in focus, etc.

Of course, the best part of using evaluation as a form of control is that explicitly no control at all has been exercised. On the surface, all that has taken place is evaluation. In this sense, evaluation is the most powerful sword – one that is never revealed.[2]

Indeed, in Finland the sword of evaluation is kept very firmly in its sheath. Finland is possibly one of the most decentralised countries in Europe as regards the administration of education. Municipalities are generally free to organize evaluation in any way they wish, as shown in Figure 1. There are various approaches to the evaluation of schools providing compulsory education in Europe. On paper, the Finnish evaluation system most closely resembles that of the UK (EnglandWales). Evaluation of education is mainly conducted at the local level, by the education provider (municipality).

FIGURE 1. The Finnish lines of QAE (Eurydice 2004)

1WHAT IS “QUALITY” IN QUALITY ASSESSMENT AND EVALUATION?

In its broadest meaning, the term QAE can be used to refer to all types of evaluation taking place in schools. As depicted in Table 1, QAE can be divided into internal or external evaluation, depending on whether the evaluation is being conducted by school personnel “from the inside” or by municipal or government officials, or by international players such as the OECD, “from the outside”. Also, the evaluation can be either informal or formal. Asking pupils whether they have been content with the quality of teaching can be considered the simplest form of informal evaluation. Tests and evaluation studies are examples of more formal evaluation.

TABLE 1. Types of quality assessment and evaluation

Internal / External
Informal / Written or oral feedback from pupils to teachers
Assessment by other school professionals (e.g. nurse, speech therapist, janitor) / Newspaper articles
TV documentaries
Formal / Tests / Evaluation studies
Quality indicators

The quality in QAE is a question in itself. If taken seriously, the point of view greatly affects the evaluation outcome, i.e. whether something is (high) quality or not. The points of view often used are those of producer, client, process and product. The producer would be the school/municipality, clients would be the pupils, teaching would be the process, and the product would be the learning outcomes.

In practice, however, the definition of quality in quality assessment studies is usually very simple. Quality is defined based on standards. In a way, we can speak of “McDonald’s quality” in the sense proposed by Ritzer. If certain set criteria are met, then there is (high) quality. According to Ritzer (1996, 9–11), “McDonald’s quality” consists of four things:

1. Efficiency
2. Calculability
3. Predictability
4. Control

Efficiency in education means that knowledge is smoothly transferred from teachers to pupils. There are streamlined, marketable education products. Calculability refers to the need of quantification of learning outcomes. Grades, degrees, student numbers, etc. need to be calculated. Predictability is essential; there have to be standards for education products. No matter where education is acquired, the level of learning is expected to be the same. Control is, in Ritzer’s thinking, mostly related to employees. They are controlled by rigorous standardization of practices, such as the temperatures in which hamburgers are prepared and served in the McDonald example.

Ritzer (1996, 125) argues that in the end, what McDonaldization really offers is an illusion of efficiency. Likewise, in many areas of entertainment, education and working life illusions are being manufactured. What really stands behind the facades does not matter all that much. What does matter is the image that is projected outwards.

2RESEARCH QUESTIONS AND DATA OF THE SURVEY

Quality Assurance and Evaluation is a multifaceted phenomenon. The focus of our research project lies in taking in the phenomenon and interpreting it in regard to the Finnish school system.The purpose of this study is to find solutions to the most crucial issue in the Finnish education, i.e. how to provide high-quality education equal to all and simultaneously meet the universal demands of modernizing education in order to enhance Finland’s competitiveness on the international markets.

Our research questions include:

1. Do the Finnish primary and lower secondary schools have their own models of Quality Assurance and Evaluation?
2. How have the Finnish schools received the international model of Quality Assurance and Evaluation?Has the model been applied to fit the Finnish school system, and if it has, how has it been realized?
3. Which are the intended effects of QAE at the grass-roots level? And, on the other hand, does it have some external consequences, and if so, what are they?

In this paper we examine how Finnish teachers and head teachers perceive the quality evaluation of education, its implementation and consequences in regard to their own work and position as teachers.Our research questions in relation to this are the following:

1. What do teachers think quality is?
2. How do their perceptions of quality translate into the quality instruments that they use? Do they connect?
3. How do their pedagogical concerns relate to their managerial concerns?
4. What is the role of the local / what are the connections with the centre/ why is there complete absence of internationalisation?
5. Do they use self-evaluation (qualitative data) more than numbers? What instruments do they use?
6. What are their views on professional autonomy, trust, respect?
7. What are their views on publication of data /league tables?
8. What are their views on parental choice?
9. Is there any correlation of length of service/ age and views on quality/ tools used?

The questionnaire material used in the study, “Laatu ja arviointi peruskouluissa – kysely opettajille [Quality and evaluation in the Finnish primary and lower secondary school – a questionnaire for teachers]”, is only a part of the more later international research project entitled Fabricating Quality in European Education. The questionnaire material was gathered using the web-based Webropol system between November 2007 and January 2008. It was sent to 8595 school teachers chosen as a cluster sample from 10 regions of Finland. A total of 1526 teachers replied, which meant a response rate of 17.8%. The Webropol program allows the researcher to monitor in real time the number of respondents who open the questionnaire on their computer screen and the number who actually answer the questions. If we examine the situation from the point of view of those teachers who took time to look at the questionnaire (in total 2574 persons), the response rate rises almost to 60%. On the other hand this means that 40 per cent of those who opened the survey did give up. In addition, the teachers responded admirable to the open-ended questions on the questionnaire, but the analysis of these answers will have to wait until a later phase of the research. At this stage the material has only been analysed tentatively using probability distributions and cross-tabulation. The large non-response figures most likely stem from the fact that some of the teachers do not use the e-mail address assigned by their employer, but have opened their own e-mail accounts.

In the first round, the questionnaire was sent electronically on 16 November 2007, and the teachers were asked to respond by 30 November 2007. The first repeat round was sent on 4 December 2007, with a deadline of 14 December 2007. A second repeat round was sent on 9 January 2008 with a deadline of 19 January 2008. In practice, both the first and second repeat round were during the most hectic times for teachers at work. Activities connected with end of the grading period for the autumn semester and preparations for the Christmas festivities at school which correspond with this time period result in considerable extra work for teachers in Finland. This may have had an effect on the over-all low response rate. The purpose of the questionnaire was to clarify the views of teachers on the evaluation of quality and its effect on basic education. Factors used to examine views on the evaluation of quality and its effects include teachers’ background (gender, age and teaching experience) and school background (size of school, location of school and social background of pupils).

3THE STRUCTURE OF THE GROUP OF TEACHERS AND ITS REPRESENTATIVENESS

The majority of the teachers responding to the questionnaire were women (74.7%). One-fourth of the respondents were men (25.3%). In 2004, there were a total of 44623 primary and lower secondary school teachers in Finland, of which most were women (72.3%), and a little more than one-fourth of Finnish teachers (27.7%) were men. Thus, a comparison with the statistics on teachers available from the Finnish Statistics Bureau, StatFin, for 2004 shows that the material was not badly skewed in terms of gender (Table 2).

TABLE 2. Gender and age in this Finnish teacher survey and in StatFin statistics

Teacher Survey / StatFin statistics3 2004
Gender / Frequency / Valid percent / Frequency / Percent
Female / 1117 / 74.7 / 32 258 / 72.3
Male / 377 / 25.3 / 12 365 / 27.7
Total / 1494 / 100.0 / 44623 / 100.0
Age
– 29 / 96 / 6.4 / 5 083 / 11.4
30–39 / 376 / 25.2 / 12 582 / 28.2
40–49 / 498 / 33.4 / 13 204 / 29.6
50–59 / 458 / 30.7 / 12 969 / 29.0
60-
Total / 64 / 4.3 / 785 / 1.8
1492 / 100.0 / 44623 / 100.0

The average age of the teachers responding to the questionnaire was 44, with a range of 21 to 75 years. Almost two-thirds of the teachers responding to the questionnaire belonged to the age class 40–49 (33.4%) or 50–59 (30.7%), while every fourth teacher was 30–39 years old. Of the respondents, 6.5% were aged 29 or under, and 4.3% 60 or over. When comparing age range of the questionnaire respondents to the statistical averages for 2004 provided by StatFin[3], it was found that the over 40 age group was over-represented and those under that age under-represented. However, the differences are not very significant.

The questionnaire was sent to schools that were located in 10 regions of Finland. The following regions were over-represented in relation to their population: Keski-Pohjanmaa, Keski-Suomi, Pirkanmaa, Pohjois-Karjala and Varsinais-Suomi. On the other hand, Etelä-Pohjanmaa and especially Uusimaa were under-represented in relation to their population. The differences in this comparison are partly due to differences in the age structure of the population as a whole compared to that of teachers.Uusimaa is the overwhelming biggest regions of Finland including Helsinkiand the whole capital area. It is pretty obvious that there might be big differences of Quality Assurance and Evaluation development between different regions and especially between the big cities and the rural areas.

TABLE 3. Regional distribution of respondents

Count / Percent of valid / Population
31.12.2006[4] / Distribution of population%
Etelä-Karjala / 63 / 4.2 / 135255 / 4.0
Etelä-Pohjanmaa / 49 / 3.3 / 193585 / 5.7
Kainuu / 43 / 2.9 / 84350 / 2.5
Keski-Pohjanmaa / 85 / 5.7 / 70672 / 2.1
Keski-Suomi / 154 / 10.3 / 269636 / 7.9
Lappi / 81 / 5.4 / 184935 / 5.4
Pirkanmaa / 219 / 14.6 / 472181 / 13.8
Pohjois-Karjala / 85 / 5.7 / 167519 / 4.9
Uusimaa / 464 / 31.0 / 1373600 / 40.3
Varsinais-Suomi / 252 / 16.9 / 457789 / 13.4
Total / 1495 / 100.0 / 3409522 / 100.0

The respondents has an average of 17 years work experience (s.d. 11.97), with a range form less than one year to 40 years (Chart1).

CHART 1. Years working in teaching profession

Many respondents had more than one teacher training certification. Almost half of them (49.4%) were qualified as subject teachers (Chart 2). A little less than half of them (43.7%) had been trained as class teachers. Almost one-fifth of the respondents had training as a special education teacher (16.5%), and one out of ten (11.2%) had some other training. The teachers responding to the questionnaire had the possibility of choosing more than one alternative concerning their training.

CHART 2. Qualification(s) for teaching

Almost one-half of the teachers (45.5%) were working in their school as subject teachers (Chart 3). A little over one-third of the respondents (35.1%), on the other hand, were working as class teachers. Only 4.3% of the teachers who responded to the questionnaire were working as the head teacher or principal of their school. In addition to the above, 15.1% reported working in some other capacity in their school, mainly as a special education teacher or as a study advisor.