The relationship between school climate & head teacher leadership, and pupil attainment: evidence from a sample of English secondary schools
Paper presented at British Educational Research Association Annual Conference, Herriot Watt University, Edinburgh September 2003.
Rosalind Levačić, Institute of Education, University of London
Lars Malmberg, Department of Educational Studies, University of Oxford
Fiona Steele, Institute of Education, University of London
Rebecca Smees, Institute of Education, University of London
Address for correspondence:
WORKING PAPER: PLEASE DO NOT QUOTE WITHOUT PERMISSION
Introduction: school effectiveness characteristics
A central l issue in school effectiveness research (SER) is the identification of the characteristics of effective schools or the factors associated with school effectiveness. Ultimately the goal is to find school climate processes or factors that are causally related to desired student outcomes, both cognitive and affective. The context-input-process-outcome model of school effectiveness (Scheerens, 1997; Scheerens, 1999; Teddlie and Reynolds, 1999) has emerged as the accepted framework for SER. School processes are also important for research on education production functions since they mediate between the input of resources and the production of outcomes and thus can explain differences in school efficiency. The generalised model of school effectiveness specifies a complex set of interrelationships between four main groups of factors – the school’s external environment, the school’s own organisational climate, classroom processes and the motivations, attitudes, behaviours and abilities of the individual students. The challenge of SER is to specify essential relationships in a sufficiently simple form for statistical estimation while not biasing the findings by omitting key variables or linkages. School and classroom climate processes must be captured in a measurable form for analysis using either surveys or researcher recorded observations.
This paper reports an attempt to create school climate constructs and test their relationship with pupil attainment. This covers the following:
· methodological issues and the current state of research relating school climate and leadership to pupil attainment;
· the development of a set of instruments for measuring school climate;
· their administration to a sample of secondary schools in England;
· the use of factor analysis to create school climate constructs from the responses;
· multilevel modelling to test for the statistical significance of the school climate constructs in explaining student attainment at KS3 and GCSE, after controlling for prior attainment, other pupil characteristics and school context.
Review of research and methodology
As Sammons et al. (1995 p. 1) note, the number of derivative publications about SER exceeds the number of original empirical studies. Their review of the Key Characteristics of Effective Schools[1], which cautions against the dangers of interpreting correlational relationships as causal or placing too much confidence in evidence from outlier studies, was influential in distilling a list of ‘eleven factors for effective schools’ from the research evidence mainly from the US, UK, Australia and the Netherlands. This list and a similar one from Levine and Lezotte (1990) were combined by Teddlie and Reynolds (1999) to produce a list of nine processes of effective schools. As Teddlie and Reynolds (1999) make clear, the key processes that have been distilled out in these influential reviews, are derived from about half a dozen major studies. These include a study of 78 Michigan elementary schools (Brookover et al., 1979; Brookover et al., 1978), the Louisiana longitudinal study (76 schools) (Teddlie et al., 1984; Teddlie and Stringfield, 1993; Teddlie et al., 1989), (Rutter et al., 1979) Fifteen Thousand Hours study of 12 London secondary schools and (Mortimore et al., 1988) School Matters: the Junior Years (50 London primary schools).
These studies used the same basic methodology for identifying ‘effectiveness factors’. The sampled schools’ relative effectiveness in value-added terms was estimated statistically from data on pupil attainment and pupil background. Data on school and class process were collected by survey instruments and/or researcher observation. Two alternative methods of identifying ‘effectiveness factors’ are to be found in the studies. One method involves selecting a sub-sample of schools that are placed into groups according to their being more effective or less effective and having low or higher SES pupils. Differences are then identified in the process variables found in effective schools, controlling for composition, compared to ineffective schools. The other method is to obtain measures of school processes and to statistically test the relationship between the process variables and pupil attainment, controlling for pupil background and school context. The school effectiveness characteristics are derived from a handful of major studies conducted many years ago, which mostly predated the use of multilevel modelling and used the first method to identify ‘effective school climate factors’. The school effectiveness characteristics distilled out in these major surveys include both class level and school level constructs though the distinction between the two levels in the list of characteristics thus derived is not clear cut.
Obtaining good quality evidence on the factors related (correlationally or causally) to school effectiveness requires an expensive and time consuming research design which tracks several thousand students in schools over a number of years. A major recent study is the LOSO project in Flemish Belgium (Van Damme et al., 2002) which has followed 4700 students in 275 secondary schools since 1991/2. The study collected data from teachers on the learning climate of their classroom and their teaching practices. School climate constructs were created from questionnaires to pupils and teachers about the school’s organisational environment. The main findings are that after controlling for pupil variables and school context, class composition, class climate variables affected attainment and were themselves interrelated. The school climate variables explained very little of the variance in attainment: only the construct ‘paying attention to differences between students’ was positive and significant.
Smaller scale studies usually utilise existing data, supplemented by school surveys. A number of smaller scale studies of school effectiveness using multilevel modelling have also included measures of school and classroom climate. A large proportion of these are from the Netherlands and offer only limited support for the impact of the ‘key characteristics of effective schools’ on pupil attainment (Teddlie and Reynolds, 1999), the evidence being stronger for primary schools than for secondary.
A related branch of research is on the effects of school leadership on student attainment. While school leadership appears as one of the school effectiveness characteristics and is expected to impact on school process variables, it has often been studied separately or as the major focus in its own right. The majority of research on leadership is qualitative and thus cannot rigorously establish a relational or causal effect on pupil attainment. Despite this lack of hard evidence, government policies and educational leadership programmes, such as the National College of School Leadership in England, are predicated on the assumption that the quality of leadership has important effects on pupil attainment. Only a relatively small proportion of school leadership studies have a used a research design which enables the ‘effect’ on pupil attainment of leadership to be estimated. A recent literature review by Hallinger (1996) and meta-analysis by Witziers et al., (in press) selected only studies meeting this specification[2]. Most studies estimated a direct effect of leadership on student attainment and only a few utilised a model in which the effects of leadership on attainment are indirect. There is also considerable lack of consensus on the appropriate conceptualisation of school leadership (possibly reflecting the wider range of general management literature from which educational leadership research draws than does school effectiveness research). As Witziers et al. note ‘consistency in the way concepts are operationalised is not the strongest feature of leadership research’. Their overall conclusion from a meta-analysis of 37 studies is that the effect size of leadership is positive but small (in the order of explaining one percent of the variation in student attainment). Dutch studies mainly show no effects of leadership in secondary schools. An important methodological issue examined by Witziers et al. is the moderating effect of school contextual variables on leadership - its effects tend to be smaller in models including school context variables and also when teacher variables are included. Theories of leadership consider both how leaders affect school and class climate and the reciprocal effects of climate in modifying leadership behaviour.
Operationalising school climate in a study of 20 English secondary schools
A major reason for the paucity of studies including school and class climate constructs compared to those which just estimate school effects including fixed pupil and school level variables is the expense of collecting such data and the reluctance of hard pressed schools to participate in surveys, especially of sensitive nature. However, a set of school climate and leadership constructs was collected from a sample of 20 English secondary schools in the course of a DfES funded pilot study on the effects of resourcing on pupil attainment, in which the school climate and leadership variables were included as controls[3].
A set of related instruments on school climate were developed to be administered to different types of stakeholder. The distinct groups identified were Year 9 and Year 11 pupils who responded to the same instrument; teachers, the head teacher and/or deputy, support staff and governors. Each of the adult groups had a slightly different version of the school climate instrument reflecting a priori assumptions about the aspects of school climate about which they could be expected to be knowledgeable. For the adults the teachers’ instrument was the starting point and the others were adjusted versions of it. The items for the school climate questionnaire were selected after comparing a number of instruments that had already been used in other research studies or for diagnostic purposes for school improvement. The content of the various instruments was mapped to a set of school climate constructs drawn from these questionnaires, which corresponded quite closely to the eleven effective characteristics of Sammons et al. The instruments reviewed in order to construct the one used in RAPA study are listed in the Appendix in Table A1. The criteria for selecting constructs and items were:
developed and used in previous research with some evidence of association with pupil attainment;
good reliability and validity tests (in particular Cronbach’s alpha reported) and a good linkage of items to constructs;
construct validity – appropriate for the stakeholder to whom it is administered;
ability to administer same construct to different stakeholders;
the instrument should not exceed 30 minutes to complete;
obtaining permission to use the instrument and at low cost.
The Assessment of School Climate Instrument (Grosin and McNamara – Stockholm and M? Universities) was the one which most heavily influenced the constructs and items selected. This instrument has been worked on since the 1990s and is mainly used for school evaluation in Sweden and the school ratings are used to inform school improvement work. The instrument has both a teacher and a pupil version - for Year 5 and 6 pupils (12-13 years of age). It is available on a website so that school staff and pupils can respond to it on line. The instrument is based on 30 years of school effectiveness research – hence its constructs map quite well to the characteristics of effective schools. Another instrument was influential – the Louisiana ABC + model (Teddlie and Stringfield) developed for school improvement work but also derived from one of the longitudinal classic studies. Permission was obtained to use both instruments. Questions were modified, omitted or added so as to reflect the English school context and age of the pupils[4].
Seven a-priori constructs, covered by 41 items, were selected for the pupil school climate questionnaire. For the teacher questionnaire, 10 a priori constructs were selected including headteacher leadership, and represented by 84 questions in total. The support staff questionnaire included nine constructs (pupil performance monitoring and rewards was omitted and teachers replaced by staff in four of the other constructs). The governors’ questionnaire included 8 constructs[5]. As in the AISC instruments, respondents are asked whether they ‘agree strongly’, ‘agree’, ‘disagree’ or ‘disagree strongly’ with statements such as ‘The teachers in this school show respect for the students’ and the responses are coded from 4 (agree strongly) to 1. Further details of the items and constructs are given in the discussion below and in tables A2 and A3 in the Appendix.
Administration of the school climate instrument
As this was a pilot study, which focused on the relationship between school resourcing and pupil attainment, sample size was restricted to 20 secondary schools in England. All secondary schools in England were stratified into 8 categories according to high or low school size, high/low percentage of pupils eligible for free school meals (FSM) and high/low resourcing (see Table 1). High or low refers to above or below the median. Data on size and FSM were available from the Annual Census of Schools 2001 and data on budget revenue per pupil were obtained from Section 52 statements for 2001-02 held at the DfES. Schools that had been recently visited for fieldwork and schools in special measures were excluded. The proportion of schools sampled from each of the six categories reflected the proportion of each category in the whole school population.
As the sampled schools were to be visited by 5 fieldwork teams[6] the selection procedure ensured that each team would have a pair of schools to visit in a single government office area[7]. A 1 in 10 acceptance rate for participation in the study was assumed and therefore 200 schools were initially sampled and given to the teams in batches of 10. Schools in each batch were contacted by telephone and invited to participate[8].
Table 1 Stratified sampling frame for selecting secondary of schools
Resourcing / Free School Meals / School size / Secondary (excluding middle schools deemed secondary)High / High / High / 392
Low / 635
Low / High / 246
Low / 376
Low / High / High / 355
Low / 328
Low / High / 719
Low / 515
A field work team of two visited each school for two and a half days in the summer term of 2002. The timing resulted in a reduced number of Year 11 pupils being available for responding to the surveys because of GCSE exams and two schools did not provide any Year 11 pupils. A mixed ability form group in Year 11 and in Year 9 in each school was administered the questionnaires in class with the field worker present. A number of instruments was administered but in this paper we report only on the school climate findings. The resulting dataset used for analysis contains 351 Year 11 students and 519 Year 10s, 124teachers, 29 heads and deputies, 77 support staff and 45 governors[9].