1

Effectiveness

and

improvement:

school and college

research compared

Dr Paul Martinez

Paper presented at the Annual Conference of the British Educational Research Association, Leeds University, 13-15 September 2001

Dr Paul Martinez

Development Adviser

Learning and Skills Development Agency

Robins Wood House, Robins Wood Road

NOTTINGHAM NG8 3NH

Telephone: Nottingham (0115) 929 9121

Fax: Nottingham (0115) 929 3505

E-Mail:


Effectiveness and improvement:
school and college research compared[1]

Abstract

There are relatively well established traditions of research in both school effectiveness and school improvement. These traditions are quite mature in that they have distinct research methodologies, agreed terms of references and discourse and, not infrequently, some consensus around advice offered to both policy makers and practitioners. There are, moreover, some signs of convergence between the hitherto distinct traditions.

The position of colleges is quite different. For a variety of reasons, colleges have provided less of a focus for educational research. At first sight, moreover, there is no equivalent to the school effectiveness and improvement traditions.

This article reviews the research that is currently available to argue that:

  • there is more college effectiveness and improvement research than has sometimes been supposed
  • the theoretical foundations and methods of this research resemble but in some crucial ways are distinct from the research traditions focused on schools
  • some specific research issues and priorities need to be addressed in college based research to enable it to make a profound contribution to educational theory, policy and practice
  • there are some opportunities, nevertheless, for school researchers to benefit from college research on retention, achievement and improvement issues.

Background

Over the last thirty years there has been an explosion of interest in two related areas of research: school effectiveness and school improvement. The first attempts to answer the question: What are the factors that are associated with effective schools? The second has a focus on the ways that schools improve to become more effective. Both traditions are inspired by the belief that schools “can make a difference”. In other words, they are premised on the belief that some schools and some departments within schools perform better than others and that teachers, heads of departments and head teachers can develop and implement measures to improve performance or, in the alternative, to maintain existing high levels of performance.

This research effort is substantial, growing in volume, supported by a number of dedicated research centres in universities and international in scope. Similar research efforts are being undertaken throughout the English speaking world and beyond. Recent and comprehensive introductions to this work can be found in Macbeath and Mortimore (2001), Gray et al (1999), Sammons (1999), Mortimore (1998) Reynolds (1990), Sammons et al (1997), Reynolds et al (1997). A convenient introduction and overview written for an intended further education audience has been produced by Somekh et al (1999).

In the context of this considerable research effort in the school sector, a number of FE researchers have identified the absence of a research culture in further education (Brotherton 1998; Elliot 2000). Indeed, a recent article noted the “dearth of research to date on college effectiveness” (Cunningham, 1999, p403).

The purpose of this article is to compare and contrast the literatures of school effectiveness and improvement with similar research in the college sector. It does so by way of:

  • a brief summary of the research base and main conclusions from the literature on schools
  • a more comprehensive review of research on colleges
  • a consideration of key differences in focus between school and college research
  • some suggestions concerning opportunities for exchange between school and college based research traditions.

In terms of scope, the article concentrates on British research on schools, colleges, adult education services and government funded, work based training schemes. It excludes research that looks at improvement and effectiveness issues within higher education.

School effectiveness research

The literature of school effectiveness is firstly concerned with the outcomes of education and making comparisons between schools. In the words of Somekh:

“ a more effective institution is typically defined as one whose students make greater progress over time than comparable students in comparable institutions” (1999, p 25).

School effectiveness studies generally:

  • seek to answer the question: what factors are associated with an “effective school”?
  • define effectiveness in terms of the whole school
  • focus on quantitative measures of outcomes, such as academic performance and attendance
  • seek quantitative measures of inputs and processes that can be related to outcomes
  • seek to compare schools on the basis of quantitative measures, and identify the variables which schools can control and which distinguish high from low performing schools
  • employ relatively sophisticated research techniques including multi level modelling and value added methodologies.

There appears to be a certain amount of agreement over issues thrown up by research to date, and how these might be addressed. Thus, problems associated with a “top-down”, whole institutional focus with its implicit reliance on the views of the head and senior teachers (Ouston 1999) are being addressed by work at departmental and subject level (Sammons, Thomas, Mortimore 1997, Harris et al 1997, Busher and Harris 2000). The difficulty caused by the failure of quantitative research to replicate findings from qualitative research concerning the role of school leaders (Scheerens and Bosker 1997) appears to have been resolved by the development of more sophisticated conceptual models and the application of robust quantitative research techniques (Hallinger and Heck 1998).

The growing maturity of the field can be illustrated by reference to the increasingly nuanced discussion of issues such as school context and pupil intakes. In the domain of research, at least, the old arguments around the impact of economic and social deprivation on school performance have given way to more fruitful discussions of how best to make like-for-like comparisons which will control for independent variables as far as possible (Mortimore 1998, Sammons 1999). Indeed, the school effectiveness field might be said to be mature in that meaningful discussions of methodology can take place on the basis of a substantial amount of shared theory (eg Ouston 1999).

Effectiveness research has paid attention to all phases of schooling from nursery to sixth form. It has an international dimension with British researchers increasingly engaged in a dialogue and in collaborative research with researchers from abroad, notably North America, Europe and Australasia. Further evidence of the relative maturity of school effectiveness research can be found in the meta-studies that have begun to appear. These studies typically propose theoretical frameworks based on syntheses and evaluations of scores and sometimes hundreds of research projects (Creemers 1994, Scheerens and Bosker 1997, Wang, Haertel and Walberg 1993).

In terms of the messages coming out of this research, there appears to be a broad consensus around a number of factors that have a significant impact on school outcomes. Different researchers place their emphasis slightly differently but the following list of factors drawn from Reynolds et al (1997) and Mortimore (1998) is reasonably typical:

  • leadership
  • shared vision and goals
  • a learning ethos and environment
  • high quality teaching and learning
  • high expectations by staff of pupils
  • positive reinforcement
  • close monitoring of pupil progress
  • pupil rights and responsibilities
  • purposeful teaching.

Finally, and perhaps most importantly the fundamental issue raised by Reynolds over a decade ago that effectiveness research “has had much more to say about what makes a “good” school than about how to make schools “good” (1999 p23), is being addressed by a growing convergence between school effectiveness and school improvement research (eg Sammons 1999, Gray et al 1999. Macbeath and Mortimore 2001).

School Improvement Research

School improvement research is concerned with the question of how schools might become better or more effective.

”School improvement is about raising student achievement through enhancing the teaching and learning process and the conditions that support it” (Hopkins et al 1994 p xx).

School improvement too has an extensive literature to which Stoll and Fink provide a convenient introduction (1996).

The great strength of school improvement research and presumably the source of its continuing attraction for practitioners is that it is:

  • motivational and inspiring
  • richly illustrated with examples of “what works”
  • representative of the views of practitioners.

There are some indications, however, that school improvement is a less mature field of research than school effectiveness. Many of the publications take the form of case studies of action research (eg Bowring-Carr and West-Burnham 1999) and suffer the problems associated with such an approach, notably difficulties of generalisation. More generally, some school improvement research has been criticised for being “lightly empirical and naively quantitative”, too reliant on ex post facto explanations and poorly predictive in terms of “what works” (Gray 2000, p 1).

These weaknesses are currently being addressed through efforts to:

  • apply more quantitative research techniques
  • make more explicit and consistent reference to work on school effectiveness
  • develop and apply more rigorous yardsticks of improvement including tests and exam results (eg Gray et al 1999; Macbeath and Mortimore 2001).

What improving schools seem to have in common are that they share:

  • a proactive and shared approach to planning
  • an ethos or culture that favours improvement
  • leadership throughout the school which focuses on the quality of teaching and learning and promotes and facilitates professional discussion around improvement
  • specific interventions to boost exam performance (Somekh et al 1999).

Research on college effectiveness

There is quite a substantial volume of research which addresses issues of college effectiveness (CE), but researchers and practitioners may have to look quite hard to discover it, since most of it is not published at all or because it exists in the “grey” literature of unpublished research dissertations and presentations and papers at conferences.

In comparison with effectiveness research on schools, most of this research addresses issues of effectiveness only indirectly. Indeed, it could be said to be addressing two more preliminary questions: what are the causes of student failure and, by extension, what variables are most significant in accounting for student outcomes (attendance, retention, achievement and attainment)? Notable exceptions to this general observation can be found in work on value added and on differential achievement in colleges whose students are recruited predominantly from areas of high social deprivation.

The discussion which follows attempts to:

  • provide an introduction to some of the research which is not in or is only partly in the public domain
  • summarise the findings of the research
  • assess the strengths and weaknesses of CE research to date.

Research within individual institutions is by far the largest category of research. It is being generated by individual colleges and adult education services as they attempt to identify the reasons for drop-out and exam failure in order to develop improvement strategies. Because it is usually intended for internal consumption and use, the work is largely unknown and unseen outside the originating institutions.

Institutionally based research takes many and diverse forms but the most common include research undertaken as part of a programme of post graduate study (MA, MEd, MBA and, occasionally, DEd); surveys of withdrawn students often accompanied by staff surveys; reports produced for management purposes (typically combining analyses of data and student surveys); research commissioned by a college from an external agency.

The questions addressed most frequently in college effectiveness research are:

  • What causes student withdrawal?
  • Which causes of withdrawal are within the influence or control of colleges?
  • What makes the most difference to student completion and withdrawal?
  • Where should colleges concentrate their energies to make improvements?

At the beginning of the 1990s, the prevailing view, faithfully reflected in an authoritative report from HMI (1991), was that drop out was largely due to factors external to colleges. The main thrust of CE research since then has been to displace that view.

Demographic factors

In Britain, withdrawn students do not have a markedly different profile from completing students in terms of age, ethnicity or gender (Martinez 1995, Martinez 1997a, Martinez and Munday 1978, Stack 1999).

Unlike schools, colleges do not have entitlement to free school meals to serve as a convenient indicator of social class. The proxy indicator used in the further education sector is therefore the relative economic and social deprivation of the electoral ward where a student lives. Research has demonstrated, however, that social deprivation measured in this way correlates poorly with retention and achievement across the college sector as a whole (Davies and Rudden, 2000p2). A relationship has been identified, but only in the 10% of colleges which recruit the highest proportion of their students from deprived postcodes. Even in this minority of colleges, variations in the demographic composition of the student intake seem to account for no more than 50% of the variation in college performance as measured by the achievement of qualification aims (Davies 2001). The only study which has asserted a significant “postcode impact” (Vallender 1998) did not consider any intervening variables, notably mode of attendance, level of programme or subject/curriculum area.

Student Motivation

While college improvement (CI) research has shown consistently that efforts to improve or maintain student motivation can lead to better retention and achievement (Martinez 1997 and 2000), research on college effectiveness suggests strongly that the initial motivations of students as expressed by their reasons for enrolling, aspirations, expectations of college etc, do not vary significantly between students who subsequently stay and students who leave (Martinez 1995, Martinez 1997a, Lamping and Ball 1996, FEDA 1998, Kenwright 1997).

A detailed study that explored student self-esteem and action-control beliefs with a relatively small sample of successful and unsuccessful students, did not find any marked differences between them (Stack 1999).

Student Decision Making

Medway and Penney (1994) were among the first to suggest that the student decision making process could be characterised as a continuous weighing of the costs of continuing with or abandoning a programme of study and that decisions to leave resulted “from rational decisions to respond to the difficulties [students] faced” (ibid p38). These early conclusions have been borne out by subsequent research. Using a variety of methods and with samples of up to 9,000 students, CE research has shown that college students have complex and multiple reasons for withdrawing from programmes of study and that decisions to withdraw can be seen as rational and positive from the point of view of students (Searle 1998, Crossan 1996, Martinez 1995, Freeman 2000, Bloomer and Hodkinson 1999, FEDA 1998, Adamson and McAleavy 2000, Adamson et al 1998, Martinez and Munday 1998).

Several studies show that students usually leave courses for several reasons (Vick 1997, Medway and Penney 1994, Martinez 1995, Kenwright 1997). One implication of this finding is that the widespread practice of recording only one, or the “main” reason for student withdrawal by colleges, officially sanctioned by FEFC (FEFC 1996 p4), misrepresents the student decision making process and gives a false picture of reasons for withdrawing (Martinez 1995, Hooper et al 1999, Kenwright 1996).

In terms of the reasons given by students for withdrawing the conclusions of a number of different studies are remarkably consistent.

Causes of dropout fall into three broad categories college-, work- and personal/family-related (Martinez 1995, Strefford 1999, Kenwright 1996, FEDA 1998, Adamson and McAleavy 2000, Davies et al 2000, CSET Lancaster University 1994, Bale 1990, BTEC 1993).

College Related Issues

Studies which limit themselves to surveys of withdrawn students can indicate the range of causative factors and can identify those factors that are college-related (Adamson and McAleavey 2000, Adamson et al 1998, Bannister 1996, Barrett 1996, Davies et al 2000, Freeman 2000, Gill 1998, North Tyneside college 1997, Hall and Marsh 1998, Longhurst 1999, Strefford 1999). Depending on the sample size and the degree of sophistication of the research design, this can produce valuable information and insights. However, the absence of control groups of completing and successful students can make it difficult to:

  • control and interpret attitudinal data derived from samples of withdrawn students
  • understand why the great majority of students complete their programmes
  • identify what is making the most difference to completion and withdrawal.

These issues have been addressed in a number of larger scale studies (Martinez 1997a, Martinez and Munday 1998, FEDA 1998, Davies 1999, Responsive College Unit 1998, Medway and Penney 1994, Kenwright 1997). These studies show that withdrawn students are most strongly differentiated from completing students by:

  • their evaluations of and attitudes towards college-related issues, and
  • lower levels of satisfaction with certain aspects of their experience of college.

Specifically, withdrawn students tend to be less satisfied than completing students with:

  • the suitability of their programme of study
  • the intrinsic interest of their course
  • timetabling issues
  • the overall quality of teaching
  • help and support received from teachers
  • help in preparing to move on to a job or higher qualification.

Withdrawn students are, moreover, much less willing to recommend the college to others (Martinez and Munday 1998, Martinez 1997a, Davies 1999, FEDA 1998, Kenwright 1997, Medway and Penney 1994).

The same studies demonstrate that withdrawn students are not strongly differentiated from completing students by:

  • the extent of their satisfaction with college facilities (canteen, toilets, classroom accommodation, equipment, library, workshop accommodation etc)
  • their personal circumstances (the incidence of personal, family or financial difficulties, their travel costs, and ease of their journey to college).

Further, the incidence of financial hardship does not seem to be strongly associated with decisions to drop out in order to gain employment (Martinez and Munday 1998 p 29). The Responsive College Unit with a sample of almost 6,000 students came to virtually identical conclusions using a longitudinal research design (1998). It found, in addition, that neither part-time work nor “external time commitments” correlated strongly with drop out.