Targets, Travelling Hopefully and True Success

Paper presented at the British Educational Research Association Annual Conference, University of Warwick, 6-9 September 2006

Targets, travelling hopefully and true success? Assessment practice in an initiative aimed at disengaged young people

David James and Jonathan Simmons, University of the West of England, Bristol.

To travel hopefully is a better thing than to arrive, and the true success is to labour (Robert Louis Stevenson El Dorado 1881)

Address for correspondence:

David James

Faculty of Education

University of the West of England, Bristol

Frenchay Campus

BRISTOL BS16 1QY

Tel 0117 328 4215

Email
Introduction

In addition to celebrating a range of improvements in primary and secondary schooling, the recent UK Government White Paper 14-19 Education and Skills repeats a familiar lament when it comes to post-compulsory education:

Numbers staying on post-16 have improved but are still too low – far down the international league table. Many employers are not satisfied with the basic skills of school leavers going directly into jobs. Some young people drift outside education, employment or training between the ages of 16 and 19. The most able young people are not as fully stretched as they could be.’ (Department for Education and Skills, 2005, p. 4)

The White Paper mentions the global interconnectedness of modern economies, and, less convincingly, the relationship between levels of qualification and ‘productivity’ (signalled by gross weekly earnings for individuals), and that between truancy and offending behaviour. It also mentions the low credibility and status of vocational qualifications and the lack of a clear identity for vocational qualifications. It offers a ‘significant programme of change’ spread over the period 2005 to 2015 (p. 93) and a series of new proposals to tackle the identified problems in the English context. GCSE and A levels are to be retained as ‘cornerstones of the new system’ (p. 6). New ‘specialised Diplomas’ will be made available at levels 1, 2 and 3, and employers will be ‘put in the lead’ through Sector Skills Councils. The ‘functional elements’ of English and maths will gain a new emphasis. GCSE achievement and attainment tables will be ‘toughened’ so that the ‘Diploma standard’ becomes the norm (i.e. the measure will become the percentage of young people achieving 5 A* to C grades including English and maths). These and other changes will be underpinned by ‘A sharp accountability framework’ (p. 86).

Part of the document concerns itself with ‘engaging all young people’. The headline promise is to ‘…provide more opportunities and incentives for teenagers who have not achieved level 2 by 16 to do so post-16 and support them in achieving level 1 or entry level qualifications as steps on the way’ (p. 6). The rationale is as follows:

The economic and social costs of young people being in the NEET group are high and they are the young people who we most need to re-engage in education and training. At the end of 2003, around 9% or 16-18 year olds were estimated to be NEET at 16, 17 and 18 – with a further 4% in the group at two out of three of these ages…Part of the solution will be to offer these young people qualifications and a curriculum that they want to pursue post-16…’ (p. 67)

This paper is principally concerned with (a) the nature of the ‘offer’ of qualifications and curriculum that young people who are ‘disengaged’[i] with regard to the processes and/or values of mainstream schooling, both within but also outside the ‘NEET’ category, may want to pursue, and (b) the arrangements under which this can work. What form can this take? How should the learning and assessment be conceived? What role can targets play in realising this goal? A number of insights with a bearing on these questions can be taken from the evaluation of a development project located mainly in Bristol, a local authority at the bottom of the GCSE league tables[ii]. Entitled Increasing Participation and Attainment levels of 14-19 Year Olds in the West of England, the project commenced in October 2002 and concluded in August 2004, with its main activity in the 17-month period January 2003 to May 2004. It was funded by the Learning and Skills Council (West of England)[iii] and was managed and operated by the Award Scheme Development and Accreditation Network (ASDAN), an educational charity[iv]. The project had one overall aim, which was to ‘make a significant contribution to the LSC strategic objective of enabling 85% of young people in the region to attain a Level 2 qualification by the age of 19’. It also had six objectives, closely linked to this aim, namely:

• To have run staff training programmes for all schools, colleges, off-site provision and training providers (including employers offering on the job training) in the West of England region. These will focus on the full range of ASDAN curriculum programmes (FE and Youth Awards, Universities Award, Foundation for Work Award, Foundation Training Award, Youth Achievement Awards), and other appropriate materials which have a track record of boosting achievement;
• To have helped staff implement and deliver appropriate programmes in their respective establishments;
• To have helped staff make appropriate links between these programmes and relevant level 1 and 2 qualifications (key skills, career planning, community volunteering);
• To have enabled staff to manage the assessment requirements associated with these qualifications;
• To have enabled the target group of young people to achieve a level 2 qualification;
• To have evaluated the impact on staying on rates and achievement levels.

It is important to note at this point that ASDAN materials and processes represent a particular type of educational provision, with its roots in the curriculum development model promoted by the Schools Council (1964 – date**), in ‘mode 3’ (teacher-devised) public examinations in secondary education, and in a ‘social and community’ interpretation of the Training and Vocational Educational Initiative of the early 1980s. It has at times been declared to be in opposition to centralised control of the curriculum (see Crombie-White, 1995; 1996). It was developed by three consortia of teachers (in Devon, Avon and Berkshire) in the late 1980s and grew rapidly during the early 1990s, both in terms of numbers of learners registered and in the range of programmes and qualifications available. It now serves some 4,500 centres, and its qualifications are approved by the Secretary of State under Section 96 of the Learning and Skills Act 2000. Several key reviews (including the Dearing review of 16-19 qualifications, the Tomlinson review and the 14-19 White Paper) have referred in positive terms to the capacity of ASDAN programmes and qualifications to recognise and reward a broader range of achievements than most mainstream provision. ASDAN declares that the common features of its programmes and qualifications are that:

• They are learner centred, offering opportunities for a negotiated curriculum which is modular and activity based
• They encourage candidates to develop responsibility for their own learning through a process of action planning and review
• They assist in the personal and social development of the individual through a focus on Key Skills for which the QCA (Qualifications Curriculum Authority) national standards provide a template for assessment and accreditation
• They recognise achievement across the school/college curriculum, as well as in the home, the community and the world of work (ASDAN, 2006)

The Qualifications and Curriculum Authority describe the essence of the process as follows:

ASDAN awards are structured around modules, which are based on topics or areas of interest. Each module comprises a challenge or series of challenges. Challenges are activity-based and involve a task or number of tasks. Students are required to produce evidence of completion. The work is moderated and verified internally at a centre (a school or college) and externally by an ASDAN regional coordinator. (QCA, 2006).

Whilst they do depend on a form of criterion referencing, ASDAN programmes and qualifications embody a particular construction of achievement that departs from mainstream assessment regimes in the extent to which it is continually professionally reconstructed. This means it is unlikely ever to enjoy the status of GCSEs of A levels (see Torrance et al, 2005, pp 10-15).

The scope and methods of the evaluation

The development project was evaluated by the University of the West of England, Bristol and a report produced in September 2004 (James with Hamilton, 2004). Subsequently, ASDAN and the LSC commissioned a small-scale qualitative research study as an extension to the evaluation, focussing on a small number of providers and learners in order to develop a more intimate understanding of practices and the nature and consequences of engagement with ASDAN activity, materials and processes. This took place between November 2004 and April 2005 and a report was produced in May 2005 (James and Simmons, 2005). Taken together, the main evaluation and its extension utilised a combination of qualitative and quantitative methods, drawing upon:

• Survey forms from 27 providers, detailing as far as possible attainments, attendance, reading age, exclusions, outcomes and a range of other information for over 600 young people, in some cases offering ‘baseline’ measurements against which project outcomes could then be compared;
• Observation forms from 25 providers, offering a range of experience related to the project but with a particular focus on changes observed in young people during their association with it;
• Fifteen in-depth interviews with key staff across a sample of providers. These were chosen to reflect the range of providers – school-based, college-based, youth service, private trainers and other schemes;

• Six telephone interviews with key staff to explore specific points.

• Interviews with 29 participating young people located with 15 of the providers;
• Perusal of documents, including a range of materials generated by ASDAN, some generated by individual staff, student portfolios and other work, general correspondence, and LSC documentation;
• Observation of three ASDAN workshops for providers;

• Attendance at four project review meetings.

Whilst independent of ASDAN, the evaluation relied in part on data gathered by ASDAN itself in monitoring its work for the specific development project. The design of the evaluation offered some triangulation of qualitative data (for example, an event, process or change could be separately described by a provider and a young person, or more than one young person, in more or less congruent ways, or could be checked against a researcher’s observations or a document). However, the quantitative elements inevitably relied on key staff in each provider gaining access to secondary data, with very limited scope for verification by ASDAN staff and by the independent evaluation. The variation in the capacity of individuals – even in schools - to supply information on prior attainments, or attendance for each of the learners within the project was much greater than expected. Furthermore, for some of the non-school providers it was a matter of principle to disregard most information about individual learner history, because they wished to offer the young people in their care a ‘clean slate’. This meant that there were many gaps in the quantitative data.

A target and other learning outcomes

In keeping with the tenets of the ‘democratic’ (MacDonald, 1987), and ‘case study’ (House, 1980; Stake, 1980, 1994) approaches in evaluation, the work focused on activities as much as on intentions. Accordingly, there were two categories of ‘main findings’ within the evaluation. The first related directly to the stated aims and objectives of the development project, and the extent to which these were fulfilled, whilst also pointing to an area of ambiguity amongst them. The second took the form of commentary on what else might be learnt from the evaluation. This included both substantive and methodological issues.

Main findings in relation to the aims and objectives

As was stated earlier, the project set out to ‘make a significant contribution to the LSC strategic objective of enabling 85% of young people in the region to attain a Level 2 qualification by the age of 19’. It encompassed 616 young people working within 25 active providers, of whom 17 were secondary schools (the others were a mixture of established and new training providers of various kinds, a Further Education College, a youth service, and a pupil referral unit). All providers were operating in Bristol or within a few miles of the city.

Taking the 25 providers as a whole, the end of the project period saw learners gaining 145 Entry-level outcomes, 88 Level-1 outcomes, and 7 Level-2 outcomes. In most cases there was evidence that these outcomes would not have been attained without the existence of the project. With 23 of the 25 providers there was also clear evidence of other significant outcomes for young people. These outcomes included raised levels of confidence, marked positive changes in social skills, and other positive spin-offs for study and life. With most providers, there were also clear examples of successful progression for young people. In 9 of the 25 providers there was particularly strong evidence of progression.

There is not the space here to go into detail with regard to all the objectives of the development project, though it is worth mentioning that they were largely judged to have been met. It is worth noting that there was near-unanimity amongst both learners and participating staff in their assessment of the strengths of ASDAN materials and systems. These were valued most highly for:

• their flexibility (particularly in terms of pace and timing);
• their compatibility with a range of curricula and pedagogic practices;
• their capacity to tap in to the interests of learners;
• their capacity to instill organised study habits;
• the sense of achievement they made possible;
• the development of confidence, independence and self-esteem that they seemed to help foster in young people;
• the relatively clear opportunities they offered for progression.

The characteristics of ASDAN materials and mechanisms were pivotal in the success of the development project and were valued by many practitioners. However, the evaluation also came to the view that it was important not to see ASDAN materials in a technocratic way, as a free-standing ‘fix’ or panacea. Contextual factors like professional values, continuity of staffing and positive institutional support were also crucial to the successes with learners. For some staff, there was also a clear sense that ASDAN materials and systems provided them with an important source of legitimation for working in ways that were a little at odds with dominant systems and assumptions. For these staff, ASDAN materials and systems had an affinity with core professional values and a sense of autonomy that other areas of their practice did not fully acknowledge or realise. There were interesting similarities here with the external sources professional sustenance discussed by Colley et al in relation to staff in Further Education colleges (Colley et al, forthcoming).

For the present discussion, it is worth pausing on the fifth and sixth objectives of the project. The fifth was ‘to have enabled the target group of young people to achieve a level 2 qualification’. The degree of success with this objective depended on its precise meaning. If it was taken to mean the completion of a recognized level 2 qualification within the project period, then the project (with only 7 [a mere 1%!] of its 616 initial participants in this category) must be judged to have failed. If, however, the objective was taken in its wider sense (i.e. with the emphasis on to have enabled), then the project represented a considerable success. Given that the over-arching aim specified ‘by the age of 19’, and that the majority of the young people involved had some years to go before they reached this age, the evaluation report argued that this latter interpretation was, by far, the more reasonable one.

The sixth objective was ‘to have evaluated the impact on staying on rates and achievement levels’. As noted, there was good evidence of progression in 9 of the 25 providers, and some evidence in most of them. Highly successful participation, of which clear progression was a feature, depended upon:

• particular professional values on the part of teachers or key staff;
• the quality and nature of relationships formed with young people;
• some predictability in the regularity and continuity of contact between staff and young people;
• a positive institutional acknowledgement of the work;
• clear location of responsibility for the work within an establishment;
• continuity of staffing.

The evaluation was persistent in trying to weigh up the extent to which certain processes and outcomes were generated by the project and would have been unlikely to have happened without it. The weight of evidence pointed clearly to overall success with the fifth objective in its broader meaning. Whilst the data from providers showed that attrition was high, there was no doubt that the project had a positive impact on the achievements and prospects of hundreds of young people, not least in terms of learning how to operate successfully in a framework that offers accreditation. A proportion of this positive impact was visible in the completion of awards that were themselves significant building blocks towards level 2 outcomes.