GW Final Evaluation Report
Evaluation of the Gateway Pilot
Final Report
March 2003
March 2003
ISBN 0-478-08738-1
© Tertiary Education Commission Te Amorangi Mätauranga Matua
National Office
44 The Terrace
Wellington
New Zealand
PO Box 27-048
www.tec.govt.nz
Contents
List of Tables
Acknowledgements
Executive Summary
Introduction
Background
Delivery approaches
Evaluation of the Gateway Pilot
Findings - Quantitative
School participation
Delivery arrangements
Student participation
Participation by Mäori students
Participation by Pacific students
Industry placement areas
Student credit achievement
Post-Gateway student destinations
Findings - Qualitative
The schools
Significant enhancements in Gateway during 2002
Significant developments and issues
The students
Reasons for participating
Organising and preparing for work placement
Being in the workplace
Perceived benefits
Gateway graduates experiences
The employers
Reasons for participating
Perceived benefits
Satisfaction with services of the Gateway co-ordinator
Advice for potential Gateway employers
Future involvement
Gateway delivery approaches
Strengths of the models
The facilitated model
The brokerage model
The mixed model
Gateway funding
Allocation to schools
School expenditure of Gateway funds
Cost analysis
Discussion
School-industry relationships
Benefits for students
Students’ views
Schools’ views
Benefits for Mäori students
Benefits for Pacific students
Benefits for schools
The future - collaborative arrangements
Structured workplace learning - success factors
Whole-school approach
School focus and strategy
Flexible school structures
The co-ordinator role
Conclusion
Appendix 1: Gateway pilot schools
Appendix 2: Industry areas of Gateway placements
Appendix 3: Gateway expenditure areas
List of Tables
Table 1. Methods and data sources (2002 data)
Table 2. Pilot schools by size and decile
Table 3. Schools’ delivery arrangements
Table 4. Student age on entry to Gateway, by number (#) and proportion (%)
Table 5. Student self-identified ethnicity, by number (#) and proportion (%)
Table 6. Age of Mäori students on entry into Gateway, by number (#) and proportion (%)
Table 7. Age of Pacific students on entry into Gateway, by number (#) and proportion (%)
Table 8. Student credit achievement by National Qualifications Framework (NQF) level
Table 9. Average credit achievement by student ethnic group
Table 10. Actual credit achievement by student ethnic group
Table 11. Gateway placement outcomes, by number (#) and proportion (%)
Table 12. Motivations for participation
Table 13. Number of employers with whom students placed
Table 14. Benefits derived from Gateway
Table 15. Range of benefits identified by employers
Table 16. Services of Gateway co-ordinators
Table 17. Cost per credit
Acknowledgements
We acknowledge the contributions of students, teachers,
co-ordinators and brokers involved in the 24 Gateway pilot schools.
We also acknowledge those employers who participated in the Gateway Pilot and provided valuable information for this evaluation.
Executive Summary
- Gateway provides senior school students (years 11-13) with a range of structured learning opportunities in workplaces. These learning opportunities are integrated with students’ school-based studies.
- For some students, Gateway provided the opportunity to embark on study towards a national qualification in their career of choice. For others, it allowed more general skills to be developed and applied in a work context, and helped these students better understand the relevance of such skills to their classroom learning.
- Several features distinguished Gateway from other school to work initiatives.[1] These features included:
– A workplace-learning component incorporated into the student’s overall study programme.
– Work placement relevant to the student’s interests and aspirations, and integrated with the student’s study programme or vocational goals.
– Formalised learning arrangements for learning and work between the school, employer and student before the student entered the workplace.
– The workplace learning assessed against unit standards or achievement standards on the National Qualifications Framework.
– An individual learning plan developed for each student detailing the learning and unit standards to be achieved by the student in the workplace.
– Workplace learning assessment usually occurred in the workplace.
– Student achievement from their participation in Gateway measured through unit and achievement standards completed.
- This report is a summative evaluation of the Gateway Pilot.[2] An interim process evaluation was conducted at the end of 2001. [3] This report builds on the findings of the interim evaluation.
- The 2002 interim report was based on a 50% sampling of 2001 Gateway students (545) who participated in qualitative interviews, focus groups or written surveys. Data was also collected from surveys completed by 218 employers and qualitative interviews of school co-ordinators/teams and brokers. Quantitative data was drawn from labour market outcomes reports and schools’ Gateway information.
- For the summative evaluation (2002 data), 457 students (40% of the total participating in Gateway) were interviewed or surveyed. In addition, 237 employers and most principals, co-ordinators/teams, and brokers participated in either qualitative interviews or written surveys. Quantitative data was also drawn from the same sources used in the 2002 interim report.
- The Gateway pilot (2001-2002) involved 22 schools from 24 decile one to five state-funded secondary schools. The pilot schools were contracted by Skill New Zealand - Pükenga Aotearoa[4] to trial three different delivery approaches - the facilitated, brokerage, and mixed models (a combination of the facilitated and brokerage models).
- Overall, slightly more schools used the facilitated model in which the school developed and managed Gateway itself. In 2002, the number of schools using the mixed model increased significantly. This trend is expected to continue as schools strengthen their relationships with enterprises and industry. Only one school used the brokerage model, that is, the school contracted with a private training establishment to manage Gateway on its behalf.
- As only one school used the full brokerage model, it is not possible to make meaningful comparisons with the facilitated and mixed models. However, the strengths of each of the delivery models are identified and the features of an effective school-broker relationship are discussed in this report.
- Schools appear to have selected approaches that met their particular needs to deliver Gateway, rather than on any proven effectiveness of a particular approach. Schools who could and wanted to build their capability in the areas of school-industry education found the facilitated model valuable for this purpose. On the other hand, schools that wanted to build some internal capacity while benefiting from external well-established arrangements opted for the mixed model approach.
- At the end of December 2002, 2,170 students had participated in Gateway since the programme began in 2001. In its first pilot year, 1,008 students participated in Gateway. There was a 15.3% increase in student participation numbers in 2002. Sixty-five students (14%) participated in both years of the Pilot.
- During the Pilot, 16-year-olds (mostly year 12 students) made up the greatest proportion (46%) of students on entry to Gateway. The average age for Mäori Gateway students was 16 years and for Pacific students17 years. Päkehä students comprised 45% of the student participants, Mäori 36%, Pacific 14%, and other groups 5%.
- Students were placed on work placements in 50 industries, covering such diverse areas as information technology, boat building, early childhood, and hospitality. During the Pilot, the six most popular industries for placements were hospitality, automotive, retail, tourism, engineering, and building.
- Sixteen new industries were involved in 2002 including film, electronic media, marine, and health.
- The second year of the Pilot saw the profile and status of Gateway grow as shown by unsolicited student demand, waiting lists, and parents requesting a Gateway placement for their son or daughter.
- Schools built on the results from the first year of the Pilot and achieved a 60% increase in units and achievements standards in the second year. In 2002, students achieved 11,322 credits compared with 6,951 credits in 2001. Half of all credits attained by Gateway students during the Pilot were at Level 2 of the National Qualifications Framework.
- More students gained employment (261, up 5%) and further training opportunities (757, up 20%) after Gateway in 2002 than in the previous year. Eighty-six students were subsequently offered apprenticeship training.[5] Overall Mäori and Pacific students achieved higher average credit achievement than other ethnic groups.
- Follow-up interviews with 2001 Gateway graduates indicated that just over half students were continuing tertiary studies and were employed in the same areas as their Gateway placements.
- The positive employer support for Gateway identified in the interim report continued into the second year. The majority of employers involved in Gateway were from small enterprises (up to 49 employees).
- Employers reported that Gateway had provided them with opportunities to be exposed to potential employees and to gain experience in training. They also reported satisfaction in helping develop young people in their industry areas or towards employment generally.
- Employer support appeared more pronounced in provincial centres and smaller towns. The reasons for this were not explored specifically in the evaluation. Such support may be due to the higher profile of schools in smaller communities than in larger urban environments. This may engender more opportunities for community involvement to address concerns about local young people.
- The development of school-industry relationships as a result of Gateway was evidenced by a range of factors, including the size and scope of schools’ employer databases, the extension of schools’ accreditation scope, and the involvement of subject teachers in workplace learning and assessment. Schools also reported that positive relationships were developed with regional Industry Training Organisation (ITO) personnel, Modern Apprenticeships Co-ordinators, and industry assessors.
- Schools identified a wide range of benefits for students on Gateway activities. These benefits included educational gains as well as attitudinal, personal and career development outcomes. Examples of these benefits included students having broadened horizons, increased self-esteem and confidence, heightened aspirations, and more employment and tertiary training opportunities.
- Schools also identified a range of benefits for themselves. These included increased student retention and improved attendance, enhanced school profile and relationships with local communities, and increased student motivation for learning (as a result of greater relevance and integration with the overall school study programme).
- In 2002 more school staff became involved in Gateway. Schools reported increased buy-in from mainstream teaching staff because of the benefits they perceived for students. Subject teachers also increased their involvement in Gateway particularly through designing workplace learning and/or workplace assessment.
- Overall, students reported that Gateway had given them the opportunity to explore their industry and career interests as well as gain new skills and qualifications (unit or achievement standards).
- The benefits of Gateway participation most highly ranked by students were learning new skills, gaining work skills, gaining knowledge about career options, and improving motivation.
- The Gateway Pilot identified several factors contributing to successfully implementing a structured workplace learning programme in a school. These factors included a whole-school approach to implementation, a school focus and strategy that is externally focused and/or achievement-oriented, flexible school structures, and the pivotal nature of the co-ordinator role.
- The total budget for the two years of the Gateway Pilot was $4 million. There were six broad categories for Gateway expenditure: infrastructure establishment; on-going co-ordination costs; student-related costs; assessment costs; broker costs; and employer reimbursements. Ten percent of the total Pilot budget was allocated to the TEC for operational costs.
- Over the two years of the Pilot, the average activity cost per student was $1,658 (total contracted $ participation total) while the average cost per credit was $197 (total contracted $ total credits achieved). These averages (cost per student and cost per credit) decreased over the two years and further reductions are expected as delivery improves. In addition, cost efficiencies in clustering arrangements (for example, more schools working together in areas of common benefit) may lead to further reductions in overall cost.
- Pilot schools welcomed the 2002 Budget announcement to continue Gateway beyond the pilot phase and have enabled them to plan for Gateway in the longer term.
1
Final Report
Introduction
This report is a summative evaluation of the Gateway Pilot. An interim process evaluation was completed at the end of 2001. The summative evaluation builds on the findings of the interim evaluation.
The 22 pilots commenced at the beginning of 2001 and finished at the end of the 2002 school year. Twenty-four state-funded secondary schools[6] (from decile 1 to 5 schools) were contracted by Skill New Zealand - Pükenga Aotearoa to pilot Gateway.
Gateway was designed to test approaches to providing structured work-based learning for senior secondary students. The key objectives of Gateway were to:
- Provide opportunities for students to participate in work-based learning, including assessment and recognition of that learning, through the achievement of credits on the National Qualifications Framework (NQF).
- Integrate work-based learning with students’ wider courses of study.
- Test different arrangements for work-based learning, including different environments and different groups of students.
Background
Gateway provided senior secondary students (years 11-13) with a range of structured learning opportunities in workplaces. These learning opportunities were to be integrated with students’ school-based studies.
For some students, Gateway provided the opportunity for them to embark on studies towards a national qualification in their career of choice. For others, it allowed more general skills to be developed and applied in a work context and helped students to better understand the relevance of such skills and their classroom learning.
Current secondary school alternative education initiatives allow schools to organise a broad range of non-conventional programmes for their students. There are also several initiatives, for example the Secondary Tertiary Alignment Resource (STAR),[7] designed to strengthen school-business links and provide opportunities for learning beyond the traditional classroom.
In the Pilot’s first year, many schools experimented with Gateway provisions and some over-lap with similar initiatives was unavoidable. However, the differences between Gateway and similar initiatives have become more defined. By the end of the Pilot’s second year, schools seemed clearer about Gateway’s purpose and aims and its unique contributions to school-business education, compared with the first year.
The features distinguishing Gateway from similar initiatives included:
- A workplace-learning component incorporated into the student’s study programme.[8]
- Work placements relevant to students’ study programme or vocational goals and the workplace learning integrated accordingly.
- The school, employer, and student formalising the placement before the student entered the workplace.
- The workplace learning assessed against unit standards or achievement standards towards the National Certificate of Educational Achievement (NCEA) or other national certificates.
- An individual training plan developed for each student detailing the learning and unit standards to be achieved in the workplace.
- The workplace learning assessment usually occurring in the workplace.
- Student achievement from their participation in Gateway measured.
Schools were required to report on student achievement from their participation in Gateway, including:
- The achievement of credits towards the NCEA or other national certificates.
- Progress towards positive destinations in terms of further education, further training, or employment (reported three months after the end of the school year).
Delivery approaches
The Gateway Pilot was intended to trial three delivery approaches. In the facilitated model, the school developed and managed Gateway itself. In the brokerage model, the school contracted with one or more parties (named ‘‘brokers”) to develop and manage Gateway on its behalf.[9] In the mixed model, which was a combination of these two models, the school retained the management of Gateway while contracting with one or more brokers to manage one or more industry areas.
Evaluation of the Gateway Pilot
The evaluation of the Gateway Pilot involved an interim process evaluation at the end of 2001 and a summative evaluation at the end of 2002.
The evaluations’ overarching aims were to:
- Examine the benefits gained by students from their participation in Gateway.
- Identify the areas of strength and best practice for each Gateway model implemented.
- Examine the relationships between industry and schools developed through Gateway.
- Provide data for comparison with other arrangements for work-based learning available in schools, in particular STAR.
Key Findings from the Interim process evaluation
The key findings of the interim process evaluation were:
- There was a high level of endorsement for Gateway from all participants – students, schools, employers, and other stakeholders such as parents.
- The provision of integrated learning opportunities in the workplace for school students required:
– Flexible school structures and systems that could accommodate students moving from an internal learning environment to an external one, that is, the workplace.
– A teaching staff that valued the workplace as a learning environment and recognises its usefulness in reinforcing and extending school-based study.
– A whole-school approach led by the school’s senior management, supported by teaching staff, and integrated into the senior school.
– School capabilities to work with enterprise and an internal infrastructure for designing, monitoring, and assessing integrated school-work learning programmes.
- There was a significant level of support from employers for Gateway. Gateway appeared to be accessing previously untapped goodwill on the part of employers towards their local school and its students. Given that Gateway involved additional obligations and effort on the part of employers compared with work experience, such support may be regarded as particularly noteworthy.
Summative evaluation - methods and data sources
The methods and data sources used for the summative evaluation are summarised in Table 1.