30 Nymphe Street
Kensington
Johannesburg 2094
South Africa
Phone +27 (0) 11 622 7075
Fax +27 (0) 11 622 7589
Cell +27 (0) 83 255 0109
Eric Schollar and Associates c.c.
reg. no. CK 94/36640/23 VAT reg. no. 414 014 7069
THE ZENEX FOUNDATIONBASELINE STUDY OF THE EVALUATION OF THE PRIMARY MATHEMATICS RESEARCH PROJECT UNDER THE MANAGEMENT OF THE LIMPOPO DEPARTMENT OF EDUCATION IN VHEMBE DISTRICT
Eric Schollar
6 April 2011
CONTENTS
History and objectives of the PMRP in Limpopo ………………………………………………. / 1Transfer of the programme to the Limpopo Department of Education ………………………… / 1
Evaluation of the programme under departmental management ……………………………….. / 3
Findings: Action-research: Development and outcomes of the transfer process ………………. / 4
Findings: Status of the programme in schools at June 2010 …………………………………… / 9
Impact evaluation study design ………………………………………………………………… / 11
Longitudinal tracking of the field trial cohort: 2007 to 2010 …………………………………... / 14
Baseline data: 2010 ……………………………………………………………………………... / 16
Years two and three of the evaluation …………………………………………………………. / 19
Appendix One: Establishing the formal status of the PMRP in Limpopo ……………………… / 20
Appendix Two: Steering Committee: Agenda for handover meeting and participants ………… / 21
Appendix Three: Programme notes for management and quality support group ………………. / 23
Appendix Four: Monitoring information by circuit and school ………………………………... / 35
Appendix Five: Numbers of schools and learners tested by circuit ……………………………. / 36
Appendix Six: Item analysis of instruments ……………………………………………………. / 39
Appendix Seven: Raw scores and percentages by circuit ……………………………………… / 41
ESA
Baseline Study of the Evaluation of the PMRP under Departmental Management
1. History and Objectives of the PMRP in Limpopo
The Primary Mathematics Research Project was initiated in 2004, with the endorsement of the National Department of Education. Its brief was (1) to research the causes of the persistence of very poor outcomes in mathematics in South African primary schools - despite the introduction of the new curriculum in 1998 and the ever increasing allocation of state revenue to education, and (b) to develop a developmental programme in response to these findings. The Limpopo Department of Education provided its support in 2007 to the field testing of the programme developed by the PMRP, locating it as a research project of the department within the provincial Learner Attainment Strategy under the direction of the Senior General Manager: Curriculum.
Subsequent to the significant improvements in learner performance measured during the field trial, ESA was invited in November 2008 to present a proposal to the Executive Management Meeting of the LDoE for the piloting and researching of the most effective available methods of transferring the operational management of the PMRP to departmental officials. The underlying intention was to examine the feasibility of potential large-scale replication of the PMRP programme in the province and, as such, the PMRP remained a research project of the department within the Learner Attainment Strategy under the direction of the SGM.
The two essential objectives of the programme that resulted from this proposal are to:
v maintain the same, or similar, levels of impact on learner performance as those measured in the 20 schools in the project group during the field test on a much larger scale – i.e. in 125 schools.
v transfer the routine operation and management of the programme to the Limpopo Department of Education over the first one to two cycles of programme implementation, training and mentoring without significant loss of programme quality and, hence, lower levels of impact on learner performance.
The proposal also provided for a partnership between the LDoE, ESA and private sector agencies in researching and piloting a workable transfer strategy:
v The first cycle of delivery of the PMRP would be implemented and managed by ESA through funding from the external partners – in this case, the Zenex Foundation, the Anglo American Chairman’s Fund and XSTRATA South Africa. The external funding would provide for training of departmental officials, SMT members and teachers, as well as a complete supply of Teachers Manuals, Learner Workbooks and Diagnostic Tests for each participating school; Grades 4 to 6.
v At the end of each first cycle of delivery, operational management of the programme would be handed over to the Limpopo Department of Education – and the department would supply the Learner Workbooks for incoming Grade 4 classes required to sustain the programme during delivery Cycles Two and Three.
2. Transfer of the Programme to the Limpopo Department of Education
Transfer of the operational management of the PMRP programme was completed by December 2011 and the PMRP is currently operating in the whole of the Malamulele Cluster in the Vhembe District of Limpopo – in Malamulele North East, East, West, Central and Vhumbedzi Circuits. Delivery in 2010 commenced on 14 February with the delivery and distribution of Learner Workbooks and Teacher Manuals, training commenced on 18 February, and school support/mentoring visits on 7 March. Participating in the programme there are:
v 125 schools
v 370 teachers
v Over 20 000 learners
v 5 Circuit Managers
v 125 Principals/SMT members
v 3 Vhembe District Mathematics Advisors
v 5 Circuit PMRP Teacher Committees
v 1 Area PMRP Teacher/SMT Committee
The original transfer strategy centred on the department’s district advisory service as the logical institutional locus-of-control but this proved unworkable; there were only three GET mathematics coordinators for the 27 circuits and 692 schools in Vhembe! Nonetheless, all three District Curriculum Advisors - one of whom is the Vhembe GET Mathematics Coordinator and the DoE PMRP Project Manager, as well as the District FFLC Coordinator – were trained and carried out school support and monitoring visits during the first and second terms of 2011. One has since retired and has not yet been replaced.
The adapted transfer strategy was shifted to focus on local-level structures and individuals to provide operational management after the withdrawal of ESA. These structures and individuals were identified and developed by the departmental Project Manager and by ESA 2010 and 2011; ‘formal’ handovers to these structures were made in October 2010. The programme is currently managed by a three-tier structure:
v The Vhembe Steering Committee comprised of the District Curriculum Coordinators, Circuit Managers and a Principal/AMESA representative. See Appendix Two for the Agenda of the institutional transfer meeting on 5 October 2010, along with a list of the members. This is primary an ‘enabling’ structure.
v The PMRP Area Committee comprised of two Principal/SMT members and two teacher members from each of the five circuits, together with the circuit managers and the departmental Project Manager of the PMRP. This is primarily a representative forum and communication structure.
v A Management and Quality Support Sub-committee of the Area Committee comprised of two leader-Principal/SMT members and two Key-Teacher members from each of the five circuits. This is primarily a programme management and support structure. See Appendix Three for the training notes provided to this sub-committee at the operational transfer meeting on 27 October.
2.1. Relationship of the PMRP to Other Departmental Programmes
The PMRP programme is conceptually and operationally integrated with national and provincial programmes.
It is effectively a successor programme to Khanyisa in Vhembe, especially in terms of the Limpopo Numeracy Strategy, and much of its design was influenced by ESA’s participation in and review of Khanyisa.
With regard to the Foundations for Learning Campaign, the PMRP programme supports the fundamental objectives of the FFLC by providing a solution to the universal problem of ‘multi-grade’ classes in the great majority of our schools in which the actual ability level of many learners can be two, three or even four grade levels below the minimum expected. It provides a ‘catch-up’ programme starting with diagnostic testing to support the acquisition of competencies learners have failed to grasp in earlier grades. Only once learners have mastered missed content can they be expected to successfully deal with the correct Assessment Standards for their grade level – the ‘core’ objective of the FFLC.
Dr. Sambo of the LDoE has suggested the introduction of the programme to Dinaletsana schools. Preliminary discussions have started and ESA has offered to provide support to the implementation of the programme in these schools/circuits using the same transfer model currently applied.
3. Evaluation of the PMRP under Departmental Management
The Zenex Foundation approved a proposal for the evaluation of the PMRP and its associated transfer strategy commencing in Term Three of 2010. The summative quantitative impact component of the study is based on longitudinal tracking of the 2010 Grade 4 classes in all 125 schools from entry into the programme to exit at Grade 6 – the data will provide very reliable information about nature and degree of impact on learner performance during the period that the programme is operated by the department itself. This is a potentially ground-breaking study in South African mathematics education and institutional development.
The degree of impact on learner performance will be dependent on the coherence, maintenance and effectiveness of programme delivery in classrooms by the current institutional (departmental) arrangements.
Figure 1: Relationship between strategic approach and delivery system in achieving impacts
Strategy on which intervention is basedEffective / Ineffective
Delivery of intervention / Effective / Predicted impact likely / Predicted impact unlikely
Ineffective / Predicted impact limited / Predicted impact unlikely
The overall process or operational objective of any intervention is to ensure that BOTH the strategic/theoretical underpinning AND the delivery of the intervention are effective. Many of the larger scale interventions in recent years have shared the idea of a systemic intervention design operating, to one degree or the other, across all levels of the school system; province, district, circuit, school, classroom and community. IMBEWU in the Eastern Cape was one of the earliest and KHANYISA in Limpopo one of the most recent – EQUIP of the NBI, is another example, as are the DDSP and IEP of USAID.
In all of these interventions, the provincial departments of education lacked the capacity to manage and deliver the intervention design without external assistance and extensive use was made of external agencies, especially NGOs, to provide field staff to the programme. Attempts to build capacity in education departments during the period of intervention delivery were almost always based on the training and mentoring of officials by the external ‘co-facilitators’ – as they were termed in Khanyisa. It was assumed that these departmental officials would participate consistently, learn ‘on-the-job’ and be able to assume operational management of the programme once the intervention itself came to an end and the external facilitators withdrew. They would also learn how to maintain and improve the quality of the delivery of the programme at classroom level. In the event, this proved much more difficult to achieve than expected. Instead, officials and teachers alike tended to regard the interventions as limited-term NGO ‘projects’ that had a discrete life. External NGO facilitators were generally left, in practice, to deliver the programme on their own but typically lacked the authority required to really establish coherent and organized programmes of instruction in schools. The consequence is that most of these interventions failed to achieve systemic change in ‘lodging’ the programme within the routine operations of the departments, schools or teachers concerned. This is not to say that none of these interventions achieved change, or that no lasting effects can be discerned, but to recognize that enduring systemic and organizational change was not achieved as intended by the programme design.
This has been the central operational challenge for the PMRP since its inception in Limpopo. Consequently, and at the express instruction of the SGM: Curriculum, the programme has always been accompanied by a strong emphasis on action-research; to explore, identify and develop the capacity of an education department to establish as a routine element of its regular functioning, an intervention programme without the need for long-term external support. ESA field researchers have visited all of the schools/teachers three times in the original pilot districts (Malamulele North East, East and Central) and twice in the 2010 circuits (Malamulele West and Vhumbedzi); 323 school visits over 2009 and 2010. At each visit, the SMT was interviewed as well as all of the Grade 4 to 6 teachers; lessons were observed on each visit. Regular contact with Circuit Managers (some are more ‘active’ than others) has been maintained by ESA and the Project Manager, along with key SMT members and teachers as they emerged. The early meetings of the Area Committee focussed on participants themselves defining the processes they need to develop; the emergence of the Management and Quality Support sub-committee is the best example of the result.
Also significant is that the Area Committee established an annual Mental Maths Competition in support of Foundations for Learning for all PMRP schools in 2010. The members of the Quality Support sub-committee set the test instruments, organized the event, helped coordinate transport and so on.
The culmination of the action-research process was the development of a set of constraints to transfer, together with a proposed protocol for dealing with them, that was submitted to the SGM: Curriculum, as well as the Steering and Area Committees. Also developed was a monitoring system aimed at providing basic programme information to management; its initial focus has been to place each of the schools in one of three categories. Both of these developments are reported in the following section.
This section makes extensive use of the reports that ESA submitted to the department, and to the ZF, at the end of Phase IV (the transfer phase) in June 2010. To illustrate the ‘sequential’ and developmental nature of the operational process with which they were dealing, tenses are left as they were in the originals. Combined with the monitoring information, these reports constitute the formative, contextual, managerial, programmatic and qualitative components of the evaluation.
3.1. Findings: Action-research: Development and Outcomes of the Transfer Process
3.1.1. Constraints to Transfer in June 2010
There have been five major constraints to the overall objective of the transfer of the operational management of the PMRP to the Limpopo Department of Education.