Table of Contents
INTRODUCTION
EVALUTION OBJECTIVES
EVALUTION DESIGN
SAMPLE
OUTCOMES
TIMELINE
BASELINE STUDY OBJECTIVES
ADHERENCE TO ASSIGNED STANDARDS
BALANCE BETWEEN TYPE 1 & 2 SCHOOLS
STUDENT CHARACTERISTICS & KNOWLEDGE
TEACHER CHARACTERISTICS & KNOWLEDGE
SCHOOL INFRASTRUCTURE
SCHOOL FINANCING
TEACHER ATTENDANCE
PTA, SCHOOL MANAGEMENT COMMITTEES & SCHOOL IMPROVEMENT PLANS
CLASSROOM SIZE, ORGANIZATION & RESOURCES
TEACHERS: BACKGROUND, INCENTIVES, SUPPORT & MOTIVATION
PLACE OF BIRTH, RESIDENCE & MEANS OF TRANSPORT
TRAINING CONTENT, RELEVANCE AND PERCEPTION OF QUALITY
TEACHER KNOWLEDGE
TEACHER ATTENDANCE
SALARY
PEER SUPPORT, MENTORING & SUPERVISION
SATISFACTION & MOTIVATION
THE CLASSROOM ENVIRONMENT
STANDARDS
LEARNER ATTENDANCE
TEACHER BEHAVIOR
STUDENT BEHAVIOR & ENGAGEMENT
TIME ON TASK
INSTRUCTIONAL TIME PER YEAR
DETERMINANTS OF STUDENT PERFORMANCE
TABLES ANNEX
List of Figures
Figure 1 Relationship between GDP per capita & pupil to teacher ratio
Figure 2 Sample selection process
Figure 3 Map of sampled districts
Figure 4 Three measures of teacher absence, by district
Figure 5 Delays in salary payment by district
Figure 6 Anatomy of a class: distribution of tasks every 30 seconds
Figure 7 Expected number of instructional hours per student per year, by district
Figure 9 Returns to experience and education
List of Tables
Table 1 Comparison of IPTE & ODL training programs
Table 2 Description of instruments & datasets
Table 3 Percentage of teachers interviewed, by standard and teacher training type
Table 4 Distribution of ODL & IPTE-3 teachers by standard
Table 5 General school characteristics
Table 6 General staff characteristics
Table 7 Student enrollment
Table 8 Student characteristics, by intervention group
Table 9 Student & teacher proficiency, by intervention group
Table 10 Teacher proficiency, by training type
Table 11 Staff characteristics
Table 12 School infrastructure: access to electricity, water & sanitation
Table 13 School infrastructure: facility characteristics
Table 14 School resources: institutional financing
Table 15 School resources: in-kind contributions
Table 16 School resources: parent contributions
Table 17 Teacher attendance
Table 18 Presence and Role of PTA and School Management Committees
Table 19 School improvement plans & teacher perceptions of most important factor to drive quality improvements
Table 20 Reasons for teacher absence
Table 21 General characteristics of classrooms observed
Table 22 Learning environment & classroom resources
Table 23 Distribution of classroom time by task performed by teacher
Table 24 Background, contract type, experience & education
Table 25 Place of birth, residence & means of transportation
Table 26 Distribution of teachers by standard, expected duration teaching standard, teaching approach
Table 27 Teacher training: content, relevance and perception of quality
Table 28 Teacher absence
Table 29 Teacher salary
Table 30 Peer Support
Table 31 Supervision by head teacher
Table 32 Supervision by community
Table 33 Supervision by PEA
Table 34 Supervision by DEM
Table 35 Supervision by Inspectorate
Table 36 Mentoring
Table 37 Satisfaction with school assignment
Table 38 Teacher satisfaction
Table 39 Determinants of Teacher Satisfaction
Table 40 Classroom observations: general characteristics
Table 41 Classroom characteristics: Learner absence and tardiness, use of books and learner motivation
Table 42 Classroom observations: teacher engagement
Table 43 Classroom observations: teacher actions
Table 44 Distribution of classroom time, by task
Table 45 Determinants of English proficiency (z-scores)
Table 46 Determinants of Math proficiency (z-scores)
INTRODUCTION
Figure 1 Relationship between GDP per capita & pupil to teacher ratioSource: World Bank World Development Indicators, 2011
Under the Africa Program for Education Impact Evaluation (APEIE), the World Bank is financing a number of impact evaluations to build evidence of what works to improve education quality in Sub-Saharan Africa. The program also seeks to build in-country capacity to develop and implement rigorous evaluations of policy interventions aimed at improving schooling outcomes and particularly students’ learning achievement. Currently, the Ministry of Education, Science and Technology (MoEST) andthe World Bank are undertaking an impact evaluation of a distance training program in Malawi – the focus of this report.
Table 1 Comparison of IPTE & ODL training programsIPTE / ODL
Length of Program
- Teacher Training College (TTC) component
- In school (residential) component
[in rural schools selected to be training schools] / 2 years
[in rural schools from their community with acute need for qualified teachers]
Selection Criteria
- Education
1 credit in English
1 pass in Mathematics
1 pass in any science subject / MSCE:
1 pass in English
1 pass in Mathematics
1 pass in any science subject
- Origin
Allowance
- In TTC
- In School
Malawi has for many years suffered a chronic shortage of teachers, resulting in an extremely high pupil: teacher ratio (PTR) which stands at 89:1, one of the highest in the world (Figure 1). Furthermore, this figure masks large differences between rural and urban areas, where PTRs can range from 150:1 to 35:1, respectively. To provide an immediate solution to the critical shortage of teachers, the Ministry of Education, Science and Technology (MoEST) has developed an innovative Open Distance Learning (ODL) program providing training for student teachers in the schools where they are needed.
As opposed to conventional teacher training programs such as the Inter Primary Teacher Education (IPTE) program, the ODL program recruits teachers from districts or zones with an acute need for qualified primary teachers (based on above average pupil to qualified teacher ratios) and who wish to work in their district or zone of residence or birth. In addition, rather than spending a significant portion of their training at a college, ODL teachers primarily learn and develop skills “on-the-job” in the classroom, with the support from tutors who provide in-service training and mentorship. Table 1provides a more detailed comparison of the IPTE and ODL programs. Otherwise, the ODL programs follow the same curriculum as IPTE using manuals derived from IPTE manuals.
EVALUTION OBJECTIVES
The overall goal of this evaluation study is to assess the ability ofdistance training as implemented by ODL to improve the quality of education provided to underserved populations in Malawi. This will be achieved by comparing outcomes associated to new teachers trained through ODL and through conventional Teacher Training Colleges.
The evaluation will consider two primary research questions:
(1)What is the impact of using ODL, instead of conventionally trained teachers, on the educational attainment and performance of students in underserved populations in Malawi?
(2)How does the quality of teaching delivered by ODL and conventionally trained teachers compareover time? Since ODL teachers that are still in training will be compared to teachers having graduated from IPTE, does the relative performance of ODL teachers vary over time?
By providing a rigorous evidence base, this impact evaluation will be informative not only to Malawi, but will also provide lessons learned for other Sub-Saharan countries considering the use of ODL types of program to train teachers.
EVALUTION DESIGN
In order to estimate the impact of the ODL program on teaching quality and student learning outcomes, the evaluation pursued a randomized control design,assigning schools randomly to one of two modes of allocating new teachers to standards. In Type 1 schools, incoming ODL teachers are assigned to standards 3 and 5 (new IPTE teachers to 4 and 6) while in type 2 schools, incoming ODL teachers are assigned to standards 4 and 6 (new IPTE teachers to 3 and 5). In this way, comparisons of student outcomes in standards taught by ODL teachers in Type 1 schools with student outcomes in standards taught by IPTE teachers in Type 2 schools (and vice-versa) will provide the basis for estimating the effect of ODL teachers on student outcomes. Randomization helps ensure that these comparisons are valid so that the estimated program effect can be attributed to ODL teachers and not to differences in the characteristics of the students (e.g. skills, socioeconomic status) or schools (e.g. level of resources, community support, leadership). In order for the evaluation strategy to hold, in each academic year, the number of IPTE teachers assigned to study schools will approximately match the number of ODL teachers assigned to the same set of schools.
SAMPLE
Figure 2 Sample selection processParticipating schools were selected among 15 districts (out of a total of 34 in Malawi) with the highest pupil to qualified teacher ratio based on Education Management Information System (EMIS) data in 2009 (Figure 3). Within each district, schools where teachers from the latest cohort of both IPTE and ODL programs were teaching were chosen to take part in the study. Overall 425 schools were selected for the study representing over 1400 IPTE and ODL teachers (Figure 2). Selected schools were then randomly assigned to Type 1 and Type 2 modalities as explained above. Among the 425 sampled schools, only 6 (3 type 1 and 3 type 2) were not visited due to logistical constraints (inaccessible in reasonable amount of time, staff sit-in, and holiday).
Figure 3 Map of sampled districtsSource: (GADM 2012)
OUTCOMES
The main outcome of interest is whether the ODL training resulted in any improvement in teaching quality and pupil learning. Intermediate indicators are also measured, including enrollment, attendance and drop-out rates.
Learning is measured by learner performance on standardized tests that measure knowledge in basic competencies in Math and English. Teaching quality is assessed through surveys and classroom observations designed to measure specific characteristics of teaching behavior such as effort, attendance, knowledge, capacity and pedagogical approach as measure by time on task, interactions with students and use of materials.
TIMELINE
In May 2010, representatives of the MoEST, who attended the Africa Program for Education Impact Evaluation (APEIE) workshop in Ghana, proposed that a rigorous impact evaluation (IE) of the Malawi ODL program be carried out. The team worked with education and evaluation experts during the workshop to design the evaluation strategy. Sampling was conducted in September 2010 using available MoEST data regarding past IPTE and ODL teachers: EMIS data 2008, 2009, 2010, and District level assignments of IPTE cohorts 3 (having reported in schools around December 2009) and School-level assignment of ODL cohort 1 (having reported in schools in May 2010).
From October to December 2010, a series of head teacher meetings were held to inform authorities (Primary Education Advisers, District Education Managers of all 15 districts and head teachers of the 425 selected schools) participating in the study of the objectives and design of the impact evaluation study. In particular, head teachers were instructed on thestandards to which relevant teachers needed to be assigned. In general, both the head teachers, PEAs and DEMs reacted very positively to the study. This was corroborated by the fact that almost all attending head teachers were willing to think through and implement the necessary teacher’s reassignment following the moderator’s presentation.
Table 2 Description of instruments & datasetsInstrument preparation…// Pilot… // Instrument revisions based on pilot…
Baseline data used in this report was collected in October of 2010. As the evaluation study is designed to follow teachers through the full length of ODL training (in order to assess how ODL performance relative to IPTE teacher performance evolves over time), follow up data collection will beadministered every one or two yearsover six years around the end of the school year(in May or June) so that the study follows three intakes of ODL teachers (in 2010, 2011, 2012) throughout the entire course of their training. This characteristic of the study is important as ODL teachers are assigned to classrooms without complete training while IPTE teachers are fully trained when they are assigned to classrooms. A summary of the data collection timeline is presented inXXX and a description of the instruments and resulting data sets used in this study are presented in Table 2. Apart from teacher and learner tests mentioned previously, the baseline data collection consisted of interviews with head teachers, interviews with a subset of teachers in standards 3 through 6 and in-depth classroom observations. While all head teachers at the sampled schools were targeted for an interview, at each school only 1 teacher was interviewed per standard. The study’s first priority was to interview the ODL or IPTE-3 teacher that followed the study assignment, followed by an ODL or IPTE-3 teacher that did not adhere to their assignment, followed by any other teacher in that standard. Table 3 displays the percentage of teachers interviewed by standardand training type. Among ODL assigned standards (according to random assignment protocol), between 74 and 84 percent of teachers interviewed were ODL; among IPTE assigned standards, between 54 and 65 percent of teachers interviewed IPTE.
Table 3 Percentage of teachers interviewed, by standard and teacher training type
Standard / ODL / IPTE / Other3 / 80 / 56 / 36
4 / 74 / 54 / 32
5 / 83 / 65 / 48
6 / 84 / 58 / 38
Multiple / 100 / 98 / 44
BASELINE STUDY OBJECTIVES
The baseline study presented in this report has two major objectives:
(1)To assess the success of the randomization by comparing student, teacher and school characteristics using data collected for the baseline study between type 1 and type 2 schools.
(2)To assess the linkages between student, teacher and school characteristics with student learning outcomes.
ADHERENCE TO ASSIGNED STANDARDS
Table 4 Distribution of ODL & IPTE-3 teachers by standardType 1 / Type 2
Standard / ODL / IPTE-3 / ODL / IPTE-3
1 / 0.3 / 2.6 / 0 / 1.7
2 / 0.9 / 1.8 / 1.3 / 1.7
3 / 52.2 / 2.3 / 4.8 / 21
4 / 7.8 / 34.2 / 60.3 / 0.7
5 / 26.5 / 4.7 / 1.61 / 26.7
6 / 0.3 / 33.6 / 27.1 / 4.7
7 / 9.2 / 3.5 / 0.3 / 24.7
8 / 0.3 / 3.5 / 0.3 / 4.7
Multiple / 2.6 / 13.7 / 4.3 / 14.3
N / 347 / 342 / 373 / 300
Note: Does not include 2 observations with missing information regarding standards.
Non-adherence to assigned standards (for example, if a large percentage of ODL teachers end up teaching standard 4 instead of standard 3 in type 1 schools) could potentially bias impact estimates. In total, 722 new ODL teachers and 649 IPTE Cohort 3 (IPTE-3) teachers recently completingtraining were assigned to sample schools in 2010. The experimental design in theory called for 100 percent of ODL teachers to be assigned to standards 3 or 5 in type 1 schools and standards 4 or6 in type 2 schools[1]. As documented by the teacher roster, 88 percent of ODL teachers in both type 1 (with 348 ODL teachers) and type 2 (with 374 ODL teachers) adhered to expected assignments. With respect to IPTE cohort 3 graduates, the degree of adherence was lower: in type 1 schools (with 346 IPTE-3 teachers)76 percent adhered to assigned standards and in type 2 schools (with303 IPTE-3 teachers) 75 percent adhered to assigned standards. Three percent of ODL teachers were assigned to multiple standards compared to over 13 percent for IPTE-3 teachers. In practice, the fact that some ODL teachers are teaching standards other than 3 or 5 in Type 1 schools and standards 4 or 6 in Type 2 schools is expected given the potential imbalance of teachers and classrooms in a given school. For example, in a Type 1 school, if there is only one standard 3 and one standard 5 class but 3 ODL teachers, it would be ok for one of the ODL teachers to be in standard 1.
Taking this possibility into account, another “weaker” measure of non-adherence, is the percentage of IPTE-3 teachers teaching ODL assigned standards (12.4 percent) and the percentage of ODL teachers teaching IPTE-3 assigned standards (14.5 percent).
BALANCE BETWEEN TYPE 1 & 2 SCHOOLS
As schools were randomized to distribute observed and unobserved characteristics equally between type 1 and 2 schools and ensure unbiased estimates of the ODL program’s impact, in addition to describing school, staff and classroom organization and conditions, this section also assesses the extent to which the randomization was successful by looking at the balance of characteristics between type 1 and 2 schools.
On the whole, type 1 and 2 schools are well balanced. Among more than 300 variables for which balance was assessed (Table 5through Table 23) approximately 30(10 percent) were statistically different between the two groups. In terms of ownership, location and staff levels relative to enrollment, the groups areidentical on observed characteristics: approximately 62 percent are faith based, 28 percent are government-run and less than 1 percent are private for profit, the vast majority are in rural areas, have upwards of 130 pupils per classroom and 90 pupils per teacher, have between 10 and 11 teachers with vacancies for between 4 and 5 additional teachers[2](Table 5 and Table 6) enroll around 950 students (split almost equally between boys and girls) with the largest fraction of students (25 percent) enrolled in standard 1 and gradually falling to 6 percent enrollment in standard 8 (Table 7).
STUDENT CHARACTERISTICS & KNOWLEDGE
Background characteristics of students in standards 3 through 6 (Table 8) are also statistically indistinguishable between type 1 and 2 schools with the exception of gender and language spoken at home. Students in type 1 schools are slightly more likely to be male and more likely to speak Chichewa at home compared to students in type 2 schools. Given their small magnitude, it is unlikely these differences are educationally significant on average. In the sample overall, the average age of the students tested is 12.3 years (90 percent of students are over-age for their standard), have 5 siblings and overwhelmingly speak Chichewa at home (88 percent of students). About one in four students is repeating the same standard they were in last year (Table 8) – signaling significant gaps in student learning.
Student performance is alarmingly low in both English and Math as measured by the baseline student assessment(Table 9) the average percentage of correctly answered questions varies from a low of 12 to 34 percent. Students across all standards perform better in Math than in English. Standard 3 students in type 2 schools tend to perform worse in English and Math relative to type 1 schools. No statistically significant difference in testing performance is detected for students in other standards.
TEACHER CHARACTERISTICS & KNOWLEDGE
In terms of staff composition, background, experience and education (Table 11) type 1 schools have a larger percentage of female teachers (35 vs. 33 percent), teacher assistants (3.3 vs. 2.4 percent) and a lower percentage of staff with ODL training (14.5 vs. 16.5 percent). Despite these differences (that are relatively small in magnitude), type 1 and 2 schools are more similar than different: staff are 34 years old on average, teachers comprise 70 percent of all staff positions and 79 percent of staff have completed upper secondary education (less than 0.1 percent have started a university diploma). In terms of training,27 percent of staff completed MITEP, 29 percent completed or are currently in IPTE, 19 percent completed1 or 2 year programs, 15 percent are in ODL and 3 percent completed MASTEP.