Document of
The World Bank
Report No:ICR00003977
IMPLEMENTATION COMPLETION AND RESULTS REPORT(IBRD-81260)
ON A
LOAN
IN THE AMOUNT OF US$40 MILLION
TO THE
REPUBLIC OF CHILE
FOR A
TERTIARY EDUCATION FINANCE FOR RESULTS PROJECT III
APRIL 27, 2017
Education Global Practice
Latin America and the Caribbean Region
CURRENCY EQUIVALENTS
April 19, 2017
Currency Unit = Chilean Peso
CLP646 = US$1
FISCAL YEAR
January 1 – December 31
ABBREVIATIONS AND ACRONYMS
CPS / Country Partnership StrategyCRUCH / National Council of Rectors of Chilean Universities
DFI / Department of Institutional Financing
DIVESUP / Division of Higher Education
GoC / Government of Chile
ICT / Information Communications and Technology
IP / Professional Institute
IPPF / Indigenous Peoples Planning Framework
MECESUP (3) / Tertiary Education Finance for Results Project (III)
MINEDUC / Ministry of Education
NPV / Net Present Value
OECD / Organization for Economic Cooperation and Development
OP/BP / Operational Policies and Bank Procedures
PA / Performance Agreement
PAD / Project Appraisal Document
PDO / Project Development Objective
PM / Improvement Plan or Small Project (Plan de Mejoramiento)
PMI / Institutional Improvement Plan (Plan de Mejoramiento Institucional)
RF / Results Framework
R&D / Research and Development
SIES / Tertiary Education Information System
TEI / Tertiary Education Institution
TTC / Technical Training Center
WBG / World Bank Group
Senior Global Practice Director:
/Jaime Saavedra Chanduvi
Practice Manager:
/Reema Nayar
Project Team Leader:
/Diego Ambasz
ICR Team Leader:
/Diego Ambasz
CHILETERTIARY EDUCATION FINANCE FOR RESULTS PROJECT III
CONTENTS
Data Sheet
A. Basic Information
B. Key Dates
C. Ratings Summary
D. Sector and Theme Codes
E. Bank Staff
F. Results Framework Analysis
G. Ratings of Project Performance in ISRs
H. Restructuring
I. Disbursement Graph
1. Project Context, Development Objectives and Design
2. Key Factors Affecting Implementation and Outcomes
3. Assessment of Outcomes
4. Assessment of Risk to Development Outcome
6. Lessons Learned
7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners
Annex 1. Project Costs and Financing
Annex 2. Outputs by Component
Annex 3. Economic and Financial Analysis
Annex 4. Bank Lending and Implementation Support/Supervision Processes
Annex 5. Beneficiary Survey Results
Annex 6. Stakeholder Workshop Report and Results
Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR
Annex 8. Comments of Co-financiers and Other Partners/Stakeholders
Annex 9. List of Supporting Documents
Annex 10. Evaluation of Overall Outcomes Using a Split Ratings Approach (Pre- and Post-Restructuring)
Annex 11. The Causal Chain for the MECESUP 3 Project
MAP
1
Data Sheet
A. Basic Information
Country: / Chile / Project Name: / Tertiary Education Finance for Results Project IIIProject ID: / P111661 / L/C/TF Number(s): / IBRD-81260
ICR Date: / 09/08/2016 / ICR Type: / Core ICR
Lending Instrument: / SIL / Borrower: / GOVERNMENT OF CHILE
Original Total Commitment: / USD 40.00M / Disbursed Amount: / USD 40.00M
Revised Amount: / USD 40.00M
Environmental Category: C
Implementing Agencies:
Division of Higher Education (DIVESUP), Ministry of Education
Co-financiers and Other External Partners: N/A
B. Key Dates
Process / Date / Process / Original Date / Revised/Actual Date(s)Concept Review: / 06/21/2011 / Effectiveness: / 06/20/2013[1]
Appraisal: / 11/29/2011 / Restructuring(s): / 09/16/14 and 07/13/2015
Approval: / 03/13/2012 / Mid-term Review: / 03/16/2015 / 03/16/2015
Closing: / 10/31/2016 / 10/31/2016
C. Ratings Summary
C.1 Performance Rating by ICROutcomes: / Moderately Satisfactory
Risk to Development Outcome: / Low or Negligible
Bank Performance: / Moderately Satisfactory
Borrower Performance: / Moderately Satisfactory
C.2 Detailed Ratings of Bank and Borrower Performance by ICR
Bank / Ratings / Borrower / Ratings
Quality at Entry: / Moderately Satisfactory / Government: / Moderately Satisfactory
Quality of Supervision: / Moderately Satisfactory / Implementing Agency/Agencies: / Satisfactory
Overall Bank Performance: / Moderately Satisfactory / Overall Borrower Performance: / Moderately Satisfactory
C.3 Quality at Entry and Implementation Performance Indicators
Implementation Performance / Indicators / QAG Assessments (if any) / Rating
Potential Problem Project at any time (Yes/No): / No / Quality at Entry (QEA): / None
Problem Project at any time (Yes/No): / No / Quality of Supervision (QSA): / None
DO rating before Closing/Inactive status: / Satisfactory
D. Sector and Theme Codes
Original / ActualSector Code (as % of total Bank financing)
Public administration – Education / 5 / 5
Tertiary education / 95 / 95
Theme Code (as % of total Bank financing)
Education for the knowledge economy / 100 / 100
E. Bank Staff
Positions / At ICR / At ApprovalVice President: / Jorge Familiar / Hasan A. Tuluy
Country Director: / Alberto Rodríguez / Susan G. Goldmark
Practice Manager/
Manager: / Reema Nayar / Chingboon Lee
Project Team Leader: / Diego Ambasz / Michael Crawford
ICR Team Leader: / Diego Ambasz
ICR Primary Author: / Nicole Mammoser
F. Results Framework Analysis
Project Development Objectives (from Project Appraisal Document)
The Project Development Objective was to improve quality and relevance for students in tertiary education by strengthening the link between funding of tertiary education institutions and accountability for performance.
Revised Project Development Objectives (as approved by original approving authority)No revisions were made to the PDO.
PDO Indicators
Indicator / Baseline Value / Original Target Values (from approval documents) / Formally Revised Target Values / Actual Value Achieved at Completion or Target Years
Indicator 1 / Number of full-time equivalent (> or = 44 hours/week) faculty members who hold PhDs at all accredited Tertiary Education Institutions (TEIs)
Value (quantitative or qualitative) / Original: 4,148
Revised: 5,109 / 4,900
(at 12/31/2015) / 7,500
(at 10/31/2016) / 8,332
Date achieved / 5/15/2011 (Revised) / 12/31/2011 / 12/31/2015 / 10/31/2016
Comments (inc. % achievement) / Original target surpassed; 344% of target achieved. (Uses actual value at Restructuring #1: 6,734.)
Revised target surpassed; 135% of target achieved.
Indicator is calculated as the sum of total working hours of academics holding a PhD divided by 44 (the number of hours in a full-time work week).At Project close, the number of full-time equivalent faculty members who hold PhDs at TEIs supported by the MECESUP 3 Project account for 95% of the actual value achieved, or 7,915 faculty members.
Baseline and target values were revised at the Project’s first Restructuring on September 16, 2014 to reflect more recent and complete data in the Tertiary Education Information System database (SIES).
Indicator 2 / Retention rate measured as first-year undergraduate students that remain at the institution in the second year
Value (quantitative or qualitative) / 68.4% / 72% / 71.3%
Date achieved / 4/30/2011 / N/A / 10/31/2016
Comments (inc. % achievement) / Target partially achieved; 80.6% of target achieved.
Indicator is calculated as first-year students in Year t who continue to study at the same institution in Year t + 1, divided by the first year students in Year t, multiplied by 100. Indicator considers students in programs at Technical Training Centers that are 4 semesters or longer, students in programs at Professional Institutes that are 6 semesters or longer, and students in university bachelor’s degree programs that are 8 semesters or longer.
Note that the first-year retention rate increases to 73.8% (surpassing the 72% target) for TEIs that received support from the MECESUP 3 Project, compared to 67.4% for TEIs that did not receive Project support. Further, the retention rate increases to 79.5% when taking into account first-year students who transfer to a different TEI to pursue their second year of study (i.e., who are retained in the system at a different TEI).
Indicator 3 / Percentage of students in teaching degrees with redesigned curricula, including practical training, at TEIs with Performance Agreements
Value (quantitative or qualitative) / N/A / 25% / 70.5%
Date achieved / N/A / 12/31/2015 / 10/31/2016
Comments (inc. % achievement) / Target surpassed;282% of target achieved.
This PDO Indicator was introduced at the July 2015 Restructuring in order to better reflect improvements in relevance for students enrolled in teacher training programs at TEIs implementing Teacher Training Improvement Institutional Improvement Plans (PMI).[2]
Indicator is calculated as the number of students pursuing teaching degrees in programs with redesigned curriculathat include a teaching practicum at TEIs implementing a PMI divided by the number of students pursuing teaching degrees at TEIs implementing a PMI.
A very high percentage of students are studying in teacher training programs that have added a teaching practicum to their curriculum (70.5% versus the targeted 25%). This is due to the fact that the majority of TEIs that implemented a Teacher Training Improvement PMI included a practicum requirement for all matriculated students, not just those entering as first-year students once a PMI was underway.
Indicator 4 / Percentage of students in technical and professional degrees[3] with redesigned curricula at TEIs with Performance Agreements
Value (quantitative or qualitative) / N/A / 50% / 89.4%
Date achieved / N/A / 12/31/2015 / 10/31/2016
Comments (inc. % achievement) / Target surpassed;179% of target achieved.
This PDO Indicator was introduced at the July 2015 Restructuring to better reflect improvements in relevance for students enrolled in technical and professional degree programs at TEIs implementing Technical/Professional Training Improvement PMIs.
Indicator is calculated as the number of students pursuing technical and professional degrees in programs with redesigned curricula at TEIs implementing a PMI divided by the number of students pursuing technical and professional degrees at TEIs implementing a PMI.
A very high percentage of students are studying in technical or professional degree programs that have redesigned their curricula (89.4% versus the targeted 50%). This is due to the fact that the majority of TEIs that implemented a Technical/Professional Training Improvement PMI integrated enhanced curricula into the programs being pursued by all matriculated students, not just those entering as first-year students once a PMI was underway.
(b) Intermediate Outcome Indicators
Indicator / Baseline Value / Original Target Values (from approval documents) / Formally Revised Target Values / Actual Value Achieved at Completion or Target Years
Indicator 1 / Number of signed Performance Agreements(supporting both PMs and PMIs)
Value (quantitative or qualitative) / N/A / For Teacher Training: 12;
for Academic Innovation: 12;
for Technical/ Professional Training: 10;
for Small Projects(PMs): 24
(58 total Performance Agreements) / For Teacher Training: 6;
for Academic Innovation: 6;
for Technical/ Professional Training: 5;
for Small Projects(PMs): 12
(29 total Performance Agreements) / For Teacher Training: 19;
for Academic Innovation: 24;
for Technical/ Professional Training: 17;
for Small Projects(PMs): 119
(179 total Performance Agreements[4])
Date achieved / N/A / 12/31/2013 / 12/31/2015 / 12/31/2015
Comments (inc. % achievement) / Original target surpassed; 309% of target achieved.
Revised target surpassed: 617% of target achieved.
Due to incomplete data at Project appraisal in early 2012, target values for signed Performance Agreements by project category were only set through 2013. Target values were revised at the September 2014 Restructuring (through 12/31/2015) to reflect more detailed information collected subsequent to appraisal.
The total number of signed Performance Agreements significantly exceeded the revised target for this indicator (179 versus the target of 29), due to the large number of Performance Agreements for Improvement Plans (PMs, referred to in the PAD as “Small Projects”) signed (119 versus the target of 12). Revised targets for signed Performance Agreements for the three PMI categories were exceeded as well: 19 for Teacher Training Improvement, 24 for Academic Innovation, and 17 for Technical/Professional Training Improvement.
Indicator 2 / Rate of fulfillment of indicators included in Performance Agreements
Value (quantitative or qualitative) / N/A / N/A / 70% / 71.3%
Date achieved / N/A / N/A / 10/31/2016 / 10/31/2016
Comments (inc. % achievement) / Target surpassed;102% of target achieved.
In mid-2014, data for Performance Agreement indicatorsshowed a 55% fulfillment rate. As a result, the target value for the fulfillment rate of Performance Agreement indicators at Project end was set at 70% at the first Restructuring in September 2014.
The 71.3% fulfillment rate for the indicators included in all Performance Agreements incorporates both Performance Agreements that had completed implementation and those that were still in execution at 12/31/2016.
Indicator is the average percentage of fulfillment of indicators included in Performance Agreements. It is calculated as the sum of percentages of fulfillment of Performance Agreement indicators, divided by the number of indicators. (Each indicator can range from 0 to 100%.)
Indicator 3
(Core Indicator) / Direct Project beneficiaries (number), of which female (percentage)
Value (quantitative or qualitative) / N/A / N/A / Set at
September 2014 Restructuring:
275,000 beneficiaries/
51% female
Revised at
July 2015 Restructuring:
280,000 beneficiaries/
51% female / 279,883 beneficiaries /
51% female
Date achieved / N/A / N/A / 10/31/2016 / 10/31/2016
Comments (inc. % achievement) / September 2014 target surpassed; 102% of target achieved.
July 2015 revised target substantially achieved; 99% of target achieved.
Due to incomplete data at Project appraisal in early 2012, baseline and target values were not set. The target value was set at the September 2014 Restructuring to reflect more detailed information collected subsequent to appraisal.
The target value for the number of Project beneficiaries was revised at the July 2015 Restructuring to reflect the fact that a higher number of Performance Agreements were signed by TEIs than had been anticipated at the September 2014 Restructuring. Although the Project fell just 117 beneficiaries short of the revised target, it met the goal of 51% female Project beneficiaries. This is due to the high number of females impacted by the PMs and PMIs underway in 2014 and 2015, many of which focused on teacher training efforts that predominantly affected women.
Beneficiaries included students, academics and administrators at TEIs that implemented at least one Performance Agreement.
Indicator 4 / Number of students completing remediation programs in TEIs with Performance Agreements
Value (quantitative or qualitative) / Original: N/A
Set at
September 2014 Restructuring:
33,925 / N/A / Set at
September 2014 Restructuring:
50,000
(at 10/31/2016)
Revised at
July 2015 Restructuring: 65,000
(at 10/31/2016) / 69,497
Date achieved / 12/31/2011 / N/A / 10/31/2016 / 10/31/2016
Comments (inc. % achievement) / September 2014 target surpassed; 221% of target achieved.
July 2015 revised target surpassed; 114% of target achieved.
Due to incomplete data at Project appraisal in early 2012, baseline and target values were not set. The baseline and target values were set at the September 2014 Restructuring to reflect more detailed information collected subsequent to appraisal, and the end target was revised at the second restructuring in July 2015.
Note that 30,503 students completed remediation programs during the same period at TEIs that did not implement Performance Agreements, which is less than half the number of students benefitting from remediation programs at TEIs with Performance Agreements.
Indicator 5 / Number of graduates from domestic PhD programs supported by the Project
Value (quantitative or qualitative) / Original: 433
Revised: 103 / 530
(at 12/31/2015) / 200
(at 10/31/2016) / Original: 685
Revised: 208
Date achieved / 12/31/2011
(Revised) / N/A / 10/31/2016 / 10/31/2016
Comments (inc. % achievement) / Original target surpassed; 260% of target achieved.
Revised target surpassed; 108% of target achieved.
Indicator’s name and its baseline and target values were revised at the September 2014 Restructuring to reflect only graduates of PhD programs at TEIs supported by the MECESUP 3 Project(not PhD graduates of all accredited TEIs).The Project funded a total of 12 PMs and PMIs that supported domestic PhD programs.
Indicator 6 / Percentage of teaching degrees with redesigned curricula in TEIs with Performance Agreements
Value (quantitative or qualitative) / N/A / 60% / 71.2%
Date achieved / N/A / 8/9/2016 / 10/31/2016
Comments (inc. % achievement) / Target surpassed; 119% of target achieved.
This indicator was introduced at the July 2015 Restructuring to reflect progress toward PDO Indicator 3. This indicator monitored the redesign of teacher training curricula in TEIs with Performance Agreements.
Teacher training curricula redesign incorporated input from MINEDUC on weaknesses in primary and secondary learning outcomes, as well as from external consultants who advised TEIs on curricula integration of current international best practices for classroom teaching across various subjects and grade levels.
Indicator 7 / Number of technical and professional degrees with redesigned curricula in TEIs with Performance Agreements
Value (quantitative or qualitative) / N/A / 150
(at 10/31/2016) / 202
Date achieved / N/A / 10/31/2016 / 10/31/2016
Comments (inc. % achievement) / Target surpassed; 135% of target achieved.
This indicator was introduced at the July 2015 Restructuring to reflect progress toward PDO Indicator 4. This indicator monitored the redesign of technical and professional training program curricula in TEIs with Performance Agreements.
Technical and professional curricula redesign incorporated input from consultation on labor market needs with private sector and civil society representatives, in alignment with international best practices.
Indicator 8 / Number of policy notes and studies carried out
Value (quantitative or qualitative) / 0 / 3
(at 12/31/2015) / 3
(at 10/31/2016) / 3
Date achieved / 12/31/2011 / 10/31/2016 / 10/31/2016 / 10/31/2016
Comments (inc. % achievement) / Original target not achieved; 33% of target achieved.
Revised target achieved; 100% of target achieved.
The date for this indicator to be achieved was revised at the July 2015 Restructuring. While one policy note was completed at year-end 2015, the timelines for both the Project’s studies were pushed out to Project close in 2016. This indicator is cumulative, so the total number of policy notes and studies carried out during the Project remains three.
This indicator pertains to Component 2 of the Project (Policy Support and Project Management).
G. Ratings of Project Performance in ISRs
No. / Date ISRArchived / DO / IP / Actual Disbursements
(USD millions)
1 / 06/24/2012 / Satisfactory / Satisfactory / 0.00
2 / 02/04/2013 / Satisfactory / Satisfactory / 0.00
3 / 12/07/2013 / Satisfactory / Moderately Satisfactory / 0.00
4 / 06/29/2014 / Satisfactory / Moderately Satisfactory / 13.27
5 / 01/05/2015 / Satisfactory / Moderately Satisfactory / 13.27
6 / 08/05/2015 / Satisfactory / Moderately Satisfactory / 13.27
7 / 01/28/2016 / Satisfactory / Moderately Satisfactory / 33.34
8 / 08/09/2016 / Satisfactory / Moderately Satisfactory / 33.34
9 / 10/09/2016 / Satisfactory / Moderately Satisfactory / 39.49
H. Restructuring (if any)
Restructuring Date(s) / Board Approved PDO Change / ISR Ratings at Restructuring / Amount Disbursed at Restructuring in USD millions / Reason for Restructuring & Key Changes MadeDO / IP
09/16/2014 / No / S / MS / 13.27 / The September 2014 Restructuring mainly amended the Results Framework with updated data that was not available at Project Appraisal in early 2012. Disbursement estimates were also corrected to reflect delays to Project effectiveness and implementation.
07/13/2015 / No / S / MS / 13.27 / The July 2015 Restructuring amended the Results Framework to increase the coherence of the Project’s results chain based on a review of Performance Agreements under implementation.
I. Disbursement Profile
1. Project Context, Development Objectives and Design
1.1 Context at Appraisal
- Country background. At the time of project appraisal for MECESUP 3 during late 2011 and early 2012, Chile was an upper middle income country that had recently acceded to full membership in the OECD. It was a politically stable country with strong institutions and a solid record of economic growth during the previous two decades. The Government that took office in March 2010 was pursuing goals similar to those of previous governments: promoting reforms to boost growth, investing in human capital, and tackling poverty and inequality.[5] Within this context, Chile was being challenged to reverse declining relative growth performance[6] and insufficient innovation capacity in order to grow the size and wealth of its middle class.
- In early 2012, the Government of Chile (GoC) was pursuing a development agenda aimed at achieving high income status by 2018. Areas of strategic emphasis to reach this goal included improving job creation and job quality and investing in human capital. Increased policy attention to education at all levels was recognized as a key driver of economic and social progress. And with approximately one million students in tertiary education,[7]the Government simultaneously faced an emboldened student population impatient for improved educational quality and opportunity.[8]
- The World Bank Group’s (WBG) Country Partnership Strategy (CPS) with Chile for the period FY2011-2016 identified three areas where Bank activities would be carried out, one of which was job creation and equity improvement.[9] The Bank pledged to support Government efforts that focused on improving the quality of education as a means tofoster job opportunities, develop labor skills, and enhance productivity.