Document of
The World Bank
Report No:ICR00004042
IMPLEMENTATION COMPLETION AND RESULTS REPORT (ICR)ON A
SMALL GRANT[1]
IN THE AMOUNT OF US$2.67 MILLION
TO THE
REPUBLIC OF ARMENIA
FOR
Strengthening the LivelihoodS and Voice of Poor and
Vulnerable PERSONS in Armenia project
February 15, 2017
Social Protection and LaborGlobal Practice
Europe and Central Asia Region
1
CURRENCY EQUIVALENTS
(Exchange Rate Effective: January 1, 2017)
Current Unit / = / Armenian Dram (AMD)AMD 487 / = / US$1
FISCAL YEAR
January 1 – December 31
ABBREVIATIONS AND ACRONYMS
DRC / Disability Resource CenterFM / Financial Management
ICR / Implementation Completion and Results Report
ISR / Implementation Status and Results Report
JSDF / Japan Social Development Fund
MLSA / Ministry of Labor and Social Affairs
MA / Mission Armenia
M&E / Monitoring and Evaluation
NGO / Nongovernmental Organization
OTJ / On-the-Job
OM / Operational Manual
PDO / Project Development Objective
PSC / Project Steering Committee
PWD / Person with Disabilities
SC / Steering Committee
SEA / State Employment Agency
Regional Vice President: / Cyril E. Muller
Country Director: / Mercy MiyangTembon
Senior Global Practice Director: / Michal Rutkowski
Practice Manager: / Andrew D. Mason
Project Team Leader: / Marina Petrovic
ICR Team Leader: / Maddalena Honorati
1
REPUBLIC OFARMENIAStrengthening the Livelihood and Voice of Poor and Vulnerable People in Armenia
TABLE OF CONTENTS
DATA SHEET
A. Basic Information
B. Key Dates
C. Ratings Summary
D. Results Indicators
I. Project Development Objectives and Design
II.Key Factors Affecting Implementation
III. Assessment of Outcomes
IV. Risk to Sustainability
V. Lessons Learned
ANNEXES
Annex 1. Team Composition
Annex 2. Implementing Agency Background
Annex 3. Outputs by Component
Map: IBRD 33364
1
DATA SHEET
A. Basic Information
Region: / Europe and Central Asia / Country(ies): / ArmeniaGlobal Practice: / Social Protection and Labor / Joint with IFC / No
Recipient: / Mission Armenia, on behalf of the Government of the Republic of Armenia / Implementing Agencies: / Mission Armenia
Sector(s)/Theme(s): / 100% Social Protection
Theme Code (as % of total Bank financing)
Income Support for Old Age, Disability & Survivorship / 30
Other social protection and risk management / 30
Participation and civic engagement / 20
Social Safety Nets/Social Assistance & Social Care Services / 20
Executed by: / Recipient
Project ID: / P125791 / TF Name and Number(s): / TF013894
Original Grant Amount: / US$2,670,000 / Donor(s): / Japan
Revised Amount: / US$2,670,000 / Disbursed Amount: / US$2,670,000
Environmental Category: / C / Safeguards Triggered: / n.a.
B. Key Dates
Process / Date / Process / DateConcept Review: / 17-Dec-2012 / Effectiveness: / 15-Mar-2013
Appraisal: / Restructuring(s): / 19-Oct-2015
Approval: / 01-Mar-2013 / Closing: / 01-Mar-2016
Revised Closing Date: (if any) / 01-Sep-2016
C. Ratings Summary
Outcomes: / SatisfactoryImplementation: / Moderately Satisfactory
Risk to Sustainability: / Moderate
(a) Project Development Objective Indicators[2]
D. Results Indicators
Indicator / Baseline Value / Original Target Values(from approval documents) / Formally Revised Target Values / Actual Value Achieved at Completion or Target Years
Indicator 1: / Number of graduates in sustained wage employment (Number, Custom)
Value
(Quantitative or
Qualitative) / 0 / 200 / — / 203
Date achieved / May 17, 2013 / March 1, 2016 / — / August 31, 2016
Comments
(including %
achievement) / Target surpassed.
Indicator 2: / Number of businesses in sustained operation (Number, Custom)
Value
(Quantitative or
Qualitative) / 0 / 40 / — / 90
Date achieved / May 17, 2013 / March 1, 2016 / — / August 31, 2016
Comments
(including %
achievement) / Target surpassed.
Indicator 3: / Increase of number of individuals helped by Mission Armeniato get benefits (Number, Custom)
Value
(Quantitative or
Qualitative) / 88 / 1,000 / — / 1,239
Date achieved / September 30, 2013 / March 1, 2013 / — / August 31, 2016
Comments
(including %
achievement) / Target surpassed.
Indicator 4: / Retaining or increasing capacity to independently carry-out activities of daily living (Percentage, Custom)
Value
(Quantitative or
Qualitative) / 60 / 70 / — / 85
Date achieved / September 30, 2013 / March 1, 2016 / — / August 31, 2016
Comments
(including %
achievement) / Target surpassed. This indicator is clarified in the Operational Manual (OM) as: ‘Beneficiaries retain or improve their well-being, as measured by (a) Morbidity per month; (b) Increase of capacity to independently carry-out activities of daily living; and (c) Increase of number of individuals helped by Mission Armenia (MA) to get benefits’.
Indicator 5: / Community financing (Percentage, Custom)
Value
(Quantitative or
Qualitative) / 10 / 11 / — / 12
Date achieved / May 17, 2013 / March 1, 2016 / — / August 31, 2016
Comments
(including %
achievement) / Target surpassed.Community financing is defined as contributions from local governments, individuals, and from other nongovernmental organizations (NGOs).
Indicator 6: / Visits to project-supported government facilities (Number, Custom)
Value
(Quantitative or
Qualitative) / 0 / n.a. / — / 19,666
Date achieved / May 17, 2013 / March 1, 2016 / — / August 31, 2016
Comments
(including %
achievement) / This indicator was linked to rehabilitation of infrastructure, which was ongoing till the project closing date. Number is an estimate.
(b) Intermediate Outcome Indicator(s)
Indicator / Baseline Value / Original Target Values
(from approval documents) / Formally Revised Target Values / Actual Value Achieved at Completion or Target Years
Indicator 1: / Number of beneficiaries. (Number, Custom)
Value
(Quantitative
or Qualitative) / 2,100 / 4,000 / — / 4,856
Date achieved / September 30, 2013 / March 1, 2016 / — / August 31, 2016
Comments
(including %
achievement) / Target surpassed.
I. Project Development Objectives and Design
1. Original Project Development Objectives (PDO) and Key Indicators
1.The approval for this Japan Social Development Fund (JSDF)-funded project did not include the drafting of a separate Project Paper. Therefore, this Implementation Completion and Results Report (ICR) evaluation uses the PDO statement from the Grant Agreement (GA), Operational Manual (OM), and Implementation Status and Results Reports (ISRs): To pilot self-reliant livelihoods and generation of employment opportunities combined with the provision of vital services for extremely poor and vulnerable groups, including persons with disabilities, in Armenia.
2.An initial draft set of indicators was included in the Grant Proposal. Later on, these were elaborated and approved as part of the OM for the project and began to be tracked in the ISRs. For this completion evaluation, thelist of key indicators from the ISRs (shown below) will be used. These are considered consistent with the Legal Agreement and with the other project documents.
(a)Number of graduates in sustained wage employment
(b)Number of businesses in sustained operation
(c)Increase of number of individuals helped by Mission Armenia to get benefits
(d)Retaining or increasing capacity to independently carry-out activities of daily living
(e)Community financing
(f)Visits to project-supported government facilities.
3.The indicators from the ISR deviate slightly from the OM indicators. The OM indicators are thus also reviewed in the ICR, as additional evidence. Table 1compares the outcome indicators from the ISRs with the indicators from the OM.
Table 1. Results Framework - Comparison between Indicators in the ISRs and OM
Outcome Indicators from ISR / Target / Outcome Indicator from OM / Target / AchievementNumber of graduates in sustained wage employment (Number, Custom) / 200 / At least 200 of the 700 graduates of training courses are still in wageemployment three months after they were placed in jobs. / 200/700 / 203 graduates in wage employment
In addition:
- about 800 unemployed poor and vulnerable people start training. About 70 percentare person with disabilities (PWDs) and about 30 percentare other vulnerable people;
- about 550 of these poor and vulnerable people graduate from training; and
- out of the 800 people starting training, about 150 unemployed poor people and PWDs receive on-the-job (OTJ) training.
Number of businesses in sustained operation (Number, Custom) / 30 / At least 40 of the 80 new businesses are still in operation three months after start-up. / 40/80 / 90 businesses in operation
In addition, out of the approximately 700 graduates of vocational training, about 170 graduates receive additional training to start micro-businesses.
Retaining or increasing capacity to independently carry-out activities of daily living (Percentage, Custom) / 70 / Beneficiaries retain or improve their well-being, as measured by
- morbidity per month,
- increased capacity to independently carryout activities of daily living, and
- increased number of individuals helped by Mission Armenia (MA) to get benefits.
In addition,
- 15 multifunctional service centers that deliver services to poor and vulnerable people, including PWDs, were rehabilitated; and
- social workers help 80 percentof beneficiaries gain social benefits, participate in public programs, acquire disability devices (such as wheelchairs and hearing aids), receive public documents, obtain social assistance, and exercise entitlements (such as free medical care) and social rights.
Increase of number of individuals helped by Mission Armenia to get benefits (Number, Custom) / 1,000 / — / — / 1,239
Community financing (Percentage, Custom) / 11 / Community financing of services to these beneficiaries rises by at least 10 percentover the 2.5 years following effectiveness of the grant. / 10 / 12
In addition,
- 4,000 beneficiaries receive community-based social and health services, and all 4,000 beneficiaries receive social services. Of these, 3,000 receive health services, 500 bedridden beneficiaries receive services at home, and 2,050 receive hot mealsevery day.
Visits to project-supported government facilities (Number, Custom) / 10,000 / 10 percentincrease in visits by PWDs and poor people to government offices: (a) rehabilitated by the project and (b) where the project delivers service. (Percentage) / 10% / 19,666
PWDsreceive more information that helps them make decisions about their future. / No target / No data on number of beneficiaries.
12 public sites are remodeled.
Poor and vulnerable people, including PWDs, increase their participation in local government and public events in the 15 communities. / No target / About 500 poor PWDs and other poor and vulnerable people receive training to advocate for themselves and help themselves (from final quarterly report).
Intermediate Indicators from ISR / Target / Intermediate Indicators from OM / Target / Achievement
Number of beneficiaries (Number, Custom) / 4,000 / — / — / 4,856
2.Main Beneficiaries
4.The project aimed to support about 4,000 poor PWDs and other extremely poor people who live in 15 urban locations in six Armenian provinces. The beneficiaries included the following sub-groups:
- 800 receiving vocational and on-the-job training and micro-business opportunities
- 4,000 receiving vital social, health care, and nutrition services
- 700 with access to information and advocacy education
5.The beneficiaries were selected based on specific criteria decided and agreed by the Project Steering Committee (PSC), which includedthe Ministry of Labor and Social Affairs (MLSA), disabled people’s organizations, and local government representatives among others (see section 3). The criteria are described in the OM and include: (a) poverty status; (b) disability status; (c) gender equality;and (d) trainee’s willingness to contribute to the training fee as a proof of commitment and motivation to participate. Additional criteria were used to select beneficiaries for Components 2 and 3.
6.According to the OM, the definition of poverty and extreme poverty was aligned with Armenia’s National Statistical Service, which at the time of preparation defined as poor those with consumption per adult equivalent below the poverty line (AMD 33,517 per month) and as extreme poor those with consumption per adult equivalent below the food extreme poverty line (AMD 19,126 per month). MA’s project staff home visits were used as a way to verify beneficiaries’ eligibility.[3]
7.The OM, together with the Financial Management (FM) Manual, describes all procedures that were to be applied by the MA, anon-governmental organization(NGO), during project implementation.
8.In parallel, theMA—the recipient and implementing agency—benefits from seed financing and capacity-building opportunities.
II.Key Factors Affecting Implementation
Project Cost
(a) Project Cost by Component (in US$,million equivalent)
Components / Appraisal Estimate (US$) / Actual/Latest Estimate (US$) / Percentage of Appraisal1. Provision of Vocational Training and Boosting Self-Employment / 887,430 / 883,162 / 99.52
2. Provision of Vital Social Care and Services / 600,560 / 592,081 / 98.59
3. Creation of an Inclusive Environment for People with Disabilities and Enhancement of their Self-advocacy / 591,060 / 584,651 / 98,92
4. Monitoring, Evaluation, and Project Management / 590,350 / 581,763 / 98,55
Total Project Costs / 2,669,400a / 2,641,657b / 98.96
Note: (a) the column ‘Appraisal Estimate’ shows the component costs as given in the Grant Proposal. The arithmetic total of Components 1, 2, 3, and 4 is US$2,643,800. However, the grant total is US$2,669,400. This may be due to a technical error in the Grant Proposal; and (b) this total number is from the World Bank system after final reconciliation and is slightly higher than the exact sum of the component costs as reported by the recipient at closing (US$2,641,746).
(b) Financing
Source of Funds / Appraisal Estimate(US$) / Actual/Latest Estimate
(US$) / Percentage of Appraisal
IBRD Grant / 2,669,400 / 2,641,657 / 98.96
9.The original Grant Proposal mentions an expected government contribution of about US$1.7 million ‘to support the proposed activities’. The Government’s commitment was reportedly met by the spending of US$1.6 million in the form of salaries for MA local staff, office utilities, and other nonrecurring costs.[4]
Original Components
10.The grant funding was provided to MA for the Strengthening the Livelihoods and Voice of Poor and Vulnerable Persons in ArmeniaProject. This three-year project sought “to pilot self-reliant livelihoods and generation of employment opportunities combined with the provision of vital services for the extremely poor and vulnerable groups, including persons with disabilities, in Armenia.” About 4,000 extremely poor persons, particularly,PWDs of different age and gender groups in six Armenian provinces (Yerevan, Shirak, Lori, Ararat, Kotayk, and Gegharkunik), were expected to benefit from project activities under its four components.
11.The project had four components as originally designed.
12.Vocational and Self-employment Training.Vocational trainingwas delivered in the form of class-based and OTJ training to the extremely poor and vulnerable groups, including PWDs. The self-employment sub-component was aimed at boosting business creation by providing technical (business training) and financial support (small sub-grants) to help selected graduates of the vocational training jump-start micro-business andincome-generating activities. Activities under Component 1 were delivered by different providers.
- MA: Vocational training courses on specific trades[5]
- Other NGOs, private firms and colleges, on-the-job training, or public institutions: training courses and micro-business opportunities
13.Vital Social and Health Services.The project supported the operation of community-based social and healthcare services for the extremely poor and vulnerable, including homebound and disabled people. MA delivers basic services to PWDs and other poor and vulnerable groups in their communities rather than at traditional large residential institutions.Under Component 2 of the project, it was intended to deliver social services, such as counseling, to about 4,000 beneficiaries, primary health services to about 3,000 of the beneficiaries, hot meals daily to about 2,050 of the beneficiaries, and healthcare home services to about 500 bedridden beneficiaries.
14.Access to Information and Advocacy Education.The project supported the establishment of disability resource centers (DRCs), improvement of environmental and communication accessibility for PWD, and empowerment of the beneficiaries with self-advocacy skills to independently enforce their rights and privileges.Component 3 was intended to helpPWDs become more independent, address, and resolve some of their own difficulties.To achieve this, the component supported the following activities:
- Better access to public facilities and to public information
(a)Establishment of 12 DRCs by renovating and remodeling parts of multifunctional service centers (day care centers)for the dissemination of information to PWD and others
(b)Remodeling of 12 public buildings so that PWD could gain access; obtain information; and apply for documents, services, and benefits
- Advocacy education
(a)Skills training in advocacy and self-help
(b)Creation of self-help groups
15.Monitoring, Evaluation, and Project Management.The MA was responsible for the monitoring and evaluation (M&E) of the project and for reporting on the indicators to the PSC and to the World Bank. Elements of the M&E system included: (a) regular field visits by the MAto ensure that services, goods, and works were actually delivered and the quality of services, such as vocational training, and monitor contractors’ respect for the terms of their contracts with MA; (b) the creation and update of a database to track progress of the project, including information on inputs, outputs, and outcome indicators; (c) the preparation and administration of questionnaires on beneficiary achievement, satisfaction, and proposals for improvement of training and social services; (d) MA social workers’ reports on a sample of beneficiaries of health and social care services; (e)quarterly progress reports;and (f) a baseline and final evaluation reportout-sourced to external consultants.
3.Summary of Implementation and Significant Changes
Rating: Moderately Satisfactory
16.The MA, a national NGO, approached the World Bank for assistance in mobilizing funds for a project for vulnerable people, PWDs, and other extremely poor people.To support this request, the Ministry of Finance of Armenia asked the World Bank to apply to the JSDF to fund this project.The Ministry of Finance of Armenia endorsed the MA as the implementing agency with the understanding that a PSC chaired by the MLSA would be established at project inception to oversee the project implementation. The MLSA, the State Employment Agency (SEA), and other stakeholders were partners to the process.
17.The PSC, chaired by the Deputy Minister of Labor and Social Affairs, was established with representatives of public and nongovernmental sectors to ensure the Government’s and broader public participation in the design and implementation process of the pilot. The PSC included the Ministry of Territorial Administration, the Ministry of Finance, the MA Project Director, representatives of Development Partners Organizations,NGOs, and representatives of communities (local governments).
18.The project was formally restructured once, in October 2015, to extend the closing date. The project’s original closing date was March 1, 2016. At the request of the implementing agency and after a review by the World Bank team, it was decided that asix-month extension would be granted to allow for the completion of the remaining activities, including monitoring of small businesses and the final project evaluation.To evaluate the impact of project activities and its PDO, it was necessary to track beneficiaries’ employment status three months after the completion of the program, which spanned beyond the project life.
19.Throughoutimplementation, the project’s OM was revised several times to reflect emerging needs. Neither the number of project sites nor the targeted number of beneficiaries was affected by the introduced changes. The Results Framework, as reflected in the OM, was adapted during the life of the project to provide a bettermeasure ofintended outputs and outcomes.The changes modified the wording but did not change the intent of the indicators being measured. The World Bank team held internal consultations (including the JSDF Unit and Legal Department) and decided that no restructuring was needed. Therefore, the OM was amended by the recipient with a ‘No Objection’ from the World Bank.