Project Implementation Review

WORLD BANK

PROJECT IMPLEMENTATION REVIEW

FY05

OVERVIEW REPORT

January 19, 2006

Overview

1.The World Bank Group’s GEF approved portfolio[1] at the end of FY05 comprised 288 full sized projects (FSP), 101 medium sized projects (MSPs) and 37 Enabling Activities, representing grant commitments of US$2.796 billion that are associated with an additional US$13.8 billion in co-financing. GEF grant commitments increased by7.5% in nominal terms over FY04, which was lower than the rate of 13% achieved in that year and the average annual growth rate of 12% since 1991.

2.The GEF approved 27 FSPs and 9 MSPs in FY05. The average FSP size declined to $7.2 million compared with $9.1million in FY04 and the ten year average of $10 million. There was a slight increase in the growth rate of MSP commitments with a 12.5% increase in FY05 compared with 11% in FY04 but well below the average of 27% between FY01 and FY03 and continuing what appears to be a longer term decline in MSP use in the Bank. In sum, during the past four years, though the number of new projects is keeping pace with the longer term yearly average, the average GEF grant size is falling and total commitments are growing at a slower rate, 9% per annum, than during the first ten years of the GEF when it grew at 13%. However, rather than a longer term trend, the slow down in commitments seems to be caused by the after effects of afreeze of the pipeline in 2001/2002 resulting from uncertainty about replenishment, together with poor resource and pipeline management by the GEF Secretariat, which is only now beginning to work itself through the funding cycle. Furthermore, lower average grant size is due to the increase in numbers of participating countries, and therefore more projects, without a corresponding increase in the total available GEF funds.

3.The distribution of the portfolio by focal area is presented in Figure 1a for FSPs and in Figure 1b for MSPs.[2] There were slight changes from FY04 that were mostly consistent with trends in recent years. The climate change (39%) and biodiversity (35%) focal areas continue to have the largest shares of GEF commitment but each declined by one percentage point since FY04, while the share of international waters increased by one percentage point to 14% a steady growth from 10% in FY00. The share of projects in the Multi-focal area had increased in recent years and though there were six additional projects the share remained the same at 6%. The share of ODS projects also remained unchanged at 5% while land degradation and POPs projects are just beginning to enter the portfolio (less than 1% each). The MSP portfolio continues to be dominated by biodiversity projects with a few land degradation projects now being approved.

4.The geographical distribution of the portfolio is presented in Figures 2a and 2b. The most noticeable change is a fall in the Latin American and Caribbean Region’s (LCR) share to 20% from 26% in FY04. This was due to delays in submissions for Council approval as a result of institutional changes in the region. Subsequently, in FY06, the Region has significantly increased its submissions and should be back to its historical share. The main gainers were IFC (9%) up by two percentage points and to a lesser extent ECA (20%) and AFR (16%) both up by 1%. The share of the other regions remained the same. For MSPs, though LCR continues to have the highest proportion of commitments its share declined while ECA’s again increased.

6.The Bank’s active portfolio[3] in FY05 comprised 225 projects (167 FSPs and 61 MSPs) with total GEF Grant commitments of US$1.46 billion. This was an increase in commitments of less than 1% over FY04 compared with increases of 5.6% and 3.4% respectively in the previous two years. The active portfolio reflects the pattern of entries and exits in any given year. Compared with the regional distribution for the Council approved portfolio, the shares of two regions are different. EAP’s share is four percentage points higher which shows a faster rate of deliveries to the Bank Board, while MNA’s share is four percentage points less due to the delay in further processing two large solar thermal projects in Egypt and Morocco. During FY05 the Bank’s Board approved 32 FSPs, the highest number since the GEF began (31 were approved in FY04), while 14 projects closed resulting in a net increase of 18. For MSPs there were 14 approvals by Regional Management and 19 closures, which resulted in a decline by 5 projects. The portfolio is beginning to mature as 85 FSPs and 41 MSPs have now been completed and closed.

Portfolio Performance

7.Eleven indicators have been identified to measure portfolio performance. Most of these are widely used by the Bank (such as outcomes, projects at risk, realism and proactivity ratios) and reported either in the Quality Assurance Group’s (QAG) Annual Review of Portfolio Performance (ARPP) or in regional portfolio management reports, while others are GEF specific, such as elapsed time between GEF Council and Bank Board Approval. These are discussed in each section below and an explanation of each indicator is also provided. Because of the relatively small number of cases in each year for some indicators, and due to the volatility of annual results, four year averages are also presented for comparison and in assessing trends. The relatively small universe of GEF projects also makes disaggregated statistical analysis less robust, for example at the regional or focal area level, but such results can be indicative of trends. Table 1 presents a summary of these indicators.

Project Outcomes

8.The Independent Evaluation Group (IEG)[4]defines outcomes as the extent to which the project's major relevant objectives were achieved, or are expected to be achieved, efficiently. A project is rated satisfactory if it achieved, or is expected to achieve, most of its major relevant objectives efficiently with only minor shortcomings.Fourteen projects exited the portfolio in FY05. Of these only five have so far been evaluated by IEG. All five were rated in the range of satisfactory outcomes[5], including one which was rated Highly Satisfactory, the Senegal Sustainable and Participatory Energy Management Project.In comparison, 78% of projects under the ESSD network (environment, rural and social development sectors) Bankwide were rated in the range of satisfactory in FY05.The net disconnect ratio for GEF projects was zero, meaning there was no difference between the rating in the final supervision report and IEG’s evaluation rating. Given the small universe of GEF projects it is more useful to look at longer term trends.

Table 1

Summary of Performance Indicators for the GEF Portfolio

Indicator / GEF FY03 / GEF FY04 / GEF
FY05 / Bankwide ESSD FY05 / GEF Average FY02-05 / ESSD Average FY02-05
IEG at least Satisfactory Outcomes / 80 / 80 / 100[6] / 78 / 84 / 78
Net Disconnect / 0 / 0 / 0 / 3 / 0 / 9
Sustainability Likely or Highly Likely / 70 / 78 / 100 / 83 / 80 / 72
Progress towards GEO (at least satisfactory) / 93 / 96 / 89 / 88 / 92
Implementation Progress (at least satisfactory) / 89 / 89 / 89 / 85 / 89
Projects at Risk / 8 / 11 / 11 / 16 (IBRD) / 11 / 14
Commitment at Risk / 14 / 13 / na
Realism / 100 / 88 / 72 / na / 87
Proactivity / 100 / 80 / 67 / na / 89
Elapsed Time: Council to Bank Management Approval / 587 / 392 / 499 / na / 490 / NA
Elapsed Time: Bank Management Approval to Effectiveness / 208 / 191 / 150 / na / 204 / NA

9.Eighty five FSPs have been completed since 1997 with seventy six already evaluated by IEG (MSPs are not evaluated by IEG). Overall, the GEF results are more positive than comparable sectors and the Bankwide IBRD[7] portfolio. Eighty four percent of the 76 GEF projects evaluated by OED were rated in the satisfactory range. For projects which closed more recently, between FY02 and FY05, the average satisfactory rating was also 84% (Table 1 above). By comparison, the rating in the satisfactory range for projects under the ESSD network was 78% for the same periods. In the case of the energy sector the satisfactory rating was 86% for FY05 (7 projects only) and 81% for FY02-05. For the Bank as a whole, QAG reported that 76% of projects achieved satisfactory outcomes in the period FY00 to 04. The Bank’s strategic target for outcomes was 75% and even though Management still sees results above 80% as desirable and achievable, it is clear that the Bank is engaged in some risky sectors, with environment often included in this group. For example, since 1991 the averages in the main sectors in which GEF is engaged were: environment (66%), rural (65%) and energy and mining (66%). Against this background, GEF results which have been consistently in the 75% range and now above 80% areencouraging. Moreover, theseoutcomes are particularly impressive given GEF’s mandate toachieve global environmental benefits which is not typically a country’s priority, but which also requires innovation and less aversion to risks. As part of the portfolio improvement plan for FY06 the results for completed GEF projects will be examined to identify the relevant factors that might be contributing to their better performance.

10.The ratings by Region for the GEF portfoliofor FY02 – 05 are presented in Table 2 below. As mentioned above, regional results are illustrative as they are lessstatistically robust. SAR (on a small number of projects) had a rating of 100%in the range of satisfactory, followed by ECA 96%, LCR , 90%, AFR, 88%, MNA (on a small number of projects), 83%, and EAP 58%. The highest proportions of projects rated in the unsatisfactory range were in EAP, 42%, and MNA 17%. For comparison, ESSD average satisfactory ratingsfor the same period were: SAR, ECA and MNA, 85% each, EAP, 78%, LCR, 73% and AFR, 63%. For the IBRD portfolio reported by QAG for FY00 to 04, ECA, SAR and LCR achieved over 80% satisfactory or higher, EAP close to the overall average (76%) with 77%, while AFR (66%) and MNA (73%) consistently achieved below average, and also had the highest proportion of unsatisfactory projects.

Table 2

Regional Distribution of Achievement of Project Outcomes

Rating / Region
AFR (17) / EAP (12) / ECA (24) / LCR (11) / MNA (6) / SAR (5) / Total (75)
Highly Satisfactory / 12 / 16 / 4 / 7
Satisfactory / 41 / 34 / 63 / 45 / 83 / 60 / 52
Moderately Satisfactory / 35 / 8 / 29 / 45 / 40 / 25
Moderately Unsatisfactory / 8 / 4 / 10 / 7
Unsatisfactory / 12 / 34 / 17 / 9
Total / 100 / 100 / 100 / 100 / 100 / 100 / 100

11.EAP was least successful in the GEF portfolio with five out of 12 projects less than satisfactory. The region is not associated with a high proportion of projects at risk (see below), and is assessed by QAG as one of the better performing regions on project supervision. However, three of the unsatisfactory projects were in Indonesia and one in the Philippines, two countries with above average proportions of projects at risk in the overall IBRD portfolio, and one was in Lao PDR. The presence of most of these projects in high risk countries contributed to these poor results, as did the Asian financial crisis and major political and economic reforms that were being implemented in Indonesia and Philippines at the time. On the other hand AFR, which consistently achieves below average on outcomes in the ESSD and overall IBRD portfolios and has the lowest supervision ratings (along with MNA) performed well for GEF projects. One explanation given is that due to the high risks and initially poor project results, significant efforts have been made in recent years by task teams to improve adaptive management for GEF projects and the results can now be seen in the abovementioned outcomes.

12.By Focal Area(Table 3 below), for FY02 - 05, all Ozone projects were in the satisfactory range, 89% of IW projects, 82% of biodiversity projects and 80% of climate change projects. However, 38% of biodiversity projects were rated moderately satisfactory compared with 20% for climate change which suggests much greater room for improvement in the former. This could be further evidence that biodiversity projects sometimes have multiple objectives and too many components.It may also reflect the difficulty of addressing complex social issues which affect achievement of conservation goals. As observed in last year’s PIR, where a project had multiple objectives, oftentimes only some of them were fully achieved thus the project was rated moderately satisfactory. Typical of this case was the Mexico Protected Areas Project which had four objectives some of which were rated highly satisfactory and others unsatisfactory resulting in an overall rating of moderately satisfactory.

Table 3

Achievement of Outcomes by Focal Area (%)

Ratings / Focal Area
Biodiversity (39) / Climate Change (20) / International Waters (9) / Ozone (7) / Total (75)
HS / 15 / 11 / 14 / 7
S / 44 / 45 / 78 / 86 / 52
MS / 38 / 20 / 25
MU / 5 / 10 / 11 / 7
U / 13 / 10 / 9
Total / 100 / 100 / 100 / 100 / 100

13.Overall, 25% of the GEF portfolio was rated moderately satisfactory, which is defined by IEG as where the Project achieved, or is expected to achieve, most of its major relevant objectives efficiently but with either significant shortcomings or modest overall relevance. Accordingly, only a relatively small proportion of the FY02-05 projects, 16%, which is within the Bank’s strategic target,were unsatisfactory compared with 22% for the ESSD portfolio as a whole.

14.The Bank’s FY04 ARPP found the strongest correlation with outcomes to be Borrower performance in implementation, irrespective of interaction with other factors. But Bank performance in supervision was also important as this can lead to timely identification and mitigation of risk. For the GEF cohort, Bank performance was rated at 79% satisfactory by OED (Bankwide or ESSD comparable data were not available).Over time there has been an improvement in both GEF and overall Bank performance in quality at entry and in quality of supervision though the ARPPsuggests that a plateau has been reached in these indices. ENVGC findings from previous years PIR, M&E and Elapsed Time studies on factors that contribute to better outcomes, which have been reflected in the annual Portfolio Improvement Plan, are corroborated by recent QAG and IEG assessments. Many of these seem obvious and are not new but are repeated in these assessments probably because they are still not fully taken into account in project design and implementation. These are:

Lower project complexity[8]
Project readiness for implementation[9]
Mitigation of project risks[10]
A sound monitoring and evaluation system with good connectivity to the project’s objectives, outcomes and components (weaknesses in M&E is singled-out in most OED project evaluations.[11]

15.Similarly, more specific “lessons of broad applicability” identified inIEG’sGEF project evaluations in the past twelve monthsare also not entirely new and many are generic. Again, they continue to be raised in order to reinforce their continued importance[12]:

Natural Resource Governance, community access and benefit streams

the importance of good natural resource governance especially for biodiversity conservation projects;
the importance of realism about natural resource management and biodiversity benefit streams. There are high risks in a local population not seeing tangible benefits soon enough from natural resource management. An aspect of this is the need to understand the potential development/biodiversity conflicts.Resentment can be created when financing of nature conservation appears to have preference over unmet local needs, be it employment or delivery of basic services;
the establishment of environmentally and socially sustainable NR supply systems can best be achieved through the introduction of community-based NR management schemes. Because governments lack the resources and the private sector the incentive to manage forests and other natural resources, these systems constitute the least cost option and offer the highest success opportunities for this kind of intervention, while generating significant rural development impacts;
partnerships are also important to achieve successful community interventions, with government, NGOs, and the private sector, given the low capacity at community level in the challenging task of protected area management. Moreover, the private sector is usually most proficient at assessing and managing revenue generating activities.

Implementation Capacity and design Issues

Strengthening implementing agencies and increasing their effectiveness and efficiency, rather than creating external implementation units, makes implementation a mainstream function of the agency.
Expediency should not be a reason to substitute external expertise for deficient local capacity, it is better to build long-term in-country capacity which strengthens project ownership and improves outcomes
Several evaluations suggest the need to consider longer project duration than under conventional development projects: to secure community buy-in to conservation goals; new community organizations and government processes may need longer to fully mature and become sustainable; and where complex institutional change is involved.
Projects of a quasi-commercial nature need to be designed with considerable flexibility to allow for changing market conditions.

Challenges in designing energysector interventions

while electricity use can significantly reduce the monthly energy expense of low- (and other) income households, elites are oftentimes better able to capture the benefits because the services are not affordable to the poor;
to be effective GEF support for the introduction of energy-efficient technology, needs to be sustained through all stages of the product introduction cycle - design, demonstration, manufacture and marketing;

16.The Bank’s GEF portfolio improvement plan for FY06/07 will incorporate actions to address the issues mentioned in the sections above.

Likelihood of Achieving Sustainability

17.According to IEG, sustainability reflects the resiliency to risks of a project as measured by the likelihood that its estimated net benefits will be maintained or exceeded over the project's intended useful life.

18.All the GEF projects evaluated by IEG in FY05 were considered likely to be sustainable. The four year GEF average of 80% was higher than the ESSD average of 72% for the same period, but less than the result for energy and mining which was84%. Post implementation impact studies carried out by the Bank’s GEF teamand confirming previous lessons showed the three most important factors affecting sustainability were:

finance, whether the project was able to develop sustainable financing streams prior to closing in order to continue support for the operation
Government commitment either to continue supporting relevant institutional structures including provision of staff, to pass supportive policies, or to provide financial resources
Changes in the overall enabling environment. For example, privatization of a public utility does not encourage demand side management as public and private goals might diverge

Projects at Risk

19.The Bank’s Quality Assurance Group (QAG) gives the following reason for introduction of the projects at risk system in 1996: “Inherent optimism of staff and managers, a perception that supervision reporting is a bureaucratic requirement rather than having substantive importance, and honest differences of opinion about what progress can reasonably be expected in a given context, have tended to make IP and DO ratings unduly optimistic.” Projects at risk include actual problems projects, which are thosefor which Implementation Progress (IP) is unsatisfactory and/or the Global Environment Objectives (GEO) are not likely to be achieved. They also include potential problem projects, which are those Projects which are rated satisfactory on IP and/orGEO but have other risk factors historically associated with unsatisfactory outcomes.