Local Governments and Climate Change

LGCC Participatory Evaluations

Local Governments and Climate Change

Report of Participatory Evaluations

April 2013

Contents

1Introduction

2Background

2.1The LoCAL Global Project

2.2Local Governments and Climate Change Phase 1

2.3Scope of Sub-Projects Supported

2.4LGCC Phase 2

2.5Timing of the Evaluations

3Purpose of the Evaluations

4Methodology of the Participatory Evaluations

4.1Steps

4.2Selection of Sub-Projects

4.3Preparation

5Fieldwork

5.1Physical Inspection of Outputs

5.2Focus Group Discussion

5.3Summing Up

5.4Scoring By The Evaluation Team

5.5Provincial Reflection Workshop

6Findings of Participatory Evaluations

6.1Technical Quality

6.2Beneficiaries’ Perceptions of Benefits

6.3Negative Impacts

6.4Sustainability

6.5Relevance to Climate Change Adaptation

6.6Impact on Beneficiaries’ Understanding of Climate Change Adaptation

6.7Cost-Benefit Scores

7Summary Of Findings And Lessons Learned

7.1Impact

7.2Relevance to Climate Change Adaptation

7.3Weaknesses Of the Sub-Projects

7.4Key points to strengthen

8Assessment of the Evaluation Methodology

LGCC Participatory Evaluations

1Introduction

This report describes the process and results of participatory evaluations of sub-projects funded by the Local Governments and Climate Change (LGCC) project in Takeo province during March 2013. LGCC is a pilot in Cambodia of United Nations Capital Development Fund’s global “Local Climate Adaptive Living” (LoCAL) project. As such, the findings of the evaluation and the methodology used are of potential value for LoCAL country programmes elsewhere, in addition to the lessons learned which are directly applicable to the future implementation of LGCC.

The evaluations were undertaken by the Secretariat of the National Committee for Sub-National Democratic Development (NCDD-S) and the Takeo Provincial Administration with technical assistance from United Nations Capital Development Funds (UNCDF).

2Background

2.1The LoCAL Global Project

The LoCAL Global Project was designed to provide access to climate finance for local governments in least developed countries (LDC). The project is based on successful previous UNCDF experiences in public financial management; local investments and performance-based fiscal grants.

Climate change adaptation finance flows to central government agencies, and yet local governments have the mandate to deal with many of the issues directly affected by climate change such as drainage and water management, land use planning, road maintenance and other issues which effect food security, especially in the LDCs.

LoCAL addresses this issue by providing Performance Based Climate Resilience (PBCR) grants to selected local governments that show themselves able to meet a defined set of minimum conditions. PBCR Grants may be used to finance climate change adaptation measures that fall within a general Investment Menu based on a climate change vulnerability assessment, and that are identified through local participatory planning processes.

LoCAL transfers PBCR Grants to local governments through existing financial and public expenditure management systems. The performance-based nature of the grants provides local governments with an incentive to build their own capacity. In most country programmes UNCDF with its partners provide direct capacity development support for climate change adaptation while also learning lessons from LoCAL implementation and supporting policy development for climate change adaptation at local level.

LoCAL is directed by a Board consisting of participating countries and donors and is administered through a Secretariat. The Secretariat deploys specialist expertise to support LoCAL country programmes, facilitates sharing of experiences and manages the monitoring and evaluation system to ensure effective use of funds.

2.2Local Governments and Climate Change Phase 1

Phase 1 of Local Governments and Climate Change was supported by a grant of $US250,000 from the Cambodia Climate Change Alliance Trust Fund and $US50,000 funding from UNCDF. The project was implemented by NCDDS in three local administrations in Takeo Province: Doun Keo Municipality and Bati and BoreiChulsar Districts. Project implementation began in the final quarter of 2011 and ended in April 2013; i.e. the project financed a single cycle of PBCR grants in the three local administrations.

The project objective of LGCC was “to demonstrate the role of Local Governments in fostering climate change resilience and to identify practical ways to mainstream CCresilience into Sub-National planning and finance systems. An important focus is thus on how to strengthen institutions and systems at both National andSub-National levels. The entry points are provided by the recently introduced Sub-National planning and Sub-National finance systems.”

LGCC had three expected outputs which were defined as:

1. Analysis of climate change awareness and integration into Sub-National plans and investment programs;
2. Piloting of systems to mainstream climate change resilience in Sub-National Finance and Planning Systems; and
3. Policy feedback to National and Sub-National authorities.

For the purpose of LGCC the three participating local administrations were considered as including the constituent Communes of the Districts and Sangkats of the Municipality. The PBCR Grant was considered as a single grant to the District or Municipality, but Communes or Sangkats, as well as the District and Municipal Administrations, could propose activities to be financed and if successful, could implement sub-project activities. In practice, infrastructure sub-projects were implemented by Communes and Sangkats following the well-established procedures of the Commune/Sangkat Fund, while non-infrastructure activities were implemented by the District and Municipal administrations

Representatives of the District and Commune levels participated in a planning exercise to identify the key climate change vulnerabilities and priority investments for climate change adaptation. The resulting workplan and budget was approved by the District/Municipal Council and submitted to NCDD-S, which approved the workplan and budget and transferred funds for those activities to be implemented by the Communes and Sangkats into the respective Treasury Accounts. For activities to be implemented at the District level, funds were transferred into a project bank account operated by the Provincial Administration.

2.3Scope of Sub-Projects Supported

Eligible expenditures of the PBCR grant funds under the Investment Menu comprised:

1. Strategic Analysis and Planning for Climate Change Adaptation;

1. Climate Change Adaptation in Sector Service Delivery;
2. Climate Proofing of Infrastructure and Additional Infrastructure for Climate Change Adaptation.

In addition, Districts / Municipalities were permitted to spend up to 5% of the PBCR grant on administrative overheads and up to 15% of the costs of infrastructure investments on hiring technical assistance (a “Technical Support Consultant”) for design and supervision of the works.

A list of LGCC sub-projects is attached as Annex 1. A summary of grant allocations is shown in Table 1 below.

Investment Type / Doun Keo / Bati / BoreiChulsar / Total
CCA Planning / $1,384 / $4,102 / $642 / 0 / $6,128
Sector Service Delivery / 2 / $4,477 / 2 / $8,624 / 1 / $4,149 / 5 / $17,250
Climate Proofing Infrastructure
Roads / 1 / $2,629 / 2 / $14,000 / 3 / $16,629
Water Supplies / 2 / $16,100 / 2 / $16,100
Irrigation / 1 / $10,000 / 2 / $20,553 / 3 / $30,553
Drainage / 1 / $6,400 / 1 / $3,631 / 2 / $10,031
Technical Support / $3,158 / $6,468 / $3,111 / 0 / $12,737
Administration / $1,492 / $2,875 / $1,383 / 0 / $5,750
TOTAL / 5 / $29,540 / 6 / $58,722 / 4 / $26,916 / 15 / $115,178

The sub-projects were implemented between May and December 2012, with some construction activities not completed until the first quarter of 2013 due to the need to wait for wet season floods to recede.

2.4LGCC Phase 2

During 2012 funding was agreed between Swedish International Development Agency (SIDA), NCDDS and UNCDF for a second phase of LGCC, to be implemented during 2013-2014. LGCC Phase II supports the same local governments in Takeo Province as in Phase 1, plus five Districts (with constituent communes) in Battambang Province.

2.5Timing of the Evaluations

The participatory evaluations which are the subject of this report formed part of the work-plan of LGCC Phase 1. The participatory evaluations were carried out after all LGCC1 sub-projects had been completed and during the start-up phase of LGCC2. Therefore, the experience of implementing the sub-projects was fresh in the memory of the participants and lessons learned could be applied toLGCC2 implementation. However, the timing of the evaluations was such that the full impacts of some sub-projects might not be apparent (e.g. because an irrigation sub-project was evaluated before the end of the first full growing season using the sub-project outputs).

3Purpose of the Evaluations

The purpose of the participatory evaluations was to improve knowledge on local administration capacity and performance in responding to climate change adaptation

The purpose of the participatory evaluations was to improve knowledge, learn lessons and identify success in the implementation of LGCC, and thus to further the project objective of demonstrating the role of local governments in fostering climate change resilience. Specifically, the methodology of the participatory evaluations was intended to”

• allow the sub-project beneficiaries to express their views on all aspects of sub-project implementation;
• to identify strong and weak aspects of the implementation of the sub-projects;
• to identify the expected impacts (including any negative impacts) and sustainability of the sub-projects;
• to assess the relevance of the sub-projects to local needs for climate change adaptation;
• to assess the extent to which implementation of the sub-projects assisted in building awareness of climate change vulnerability and adaptation needs at local level;
• to assess whether the investments provided value for money as “no regrets” CCA investments, meaning that the immediate and ongoing benefits of the projects are broadly commensurate with the project costs; and
• to develop and field-test a simple, robust evaluation methodology that can be replicated for different types of sub-project and in different geographical areas.

Evaluation of small, diverse and geographically scattered sub-project outputs is a challenge and in particular it is difficult to use quantitative methods that practical to apply, yield results that can be meaningfully compared, and that yield useful information about the impacts of the project or activity being evaluated. Some methods that are routinely applied to large projects are too complex or expensive to be useful in this situation. When the desired impact is “climate change resilience” there is additional difficulty as resilience is the capacity to adapt to unknown future events.

The methodology of the participatory evaluations, described in detail in the following section of this report, was designed with these difficulties in mind. High priority was given to simplicity and replicability, while yielding useful information of relevance to climate change adaptation impacts.

4Methodology of the Participatory Evaluations

4.1Steps

The participatory evaluations were carried out according to the following steps:

1. Selection of sub-projects to evaluate;
2. Preparation including summary of available information on each sub-project;
3. Fieldwork;
4. Physical Inspection
5. Focus Group Discussion
6. Summing Up
7. Scoring by the Evaluation Team;
8. Provincial Reflection Workshop including:
9. Presentations by the beneficiaries based on focus group discussions
10. Presentation of scoring by the evaluation team
11. Reflection and lessons learned.

4.2Selection of Sub-Projects

Sub-projects were selected to represent the range of sub-project types implemented across the three local administrations (District or Municipality). Because of the small total number of sub-projects this could not be achieved by random sampling. Therefore it was decided that in each local administration one road sub-project, one water sub-project (irrigation or water supplies) and one non-infrastructure sub-project would be selected. Selection of the sample sub-projects was done by the national level members of the evaluation team to reduce the risk of bias (as they were less familiar with the individual sub-projects than the Provincial team members).

The following sub-projects were selected for evaluation:

District / Commune / Type / Description
Bati / Lumpong / Pond / Rehabilitation of a community pond 60m x 30m x 2m approx for water supplies and gardening
Bati / Thnowt / Canal / Rehabilitation of a canal 1450m long with one culvert and repair of road alongside canal
Bati / Pea Ream / Non-Infrastructure / Demonstration of climate resilient rice seed variety at 4 locations
BoreiChulsar / BoreiChulsar / Road / Drainage structures and raised embankment on road in flood-prone area
BoreiChulsar / KoukPhos / Drainage / Construction of culverts on line of an irrigation canal and providing access across the canal
BoreiChulsar / BoreiChulsar / Non-Infrastructure / Training on water use and hygiene and provision of water filters to XXX families in a flood-prone village
Doun Keo / Roka Knong / Road / Construction of two short lines of road crossing a flood prone area, with raised embankments and drainage structures
Doun Keo / Baray / Irrigation / Construction of water gate on an existing canal line
Doun Keo / Roka Krao / Non-Infrastructure / Raising awareness on climate change and planting tree seedlings

4.3Preparation

Preparation for fieldwork included preparing a summary of available data on each sub-project. This information was obtained from the Project Information Database of NCDDS and from other sources by the Provincial Administration. The basic information on each sub-project consisted of:

• Sub-project location;
• Implementing agency (Commune or District level)
• Sub-project name
• For infrastructure sub-projects:
• Contract Number
• Contractor Name
• Name of Technical Supervisor
• Description of Outputs
• Contract Cost
• Table of Outputs: Number, Description, Location (Village), Quantity
• Completion Date (Planned and Actual)

In addition to tabulating this information, copies of relevant documents including contract and design drawings (for infrastructure sub-projects) and work plan (for non-infrastructure sub-projects) were collected for reference in the field.

5Fieldwork

5.1Physical Inspection of Outputs

For infrastructure sub-projects the evaluation team inspected the outputs in the company of the local authorities and beneficiaries. The evaluation team attempted to determine whether the design was appropriate, the quality of the construction and whether the output was functioning correctly. The views of the beneficiaries were taken into account and were cross-checked with technical staff including the Provincial Infrastructure Adviser and the Technical Support Consultant who was responsible for design and construction supervision.

The evaluation did not include a complete “engineering” assessment of the structures or any physical tests.

5.2Focus Group Discussion

The focus group discussions were structured around a number of themes. The beneficiaries were assisted to summarise their ideas on each theme into a few sentences which one of them wrote down on flip-chart paper.

In following this procedure, it was important not to try to write down everything anybody said. The intention was to encourage an open discussion on each topic and then to summarise the views expressed into a few sentences.

The themes around which the discussion was structured were:

• What were the sub-project activities (as perceived by the beneficiaries)?
• What impacts do the beneficiaries expect (including negative impacts) as a result of the sub-project activities?
• How does the sub-project help to build climate resilience?
• What actions are needed to ensure the sustainability of the sub-project?
• What lessons can we learn that can be applied in future sub-projects?

It is emphasised that this list of “themes” is not a questionnaire requiring a set of direct, one-to-one answers. The purpose of the themes is to stimulate discussion.

The time allocated for the physical inspection and the focus group discussion was a maximum of two hours – this permitted three sub-projects to be evaluated in one day but more importantly, served to avoid excessive demands on the time of the beneficiaries.

5.3Summing Up

At the end of the focus group discussion the beneficiaries read through and reviewed the summary sentences written on the flipchart paper. Finally, the beneficiaries were asked to select one man and one woman, who should not be people with an official position of any kind (i.e. not the village chief or assistant, or similar positions) who would attend the reflection workshop and present the findings of the discussion.

5.4Scoring By The Evaluation Team

The evaluation team (meaning the representatives of NCDDS and the Provincial Administration, plus the UNCDF advisers present) carried out scoring of each sub-project as soon as possible after completing work at the sub-project site.

Scores were awarded against the six criteria in the following table.

• Technical Quality (max 10 points)
• Sustainability (max 10 points)
• Negative Impacts (max 10 points)
• Relevance to Climate Change Adaptation (max 20 points)
• Impact on Beneficiaries’ Understanding of Climate Change (max 20 points)
• Estimated Cost: Benefit Ratio (max 40 points)

Therefore, each sub-project was awarded a score out of a maximum 100 points.

For the first five criteria, the team arrived at the score by consensus. For each criteria, a table was provided defining the meaning of each score. The table for “technical quality” is reproduced below as an example. All the scoring tables are included in Annex 2 to this report.

Technical quality: Meaning of Evaluation Scores
Score / Meaning
10 / Very high technical quality: better than technical design and standards and no defects
8 / High technical quality: conforms with the technical design and standards with no defects
6 / Good technical quality overall. Some defects but these will not significantly reduce the benefits of the project
4 / Poor technical quality. The benefits of the project will be reduced because of the low quality
2 / Vey poor technical quality. The benefits of the project are much less than planned because of the poor technical quality.
0 / Extremely poor technical quality. Because of the poor technical quality the project provides very small benefits or no benefits at all.

By ensuring that the scoring conformed fairly strictly to these tables, the scores awarded could be considered as comparable between projects of different types and potentially between different evaluation teams.

Technical Quality was considered mainly as meaning that the design and construction of the (infrastructure) outputs was suitable to ensure the intended sub-project benefits and the long-term sustainability of the sub-project. For non-infrastructure projects the team attempted to award equivalent scores based on the quality and effectiveness of services provided, for example the quality of a training as described by the beneficiaries and as evidenced by the beneficiaries’ understanding of the training topic.

Sustainability was defined in terms of the level of maintenance effort the community would have to provide to ensure that the sub-project would continue to provide benefits for 10 years or longer. Therefore a sub-project scored maximum points if benefits were expected to continue for 10 years without any specific maintenance effort being required.