Terms of Reference for Project Evaluation

Transforming Environmental and Security Risks into Cooperation in the South Eastern European Region (Phase II); and Climate Change and Security in the Dniester River Basin

Project location / Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Kosovo*, Montenegro, Serbia
Project name / “Transforming Environmental and Security Risks into Cooperation in the South Eastern European Region (Phase II); and Climate Change and Security in the Dniester River Basin”
ADA project number / 8071-01/2012
Project budget / 1.300.000,00 EUR
Donor(s)/ funding sources / Austrian Development Agency (ADA)
Project duration / 1 December 2012 to 31 December 2015(and extension until 30April 2017)
Implementing agency and partners / Organization for Security and Co-operation in Europe (OSCE), on behalf of the Environment and Security Initiative (ENVSEC)

Note that the subject of the Project Evaluation is solely the Project Component 1 “Transforming Environmental and Security Risks into Cooperation in the South Eastern European Region (Phase II)”.

  1. Context of the Environment and Security Initiative (ENVSEC)

The Environment and Security Initiative (ENVSEC) was launched in 2003 at the Fifth Environment for Europe Ministerial Conference in Kyiv, Ukraine to strengthen national capacities, regional co-ordination mechanisms and international co-operation for environment and security risk reduction. Since then, the Initiative has developed into a unique multi-agency programme operating in four regions: Eastern Europe, South Eastern Europe, the South Caucasus and Central Asia. The Organization for Security and Co-operation in Europe (OSCE), the United Nations Development Programme (UNDP), the United Nations Environment Programme (UNEP), the United Nations Economic Commission for Europe (UNECE) and the Regional Environmental Center for Central and Eastern Europe (REC) have joined forces to provide holistic solutions to environmental challenges.

The increasing pressure on natural resources, the impacts of climate change, environmental degradation and unsustainable development pose multifaceted security challenges within and across borders. The ENVSEC Initiative helps to:

  • Transform these risks into opportunities for co-operation;
  • Improve mutual understanding;
  • Effectively manage transboundary environmental challenges on a co-operative basis

Following the achieved results and identified needs of Phase 1 of the ENVSEC programme “Transforming Environmental and Security Risks into Cooperation in the South Eastern European Region” (2009-2012), ENVSEC in cooperation with the Austrian Development Agency (ADA), the operational unit of the Austrian Development Cooperation, has been implementing Phase 2 of the ENVSEC programme in the period 2012-2016. Phase 2focuses on the reduction of environment and security risks and increased cooperation on environment and security issues both between and within countries in the region of South Eastern Europe (SEE), i.e. Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Kosovo*[1], Montenegro and Serbia.

More specifically, Component 1 addresses the following ENVSEC priorities in SEE:

  • A: management and reduction of transboundary risks from hazardous activities;
  • B: management of shared natural resources;
  • C:strengthening of regional cooperation on environmental governance through participatory and informed decision making and implementation processes;
  • D: Adaptation to the impacts of climate change for reducing security risks;
  • E: Regional cooperation in SEE

Pursuant to the ENVSEC practice in implementing the ENVSEC programme in SEE, activities are grouped under relevant priorities and considered as separate projects. The attached log frame matrix describes and defines for each activity the expected results and tasks.

The target group of beneficiaries of Component 1 includes policy and decision-makers as well as experts of relevant ministries (environment, foreign affairs, and other line ministries in charge of economic development, emergencies, transport and industry, agriculture), local governments, national authorities, public companies and agencies, protected area administrations, local municipalities, civil society organizations and academia. Indirect beneficiaries are the population of the recipient countries and region.

Component 1 is jointly implemented by the ENVSEC partner organisations REC, OSCE, UNDP and UNEP based on their respective areas of expertise and know-how.

  1. Purpose

Thefinal evaluation is commissioned by the REC, as ENVSEC Regional Desk for SEE and on behalf of ENVSEC, in accordance with the evaluation guidelines and templates of ADA, with the purpose of determining the relevance and fulfilment of objectives, results, development efficiency, effectiveness, impact and sustainability of Component 1. ENVSEC seeks to obtain through the final evaluation credible and useful information that enables the incorporation of lessons learned into the decision-making process of ADA and ENVSEC.

  1. Objectives

The main objective of the final evaluation is to assess and present results, conclusions, lessons learnt and recommendations related toComponent 1. The definition of appropriate standards, examination of performance against those standards, assessment of actual and expected results and identification of relevant lessons will contribute to:

  • Improving the development policies, procedures or techniques relevant to the ENVSEC Programme
  • Increasing the mutual accountability of the ENVSEC Programme
  • Enhancing the impacts of the ENVSEC Programme
  • Advising the strategic direction, design and implementation of the ENVSEC Programme

In particular, the final evaluation will analyse:

  • the design and coherence of Component 1 including the design of the log frame matrix;
  • the extent to which Component 1 has achieved its objectives and results;
  • the extent to which cross-cutting issues were applied;
  • the strengths and weaknesses of planning, management, implementation and monitoring;
  • the added value of the multi-agency approach of ENVSEC vis a vis standard approach; and
  • the effectiveness and efficiency of the regional coordination and steering mechanismof ENVSECin the SEE region.

By ascertaining results and assessing the effectiveness, efficiency, relevance and sustainability of Component 1, ADA, ENVSEC partnerorganisationsand ultimately the ENVSEC partners in the SEE region and other regions will benefit from the results of the final evaluation.

  1. Subject and focus

The final evaluation will encompass Component 1 of the project “Transforming Environmental and Security Risks into Cooperation in the South Eastern European Region (Phase II); and Climate Change and Security in the Dniester River Basin”. Considering the intervention logic, objectives and expected results of Component 1, the final evaluation will be performed atregional and national level (Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Kosovo*, Montenegro and Serbia), and at priority and activity i.e. project level.

The group of beneficiaries and other relevant stakeholders identified by ENVSEC to be interviewed comprisesin particular the following:

  1. ENVSEC National Focal Points in Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Kosovo*, Montenegro and Serbia
  2. Aarhus Centres in Albania, Bosnia and Herzegovina, Montenegro and Serbia
  3. International, regional and national participants and beneficiaries of ENVSEC events and activities organised within Component 1
  4. ENVSEC Partner organizations, ENVSEC Regional Desk for SEE and ENVSEC project managers
  5. Donors

The final evaluation will rely on the five OECD DAC evaluation criteria of relevance, efficiency, effectiveness, impact and sustainability, complemented with the additional criteria of partnership and cooperation which are relevant for the ENVSEC Programme.

  1. Main evaluation questions

Relevance
  • How relevant are the priorities and objectives of Component 1 for the region, countries and partners?
  • How consistent are expected results/outputs and outcome, immediate impact and overall goal/impact of Component 1 (Analysis of logframe matrix, programme model)?
  • To what extent did the multi-agency approach of ENVSEC contribute to ensuring the relevance of Component 1?

Effectiveness
  • To what extent has Component 1 already achieved its expected results/outputs?
  • What were the major factors influencing the achievement or non-achievement of the outcome(s)/expected results/outputs?
  • Were local experts involved in Component 1 and how did the input help to shape the final results?
  • Was Component 1 managed as planned? If not, what issues occurred and why?
  • To what extent have all stakeholders collaborated as planned?
  • Did Component 1 contribute to capacity building as planned?
  • To what extent was gender mainstreaming included in Component 1?
  • How did the multi-agency approach of ENVSEC contribute to ensuring the effectiveness of Component 1?

Efficiency
  • Was Component 1 implemented in the most efficient way (time, personnel resources) compared to alternatives? Have any issues emerged, if so which ones and why?
  • How did the multi-agency approach of ENVSEC contribute to ensuring the efficiency of Component 1?
  • How did the ENVSEC governance structure and regional mechanism affect the project implementation?

Impact
  • Did Component 1 reduce or contribute to reducing environment and security challenges in particular transboundary ones? To what extent can the results be attributed to Component 1?
  • How many women, men, girls, boys and people in total have already benefited from Component 1 (immediate impact)?
  • What exactly has already changed in the lives of women, men, girls, boys (immediate impact)?
  • Which positive and/or negative effects/impacts in terms of gender and environment can be possibly be attributed to Component 1?
  • Which institutions have already benefitted from the Component 1 and how? What has changed for whom (immediate impact)?
  • How did the multi-agency approach of ENVSEC contribute to ensuring the impact of Component 1?
  • To what extent were cooperation and synergies in project activities unlocked by the executing ENVSEC partner organisations to maximise the impact? Did the implementation of activities through individual projects have an (unexpected) impact?

Sustainability
  • If Component 1 continues, will it be integrated in local structures and/or funded by other sources?
  • Are there structures in place assuring sustainability?
  • What were the major factors which influenced the achievement or non-achievement of sustainability of Component 1?
  • If applicable, what needs to be done and/or improved to ensure sustainability?
  • How did the multi-agency approach of ENVSEC contribute to ensuring the sustainability of Component 1?
  • Was ENVSECwas able to trigger follow up support to countries in focal areas?

  1. Approach and Methods

The evaluation will be carried out by an evaluation team selected in an open call for tenders of the REC. The evaluator is expected to elaborate an inception report and a final evaluation report, resulting from desk research and interviews with 20 international, regional and national stakeholders, and phased as follows:

  1. The evaluator will be requested to commence the assessment with desk research, by screening the project documents and the implementation files (reports, evaluations of the project deliverables, contact lists for the beneficiary countries) as well as other relevant documents. A meeting with REC project team will be organised to debrief the evaluator.
  2. Based on the information collected, a draft inception report will be prepared, which will be discussed with the project team with a view to receiving comments/clarification requests. For the report, preliminary data will be analysed also taking into account horizontal (cross-cutting) issues (e.g. gender, anti-corruption, transparency), and a final methodology will be agreed upon. The evaluators will produce the inception report in compliance with the format available as an annex in the ADC Guidelines for Project and Programme Evaluation.
  3. The evaluator(s) will then conduct theinformation and data gathering, through the selectedmeans approved in the inception report, and will give due attention to, and include, the opinions of the beneficiary countries’ representatives, project team, partners, and any other stakeholders involved in their assessment. The data collection will be followed by final data analysis and the drafting of the evaluation report
  4. The draft final report will be sent to the ENVSEC partner organisations, project beneficiaries and ADA for quality control and comments. Once relevantcomments have been taken on board the report will be finalised and made available to the project stakeholders.

Data and information shouldbe obtained through different methods such as: analysis of documents, structured interviews, semi-structured interviews face-to face or by phone, group discussions, online-survey (if applicable), others. All data collected needs to be disaggregated by sex.

The Guidelines for Project and Programme Evaluations developed by the Austrian Development Agency need to be considered throughout the entire evaluation process.

  1. Timetable

Action / Responsible / Date
Submission of bid (electronically) / Contractor / 20April 2017
Contract signed and documents provided / REC and Consultant / 24 April 2017
Kick-Off meeting / REC and Consultant / 26 April 2017
Desk Study including interviews / Consultant / 15 May 2017
Submission of draft inception report / Consultant / 15 May2017
Inclusion of comments and submission of finalinception report / Consultant / 22 May 2017
Field Visit, interviews / Consultant / 31 May2017
Submission of final draftevaluation report / Consultant / 5 June 2017
Inclusion of feedback in final draftevaluation report and submission of final evaluation report / Consultant / 16 June 2017

The total number of days dedicated to the assignment is 37 working days, including on-the-spot visits, with a maximum duration of ten days.

  1. Reports

The Consultant will submit the following reports:

  1. an inception report (10-15 pages without annexes);
  2. a final draft evaluation report (about 25-30 pages without annexes), including a draft executive summary and the results-matrix format (part of the reporting requirement); and
  3. a final evaluation report (25-30 pages without annexes), the final executive summary and the results-matrix format (part of the reporting requirement).

All reports need to be written in English.

The executive summary should summarize key findings and recommendations (three to five pages) and needs to be submitted as part of the final draft report. The findings and recommendations of the draft final report and final report have to bestructured according to the evaluation questions. An outline of the report’s structure needs to be agreed upon during the inception phase.

The quality of the reports will be judged according to the following criteria:

  • Is the results-matrix format part of the report?
  • Does the report contain a comprehensive and clear executive summary?
  • Were the Terms of Reference fulfilled and is this reflected in the report?
  • Is the report structured according to the OECD/DAC criteria?
  • Are all evaluation questions answered?
  • Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
  • Does the report describe and assess the intervention logic (e.g. logframe, program theory) and present/analyze a theory of change and its underlying assumptions?
  • Are cross-cutting issues analyzed in the report?
  • Are the conclusions and recommendations based on findings and are they clearly stated in the report?
  • Does the report clearly differentiate between conclusions, lessons learnt and recommendations?
  • Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to?
  • Were the most significant stakeholders involved consulted?
  • Does the report present the information contained in a presentable and clearly arranged form?
  • Is the report free from spelling mistakes and unclear linguistic formulations?
  • Can the report be distributed in the delivered form?
  1. The Evaluation Team

The evaluation team will consist of max 4 members (international).

Key Qualifications of the team of evaluators should be:

  • Relevant academic degree (master level) in social or environmental sciences
  • A minimum of three years’ experience and expertise in the field/sector of environmental policy
  • The expert has conducted and/or participated in at least three evaluations in the last five years ideally in the relevant field
  • Knowledge of the SEE region with focus on environmental topics
  • Experience in project cycle management
  • Experience preparing and analysing a theory of change
  • Experience and expertise in evaluating cross-cutting issues
  • Experience in social science methods
  • Excellent oral and written English skills, knowledge of BSCM languages is an asset
  • Sound MS Office and IT skills

Applicants must not have been involved in the design, implementation or monitoring of this project.

  1. Application

The applicant must submit to the REC an application package containing a technical offer and a financial following instructions below. The technical offer must include a cover letter of a maximum of two pages presenting the evaluator and illustrating the evaluator’s capacities, relevant experiences and qualifications for the envisaged tasks. The cover letter should be accompanied by supporting documents (CVs, references, relevant certificates). The financial offer must be submitted on the separate email account as the two will be assessed independently.

Instructions for submission:

The technical offer should be submitted on the following email account: .

The financial offer should be submitted on the following email account:

The deadline for the submission of applications is 20 April 2017.

  1. Contacts

ENVSEC Regional Desk for South Eastern Europe, hosted by the REC: Ms. GordanaKozhuharova, Regional Director for South Eastern Europe, REC.

  1. Annexes

Annex 1 Results Matrix

Annex 2 Log frameMatrix

Annex 3 ADC Guidelines for Project and Programme Evaluation

Annex 4 ADC Template for Inception report

1

* In the entire document and its annexes, this designation is without prejudice to positions on status, and is in line with UNSCR 1244/99 and the ICJ Opinion on the Kosovo declaration of independence.