Evaluation Terms of Reference (ToR)

Project/Programme Title:CAPAQUA - Development of Educational and Research CAPacity in Eastern Africa for the Sustainable Management of AQUAatic Ecosystems

Country: Eastern Africa

Project/Programme Number:0612-00/2015

Name of partner Institutions stated in project contract: University of Natural Resources and Life Sciences, Vienna (BOKU); Egerton University (EGU); Addis Ababa University (AAU); and Bahir Dar University (BDU)

Additional partner Institutions: Ethiopian Institute for Agricultural Research, National Fisheries and other Living Aquatic Organisms Research Centre (EIAR-NFLARC);and Institute for Water Education (IHE Delft)

  1. INTRODUCTION/BACKGROUND

Project context

Aquatic ecosystems provide crucial resources and services, which are under high pressure in Eastern Africa due to the growing population, high poverty prevalence and climate change effects. Moreover, Eastern African lakes, rivers and wetlands are biodiversity hot-spots, whereas both fundamental ecological baseline-information, as well as the implementation of conservation measures is not keeping pace with environmental pressures. The increasing discrepancy among demands and availability of aquatic ecosystem resources and services is calling for innovative management solutions suitable for the Eastern African context - ideally developed, promoted and implemented by Eastern African institutions and professionals. However, the quality of higher education has become a subject of major concern among stakeholders in Eastern Africa recently.

Project description

The overall project objective of CAPAQUA 0612-00/2015is to foster the sustainable management of aquatic ecosystems and aquatic resources towards the achievement of the SDG’s. The project objective is to enhance the capacity of Eastern African HEST-Institutions to provide highly qualified graduates to the job market and to develop and implement strategies/projects towards the conservation and sustainable management of aquatic ecosystems and aquatic resources.

Strategically,the CAPAQUA project isfocussing on the support of capacity development processes at Eastern African HEST institutions at the qualitative level via enhancing the quality and relevance of postgraduateprogrammes,thestrengthening of national and regional networking andthe provision of well-educated and well-skilled postgraduates to the Eastern African job market. The target institutions and programme graduates are perceived as key-agents to initiate transformation processes towards the sustainable management of environmental assets to the benefit of Eastern African people.CAPAQUA is concentrating on Master’s programmes, since the availability of fellowships for Master’s programmes is limited in Eastern Africa. CAPAQUA is supporting joint degree programmes to strengtheninstitutional north-south and south-south collaboration and to enhance the quality of educational programmes by combining complementing thematic expertise and facilities of northern- and southern universities and research/policy institutions.Special emphasis is laid on skill-oriented course elements such as, laboratory & field-work, excursions, group-work, seminars, data analysis/interpretation, student presentations, report writing and environmental modelling. The obtained data/results are put into a broader context and/or legal frameworks to derive conclusions and recommendations with relevance for real world problems. Course modules include inter- & transdisciplinary aspects, action learning components and the involvement of stakeholder groups in order to achieve educational objectives at higher levels of learning.Student and staff mobility is considered as an important vehicletowards the exchange of knowledge, practises and resources in environmental-, research-, policy-, political-, economic-, social-, and cultural domains and ultimately for building trust and friendship across borders.

Expected project results

(1) International joint degree Master’s programme in "Limnology & Wetland Management (LWM)" successfully implemented as high-quality programme by Egerton University, IHE DELFT & BOKU and attraction of external funded students.

(2) National joint degree Master’s programme in “Aquatic Ecosystem & Environmental Management (AEEM)” successfully implemented as a sustainable high-quality programme by Addis Ababa University, Bahir Dar University & EIAR-NFALRC, delivering at least 20 highly qualified graduates.

(3) Concept and road-map to ensure sustainability of AEEM & LWM developed and suitable measures initiated.

Direct beneficiaries

  • In total, at least 86 young water/environmental professionals from primarily Eastern Africa are attending the LWM and AEEMMaster’s programmes, whereas 52 participants receive ADC/NFP fellowships and 34 students are external/self-funded.
  • The international joint degree Master’s programme LWMis implemented by BOKU, EGU and IHE DELFT with 49 students.
  • AAU, BDU, EIAR-NFALRC and BOKU implement the joint degree Master’s programme AEEM with 37 students.
  • At least 40 Eastern African experts from at least 10 different institutions contribute to the LWM and AEEM programmes.
  • One Train-the-Trainer workshop with 20 participants is implemented in Ethiopia for academic personnel of the AEEM partner institutions.

Project implementation

CAPAQUA 0612-00/2015is implementedby BOKU,IHE DELFT, EGU, AAU, BDU, EIAR-NFALRC, whereas BOKU is in charge of the overall coordination of the project. Each project partner is responsible for the implementation of specific project activities, which are outlined in the project proposal. The Eastern African partner institutions play leading roles in the implementation of all project activities to ensure local ownership, institutional capacity and regional networking in Eastern Africa.BOKU, IHE DELFT and EGU are planning,implementing, managing and steering the joint degree Master’s programme LWM. AAU is the overall coordinator of the joint degree Master’s programme AEEM and the AEEM programme is planned, implemented managed and steered by AAU,BDU and EIAR-NFALRC, with backstopping of BOKU. The organisational procedures are described within written agreements among the project partners in detail (Cooperation Agreements, Operational Agreements, and Education & Examination Regulation Agreements).

Project history

The CAPAQUA project 0612-00/2015builds on long lasting collaborations in limnology among Austria and Eastern Africa within the IPGL programme (International Post Graduate training programme in Limnology) implemented from 1975 to 2009 andthe experience gained within two former CAPAQUA projects(2009 to 2012 and 2012 to 2015).

  1. PURPOSE

The main purpose of the evaluationis to identify successes, as well as weaknessesand to provide recommendations for the project partners to improve the relevance, effectiveness, efficiency and sustainability of future CAPAQUA projects.

  1. OBJECTIVE

The main objective of the evaluation is to assess the CAPAQUA project achievements, to identify lessons learned and to provide recommendations to improve the relevance, effectiveness, efficiency and sustainability.

The specific objectives are:

  1. To analyse the coherence and relevance of the logical framework matrixfor the partner countries, partner organisations and beneficiaries;
  2. To assess the extent to which the project has achieved the expected results,objective and where possible the overall project objective;
  3. What were the major factors influencing the achievement or non-achievement of the expected project results and objective and how can the limiting factors be overcome;
  4. To evaluate the effect of the project activities on the direct beneficiaries (direct beneficiaries, as stated in the project proposal);
  5. To assess whether the project results and objective have been achieved in a cost-efficient manner?
  6. To assess the sustainability of (a) theproject results, (b)achievementswith regard to the project objective and where possible (c) the achievement of the overall project objective;
  7. To appraise the extent to which the cross-cutting issue gender mainstreaminghas been applied; and
  8. To identify lessons learned and to provide recommendations to improve the project relevance, effectiveness, efficiency and sustainability.
  1. SUBJECT AND FOCUS

Subject of the evaluation is the ADC funded project CAPAQUA 0612-00/2015 (project period from October 2015 to September 2018). Considered shall be as well the input of LWM & AEEM students (including module/course participants) and graduates over the whole LWM & AEEM programme duration from of October 2012 to May/June 2018 (6 LWM student and 5 LWM graduate cohorts; 4 AEEM student and 3 AEEM graduate cohorts), in order to assess effectiveness and sustainability aspects in a more comprehensive manner.

Geographically, the evaluation shall focus on the Eastern African region, with special emphasis on Uganda, Kenya and Ethiopia – hence, the desk study work shall emphasise on these three countries in particular and workshops plus interviewsshall be held in Uganda, Kenya and Ethiopia.

The evaluation shall focus on the OECD-DAC evaluation criteriarelevance, effectiveness, efficiency and sustainability. The OECD-DAC criteria impact is not subject of the evaluation, since a comprehensive impact assessment study is planned to be implemented in 2019/2020.With regard to the assessment of the OECD-DAC criteria efficiency, the subject of the evaluation are financial means provided by ADC.

The assessment of the academic quality of the Master’s programmes is not subject of the evaluation, since an external review on the academic quality of LWM has been implemented in 2014 (Prof. Jenny Day) and both, the AEEM and LWM curriculawere reviewedin 2015, including the involvement of stakeholders, students, alumni and Prof. Jenny Day.

  1. SPECIFIC EVALUATION QUESTIONS[1]

Relevance

  1. To what extent are the overall objective and objective of CAPAQUA still valid for the Eastern African partner countries, partner institutions and beneficiaries?
  2. How relevant are the following CAPAQUA strategy aspects for the Eastern African partner institutions and countries (Kenya, Ethiopia and Uganda)?
  1. Support of capacity development processes at Eastern African HEST institutions at the qualitative level via enhancing the quality and relevance of postgraduate programmes.
  2. Strengthening of national, regional and international networking;
  3. Focus on Master’s programmes.
  4. Establishing joint degree programmes in order to strengthen institutional north-south and south-south collaboration and to enhance the quality of educational programmes.

Effectiveness

  1. To what extent has the project already achieved its expected results and objective, or is likely to achieve them?
  2. What were the major factors influencing the achievement or non-achievement of the expected results and objective, including factors beyond the control of the project and how can the limiting factors be overcome?
  3. How effectivehas been CAPAQUA in terms of strengthening the capacityof LWM and AEEM students/alumni and CAPAQUA partner institutions at the individual, institutional and societal level, with specific reference to:
  4. The relevance and quality of postgraduate programmes, including quality assurancemeasures/procedures and the international harmonisation of postgraduate programmes;
  5. The support of internationalisation strategies/policies of the CAPAQUA partner universities;
  6. The individual and institutional teaching and research skills and outputs;
  7. The intensity of national, regional and international networking; and
  8. The provision of contributions towards theconservation and sustainable management of aquatic ecosystems in Eastern Africa?
  9. Did/does CAPAQUA educate well-skilled postgraduates and how successful are they at the job market (viastudent and alumni interviews, alumni surveys,career path studies and interviews of employing-institutions and stakeholders)?
  10. To what extent did/do CAPAQUAstudent and staff mobility programmes and international class-rooms lead to brain drain, versus stimulating brain circulation?
  11. To what extent was gender mainstreaming implemented?
  12. Which might be the most appropriate strategies and measures to improve the effectiveness of CAPAQUA in the context of the project’s objective and overall objective?

Efficiency

  1. To what extent have the followingproject outputs delivered in a cost efficient manner,including the comparison with eventual alternative approaches, with regard to:
  2. The number of beneficiaries, such as AEEM & LWM participants;
  3. The establishment of high quality postgraduate programmes;and
  4. The attainment of national, Eastern African and international institutional networking?

Sustainability

  1. Did CAPAQUA initiateany sustainable improvements, innovations, reforms,and/or transformation processes at the institutional level within the domains“higher education”, “science research”and “conservation and sustainable management of aquatic ecosystems” in Eastern Africa, such as:
  2. Curricula and the implementation of postgraduate programmes;
  3. Institutional management and processes (e.g. project management, academic quality assurance, quality management systems, etc.);
  4. Institutional south-south and north-south collaboration and networking; and
  5. Enhancement of institutional capacities towards the conservation and sustainable management of aquatic ecosystems in Eastern Africa.
  6. To which extent is there evidence of project ownership and project leadership of Eastern African institutions?
  7. What are the major factors affecting the sustainability of CAPAQUA and which might be the most appropriatestrategies and measures to improve the sustainability?
  1. APPROACH AND METHODS

The evaluation will be undertaken by an independent evaluator, or team of evaluators. Throughout the entire processthe evaluation will be undertaken in accordance with the “Guidelines for Project and Programme Evaluations”, developed by the Austrian Development Agency.The evaluation shall follow a participatory approach and engage a wide range of project beneficiaries andproject stakeholders in the process.

The evaluation shall include the input of LWM & AEEM students (including module/course participants) and graduates over the whole LWM & AEEM programme duration from of October 2012 to May/June 2018 (6 LWM student and 5 LWM graduate cohorts; 4 AEEM student and 3 AEEM graduate cohorts), in order to assess effectiveness and sustainability aspects at the individual and institutional level. Other groups of direct beneficiaries (as stated in the CAPAQUA project proposal) and the project partner institutions shall be considered as important source of information as well. It is suggested that workshops and in-depth interviews (or any other valuable method to be agreed on) with project beneficiaries and stakeholder groups shall be carried-out, as well as career paths analyses of LWM & AEEM graduates.

Data collection shall be triangulated to the extent possible to ensure the validity and reliability of findings, as well as recommendations. The evaluator shall engage in quantitative and qualitative analysis in respond to the specific evaluation questions and present the findings quantitatively and qualitatively, to the extent possible.The evaluation will comprise of document reviews/analyses, workshops, interviews, feed-back discussions,syntheses, recommendations and including gender aspects as well.It is expected that the evaluation identifies lessons learned, opportunities arising and provides recommendations at the strategic level, as well as concrete recommendations addressed to specific stakeholder groups.

The evaluation shall consist of several phases:

  1. Contract and kick-off meeting

The contract is signed and a discussion of the assignment and the finalisation of the work-plan takes place. CAPAQUA project documents, including available data are provided to the evaluator.

  1. Inception phase

The inception phase includes a desk study, wherein the evaluator shall review and analyse CAPAQUA key documents provided by the project implementing institution’sand analyses the logical framework, including its assumptions. Data is analysed and an initial set of findings is generated to plan and describe the evaluation methodology, list of persons/institutions to be interviewed, detailed work & time plan and key issues to be addressed within the following phases of the evaluation. The findings are expected to be put into a wider context via the review of secondary data/information of relevance (data/findings of other projects/programmes capturing CAPAQUA objective and overall objective, policy documents of relevance, etc.).The use of a data collection planning worksheet or a similar tool is required. Data triangulation and quality control are very important and need to be discussed in the inception report. The desk study findings are documented within a draft inception report, in which the evaluator describes the evaluation methodology, elaborates on how data will be obtained, analysed and presents the detailed work- and time plan. The draft inception report is discussed with the evaluation coordination team and major project stakeholders in order to include theirs feedback for the establishment of the final inception report.

  1. Field phase

The field phasein Uganda, Kenya, Ethiopia and Austria will only take place upon official approval of the inception report by the contractor.

The field phase will include:

  1. Focus group discussions on the specific evaluation questions between the evaluator, CAPAQUA project management, target beneficiaries andif available with ADC representatives as well.
  2. Workshops and in-depth interviews of LWM and AEEM students and alumni in Uganda, Kenya, Ethiopia and Austria to assess the effectiveness and sustainability of CAPAQUA project activities at the individual level and eventual effects at theinstitutional level.Considered shall be the input of at least 35 LWM and AEEM programme participants who attended the aforementionedprogrammes within the time period of October 2012 to May/June 2018 (6 LWM student and 5 LWM graduate cohorts; 4 AEEM student and 3 AEEM graduate cohorts). Baseline information for the in-depths interviews and workshops will be provided to the evaluator by the contractor in the form of LWM and AEEM alumni survey data.
  3. Interviewsof selectedother beneficiary groups (as stated in the CAPAQUA project proposal), the project partner institutions in Kenya, Ethiopia and Austria and key stakeholder institutions in Eastern Africa, such as the Ugandan Ministry of Water, NWSC Uganda, NAFIRRI, Jinja Uganda, Kampala City Council, Makerere University, Uganda; KMFRI, Kenya, etc. in order to assess the relevance, effectiveness and sustainability of CAPAQUA project activities at the institutional level.
  1. Presentation of key findings

The key findings are presented within a feedback workshop, after 3weeks of thefield phase at latest. The key findings are discussed with the contractor and if available with ADC representatives as well.

  1. Final Draft Report

Submission and presentation of final draft report and inclusion of comments from CAPAQUA project partners and if available from ADCas well.

  1. Final Report

Submission of final report, see reporting requirements under point 9).

  1. TIME-TABLE

The suggestedtime-plan for the evaluation is indicated in the table below: