TERMS OF REFERENCE

CONSULTANCY TO CONDUCT AFAAS SECOND MDTF END OF PROJECT EVALUATION

Contract Reference: AFAAS/ICS/2017/007

(Item 3.1.3a of the Approved Procurement Plan)

  1. GENERAL BACKGROUND

African Forum for Agricultural Advisory Services (AFAAS) is a Continental body that brings National Agricultural Extension and Advisory Services (AEAS) under one umbrella. The AFAAS’ goal is to enhance utilization of improved knowledge, technologies and innovations by agricultural value chain actors for improving productivity oriented towards their individual and national development objectives. AEAS is a key component of the innovation system, playing a pivotal role in promoting productivity, increasing food security, strengthening rural communities, and underpinning agriculture as the engine for pro-poor economic growth. AEAS is one of the key pillars for transforming rural livelihoods and contributing to Africa’s agenda 2063. AFAAS upholds the 2014 Malabo Declaration and the Comprehensive Africa Agriculture Development Programme (CAADP), whose Monitoring and Evaluation Framework is overarching for the CAADP institutions.

The AFAAS Strategy and Operational Plan- S&OP (2011-2015, extended up to 2017) has five components: (i) participation in the implementation of CAADP; (ii) information, communication and knowledge management; (iii) supporting country fora (CF); (iv) partnerships and collaborations; and (v) governance, management and funding systems. The implementation of AFAAS SOP has been supported largely through AFAAS Second Multi Donor Trust Fund, funded by the European Union and managed under the World Bank.

The AFAAS second MDTF Project Development Objective (PDO) is to support and strengthen Agricultural Advisory Services (AAS) in accordance with the Framework for African Agricultural Productivity (FAAP) principles.The expected results : i) increased country level capacity in Agricultural Advisory services (AAS); ii)increased regional and continental collaboration and partnerships in support of AAS; iii) better governance and management of AFAAS in support of AAS. Expected direct benefits of the project: i) tofacilitate development of farmer participatory knowledge systems and promote value addition, agro-processing and marketing that can better exploit economies of scale and encompass vertical, horizontal and lateral integration from production to markets; ii) to serves as a platform for sharing information, lessons, tools and approaches for efficient and effective AEAS delivery and to provide a mechanism for supporting and coordinating the development of AEAS within the CAADP framework.

In the short-to-medium term AFAAS in its S&OP was to be directly accountable for delivering on the following specific objectives:

(i)Ensure that CAADP Pillar IV sufficiently involves advisory services providers in its strategy, work plan and implementation, contributing to making these services more effective and relevant;

(ii)Ensure the availability and accessibility of appropriate and up-to-date knowledge on advisory services from a range of sources in Africa and worldwide;

(iii)Strengthen the capacity of country level advisory service stakeholders in determining own priorities and in improving their advisory service systems;

(iv)Build partnerships at national, regional and international levels between AEAS and other institutions contributing to sustained growth and transformation of agriculture; and;

(v)Build its own capacity to serve effectively as a continental African organization that can sustainably support national agricultural advisory services to continuously enhance their contribution to national, regional, continental and global development objectives.

  1. Purpose ,Objectiveand Use of the Evaluation

The AFAAS Strategic and Operational Plan (S &OP)set forth its mandate to track, assess, and widely share the lessons learned from the continental programmes and to contribute to global know-how in the field of AEAS. Similarly, AFAAS Monitoring and Evaluation (M&E) is based on the Result Framework (RF) which guides in ensuring that all activities are well aligned to support result orientation for impact. The RF ispivotal toolin AFAAS Multi Donor Trust Fund (MTDF) Project Appraisal Document. The second AFAAS MTDF ends on 31st December, 2017; therefore, it’s important that AFAAS conducts end of project evaluation to provide lessons, highlight the significant accomplishments or project potential and obtain recommendations for improvement. The objective of the end of Project Evaluation is to assess: the progress made towards achievement of intended results; effectiveness and efficiency of its implementation against expected project outputs, immediate objectives and development objective. It will also analyse the project’s achievements and identify ways through which AFAAS can be strengthened to support future programming in AEAS.

The main purpose of the AFAAS S&OP and MDTF end of evaluation is enshrined in the following:

Accountability

  • Provide credible and reliable judgements on the programmes’ results, including in the areas of programme design, implementation, impact on beneficiaries and partners, and overall results;
  • Provide high quality assessments accessible to a wide range of audiences, including donors, public and private actors in the African agricultural innovation system, government agencies, peer multi-lateral agencies, and other agricultural value chain actors.

Learning

  • Identify innovative approaches to catalyse processes toward enhancing Africa’s AEAS capacity for supporting Agricultural innovation system;
  • Identify particular approaches and methodologies that are effective in meaningfully and tangibly advancing the performance of institutions, policies, actors and the AIS in general.

Improved evidence-based decision making

  • Identify lessons learned from the experience of actors in order to influence policy and practice at national, regional, continental and global levels;
  • Inform and strengthen AFAAS planning and programming by providing evidence-based knowledge on what works, why and in what context.

Dissemination and Utilisation

Final evaluations are summative exercises that are oriented to gather data and information to measure the extent to which development results have been attained. However, the utility of the evaluation process and products should go far beyond what was said by programme stakeholders during field visits or what the evaluation team wrote in the evaluation report. The momentum created by the evaluation process (meetings with government, donors, beneficiaries, civil society, etc.) is the ideal opportunity to set an agenda for the future of the programme or some of their components (sustainability) through a Management Response. It is also an excellent platform to communicate lessons learnt and convey key messages on good practices, share products that can be replicated or scaled‐up at the country-specifically at Country Fora, regional, continental and international levels- through the Global forum for Rural Advisory Services and other fora. The evaluation team will provide inputs for the design of a complete dissemination plan of the evaluation findings, conclusions and recommendations with the aim of advocating for sustainability, scaling‐up, or sharing good practices and lessons learnt at local, national, sub-regional, continental or/and international level.

  1. SCOPE OF WORK

The scope of the project evaluation will cover all activities undertaken in the framework for the project and the evaluator(s) will compare the planned outputs of the project to actual outputs and assess the actual results to determine their contribution to the attainment of the project goal. The geographic area of intervention is Africa as a continent with specific AFAAS member countries. The timeframe of the evaluation will cover from the period of conceptualization and design to the moment when the evaluation is taking place. Specifically, the activities of the consultant will include but may not limited to the following:

  1. Review relevant documents (strategic and operational) that can influence or affect the implementation of the project; carry out interviews at AFAAS Secretariat headquarters in Kampala and with sample country Fora to evaluate the roles played and results achieved by AFAAS in support of specified programmes and to assess the extent to which AFAAS support has contributed to national and continental achievements;
  2. Develop a minimum set of criteria for assessing the project outputs and likelihood of attaining inter-immediate and development objectives;
  3. Develop research instruments (questionnaires, interview guidelines, etc.);
  4. Assess if the project pre-conditions and assumptions held;
  5. Assess the extent the project was efficiently implemented and delivered quality outputs and outcomes, against what was originally planned or subsequently officially revised as specified in the AFAAS RF, including any unplanned positive or negative consequences;
  6. Assess the extent the programme has attained development results to the targeted population, beneficiaries, participants - whether individuals, communities, institutions, etc., therefore enhancing Africa’s agricultural innovation capacity;
  7. Examine the processes and mechanisms of change through country Fora and partners’ consultation in order to identify critical factors for success and to distil lessons learned;
  8. Meeting with project staff and stakeholders on the initial findings and recommendations;
  9. Provide recommendations on how the AEAS innovations generated through country fora; communication, information and knowledge management; partnerships and collaborations can be scaled up and out.
  10. The Consultants will be required in their application, to state detailed methods they propose to apply in executing the evaluation.A mix of quantitative and qualitative methods approach is expected to be applied, to determine impact and sustainability of results achieved. In line with international best practice guidance on designing and conducting evaluations, the design stage will identify the data needs, methods, instruments and theory articulation, linking interventions to outcomes. The Secretariat reserves the right to seek further expert opinion before final decisions are taken on the methods to be applied.
  1. EVALUATION CRITERIA, QUESTIONS AND METHODOLOGICAL APPROACH

The evaluations shall be organized around the standard OECD evaluation criteria, which are relevance, effectiveness, efficiency, impact and sustainability of the programmes. This evaluation should integrate gender throughout areas of analysis and within its methodology.

  1. OECD evaluation criteria and set of questions

Relevance:

  • Were the programme objectives addressing identified needs of agricultural development in the African context?
  • What objectives did the programme advance under continental development commitments, as well as under regional legal and policy frameworks; and were these still valid?
  • Was the original programme design articulated in a coherent structure? Was the subsequent redesigning done adequately? Were the definition of goal, outcomes and outputs clearly articulated? Was the context well aligned to AEAS?

Effectiveness:

  • To what extent was the programme design coherent with the AFAAS strategic plan and CAADP priorities?
  • What has been the progress made towards achievement of the expected outcomes and expected results? What are the results achieved?
  • Were there any unexpected results /unintended effects (negative or positive)?
  • What are the reasons for the achievement or non-achievement?
  • To what extent are the intended beneficiaries participating in and benefitting from the projects?
  • Did the programme have effective monitoring mechanisms in place to measure progress towards results? To what extent did attempts (if any) to improve these mechanisms along the project bring positive change?
  • What were the efforts undertaken by the programme to change legal frameworks at the national and regional level, and what was their outcome or what could be the potential effect?
  • To what extent have capacities of partners and actors been strengthened as a result of the programme?
  • What level of information sharing is evident among Countries and partners?

Efficiency:

  • Was the programme cost-effective, i.e. could the outcomes and expected results have been achieved at lower cost through adopting different approaches and/or using alternative delivery mechanisms?
  • What measures were taken during planning and implementation to ensure that resources were efficiently used?
  • Were the outputs delivered in a timely manner?
  • Did the AFAAS and partner organizational structures, managerial support and coordination mechanisms, both at CFs and regional level, effectively support the delivery of the programme?
  • How did the programme utilize existing local capacities of tools and platforms to achieve its outcomes?
  • What extent did the knowledge management tools and platforms contribute to the efficiency of the program implementation at CF and regional level?
  • To what extent did partnership arrangements contribute to the efficiency of the program implementation?

Sustainability:

  • What is the likelihood that the benefits from the programme will be maintained for a reasonably long period of time?
  • Were the tools and platforms established demonstrate self-sustaining? What is the level of buy in by the CF and their members?
  • Were the programmes supported by national/local institutions and organizations? Did these institutions and organizations demonstrate leadership commitment and technical capacity to continue to work with the programme or replicate it?
  • Did the partners have the financial, organizational and/or HR capacities to maintain the benefits from the programme, or did they transfer such capacities to local partner organizations and/or institutions?

Impact:

  • What were the intended and unintended, positive and negative, potential and effective long term effects of the programme?
  • To what extent can the changes that have occurred as a result of the programme be identified and measured?
  • To what extent can the identified changes be attributed to the programme?
  • What is the evidence that the programme enabled the rights-holders to claim their rights more successfully and the duty-holders to perform their duties more efficiently?

Additional evaluation criteria:

  • Risk management: Identify key risks identified before and during the project implementation and to what extent they were handled and outline key issues that may inform future programme designs;
  • Governance; assess the level of compliance of governance structures. what extent are governance decisions implemented and how they shape AFAAS and its programmes;
  • Attribution of results: Assess the contribution of the project to the degree by which outcomes were achieved as a direct result of the efforts of the project (attribution), including by relating and comparing it to the contribution of other drivers and factors of the outcome.
  • Cross cutting issues: Identify cross cutting issues that were key in the process of implementation and worth considering in informing programme improvement and delivery of future programmes;
  • Lessons Learnt and sharing of experiences: Key lessons learnt throughout the period of the project which can be utilised to guide future strategies.
  1. Methods and techniques:

The evaluation will use methods and techniques as determined by the specific needs of information, the availability of resources and the priorities of stakeholders. The consultants are expected to identify and utilize a wide range of information sources for data collection (documents, filed information, institutional information systems, financial records, monitoring reports, past evaluations) and key informants (beneficiaries, staff, funders, experts, government officials and community groups). The consultants are also expected to analyse all relevant information sources and use interview and focus group discussions as means to collect relevant data for the evaluation, using a mixed-method approach that can capture qualitative and quantitative dimensions. The methodology and techniques (such as a case study, sample survey, etc.) to be used in the evaluation should be described in detail in the inception report and in the final evaluation report and should be linked to each of the evaluation questions in the Evaluation Matrix. When applicable, a reference should be made regarding the criteria used to select the geographic areas of intervention that will be visited during field visits.

The methods used should ensure the involvement of the main stakeholders of the programme. Rights holders and duty bearers should be involved in meetings, focus group discussions and consultations where they would take part actively in providing in-depth information about how the programme was implemented, what has been changed in their status and how the programme helped bring changes in their livelihoods. The evaluator will develop specific questionnaires pertinent to specific group of stakeholders and their needs and capacities (for example, illiteracy needs to be factored in, or language barriers).When appropriate, audio-visual techniques could be used to capture the different perspectives of the population involved and to illustrate the findings of the evaluation.

  1. Approach

The assignment involves both home-based work and limited regional travel to selected CFs. The type of activities the consultants might be undertake includes: (i) document review; (ii) conduct of country case studies of AFAAS work (including supervision of national consultants, as appropriate); (iii) case studies of specific AFAAS initiatives in supporting the AEAS and CAADP at continental, regional and national levels; (iv) analysis of evaluation evidence collected from different sources; and (v) report writing and editing. TheEvaluation Report shall be based on the standard outline recommended by the International Development Evaluation Association (IDEAS).

Furthermore, the evaluation will provide quantitative and qualitative data through the following methods:

(i)Desk study and review of all relevant project documentation including project documents (the approved large design grant document of the project), annual work plans, project progress reports, annual project reports, etc.

(ii)In depth interviews to gather primary data from key stakeholders using a structured methodology;

(iii)Focus group discussions with project beneficiaries, relevant partners and other stakeholders engaged with AFAAS on this project (either face to face or online);

(iv)Interviews with relevant key informants (at CF level). The AFAAS Secretariat will facilitate contact with identified individuals or institutions and obtain appointments for face to face discussions where necessary.

(v)Visits to subsidiary grantees to be guided by CFs and AFAAS Secretariat