Decentralized Evaluation Quality Assurance System (DEQAS)
Quality Checklist for Decentralized Evaluation Terms of Reference
[title of the decentralized evaluation]
OverallGeneral / Comments/Status
· Title: Clear without being too long, reflecting the subject of the evaluation
· Length: does not exceed 15 pages (excluding annexes)
· Accessibility:
· TOR written in a clear and understandable language
· TOR adequately emphasises the strategic and/or operational focus of the decentralized evaluation.
· TOR provides a good substantive overview of the subject of the evaluation.
· TOR provides sufficient information to stakeholders on how the evaluation will unfold
· TOR provides sufficient information to the evaluation team on what is expected from them.
Editing
· Template has been followed and all its elements are included in the TOR (titles)
· Acronyms are spelt out the first time they are used.
· Paragraphs and pages are numbered electronically.
· Cross-references are used.
· Tables and diagrams are used as relevant and are numbered.
· When data or quotes are used, TOR provides sources of the data/quotes (either directly below the table/graph or in footnotes)
1. Introduction
Overall: The introduction should provide key information about the purpose of the TOR and the facts relevant to the subject of the decentralized evaluation.
Expected Content / Assessment criteria / Comments/Status
· Purpose of TOR
· Factual information about the subject of the evaluation (i.e. the activity/operation/thematic area/country portfolio/transfer modality/pilot project/other being evaluated); / ü Standard text has been used.
ü Clearly indicates
1. subject of evaluation
2. Name of commissioning office
3. Period covered by the evaluation
2. Reasons for the Evaluation
Overall: Concise summary of why the evaluation is being conducted, and for who.
Expected Content / Assessment criteria / Comments/Status
2.1 Rationale and Use
· Specify why the evaluation is to be undertaken
· Specify why it is needed at this time
· Specify how the evaluation will be used by the WFP commissioning office and other stakeholders / ü Valid rationale
ü Clarity on how the timing of the evaluation meets the stated needs in decision-making processes
2.2 Objectives
· Specify the objectives of the evaluation (e.g. accountability and learning) / ü Standard text on accountability and learning used
ü Additional objectives added where appropriate and/or the accountability and learning objectives contextualised
2.3 Stakeholders and Users
· Specify the key internal and external stakeholders
· Specify intended primary and secondary users of the evaluation, both internal to WFP and external where applicable
· Indicate how beneficiaries’ perspectives will be included in the evaluation process / ü All relevant stakeholders are identified/listed
ü Demonstrates understanding of stakeholder interests and concerns
ü Identification of users is linked to the reasons for and objectives of the evaluation
ü Beneficiaries are identified ad stakeholders
3. Context and Subject of the Evaluation
Overall: Comprehensive description of the evaluation context and subject
Expected Content / Assessment criteria / Comments/Status
3.1 Context
Inclusion of information about the context in within which the subject of the evaluation has been/is being implemented, including:
· Poverty, food security
· Government policies and priorities
· Humanitarian issues
· Gender dimensions of the context
· Key external events
· Other WFP work in the area and
· work of other key actors / ü Contextual Information is focussed and concise.
ü Information is relevant and important to understanding the context for the subject of the evaluation
ü The section focuses on trend data that is relevant and important to the subject of the evaluation.
3.2 Subject of the Evaluation
· Type of activity/operation/ thematic area/transfer modality/pilot project/other subject of evaluation
· Geographic scope of the evaluation subject
· Relevant dates: Approval date; start date; expected end date
· Planned outputs at design
1. Beneficiary numbers (planned and revised) disaggregated by gender/activity
2. Amount of transfers (food, cash, vouchers)
3. Other outputs
· Main partners (Government; NGOs; Bilateral; Multilateral)
· Resources (% funded of total requirements) and key donors. If subject funded from pool funds, show resource allocated
· Key objectives, intended outcomes and activities
· Other relevant preceding/ concurrent activities/ interventions/operations
· Past evaluations/reviews related to the subject
· Maps/graphs for illustration
· Any changes in planned implementation in terms of coverage, budget, planned beneficiaries and explanations / ü Information is relevant and important to understanding the subject of the evaluation:
1. What it is?
2. When it was designed?
3. What are the key inputs ($ value)
4. What are the planned outputs (bens, mts, C&V $)
5. What is the target/scope?
6. Who is involved in the implementation?
ü Highlights relevant issues from past evaluations and reviews that are relevant to the evaluation
ü Gender dimensions explained
ü Differences between original design and final design are explained if appropriate
4. Evaluation Approach
Overall: Clarity on the scope of the evaluation and the quantity and quality of data available to support it, including any significant gaps; as well as the overall approach and aspects of methodology which the evaluation team will incorporate in their response to the TOR and in their proposed methodology.
Expected Content / Assessment Criteria / Comments/Status
4.1 Scope
· The defined scope should specify what will be included and not included in the evaluation, in relation to the activities of the subject of the evaluation
· Specify the focus of the evaluation including: time frame, issues, geographic areas, operations and types of activities and target groups / ü Clarity on what will be covered and what will not in relation to the contents/coverage of the subject of evaluation
ü Clarity on duration/period that will be covered by the evaluation
ü Clear justification for the scope and focus, including selection of activities and areas
4.2. Evaluation Criteria and Questions
· Evaluation criteria, justified as appropriate and consistent with evaluation objectives
· Specification of evaluation questions which address the selected criteria
· Gender equality and women’s empowerment mainstreamed throughout evaluation questions / ü Clear explanation of selected evaluation criteria
ü Evaluation questions are clear and relevant to the subject of the evaluation
ü Evaluation questions sufficiently address the selected evaluation criteria
ü Gender is effectively mainstreamed throughout questions
4.3 Data Availability
· Identify the main sources of information/data available to the evaluation team
· List any gaps in the data and proposed ways for the evaluation team to deal with them / ü Clear indication of the data available and their sources
ü Limitations explained and included in the TOR
ü Suggested mitigation measures
4.4 Methodology
· Standard text provided in the template.
· Present the overall methodology for the evaluation outlining the criteria, impartiality requirement, and the methods that will be employed (quantitative, qualitative or mixed)
· Identify key risks and appropriate mitigation/ management measures for the methodology approach proposed for further refinement during inception as appropriate.
· Specify how gender issues will be addressed by the methodology. / ü Standard text used.
ü Clear delineation of the overall methodology for the evaluation outlining the data collection methods and overall approach to ensure impartiality
ü Key risks and appropriate mitigation/management measures for proposed approach, satisfactorily identified;
ü Sufficient detail to understand how gender issues will be addressed by the methodology
4.5 Quality Assurance
· Standard text provided in the template of TOR.
· Identify additional measures to assure the quality of the process and product, and thus increase the credibility and impartiality of the evaluation. This may include use of external reviewers for key products / ü Standard text used.
ü Quality standards are included in the TOR.
ü Quality assurance is built into the whole process by briefly showing how different products will be quality assured
ü Requirements are spelled out for the evaluation team to ensure validity and accuracy of data
ü Reference is made to an external review, if required.
5 Phases and Deliverables
Overall: Clear statement of the overall timing and key deliverables for the evaluation.
Expected Content / Assessment Criteria / Comments/Status
· Clarify the timing of the five evaluation phases and the deliverables including milestones
· Specify if other products, in addition to the evaluation report, will be required (e.g. Power Point; Free-standing Summary, Video clip or if suggestions for innovative products are requested from the evaluation team). / ü Suggested table in annex 2 of the ToR Template is used
ü Reasonable amount of time for each of the phases/steps is provided for
ü Clearly identified deliverables with stated responsible persons
6 Organization of the Evaluation
Overall: Clear statement of the main players in the evaluation team conducting the evaluation and in WFP management of the evaluation, as well as anticipated lines of communication, timing of key deliverables and key considerations, including security, to be considered when planning to undertake the evaluation.
Expected Content / Assessment Criteria / Comments/Status
6.1 Evaluation Conduct
· Indicate how the evaluation will be conducted and led, including key aspects of communication
· Specify ethical considerations for the conduct of the evaluation
· Include the evaluation schedule
· Specific tasks and outputs are identified for team members in the evaluation schedule. / ü Specifies team structure and relationship with WFP Evaluation Manager
ü Consideration of ethical issues that may relate to the subject of the evaluation
ü Evaluation schedule is logical and provides adequate time for all phases of the evaluation, and annex 2 in the TOR is used
6.2 Team composition and competencies
· Indicate how the evaluation team will be composed (including gender, nationality, and if appropriate cultural background)
· Key competencies and knowledge of evaluation team / ü Competencies of the evaluation team are consistent with the subject of the evaluation
ü Inclusion of gender parameters for evaluation team
ü Gender expertise is included
ü Leadership competencies are spelt out for the team leader
6.3 Security Considerations
· Specify any security considerations that may be relevant / ü Security considerations are relevant to the context for the evaluation
ü Security considerations are relevant to the nature of the contracting arrangements with WFP
6.4 Roles and Responsibilities of Stakeholders
Overall: Clear statement of WFP’s expectations in terms of involvement of internal and external stakeholders and inclusion of particular groups, including beneficiaries, and both women and men and of the impartiality mechanisms established.
Expected Content / Assessment Criteria / Comments/Status
· Identify the key stakeholders of the evaluation (including beneficiaries)
· Note the specific interests/concerns of some groups in the evaluation, particularly those commonly marginalised
· Specify the intended users of the evaluation results and how results will be used.
· Include indirect stakeholders who have an active and important role in the subject/sector under evaluation, but are not directly involved in the operations subject to the evaluation.
· Identify internal (& external if used) Reference Group and establish roles and responsibilities. / ü All relevant stakeholders are identified, demonstrating impartiality.
ü Clear understanding of stakeholder interests and concerns.
ü Indirect stakeholders who may have a role are identified.
ü Clarity on measures to ensure impartiality and credibility of the evaluation, as they related to each stakeholder
ü Internal Reference Groups have been established and their roles and responsibilities have been set out
7. Communication and Budget
Overall: Clear statement of how the report will be communicated and the roles and responsibilities for communication and sharing the report.
Expected Content / Assessment Criteria / Comments/Status
7.1 Communication
· Determine communication-related roles and responsibilities
· Indicate how interactions will be conducted (meetings, teleconference, email etc.)
· Specify the need for translation and the language of each report, if not English
· Indicate if additional targeted products, apart from the evaluation report are required, for what and whom, and at what phases / ü Dissemination-related roles and responsibilities have been set out
ü Milestones for dissemination are built into the process
ü Dissemination strategy (if available) is well explained and taken into account in timeline and budget
ü Communication issues (e.g. translation, interpretation/ separate communication among stakeholder groups) are recognised
ü Communication Plan clearly developed at the Evaluation outset, with clear indication in the TOR of its main elements.
7.2 Budget
Overall: Any budgetary guidelines or limitations are clearly set out.
Expected Content / Assessment Criteria / Comments/Status
· Standard text provided in the template of the TOR.
· Identify sources of funds
· Based on the contracting approach chosen, state how the evaluation budget will be arrived at, and how total cost will be broken down
· Clarify whether the budget will include the cost of workshops or special communication efforts if needed. / ü Standard text used
ü Clarity on the contracting method to be used (procurement through open tender, LTA/SLA or HR action)
ü Clarity on the cost elements that will/should be included in the evaluation budget such as of workshops or special communication efforts included, as appropriate.
Annexes / Comment / Comments/Status
· Glossary of Terms
· Map(s)
· Bibliography
· Timeline/schedule
· Job description for individual team members (optional) / ü Assessment criteria: Complete and necessary
TOR draft QC Version August 2015 Page 4 | 7