Programme Quality TOolBox
Selection Protocol
I.INTRODUCTION
The programme quality toolbox (PQ Toolbox) provides a set of operational standards, key actions and associated tools and guidance to support quality cash transfer programming (CTP) at each stage of the programme cycle. It is both a Sphere accompanying document, as well as a component of CaLP’s Knowledge Hub, whose overall purpose is to generate CTP-related knowledge and make existing CTP knowledge accessible and effectively shared.
The programme quality toolboxis structured around 3 main axis: first, defining quality; second, doing it collaboratively; and third, measuring quality. This protocol focuses on Section 1, Defining Quality.
It is a priority that the entire process is a collaborative exercise, in order to ensure inter-agency buy-in to the toolbox from the outset. The operational standards and key actions have been developed by the CaLP technical advisor and the lead consultant with the inputs of the Technical Advisory Group (TAG). The next stage of the development of Section 1, involves the curating and populating of the toolbox so as to support practitioners in navigating the plethora of guidance and tools that have recently been developed.
Within the Section 1, defining quality, the present protocol iscentered around the identification and selection of the most appropriate guidance and tools to be used at each step of the project cycle to ensure the operational standards are met and the supporting actions implemented.
II.Selection method
The selection of the tools and guidance to be included in the toolbox is envisioned as a two stepped approach: first, a comprehensive long list of all existing materials, and second, a phased identification process to ensure transparency and adherence to the final toolbox.
The process is led by the consultancy team, under the supervision of the CaLP Technical Coordinator. The CaLP Standard Working Group is acting as the advisory group for this piece of work, revising key elements such as the structure of the Toolbox, the standards and supporting actions as well as the present document. The Working Group is also expected to test the present identification protocol to ensure there is a common understanding of the selection criteria and application of those.
The TAG members and members of the larger community of practice have been and will be consulted at critical steps during the consultancy such as for the collation of the long list.
A.The long list
The long list of tools and guidance was first put together by the CaLP Technical Coordinator and lead consultant. It was then presented in May 2017 to the Standard Working Group and in June 2017 at the TAG meeting in Geneva so as to identifyadditional materials to be part of the long list.
It is now open for inputs with the larger community of practice through the CaLPDGroup. The objective is to have a list as comprehensive as possible, organized as per the different steps of the project cycle. The search for tools and template to include in the long list will rely on community of practice inputs through those different fora.
Any guidance or tools shared by any member of the community of practice is included in the long list. There are no inclusion criteria at this stage.
B.Identification of the guidance and tools to be included
1.Type of resources to be included
To be eligible for inclusion in the toolbox the documents must be part of one of these two categories: technical or operational principles and guidelines or tools and templates.
However, some of the criteria used to assess eligibility will be the same for the identification of both the tools and guidance, therefore the first step is to asses if the document possesses the criteria below:
- Population: Documents must be related to the use of CTP to support any type of crisis affected population whatever their age, gender, disability or pregnancy status, or their specific status according to the Geneva conventions and their additional protocols (refugees, internally displaced persons), their ethnic and religious belonging or their level of income. Documents will be eligible if they are applicable to all or only some of the above mentioned populations.
- Type of crisis: Documents must mention they are applicable to humanitarian crisis context to be included in the toolbox. By humanitarian crises, it is understood to encompass various forms of crisis, which can have one or several direct origins: internal or international conflict, ethnic cleansing, genocide, large-scale epidemics, natural disasters such as earthquakes, floods, tsunamis, droughts, economic shocks and inflation or a mix of several events. The nature of the crisis will not act as an inclusion/exclusion criteria.
- Intervention: To be eligible, the document must focus on interventions that pertain to CTP used in humanitarian crisis. CTP[1]is understood to refer to all programs where cash (or vouchers for goods or services) is directly provided to beneficiaries. In the context of humanitarian assistance the term is used to refer to the provision of cash or vouchers given to individuals, household or community recipients; not to governments or other state actors. CTP covers all modalities of cash based assistance, including vouchers. This definition, and thus the toolbox, will exclude documents on remittances and microfinance in humanitarian interventions. The toolbox will also exclude documents that solely focus on the set up, the implementation and the monitoring of social protection scheme.
- Outcomes: The toolbox will include guidance and tools looking at CTP used to reach any types of outcomes (be they sectoral or multi sectoral).
- Year of publication: All guidance and tool published after 2014 will be included. A 3-year turnover is deemed to be the limit considering how quickly the sector evolved in recent years. Guidance and tool published before 2014 will be examined on a case by case basis and their inclusion will be left to the professional judgment of the consultancy manager and team.
- Language: Only guidance and tools published in English will be included.
- Sources: Only open source documents. Documents produced by NGOs, international organizations, government agencies, think tanks will be considered alike.
Therefore, the toolbox is expected to be applicable across any type of population, context and type of crisis. The selection process will not specifically look at organising resources as per those specific criteria. However, in case a tool is only applicable to one specific setting or to type of population, it will be mentioned.
Type of documents to be included: The type of documents to be included will vary depending on whether they are part of either of these categories: technical or operational principles/guidelinesor tools and templates.
- Guidance documents: These documents give instructions on the design, implementation and monitoring of CTP across the project cycle or a specific phase. These include guidance documents on technical aspects of CTP, operational guidelines and procedures. They give indications on know-how or recommendations on a specific modality of intervention, cash payment mechanism or aspect of CTP.
Documents that also provide norms, best practices, protocols, standard operating procedures across the project cycle, contexts, organizations and types of CTP will be considered. Documents in this category also include agreements, and evidence based policy,which have led to best practices, recommendations or standards on CTP. As CTP requires a market assessment and a market analysis, guidance with a focus on those aspects of CTP will be included.
- Tools: These are documents that encompass manuals andinstructive texts on specific design, implementation and/or monitoring of CTP, across the project cycle or for a specific step. Tools include templates, checklists, matrix, modules, visuals, roadmaps, maps, decision trees, tables and figures which instruct on CTP step by step. Tools that focus on methods, methodologies and approaches to guide a practitioner will also be included. Tools are specific CTP ‘how-to’. Same as for guidance documents, tools and templates, which have a focus on market assessment and analysis will be included, as they are integral parts of quality CTP.
Non-eligible documents pertain to research, academic articles, evaluations, systematic reviews and all other types of documents that do not have an instructive purpose for CTP practitioners. Eligibility rests on operational guidance and instructions. Documents that respond to the larger debates or issues surrounding CTP, such as evidence building or advocacy, will not be eligible; unless they explain how those debates affect programming and how to respond to those debates or problems practically.
2.Quality appraisal
The criteria will serve to rank documents on the quality of the help the document provides the practitioner in setting up CTP across the project cycle or at a particular project cycle stage, or for a particular type of CTP. It will be ranked based on its:
- Quality of the information / its source of information
- Document relevance and appropriateness
- Comprehensiveness of the topic covered
- Document design and clarity
The quality appraisal tool is provided in Annex 1.
3.Identification process
A first step of the identification process will consist in making sure that all documents meet the criteria to be included as described in Section 1. Types of Documents to be included.
Following step one, we will aim to appraise the quality of each single guidance and tool, following the critical appraisal tools (in Annex 1, one for guidance and one for tools). The quality of each tool and guidance identified per step will be assessed using the tools against the five quality criteria. Each criterion contains a set of indicators, which have been adapted to appraise guidance documents and tools separately. The reviewer will measure the achievement of this indicator and attribute a score per indicator. A score of 0 will be attributed if the indicator is not met, a score of 2 if the indicator is met and a score of 1 if the indicator is partly met. At the end of the process, each tool and guidance will have a quality scoring based on their respective appraisal tool.
In order to minimize the risk of personal bias, the consultancy team and the Working Group will independently complete the critical appraisal of some steps. For time purposes the WG volunteers will score documents only on some pre-determined steps as opposed to the whole protocol. They will then share, discuss and agree the assessment of each document against each of the criteria in order to determine an overall quality score.
It is expected that the final selection will be narrowed down to 2 to 3 guidance documents and 2 to 3 tools per step, selecting those who obtained the highest quality score. This is a simplistic estimate, on average, as we acknowledge some steps will require more guidance and tools than others, such as those focusing on market assessment. As such, there may be further quality appraisal criteria added in a Step 3 if needed to further narrow down the selection.
CRITICAL APPRAISAL TOOL FOR TECHNICAL GUIDANCE DOCUMENTS:
Document type / Methodological appraisal criteria / ResponseScore* / Comment / Confidence judgment
- DOCUMENT DESIGN AND CLARITY
Appraisal indicator:
Consider whether
iThe document objective is clear
iiThe structure of the document is explained (for guidance documents this includes a clear table of contents and for a tool it includes a description of how to use the tool)
iiiThe document target audience and intended use is spelled out
ivThere are illustrations and examples for most of the sections
vThe document does not use acronyms and organisation specific jargon
viThe document is no more than 80 pages long without annexes
viiThe update process is mentioned
- DOCUMENT RELEVANCE AND APPROPRIATENESS
Appraisal indicators:
Consider whether
iThe document specifies in which type of context and for which type of programmes it is intended to be used
iiThe document is clear as to which modality is being discussed
iiiThere is a track record (based on Working Group knowledge) of people across organisations and sectors using the document to deliver programme or training
ivPractitioners can easily navigate the document by picking the steps/actions relevant to their situation
vThe document is appropriate for its aims and objectives
- QUALITY OF INFORMATION
Appraisal indicators:
Consider whether
iThere is a clear description of the process/methodology followed to develop the tool or guideline
iiThere are references to publications to back statements or to provide further readings
iiiThe document quotes at least one publication from the year before its publication
ivThere are no contradictions in language and the use of CTP is clear
vThere are no technical anomalies (based on lead consultant screening and double screening from the Working Group)
viThe document provides concrete and practical information on the operationalisation of the steps
- COMPREHENSIVENESS
Appraisal indicators:
Consider whether
iEach described action and step is sufficiently detailed to allow for standalone use for the purposes outlined in the document
iiThe content is in line with the title and the intended scope
iiiThere are no references to documents critical to achieve an action that are not open source
OVERALL SCORE :
ANY FURTHER COMMENTS:
* Scoring Matrix: 0 = No; 1 = Partly; 2 = Yes
CRITICAL APPRAISAL TOOL FOR TOOLS:
Document type / Methodological appraisal criteria / ResponseScore* / Comment / Confidence judgment
- DOCUMENT DESIGN AND CLARITY
Appraisal indicator:
Consider whether
iThe document objective is clear
iiThe structure of the document is explained (for guidance documents this includes a clear table of contents and for a tool it includes a description of how to use the tool)
iiiThe document target audience and intended use is spelled out
ivThe document does not use acronyms and organisation specific jargon
- DOCUMENT RELEVANCE AND APPROPRIATENESS
Appraisal indicators:
Consider whether
iThe document specifies in which type of context and for which type of programmes it is intended to be used
iiThe document is clear as to which modality is being discussed
iiiThere is a track record (based on Working Group knowledge) of people across organisations and sectors using the document to deliver programme or training
ivPractitioners can easily navigate the document by picking the steps/actions relevant to their situation
vThe document is appropriate for its aims and objectives
- QUALITY OF INFORMATION
Appraisal indicators:
Consider whether
iThere are no contradictions in language and the use of CTP is clear
iiThere are no technical anomalies (based on lead consultant screening and double screening from the Working Group)
iiiThe document provides concrete and practical information on the operationalisation of the steps
- COMPREHENSIVENESS
Appraisal indicators:
Consider whether
iEach described action and step is sufficiently detailed to allow for standalone use for the purposes outlined in the document
iiThe content is in line with the title and the intended scope
iiiThere are no references to documents critical to achieve an action that are not open source
OVERALL SCORE:
ANY FURTHER COMMENTS:
* Scoring Matrix: 0 = No; 1 = Partly; 2 = Yes
[1]CaLP Glossary