How to Collect Data
that feed into WBI's Aggregate Results
A Guide for WBI Teams
Working on Capacity Development Initiatives
DRAFT
World Bank Institute
Capacity Development and Results Practice
March 5, 2011
Table of Contents
I.Introduction
A.How WBI teams feed data in WBI's results and Key Performance Indicators (KPIs)
B.How the guide relates to the Capacity Development and Results Framework (CDRF)
C.Why and when should WBI teams use this guide
II.Planning for the collection of results data
A.Features of evidence of results that permit aggregation with WBI's results
B.How to use the suggested data collection tools and approaches
C.Implementation tips on how to collect data at different stages of an initiative
III.How to collect results data as stakeholders empowered via ICOs
IV.How to collect results data on institutional capacity change
Annex A – How this guide relates to WBI's results agenda
Annex B – Frequently-Asked Questions
A.Policy
1.What is a TTL's accountability for demonstrating and reporting results of an initiative?
2.What is WBI's policy on participant feedback questionnaires (Level-1 evaluation)?
3.Which of the tools introduced in this guide are mandatory?
4.Are TTLs expected to report results per WBI's aggregate result indicators?
B.Main concepts and rationale for measuring WBI results in this way
5.How are WBI's results related to the CDRF?
6.Why is supporting evidence for results required?
7.Why do the examples of supporting evidence vary broadly?
8.Why use a mix of self-reports and evidence of results from external sources?
9.Why make some questionnaires anonymous and others by name?
10.What is the difference between a milestone, deliverable, process performance indicator, and result?
11.Is a change in network structure a result or a process?
12.How is "client-operated" defined?
13.How are the ICOs on altered processes and relationships counted in WBI results?
14.Why use a process performance indicator if it won't end up in WBI's results?
C.Practical implementation considerations
15.When should results data be collected?
16.When should WBI teams report results data?
17.How can results of an initiative be reported after the ACS has been closed?
18.Should a TTL report a result, if the contribution of the WBI initiative was minimal?
19.Will WBICR screen results before aggregating them to avoid reputational risks?
20.Can results be reported confidentially?
21.What happens if an initiative has results, but no supporting evidence?
22.Which media can be used for evidence of results?
23.Will survey data feed into WBI results if their questions and scales differ from WBI's?
24.Should teams collect and report results in numbers, percentages or both?
D.Results as "stakeholders empowered through intermediate capacity outcomes"
25.What does "direct" and "indirect" client (stakeholders) mean?
26.Does WBI account for results from direct and indirect clients in the same way?
27.Is any participant who took a course based on WBI materials counted in WBI results?
28.Why not use client survey questions to see if participants enhanced their skills?
29.Why have client survey questions relate to the client and not the group?
E.Results as "development initiatives advanced through increased institutional capacity"
30.What is meant by "development initiatives"?
31.What is meant by "advance"?
32.How are changes in institutional capacities demonstrated?
Acknowledgments
Under the overall guidance of Samuel Otoo, Manager of the World Bank Institute Capacity Development and Results Practice, Violaine Le Rouzic, Joy Behrens, Anne Gillies and Sara Okada produced this guide and its supporting tools. Sharon Fisher edited this guide.
The authors are very grateful toNatalia Agapitova, Une Lee, and Cristina Ling for their useful review and comments, to the members of the Working Group on WBI's Results and Key Performance Indicators for their valuable input that spurred the creation of this guide, and to the WBI management team for creating a strategy and an environment that fosters results.
Acronyms
ACSActivity Completion Summary
AISActivity Initiation Summary
AUSActivity Update Summary
BTORBack-to-office Report
CDCapacity Development
CDRFCapacity Development and Results Framework
FAQFrequently-Asked Question
FYFiscal Year
ICOIntermediate Capacity Outcome
IRISIntegrated Records Information System
KPIKey Performance Indicator
M&EMonitoring and Evaluation
PlatoWBI's Planning Tool
SAPSystems, Applications and Products in Data Processing
TTLTask Team Leader
WBIWorld Bank Institute
WBICRWorld Bank Institute Capacity Development and Results Practice
WBILTWorld Bank Institute Leadership Team
Page 1 of 46
- Introduction
This data collection guide and associated tools are designed to help WBI teams in charge of Capacity Development (CD) initiatives[1] plan for and then collect the information on the outcomes of their initiatives in a way that could be aggregated with and reported among WBI's results. The introduction (Chapter I) describes: a) how teams contribute to WBI results and Key Performance Indicators (KPIs), including what is meant by WBI results, b) how this guide relates tothe Capacity Development and Results Framework (CDRF), and c) why and when use this guide. ChapterII helps teams plan for how they will collect and use results data. Chapter III suggests data collection approaches per targeted intermediate capacity outcome. Chapter IV gives ideas on how to demonstrate results in terms of institutional capacity change. More information, including Frequently-Asked Questions, isin annex.
- How WBI teams feed data in WBI's results and Key Performance Indicators (KPIs)
The WBI results agenda aims to strengthen WBI's global leadership in the thinking and practice of results-focused capacity development. At the Institute level, WBI monitors its performance through KPIs. Information collected on the KPIs (both qualitative and quantitative) enables WBI to tell its performance story to the World Bank Group, as well as its partners, clients and the general public.
The WBI KPIs comprise indicators structured around the six characteristics of organizational effectiveness of the CDRF. These characteristics are: clarity of mission; achievement of outcomes; operational efficiency; financial viability and probity; communications and stakeholder relations; and adaptability. Under the characteristic "achievement of outcomes," WBI reports on the outcomes of its initiatives using two indicators of WBI's aggregate results[2], which are:
1.Direct and indirectstakeholders (a.k.a. clients) empowered through:
- raised awareness
- enhanced knowledge and skills
- Improved consensus and teamwork
- strengthened coalitions
- enhanced networks
- new implementation know-how
2.Development initiatives advanced through increased institutional capacity—of which through:
- Stronger stakeholder ownership
- More efficient policy instruments
- More effective organizational arrangements
These indicators are expressed in number of stakeholders and development initiatives and complemented by descriptions of observed changes to which WBI contributed.
It is the responsibility of WBI teams to:a) collect both numbers and descriptions of what changeddue in part to their initiatives;b) archive the evidence of results—results meaning the change(s) observed in relation to any of the two indicators above—, and c) report the information and data on what changed in the Activity Update Summary and Activity Completion Summary in SAP.
WBICR reviews the results information provided by WBI's teams, includes this information in WBI's results database (if they meet the required criteria), categorizes them, and provides a variety of aggregate reports on WBI's results, including the WBI aggregate results datathat are part of the KPIs.
Chart 1 summarizes the flow of information from the results data collected by WBI teams at the level of an initiativeto how the results information becomes part of the overall performance story of WBI.
Chart 1. Flow of WBI results data, from initiative to WBI's aggregate results
- How the guide relates to the Capacity Development and Results Framework (CDRF)
This data collection guide is part of the CDRF application tools and is therefore intricately related to the CDRF in various ways. The CDRF underpins WBI's strategy and supports WBI teams in designing and managing their initiatives for results. This guide follows the adaptive management approach of the CDRF. The data that teams collect following this guide flow into the WBI KPIs which are articulated around the CDRF's characteristics for effectiveness of organizational arrangements. Within the KPIs, the two aggregate results indicators of WBI have been built on what WBI teams reported in FY10 in their Program Results Summaries—a template that articulates a program's logic according to the CDRF. The CDRF also provides WBICR with a grid for aggregating initiative results reported by WBI teams based on this guide.
More specifically, the CDRF enables WBI teams to design and implement CD initiatives that produce changes in the capacity of WBI's clients, andin their institutional capacities. WBI results are expressed in terms of on whom oron whatthese capacity changes can be observed. The CDRF also guides the reporting of WBI's results by allowing WBI teams to describe the change process based on the results logic of the initiative. By using the CDRF in design, measurement and then reporting, WBI teams are better able to explain to others what difference is being made bytheir initiatives and demonstrate how they contribute towards improved capacities of its clients.
Each WBI CD initiativeis expected to contribute towards capacity development on two inter-connected levels, and the guide is structured around these levels:
- Intermediate Capacity Outcomes (ICOs)
As an intermediate step toward institutional capacity changes, an initiative must improve the capacity of stakeholders (clients)andmeasure what changes occur. WBI teams need to design initiatives that use a strategic combination of ICOs to leverage broader institutional capacity changes. Chapter III of this guide recommends means to collect results in terms of stakeholders empowered through intermediate capacity outcomes.
- Institutional Capacity Changes
Due to improved capacities achieved at the ICO level, WBI clientscan be expected to act as agents producing changes in higher-level institutional capacity areas: stakeholder ownership, policy instruments and organizational arrangements. Chapter IV of this guide provides ideas for collecting results data corresponding to institutional capacity changes.
- Why and when should WBI teams use this guide
This data collection guide is designed to help WBI teams plan for and then collect the needed measurement information on the outcomes of their initiatives in a way that could be aggregated with and reported among WBI's results. The guide supports WBI teams at several stages of their initiatives, and more generally contributes, to various degrees, to the implementation of several elements of the WBI results agenda.
At the design stage once the initiative's objectives are specified, the guide helps WBI teams identify results indicators and plan for the related data collection as they fill out their Activity Initiation Summaries and their concept notes. For this, teams should read Chapter II on planning for the collection of results data to understand the overall process. Teams should then go to Chapter III, spot in the left column of Table 2 their intended intermediate capacity outcomes, consider the corresponding data collection approach suggested and decide on their specific approach. For ideas on how to demonstrate results in terms of institutional capacity change, teams can review the examples in Chapter IV. For additional information, teams can refer to the Frequently-Asked Questions in Annex B.
Based on this plan, at implementation stage, the guide and its data collection tools help WBI teams produce the instrument(s) that would gather information on the results of the initiative (or any part of the initiative assessed)—starting with baseline data. As results data are collected throughout the initiatives, WBI teams report this information in the Activity Update Summary (AUS) and Activity Completion Summary (ACS) in SAP.
By following this guide, the WBI teams contribute to WBI's results agenda in a variety of ways. For more information on how this guide relates to the WBI results agenda see Annex A.
- Planning for the collection of results data
This chapter helps WBI teams plan how they will collect and use results data (including baseline) throughout the initiative using tools and approaches that enable the demonstration of WBI results at initiative level and in the aggregate. The chapter:a) indicates the features of initiatives'evidence of results that enable their aggregation with WBI's results, b)presents recommended data collection tools and approaches for WBI initiatives,andc) gives practical implementation tips using examples of how to collect data at different stages of an initiative given a variety of circumstances.
- Features of evidence of results that permit aggregation with WBI's results
WBI teams are to report the results of their initiatives in their Activity Completion Summaries. The last description section of the ACS template asks for links to the evidence supporting these results. Evidence can take a broad variety of shapes,be it quantitative, qualitative, or both. To enable the aggregation of an initiative's results with WBI's results, the supporting evidence must have common features across WBI. Specifically the pieces of information that make up the evidence need to have all of the following features:
- The evidence needs to indicate a change in the capacity of clients and/or in their institutional capacities. At minimum, a change is an observed difference between before and after the intervention. Evidence of change can be derived from an analysis of several documents (e.g., comparing clients' annual reports) and/or from information collected on one document (e.g., questionnaire asking participants to rate the extent to which the activity helped move forward the formulation of their action plan).
- The change should be logically associated to the contribution of the WBI initiative. WBI teams should be able to explain by themselves and/or provide documents that show or logically imply that their initiative contributed to the change in capacity that they report as result. WBI is concerned with contribution to results—not attribution of results. Therefore, teams only have to provide in their Activity Completion Summary a simple description that would help stakeholders understand the contribution of WBI's initiative to a result. [3]
- The evidence of the change in capacity must come from a documented external source, i.e., participants, clients, internal or external partners, observers, media, or external records. (Information based solely on self-report from WBI will not be aggregated with other WBI results supported by evidence.)
- The information on the change in capacity is provided in a format which can be counted as in WBI's aggregate result indicators.[4] In general, raw, unedited, unprocessed information (e.g., actual questionnaires on which participants answered, unedited videos, copies of client annual reports) strengthen the credibility of the evidence. Teams are encouraged to provide this raw information—whether or not they also have processed pieces of evidence.
The examples provided in this guide and its associated data collection tools are designed to help teams collect evidence of results in a way that could be aggregated in WBI's results. Therefore, teams are encouraged to use this guidance and tools. If WBI teams wish to explore other avenues, they should seek advice from WBICR.
WBI teams are to support the results of their initiatives with evidence for the following reasons:
- Monitoring results throughout an initiative informs decisions that help the initiative achieve greater results than if no attention was paid to tracking outcomes.
- Monitoring CD results requires using proxy measures, such as observed or reported change in behavior, to indicate whether capacity has been developed. Proxy measures are needed because one cannot determine just by looking at participants the extent to which their capacity is being developed. These proxy measures serve as evidence of results.
- Supporting reports of results of an initiative with evidence increases stakeholders' interest in the initiative, as the stakeholders of the initiatives'stakeholders are themselves interested in seeing what results were achieved in part with their resources.
- Supporting evidence also enhance the credibility of the results reported.
While collecting evidence of results requires some efforts on the part of the WBI teams, using this guide and its associated tools makes the process practical, simple, and inexpensive. More importantly, the process is meant to be helpful, as the guide recommends embedding the collection of results data in the design and implementation of an initiative, and to use the information to make decision on the next steps.
- How to use thesuggested data collection tools and approaches
This section presents the tools and approaches designed to help WBI teams collect information on the results of their initiatives to feed into WBI's aggregate results. The sectionexplains how to use each tool and approach, starting with the list of data collection tools linked to this guide for teams to customize to their needs before use, and finishing by summarizing other approaches to data collection that are mentioned throughout this guide.
The tools and approaches presented in this chapter are examples made available to WBI teams to facilitate their data collection. Teams decide whether and how they use these tools and approaches. None of them is mandated. Other tools and approaches might be preferred by a team given their circumstances. Initiative results will feed in WBI's aggregate results if their supporting evidence has the features listed in Section A above.
All data collection tools and approaches are designed to collect information that would help WBI teams do the following: