Monitoring the millennium development goals
Draft Document
Monitoring the millennium development goals
A catalogue of procedures and an assessment of statistical capacity
By Ludovico Carraro, Salman Khan, Simon Hunt, Georgina Rawle, Matt Robinson, Manos Antoninis – Oxford Policy Management
Copyright-2003, by The Department for International Development
Enquiries concerning reproduction should be sent to:
The Department for International Development
Contracts Department
Abercrombie House
Eaglesham Road
East Kilbride
Glasgow GT 75 8EA
This report has been prepared by Oxford Policy Management
The findings, conclusion and interpretations expressed in this document are those of Oxford Policy Management alone and should be in no way taken to reflect the policies or opinions of DFID.
Comments are welcome and should be addressed to Ludovico Carraro (e-mail: )
May 2003
Preface/Acknowledgements
In the compilation of this report, the authors received valuable help and support from the Paris 21 task team, in particular from Sarah Hennell of DFID, Neil Fantom of the World Bank, and Martin Dyble of Eurostat, who kept in regular touch and provided valuable data and feedback.
In addition, a number of individuals from various organisations provided data and statistics, useful advice and comments, and responses to various queries, all of which helped substantially in the production of the report. In this regard the authors would like to acknowledge the help of: Rachael Beaven from DFID; Colin Mathers, Carla AbouZahr, Mercedes de Onis, Monika Blössner, Tony Burton; José Hueb, Margie Schneider, and Abdul Ghaffar from WHO; Tessa Wardlaw, Roleand Monasch from UNICEF; Douglas Lynd, Olivier Lobe, and Anuja Singh from UNESCO; Desmond Jones from UNAIDS; Habib Khan from the Ministry of Education in Pakistan; Chet Chaulagai from the Ministry of Health in Malawi; Kristi Fair from Macro International in relation to the education data section of the Malawi DHS; Hammad Ali from the Federal Bureau of Statistics in Pakistan; Simon Scott from the OECD; Michael Minger from the ITU; and Loganaden Naiken, in relation to issues concerning the calculation of dietary energy consumption data.
Executive Summary
Background
This short study was commissioned against a background of attempts by the Millenium Development Goals (MDG) Task Force, and others, that are aimed at improving the quality of MDG indicators.
The particular aim of the study is to provide a better understanding of the monitoring process and its standards and to highlight areas of possible improvement.
There are three main outputs that have been produced as part of the study:
1. a catalogue - which documents for all 48 indicators, the Agency responsible for reporting a particular indicator; the definition of the indicator; availability and timeliness of data; the original sources of the data used in deriving the indicator; the actual construction of the indicator and the process of reporting it; the checks performed on the original source data; and a comparison of the MDG data with other international data source.
2. an international report – which documents the key quality related issues that arose out of the cataloguing exercise containing both an international overview of the issues and some suggestions that might explored to resolve them;
3. a national report – which examines MDG processes in two countries Pakistan and Malawi.
The Catalogue
The catalogue is a compilation of 48 tables, one for each indicator, that form part of a separate document. The catalogue is very much a work in progress as data, definitions, sources, processes and systems for calculation and dissemination are being constantly updated.
The International Study
The International Study comprises the first part of this present document. On the basis of the work undertaken in developing the catalogue it identifies five main issues that if addressed would substantially improve data quality:
· Definitional issues – need for consistent usage across all users;
· Data availability – can be poor and raises problems vis-à-vis the quality of regional estimates, and trends in performance;
· Issues that arise from modelling exercises – using models to estimate missing data raises issues about the value of estimates, and in particular those models that are calibrated on old data;
· Timeliness – indicators that use household surveys deliver data that is on average 3 to 5 years backdated;
· Comparability – cross-sectional and time series comparisons can be difficult not only because of the problems of consistent definitions, data availability, modelled estimates, etc., but also because of different methods used in the calculation of particular indicators (see Table 1 in Appendix 1).
The study discusses five activities that if addressed could help to overcome some of these shortcomings:
· an appropriate use of available data from household surveys - especially through the consolidation of existing survey networks and through more proactive use of surveys when data is missing and old;
· changes in the use by international organisations of data reporting questionnaires sent out to national governments – there is scope for improvements in quality control;
· changes in the use of international population data in the calculation of some indicators- in a number of instances a mix of population estimates rather than single population estimates are used for the calculation of indicators with inevitable consequences for credibility of some indicators in some countries and regions;
· changes in the management of common methodologies and definitions – for some indicators international agencies still strive to achieve consensus on definitions and uniformity in their application; and
· changes in data management practices – in particular providing footnote details on data points; the sources of data; and even access to original datasets would help to improve the interpretation and comparison of particular indicators.
The National Study
The National Study comprises the second part of this current document. It provides the pre-cursor for a larger country study to be undertaken later in 2003.
Using the experiences of Pakistan and Malawi, this brief desk study:
· makes the comparison between national used data and the internationally available data under the MDG
· compares indicators that have been selected for monitoring the PRSP with those under the MDG, and
· briefly discusses statistical capacity in both countries to monitor MDG and PRSP indicators.
Further Issues
Finally, this report identifies a number of areas in which international efforts might be focussed, and suggests some of the issues that might be taken forward in expanded country studies.
Table of contents
Preface/Acknowledgements 2
Executive Summary 3
1. Background 3
Table of contents 5
2. Introduction 7
3. International study 8
3.1. Salient characteristics of the MDG indicators 8
3.1.1. Definitional issues 8
3.1.2. Data availability 9
3.1.3. Issues that arise from modelling exercises 10
3.1.4. Timeliness 10
3.1.5. Comparability 11
3.2. Key issues to address 12
3.2.1. An appropriate use of available data from household surveys 12
3.2.2. The use of international agency questionnaires for data reporting 15
3.2.3. The use of international population data in the calculation of indicators 16
3.2.4. The importance of common methodologies and definitions 17
3.2.5. Data management 18
4. Country studies: Pakistan and Malawi 20
4.1. Introduction 20
4.2. Comparison of national and international data on MDGs 21
4.3. Comparison of indicators monitoring MDGs and PRSP in Pakistan and Malawi 22
4.4. Statistical capacity to monitor MDG and PRSP indicators 24
5. Further Issues 26
5.1. International study 26
References 28
Appendices 30
1) 48 tables with indicators 41
2) Pakistan 41
3) Malawi 41
Abbreviations 42
1. Introduction
This report summarises the preliminary findings of an on-going study that assesses the status of the capacity to monitor the Millennium Development Goals (MDG). The study has two components: first, an international assessment of how international lead agencies are monitoring the MDG; and second, some country-specific analysis on how two governments are engaged in the reporting process of the MDG as well as an assessment of their ability and commitment to monitor the MDG indicators.
The aim of the study is to get a better understanding of the monitoring process and its standards, and to highlight possible areas of improvement. Indeed, other reports have already recognised “serious and far-reaching problems in almost all of the millennium development goals indicators in terms of data availability, accuracy, coverage in order to produce global and regional estimates, and consistency over time” (sic) (Inter-agency Expert Group on MDG Indicators, 2002). However, these problems are not comprehensively documented, moreover there is no systematic account of the methodologies used by the various agencies in compiling the data. It is in these two main areas that this study wants to make a first contribution. Therefore, this paper does not comment on the international community’s progress towards reaching the goals, nor is it an attempt to discuss which indicators are better in monitoring the goals, or what the indicators aim to capture. Instead, it focuses merely on understanding processes of data reporting and compilation, and the methodologies behind them.
In particular, the international study has developed a catalogue of the various processes in place to monitor the 48 indicators of the MDG.[1] For each indicator the catalogue specifies: the agency responsible for reporting international statistics (as identified by a meeting of the relevant stakeholder institutions in March 2002); definition of the indicator; timeliness and availability of data; original sources; construction/compilation of the indicator (estimates and adjustments); process of reporting; checks performed on the original sources; and, a comparison with other international sources.
The country studies (at the moment Malawi and Pakistan) aim at presenting examples of the interaction between lead international agencies and national authorities, and at investigating available national statistical resources and whether these are appropriately used in the reporting process. Moreover, the country studies compare the set of indicators identified by the Poverty Reduction Strategy Papers (PRSP) with the MDG indicators to analyse both to what extent the international monitoring effort is shared by national targets and how national statistical production is institutionalised to produce reliable data.
This paper presents the main findings of the study divided into the international and country component, while the catalogue and the documentation material of the two country studies are included in special appendixes.
2. International study
MDG monitoring and its coordination by the United Nations Statistics Division (UNSD) began in 2001. Both monitoring activities and their coordination is still evolving with the existing framework of responsibilities providing the platform for the further development of responsibilities and systems. Indeed, since this study begun (March 2003), a number of changes in systems and methodologies have already taken place, and it is recognised that there might be further changes underway of which the present analysis is unaware. Nonetheless, this summary highlights some general points that may provide a platform of support to further innovations and improvements.
We have divided this section into two parts, firstly we provide some observations on the salient characteristics of the MDG indicators and their compilation that have arisen from this study’s work on the indicator catalogue; and secondly, a discussion of the key issues
2.1. Salient characteristics of the MDG indicators
The 48 MDG indicators combine those that have been widely used and for which data reporting systems are well established with those that are relatively new or indeed completely new to a broad international user group. As a result the pattern of data coverage and data quality is very variable across the range of indicators.
For example, for an indicator such as “the prevalence of underweight children below five years of age” the Global Database on Child Growth and Malnutrition (WHO) has been collecting and categorizing relevant data since 1986. However for an indicator such as “the proportion of households with secure tenure”, there is effectively no data available.[2]
The cataloguing exercise raises and echoes the issues raised by the Inter-agency Expert Group on MDG Indicators namely:
1. Definitional Issues
2. Data availability
3. Issues that arise from modelling exercises
4. Timeliness
- Comparability
2.1.1. Definitional issues
For some indicators – usually for indicators that are relatively new to the debate – there can be a lack of consistency in the use of a common definition. Inevitably some of the indicators related to new problems are subject to more debate and more revision compared to others for which indicators are better understood and widely used.
This is the case for “the condom use rate of the contraceptive prevalence rate”[3], “number of children orphaned by HIV/AIDS”[4], and “the unemployment rate of 15 to 24 years olds”, where several indicators rather than a single indicator are being used in practise.
Furthermore, indicators that monitor the spread of HIV/AIDS that were identified in 2001 have subsequently been changed. Nevertheless some agencies still refer to the earlier indicators, or use slightly different definitions.[5]
2.1.2. Data availability
For a number of key indicators based on country reported data or household surveys, data availability (as measured in terms of number of countries with at least one observation after 1995) is relatively poor. This raises serious doubts vis-à-vis the credibility of regional and global estimates. This is particularly true for indicators monitoring goal 2 (education), 3 (gender equality) and 6 (combating HIV/AIDS, malaria). Not only is the number of countries for which data are available relatively low, but information is also missing for particularly large countries (China and/or India).
A further issue is that of the number of observations available for the same country over time. This is essential to assess and monitor changes. In general, for those indicators for which there is a general lack of data in the period since 1995, it is also the case that there is insufficient data to make comparisons over time.
2.1.3. Issues that arise from modelling exercises
Some of the indicators raise issues because, in the absence of actual observations, estimation models, using the limited available data, are engaged in generating estimates. The two main issues are that:
- the appropriateness of data and models used in the estimation exercises especially for models of literacy and mortality rates, maternal mortality rates, and malaria prevalence, and
2. some of the methodologies for estimating missing data, and in particular approaches that provide estimates based on key informant responses, do not provide “statistical” estimates.