Youth, Population and Development in Latin America and the Caribbean

United Nations

Economic Commission for Latin America and the Caribbean, ECLAC

Twenty-eighth session

Mexico City, 3-7 April 2000

SYSTEM OF INDICATORS FOR THE FOLLOW-UP AND EVALUATION OF THE GOALS OF THE PROGRAMME OF ACTION OF THE INTERNATIONAL CONFERENCE ON POPULATION AND DEVELOPMENT FOR LATIN AMERICAN AND CARIBBEAN COUNTRIES.

A FIRST DRAFT FOR DISCUSSION PURPOSES

Partial translation of the original document LC/L.1340

This document was prepared by Mr. Arodys Robles by request of the Latin American and Caribbean Demographic Center (CELADE) –Population Division of CEPAL, as a first approach in meeting the request of the open ended meeting of the Presiding Officers of ECLAC’s Sessional Ad Hoc Committee on Population and Development.

I. INTRODUCTION

(not translated)

II. RESOLUTIONS ON WHICH THE SELECTION OF INDICATORS IS BASED

(not translated)

III. REVIEW OF EXISTING INDICATOR PROPOSALS

Several agencies have proposed minimum indicator lists to assess country situations. Some of these are briefly reviewed below.

a)Indicators to assess fulfilment of the targets established in the World Summit for Children

One of the first indicator proposals was made by WHO and UNICEF (UNICEF, 1995) in order to assess progress in meeting the targets set at the World Summit for Children. The list contains 54 indicators relating altogether to 24 of the targets agreed at the summit, mainly health and nutrition conditions among women and children. Apart from indicators on infant, child and maternal mortality, there are several specific indicators dealing with women’s and children’s nutrition, and the incidence and prevalence of certain infectious diseases, especially those that are preventable by inoculation. Lastly, there are a set of indicators on access to sanitation and health services.

b)Indicators for monitoring reproductive health (WHO, 1997)

On the subject of reproductive health, WHO evaluated a list of indicators for measuring the reproductive health situation in individual countries (WHO, 1997). The selection process used is summarized below in some detail, as it is an appropriate way of arriving at a set of indicators.

The evaluation took into account indicators proposed by a variety of agencies (WHO, UNICEF, Evaluation Project; UNFPA) (see WHO 1997), and, after identifying gaps in the capacity of indicators to describe the reproductive health situation,16 indicators were chosen based on those that were common to the different sources. The programme areas to which these indicators belonged were: family planning, maternity, prenatal care, intra-partum care, essential obstetric care, postpartum care, maternal nutrition, breastfeeding and newborn health, STD/HIV/AIDS, abortion care, adolescent reproductive health, female genital mutilation, violence against women, reproductive tract cancers, and infertility. The indicators were also classified according to the type of phenomena they sought to reflect: a propitious environment, individual empowerment with respect to one’s own reproductive health, and the provision of accessible and effective health care.

The review also considered the need for indicators to reflect the status of policies.

WHO has defined four types of indicators to monitor progress in the Health For All (HFA) programme:

Health policy indicators
Economic and social indicators related to health,
Indicators of the provision of health care
Health status indicators

All of these require quantitative information on the existence or otherwise of policies and legislation to support reproductive health targets. The WHO document (WHO, 1997) concluded that its capacity to report on changes in the reproductive health situation was limited, so further research was needed before including it in the set of indicators. Lastly, the classification also considered indicator types defined according to a logical framework such as: inputs, direct output (availability, accessibility, quality of care), process or intermediate output (use of services, knowledge and practices).

The selected indicators were reviewed by an panel of experts before drawing up a final list. The list of 16 indicators, presented in Annex 1, was then assessed in terms of the following criteria (see also box 4): ethics; usefulness; validity; specificity; sensitivity; reliability; representation; intelligibility and accessibility.

Box 4
INDICATOR SELECTION CRITERIA

Ethics

Usefulness

Validity

Specificity

Sensitivity

Reliability

Representation

Intelligibility

Accessibility of information / Data collection should maintain respect for individual rights to confidentiality, freedom of choice in supplying information, and informed consent. The context in which information is collected is also an important consideration; for example, screening for cervical cancer may be unethical if conditions do not exist to provide treatment and follow-up for the women concerned, and information in this case is unlikely to be provided under informed consent. Data collection should also be consistent with the moral values and beliefs of the population.
Given that the aim of the indicators should be to measure progress made toward a target, indicators that reflect impact are the most useful. However, many indicators are measures of mortality and so of limited use in assessing the status of reproductive health as defined by ICPD. Output and process indicators may be useful when causal links with the desired impact are known, which is not always the case. Preference should be given to indicators measuring the process that is closest to the expected result, such as behaviour or utilization of service. Another consideration in connection with the usefulness of the indicator is the availability of base level data about the situation prior to intervention or establishment of policies.
It is essential for the indicator to actually measure what it is supposed to. This requires clear definition of the object of measurement, and, in the case of measurements of risk, the population that is being referred to. The validity of process indicators, for example, depends on their direct link with the results they are designed to measure.
A specific indicator is one that only reflects changes in the result or process it is designed to measure. Changes in the value of the indicator should therefore reflect true differences in the phenomenon being measured, and not be influenced by changes in other factors. One such influence is change in coverage and quality of record systems (improvements in the quality of death registration, for example).
Sensitivity relates to the capacity of the indicator to reflect changes in the phenomenon it is designed to measure. In the case of reproductive health, many indicators relate more to adverse results rather than health status.
This characteristic relates to the capacity of an indicator to provide the same result if repeated measurements are taken at the same time using the same population. Indicators that come from surveys, for example, may be unreliable owing to problems in replies given by interviewees.
A representative indicator is one that encompasses all aspects of the relevant population. Representation is compromised if there are excluded groups, either by definition of the denominator or because the information source excludes them. National indicators also have the problem of not reflecting differences between groups, which often may be considerable.
To be understandable, an indicator should be defined simply and explicitly. Where composite measures are used, the interpretation of the indicator may not be clear.
Information should be readily available or obtainable by previously validated survey methods. Indicators based on routinely collected recorded data, however, may suffer from problems of representation and reliability.
To be understandable, an indicator should be defined simply and explicitly. Where composite measures are used, the interpretation of he indicator may no be clear.
Information should be readily available or obtainable by previously validated survey methods, Indicators based on routinely collected recorder data, however, may suffer from problems of representation and reliability.
Source: WHO, 1997.

c)Indicators for population and reproductive health programmes proposed by UNFPA

UNFPA has proposed a set of indicators for monitoring the ICPD agreements (UNFPA, 1997 and 1998). The proposal, entitled Indicators for Population and Reproductive Health Programmes, is the most wide-ranging in existence, as it tries to cover nearly all the agreements reached in the 1994 Conference. The indicators also aim to provide information on the thematic areas under which UNFPA programmes its work: namely reproductive health; population and development strategies; population advocacy, information, education and communication. Special emphasis is give to gender aspects in all areas.

One of the criteria used in the selection was that the indicators, which can be constructed from existing sources, should be relevant and robust. Other criteria used were as follows: the indicators should not impose an additional burden on the agencies producing information; they should be constructed on the basis of existing information sources; they should provide robust measures of progress toward ICPD goals and for monitoring programme outputs; the indicators should be quantifiable and capable of providing consistent repeated measurements; they should be easy for programme administrators to understand and straightforward to interpret.

The UNFPA report introduces an important idea in indicator use: recognizing that any single indicator is likely to have shortcomings, it argues that clusters of indicators should be used to summarize the status of a particular problem and monitor the results of actions. For example, to assess the sexual health situation among adolescents, the following indicators can be used: adolescent fertility rate (unsatisfied demand); the existence of legislation on minimum age for marriage (policies and administration); percentage of health centres or service delivery points (SDPs) providing reproductive health services to adolescents (coverage and access); availability of advisory services on sexual reproductive health for adolescents (administration); percentage of adolescents with knowledge about key aspects of reproductive health (IEC); percentage of students who receive education on population issues (population education). This selection of indicators fulfils the function of helping to monitor and assess programmes. The UNFPA proposal selected a short list of key indicators for population and reproductive health programmes from the proposed indicator set (see box 5).

The UNFPA indicator proposal includes a significant number of process indicators, or those providing information on the implementation of programmes. The inclusion of this type of indicator implies an emphasis on programmes and project administration and therefore requires a different type of information from output or impact indicators. The construction of indicators based on information sources originally designed to facilitate the administration and management of programmes should be seen as an important sub-product of the management system. This means including indicator construction in programme management to make it possible to adequately monitor performance.

Indicators proposed by LACWHN for monitoring ICPD progress

One of the first proposals for indicators to monitor progress in fulfilling ICPD agreements in Latin America was made by the Latin American and Caribbean Women’s Health Network (LACWHN) (LACWHN, 1998). The Women’s Health Network proposed following up ICPD implementation in five Latin American countries: Brazil, Chile, Columbia, Nicaragua and Peru. It put forward a matrix of indicators on what it considered to be key aspects of sexual and reproductive health for this purpose. The indicators are arranged along six thematic areas with three evaluation dimensions. The thematic areas are: living conditions among girl children, with emphasis on sexual violence; men’s responsibility with regard to sexual and reproductive health; participation of civil society, especially women’s organizations; adolescent access to sexual and reproductive health information and services; quality of sexual and reproductive health services; and humanized treatment of incomplete abortion. Each of these thematic areas was assessed in three dimensions: context or input indicators measuring aspects that are working to alter pre-existing situations; process indicators that report on the operationalization of changes in the country on questions of education, communication and services, through policies, programmes and resource allocation; impact or output indicators that report on the effects on the population of efforts made by individual countries. Lastly, the indicators included a set of “critical incident indicators” recording outstanding events that have occurred in certain countries.

Given that the matrix of indicators developed by LACWHN was proposed for the purpose of appraising country situations vis-à-vis ICPD goals at a given moment, it provides detailed information on the framework for implementing the agreements in each country. Most of the indicators consist of questions relating to the existence or otherwise of an appropriate legal framework, the care capacity existing in the country and the mobilization of resources.

As can be seen from the report, the result of data collection is a listing of pre-existing or recently enacted legislation, a list of programmes and plans, and actions to provide resources for such programmes. Few of the indicators are capable of assessing change in a country’s situation over time. In general, the information needed does not come from continuous records, or from censuses or surveys, but requires specific research. On the other hand, given that the aspects on which information is collected are peculiar to the administrative and legal organization of each country, comparison requires a synthesis of the situation in each country, such as was made in the report. This provides a clear assessment of the specific actions taken in each country to implement ICPD agreements.

d)Gender indicators to monitor the Beijing Platform for Action in Latin America and the Caribbean (ECLAC, 1999a)

The most recent indicator proposal for monitoring fulfilment of conference agreements are the gender indicators for follow-up and evaluation of the Regional Programme of Action for the Women of Latin America and the Caribbean, 1995-2001 and the Beijing Platform for Action. This list of indicators, prepared at the request of the Regional Conference on Women in Latin America and the Caribbean (Santiago, 1998), is intended to assist in collecting data on the situation of women as compared to men (ECLAC, 1999a).

As explained in the document, the indicators proposed depend on the availability of official statistics, except for certain indicators which require surveys on use of time. The selection criteria used include the feasibility of measuring the gap between men and women; the identification of positive values so that the indicator relates directly to positive progress; definition of an explicit numerical value for the optimal situation, in order to evaluate the gap between men and women; comparability over time and between countries; and annual or biannual measurement.

The indicators are arranged under the following topics; women and poverty, participation in micro-enterprise, access to housing; women’s education and training, schooling of girls aged 15-19 from poor households, schooling of girls aged 6-13 years, literacy; women and health, adolescent pregnancy, HIV infection, cancer detection and screening, nutrition; violence against women, sexual and intrafamily violence; existence of protection programmes and care for refugee women; women and the economy, equal pay, women in “male jobs”, differences in employment, access to credit, use of time in the home; exercise of power and decision-making, participation in the public sector, candidates for parliament, women in elective posts; institutional mechanisms for women’s advocacy; human rights, and means of diffusion.

In some cases the indicators are defined as the difference between the respective measurements for men and women, in others as two independent measurements, and, in still other cases, as the quotient between the two rates. A set of qualitative indicators is also included to provide a listing of measures or point up the existence or otherwise of programmes or laws. Other indicators relate to the magnitude of a phenomenon or the existence of information.

Although the proposal emphasizes existing sources (household surveys, health or school records), some of the measurements require additional definitions, which in certain cases makes direct interpretation of the indicator difficult. For example, the proportion of women in male occupations presupposes a classification of jobs as essentially male or female. In such cases, progress in reducing differences tends to change the way the indicator is calculated. In other cases there is likely to be a difference between the quality of information for men and women, as is the case with the remuneration of self-employed workers.

e)Framework of indicators of the Common Country Assessment (CCA, United Nations, 1999 )

The main purpose of Common Country Assessment indicator set (CCA, 1999) is to identify possible areas for priority action in individual countries.

The CCA indicator framework consists of four components:

Indicators relating to development goals and targets established in United Nations conventions, conferences and declarations. This group of indicators is based on the world listings formulated by OECD, the World Bank and United Nations, the document prepared by the Administrative Committee on Coordination (CAC) on basic social services for all, and the minimum national social data set.
Indicators from conferences and conventions relating to governance and civil and political rights.
Basic contextual indicators relating to a country’s economic and demographic conditions, which provide information needed for understanding development problems.
Thematic indicators that should make it possible to go into greater depth on important development issues, bearing in mind the peculiar circumstances of the country concerned, national priorities and needs, and multisectoral problems.

CCA indicators can be used to measure progress on many development problems, and measurements will need to be taken on at least two different occasions or, preferably, trend intervals  periods of five or 10 years, for example. In this way, changes in the degree or quality of progress on development issues can be quantified through the indicator values.

The three main criteria for selecting the indicators were as follows:

The indicators should represent relevant measures based on progress towards the objectives and targets contained in United Nations conventions, conferences and declarations; they should be clear and relatively easy to interpret; and they should provide a basis for international comparison;
The indicators should not impose an unnecessary burden on countries, and wherever possible they should be generally consistent with other world listings;
Wherever possible, the indicators should be prepared using data from well-established existing sources; they should be quantifiable and susceptible to systematic measurement over time.

As is the case with any list of indicators, the CCA framework should be dynamic and is bound to evolve over time in line with needs.