Global HIV/AIDS Program at the World Bank

Global HIV/AIDS Monitoring and Evaluation Support Team (GAMET)

Note on Proposed Strategy and Work Plan

Draft 1

December 10, 2002

Introduction

  1. Recent years have seen a rapid escalation in international and national commitment to an expanded response to the fight against HIV/AIDS, stimulated to a large degree by the very effective technical leadership of UNAIDS and its co-sponsors. Major milestones in this process include the January 2000 UN Security Council meeting on AIDS, the UN Secretary General’s Call to Action, the launching of the large scale Multi-Country HIV/AIDS Action Program of IDA credits/grants by the World Bank for Africa and a similar program in the Caribbean. Bilateral development assistance agencies have are significantly increasing their support for the fight against HIV/AIDS and specialized groups, including in particular the US Centers for Disease Control are also intensifying their programs.[1] The preparation of two, largely consistent estimates of future resource requirements for the fight [2][3] and the establishment of the Global Fund to Fight AIDS, TB and Malaria (GFATM) also reflect growing urgency and commitment to the fight at the highest levels.
  1. As summarized in a recent report from UNAIDS [4], these steps resulted in major increases in the availability of resources for the expanded response to HIV/AIDS, suggesting that on the order of $2.8 billion will be available in 2002, a major step forward from annual global spending of less than 300 million in 1996. While there is still a yawning gap between available resources and requirements, it is equally important that attention also focus on ensuring that the resources which are available are put to the most effective use. This challenge is especially difficult in countries already weakened by persistent poverty and where inadequate and abysmally resourced public sector management arrangements make it especially difficult to ensure that resources reach the community level.
  1. Recognizing this, UNAIDS and its cosponsors provided the World Bank with a budget to house a small Global Monitoring and Evaluation Support Team (GAMET) within the Global HIV/AIDS Program to facilitate cosponsor efforts to build country level monitoring and evaluation capacities and coordinate technical support in this field. The Team was initially established in June, 2002 with the appointment of a Team Leader. The GAMET is being established to fill gaps and facilitate coordination among existing sources of M and E expertise , including the M and E units in each of the cosponsoring ageinces, and key elements of the UNAIDS secretariate, inlcuding the Country Response Inforamtion System (CRIS). It should also build networks and relationships with the monitoring and evaluaition units of other UN agencies, bilateral donors and the Global Fund to Fight HIV/AIDS, TB and Malaria (GFATM).
  1. This discussion note outlines the strategy, objectives and work plan that will guide the activities of the GAMET over the next 5 years. Its purpose is to facilitate joint planning and coordination of monitoring and evaluation (M & E)[5] support activities across co-sponsoring agencies. This plan will be modified and improved based on lessons of experience.

Context: Why does UNAIDS need a Monitoring and Evaluation Support Team?

  1. A number of factors are relevant to defining the role of a team dedicated to improving monitoring and evaluation for HIV/AIDS programs and policies:

The ever-widening scope of the epidemic and its devastating economic and social impact continues to fuel the need for urgent action against HIV/AIDS and reinforces the need for a multisectoral/expanded approach. It is increasingly clear that successfully fighting the disease requires willingness to try many approaches, stimulating action by multiple implementing partners (public, private, civil society) at all levels. Improved use of systematic monitoring tools and evaluation research at the country level will contribute to the discovery and dissemination of successful interventions for a wide variety of implementing partners.

Facilitating and coordinating the expanded response is the responsibility, particularly in Africa, of newly established National AIDS Councils (NACs). These and broadly analogous coordinating bodies in other countries, are responsible for mobilizing domestic and international financial resources and allocating resources to implementing partners in support of the National HIV/AIDS Strategy and policies. These agencies are new and require assistance to establish their capacity to monitor and track the flow of resources, and to gather and interpret reports on resource use and performance. Building their capacity to lead through soliciting proposals and using reports on performance will be an important antidote to the natural tendency for these agencies to conflate their facilitation role with the implementing responsibilities of key partner agencies (particularly Ministries of Health).

Donors, UNAIDS partners and countries themselves recognize that in every country a numerous agencies are implementing small scale efforts to prevent, treat or mitigate the impact of HIV/AIDS. There is an urgent need to ensure that countries have the capacity to identify which interventions are working so that their lessons can be shared and ‘scaled up and out’, which need modification to be made more effective, and which are less successful and should be discontinued. Doing so requires the collection and use of information on results, the development of an appreciation of the role of monitoring and simple monitoring tools appropriate to the needs of particular implementing partners.

Experience across other development sectors as well as within the health system as a whole repeatedly attests to the very high costs – to government and donor agencies alike, of multiple, donor-driven approaches to assessing and reporting on results. UNAIDS and its partners have already worked to better harmonize approaches to monitoring and evaluation HIV/AIDS programs [6]. They also worked to define a common framework and system of targets for assessing progress against the disease through the United Nations General Assemble Special Session on HIV/AIDS (UNGASS). [7] Enhancing the already strong commitment to harmonization of monitoring and evaluation procedures across the many donors and agencies active in the provision of technical and financial assistance for HIV/AIDS should reduce programming costs. Moreover, the recent growth in resources and expansion in the number of actors in HIV/AIDS risks duplication, maldistribution of effort and burdensome and sometimes conflicting advice to NACs. In the absence of capacity building efforts to shape NAC use of these resources, NACs and their implementing partners may become overwhelmed by divergent advice. Improved monitoring at the country level and coordination of technical resources at the international level will facilitate transparency and coordination of efforts and enhance country level commitment to achieving results ‘on the ground’.

Systems and capacity to learn from results at the country and local levels are woefully inadequate. Few countries, particularly those most seriously affected by the pandemic, have adequate data on population characteristics (censuses are typically out of date, and, for example, no country in Sub-Saharan Africa has a working vital registration system) and systems for biological and behavioral surveillance of the epidemic are also weak. Low pay scales in and out of government make it difficult to attract staff with adequate data analysis and interpretation skills to monitoring or evaluation positions, in the few places that they exist, and few planning and budgeting systems operate in such a way as to encourage the use of data on results in decision making.

The ‘accountability dynamics’ for development assistance naturally tend to emphasize the concerns and interests of the bodies that authorize development assistance (e.g. parliaments and executive branches of governments in the case of bilateral agencies, shareholders and global governing bodies in the case of multilateral agencies). As a consequence, efforts to improve the availability of data relevant for assessing performance and evaluating the impact of interventions tend to focus on the collection of nationally representative data, and to neglect the establishment of systems for assuring learning and accountability among stakeholders at provincial, district and local levels. As donor agencies themselves have become more ‘results oriented’ in recent years, it has become increasingly obvious that lack of monitoring and evaluation capacity at local levels is a major constraint to tracking and improving overall development effectiveness.

Despite growing recognition that donors can’t steer by outcomes if countries and implementing agencies aren’t able to do so -- donors and countries have very little experience so far in building capacity for country level monitoring and evaluation . Although donors and governments have spent many millions of dollars and years of staff time on developing health information systems and a range of service delivery statistical systems, many lie idle for lack of adequate leadership and resources, and most fundamentally, limited work to focus on using the results from information systems to improve planning and budgeting.

Last, pressures on donor agencies and their national level partners to be able to show specific results, always high, are growing, particularly as knowledge of the enormous impact of the pandemic and frustration with the pace of progress against the disease increases. Failure to help countries show that the increased resources for HIV/AIDS have been well spent and are making a difference puts renewal of existing allocations and efforts to raise additional resources at risk. Sound monitoring and evaluation is vital to sustain and grow the funding levels.

Role and Strategic Objectives for the Global Monitoring and Evaluation Support Team

  1. In view of these issues, UNAIDS and its partners have decided to house a global M and support Team at the World Bank. The GAMET Support Team will have the following key strategic objectives :
  1. Objective I: Strengthen Monitoring and Evaluation Capacity at the country level. The Team will work to ensure that all country’s participating in the program[8] have the necessary infrastructure and systems to enable them to comprehensively monitor and evaluate HIV/AIDS policies and programs. While the Team is concerned that all countries with large scale HIV/AIDS programs have the necessary capacity, its work will concentrate in the first instance on a group of approximately 30 priority countries, to include those participating in the Bank’s MAP programs in Africa and Latin America, the 25 countries that USAID has identified for ‘rapid scale up’ activities, and any other countries identified as priority concerns by UNAIDS and its cosponsors (see Annex II for a proposed list of priority countries).
  1. Support to the 38 priority countries will be introduced in three phases, with 10 countries receiving intensive capacity building support each year. Thus, 10 countries will be receive intensive support in Year 1, a further 10 countries will receive intensive support in Year 2 and another 10 countries will receive intensive support in Year 3. While intensive support will be phased in as described, all MAP recipients will receive basic M&E support as rapidly as possible, to enable them to undertake the financial and program activity monitoring required for their grant-making roles.
  • Performance indicatorsfor progress toward this objective will be based on assessments of the number of countries who have established core and enhanced M & E systems each year(see attachment I for a definition of core and enhanced systems). Other performance indicators will include the development of training curricula, the completion of the targeted number of training courses for M & E specialists and M & E partners and the identification of universities/training institutions to provide ongoing M & E training courses.
  1. Objective II: Support further harmonization of approaches to and coordination of technical assistance in support to Monitoring and Evaluation The Team will build on the excellent work that UNAIDS has led toward the development of a unified approach to monitoring and evaluation , promoting continued effort to harmonize of M & E approaches and resources across donor and technical agencies. It will do this by providing information on ‘who is doing what’ to support improved monitoring and evaluation, facilitating coordination of contributions of the multilateral and bilateral agencies to the establishment at the country level and encouraging donors to support the establishment of one overall M & E system with high quality components rather than numerous under-funded monitoring and evaluation exercises of variable scope and quality.
  • Performance indicators for tracking achievement of this objective will be 1) assessments of the Team’s contribution to coordination by UNAIDS co-sponsoring agencies, to be conducted on an annual basis, and 2) feedback from country client groups (initially NACs and key implementing partners, using a structured coordination assessment tool, administered annually.)
  1. Objective III. Promote use of monitoring and evaluation tools in HIV/AIDS policy and program decision making. Experience across development sectors, as well as within the HIV/AIDS field provide numerous examples of situations where capacity to monitor programs, and/or to evaluate their efficiency and effectiveness of programs are underutilized due to inadequate linkages between the availability of data and its actual use in decision making. While few decision makers at country or donor level would disagree that improvements in monitoring and evaluation are desirable, experience suggests that knowledge and practice on how to strengthen capacity to monitor and evaluate, and more fundamentally, how to ensure that monitoring or evaluation results are used to improve decision making, is limited. It is evident, however, that mere exhortation to improve the situation is not an adequate response. In view of the many uncertainties about how best to proceed, the Team will seek to promote cross country learning, and on a selective basis within countries, cross implementing partner learning, about how to use the results of monitoring systems, and as feasible, evaluations of particular interventions in determining HIV/AIDS policy and program priorities.
  1. The Team will use a number of methods to promote better use of data in decision making, including information technology.
  1. The Team will develop a standardized website template for NACs. This template will be available on CD and may also be downloaded from the net. It may be installed with various graphic display levels, depending on the bandwidth available in different countries. It will be supplied with instructions for easy customization, for example, of the home page, logo and contact details. However, the core architecture, events and fields will be standardized and comparable across NACs. The website will provide NACs with an important national and international information sharing tool, with respect to overall program management and monitoring and evaluation. For example, the website will contain a detailed monitoring and evaluation section, reflecting the structure of the core and enhanced M & E system, with the overall system, surveillance, research, financial monitoring and program monitoring components. NACs will be encouraged to post M & E outputs for each of these components on the website. For example, NACs will be encouraged to post biological surveillance reports, behavioral surveillance reports, relevant research studies, financial expenditure information and program activity information, capturing all the services delivered by NAC implementing partners. They will also be encouraged to post management tools, such as work plans, budgets and terms of reference for each of the M & E components, job descriptions for M & E specialists and examples of M & E reports. There will be a major section devoted to the effective interpretation and use of M & E data. There will also be an “ask the expert” forum and related discussion forum, where program management and M & E information may be sought from information and peers. There will also be an opportunity for civil society actors to post observations and criticisms.
  1. The website will enable NACs to share information effectively with national partners, by providing detailed and timely information on such topics as obligations, disbursements, utilization rates and services delivered. It will also encourage NACs to undertake international benchmarking, by for example, encouraging them to make cross-country comparisons of:
  • Overall financial utilization
  • Financial utilization by components
  • Number of public sector, private sector and civil society agreements concluded and disbursements made
  • Number of services delivered by public sector, private sector and civil society partners
  • Aggregate services stimulated through NAC support
  1. Such cross-country comparisons will motivate NACs to develop functioning M & E systems and to use these systems to compare financial and program performance and to use M & E data and benchmarking comparisons to improve performance.
  1. Other methods may include the development of a program of small grants to encourage evaluation research by country research institutions, sponsoring regional events to encourage sharing of experience , and “Results Fairs” to create incentives for groups to show case the effectiveness of their work, and promotion of ‘peer to peer’ training/learning efforts[9] for project/program teams to share and learn from each other, and the adaptation of organizational development/change management techniques, such as the ‘rapid results’ approach to helping NAC M & E officers better understand their role through gaining experience in the use of results data to improve performance.[10] A key objective of the Team will therefore be to identify and do ‘what it takes’ to promote greater use of M & E capacity in decision making.
  • Performance indicators for achieving this objective will be
  • the number of countries with documented experience in using monitoring data or evaluative information to systematically improve resource allocation, service provision and program effectiveness.
  • the frequency and quality of inter-country (‘peer to peer’) reviews of HIV/AIDS program performance, and
  • The number, characteristics of the IT and communication capabilities of participating countries

Proposed Work Program: FY03 and FY04

  1. This ambitious set of objectives will take some time to achieve, and there will likely be a need to revise the proposed approach as experience unfolds. To ensure focus on these core objectives the Team will organize its work over the coming 2-3 years along three lines of activity with specific deliverables identified on an annual basis through review by (MERG? Another set of reps of co-sponsoring agencies?) and by the Advisory Committee which has been established to provide oversight over the activities of the Global HIV/AIDS program. The Team’s work programs will be structured to support three lines activity, with specific activities and deliverables identified for each year in each of the Team’s Annual Work Plans.
  1. Details of the Team’s work program for the next two fiscal years are supplied at Tables 1-2.
  1. Activity 1: Track and strengthen country level capacity to use monitoring and evaluation tools to guide decision making on HIV/AIDS. Activities in this category will be directed in the first instance at the establishment of M & E staff and supporting consultancies for the NAC (or other nodal body) for the priority set of countries. To accomplish this, the Team will oversee the establishment and operation in year 1 of a ‘ mobile support team’ of consultants to be based primarily in Africa who share a common understanding of the Team’s approach and areas of emphasis. Each member of the mobile support team (MST) will be assigned responsibility for a ‘portfolio’ of 4-5 countries which will be their primary areas of concentration. Over time, this group will be expanded to include experts and staff from the focal countries who show an interest in promoting and facilitating use of data from monitoring and/or evaluation studies in HIV/AIDS program decision making. (The activity will also be planned to link with the establishment of the Technical Resource Network that UNAIDS is supporting).
  1. GAMET will support the establishment of the MST and will also encourage participation of key technical groups in the MST as and when feasible. Participation in the MST will facilitate linkage between the capacity requirements which develop from work on preparing a national M and E strategy and key technical resources. Technical agencies will be invited to identify specific representatives who would could be available to work along with the MST, ideally on a regular basis to ensure team continuity.

Establish and Operate Mobile Support Team (MST) The MST will focus on promoting, in conjunction with other stakeholders, the following activities: