NICARAGUA

Non-Lending Technical Assistance (NLTA)

Current Monitoring & Evaluation Systems in Select Ministries

Diagnostic Report

June, 2010

  1. Introduction
  1. This diagnostic report describes activities of a non-lending technical assistance (NLTA) conducted upon request from the Ministry of Finance (MHCP) of Nicaragua. It details the monitoring and evaluation systems currently in place for following and assessing the implementation of governmental emblematic programs from select ministries. The objectives and content of this diagnostic report were agreed upon with the MHCP in an initial meeting held on January 2010 and through subsequent discussions with the Director of Medium Term Expenditures Operations, who is the duly delegated counterpart from the Government of Nicaragua (GoN) for this task. Several missions were also held between February 2010 and May 2010 to meet with the representatives from various Ministries to discuss potential World Bank support for the Monitoring and Evaluation Systems Improvement in Nicaragua.
  2. The GoN is engaged in a long-term strategy for improving its emblematic programs outcomes and its poverty reduction attainment on the population. With guidance from MHCP, the World Bank selected four Ministries for this diagnosis: Education (MINED); Health (MINSA); Agriculture and Forestry (MAGFOR); and Family, Adolescence and Childhood (MiFamilia). In consultations with the Ministries, strategic programs were then identified for the diagnosis of their monitoring and evaluation systems. This document reports on the M&E systems currently in place for 7 programs: (1) School and text book distribution – MINED; (2) PINE – MINED; (3) Immunizations – MINSA; (4) Hambre Cero, distribution of animals – MAGFOR; (5) Hambre Cero, distribution of seeds – MAGFOR; (6) Working Children program – MiFamilia; and (7) the Right to a Name program – MiFamilia.
  3. M&E systems can contribute to poverty reduction and social development by enhancing efficiency in the use of scarce resources allocated to Ministries and social projects. Currently, M&E systems in Nicaragua compare poorly with those in other countries in the region, although there is significant heterogeneity in quality among systems in the country and even within Ministries. The GoN recognizes that to increase its outputs at a local level, it is necessary to start at the Central level by improving or, in some cases,establishingM&E systems to support more effective planning and program implementation.
  4. The current document builds upon and expands on the findings and recommendations of the Modernization of the Public Sector Program implemented by the World Bank in Nicaragua in 2004 – 2009. This operation helped to implement measuresto support budget planning and the development of M&E capabilities for the MHCP. Furthermore, the project improved the budget formulation process by facilitatingthe development of a medium-term expenditure framework. Additionally, it pointed out the need for an M&E system to track the implementation and results of the Human National Development Plan of Nicaragua.
  5. This activity carried out for this report also complements World Bank investment operations in Nicaragua. For instance, the school and text book program implemented by MINED this year was part of the larger Education Sector Project (PASEN) financed by a World Bank loan (#39780).Similarly, the school feeding program in Nicaragua recently received trust fund support from the World Bank (TF 93698) to supplement food purchases in light of increases in food prices.
  6. This diagnostic report details the M&E and information-flow systems currently in place for the programs listed above. A summary of each program studied is given in the next section. The report is then organized by procedural steps, from procurement and quality control (Section III) to distribution (Section IV), targeting (Section V), service quality and delivery (Section VI), and follow-up and impact evaluation (Section VII). A logical framework organizing the information by Ministry is provided in Annex A.
  1. Scope and Methodology

In order to tackle the gap in M&E practices in government institutions, this diagnostic study describes the current status of M&E systems for the selected programs, identifying gaps in the flow of information that prevent the government from monitoring and assessing program performance. The framework used in this study is derived from an understanding that M&E systems should answer the following basic questions for these programs:

  1. Financial management:

-How are government services/goods procured?

-Are procurement processes transparent?

-Are internet auctions used?

-How does the government certify that resources are being used for programmed expenditures?

-How do prices/rates paid compare to market prices?

  1. Program implementation:

-Is the right target population being reached?

-Are services/products being delivered appropriately (right time and location)?

-Are services/products being utilized by clients/targeted population?

-Are programs being implemented as planned?

This study thoroughly describes how information is currently gathered and analyzed to evaluate program financial management and implementation.

Following the discussion of this study with the GoN, the next step will be to organize Logical Framework workshops for each of the selected programs with government officials and program managers. These workshops will help define each program's objectives, target population, services and products to be delivered, program activities, the responsible parties, and required inputs. Typical M&E questions will be elicited that analyze the transformation chain, from inputs to activities, to outputs and to outcomes. These questions will drive the data gathering and auditing activities described below.

After the Logical Framework workshops are held, the following step will be to help the GoN to establish data gathering protocols to answer each of the evaluation question elicited. For instance, to check if a program is reaching the intended target population, the GoN could request that specific questions are added to the national household survey. To check if services and outputs are delivered on time and adequately, periodic random spot surveys may be carried at a sample of service providers (schools, health centers, extension offices, etc.). To check if services and goods were procured properly, interviews with providers (both whom won and lost in the bidding process) would be carried to assess if bidding process were transparent and fair. These quality assurance surveys serve to not only verify the quality of service delivery, but also to create incentives for front line service providers to comply with program norms.

The final step will be to help the government analyze the data gathered and prepare brief evaluation reports about the quality of program implementation. These reports will pin point where the bottle necks are and what must be done in terms of program operation to improve implementation. They will also identify where program designs must be change to adapt to local conditions and resource constraints

  1. Ministries and Programs
  1. As described above, this diagnostic covers projects in four Ministries selected under guidance from MHCP: MINED, MINSA, MAGFOR and MiFamilia. Through consultations with MHCP and budgeting representatives and program directors from each of these Ministries, one to two strategic projects in each Ministry were chosen to undergo the current diagnostic study.
  2. From MINED, two programs tied to primary education were selected: (1) school and text book distribution and (2) PINE (Integrated School Feeding Program). Both of these projects have received recent support from the World Bank, and follow the distribution chain from the central level down to schools and their students. While the budget is not provided for each of these activities, it is worth noting that the current budget allocation for primary education in 2010 is around3.54 billion Nicaraguan Córdobas (US$ 177 million).
  3. In MINSA, this diagnostic study focuses on the PAI (Expanded Immunization Program), a cold-chain distribution system for immunizations that was primarily donor-run until recent years but is now fully implemented by the GoN with support from PAHO in procurement. 1.91 billion Nicaraguan Córdobas (US$ 95 million) are allocated for basic healthcare (which also includes other services such as maternal and infant health care) expenditures in 2010.
  4. The Hambre Cero program run under MAGFOR is one of the most frequently highlighted social programs by the government. Through this program, the GoN provides animals and seeds to women in poor rural families in order to promote productivity and self-sustenance in rural areas. The 2010 budget allocation for Hambre Cero alone in 2010 is of 280 million Nicaraguan Córdobas (US$ 14 million).
  5. MiFamilia is the Ministry responsible for activities that protect and uphold the basic rights of Nicaraguan citizens. As such, many of its principal activities are classified under Services to Restore the Rights of Boys, Girls, Adolescents and the Elderly, registered with MHCP as Programa AMOR. For this diagnostic, we focused on two sub-programs within Programa Amor: (1) Working Children and (2) Right to a Name. The former aims to get working children to attend school, while the latter assists families in securing registrations and IDs for their children. Programa AMOR as a whole has a budget allocation of 202 million Córdobas (US$ 10 million) for 2010.
  1. Procurement and Quality Control
  1. Nicaragua Compra (nicaraguacompra.gob.ni) is the government online portal through which GoN Ministries run their procurement process, posting calls for bids and accepting proposals (for an example, see Annex B). The calls for rice, beans, cereal, flour, and oil for the school feeding program run by MINED are made through Nica-Compra, though the program is currently piloting local acquisition of food products as well. Calls are made three times a year, and the food is ultimately purchased through the national commodity market. Once sellers deliver the food to MINED’s facilities in Managua, samples are taken for quality testing, and payments are made only once it is certified that the food products meet required standards. MINED representatives reported that receiving food that did not comply with standards is rare; the most recent instance was in 2008.
  2. For its Hambre Cero program, MAGFOR advertises its call for bids on Nica-Compra as well as in newspapers, fliers, and postings in government offices and churches. Bids for seeds are placed and accepted at the national level, but MAGFOR has exclusion to take bids for animals locally given their high transportation costs and risk of injury or damage. In the latter case, MAGFOR relies on its decentralized representation, running parallel bidding processes in each zone. MAGFOR also ensures that, for animals, their transportation from the seller to the program beneficiary is included in the biding price. This is actually the result of a recent change. Until two years ago, bids were taken for the animals alone and MAGFOR carried out the distribution. This proved to be costly and time-consuming, as it required purchasing vehicles and driving them across regions to pick-up and deliver animals.
  3. As with MINED's school feeding procurement process, MAGFOR also runs quality tests before closing the purchase deal. MAGFOR has 372 technicians distributed throughout the municipalities, who are responsible for verifying that each animal meets health standards and have the proper immunizations and that each cow purchased for distribution under the Hambre Cero program is pregnant. Once this is verified, the animals are distributed by the seller to the program beneficiaries and the payment is made. Although both programs described above use Nica-Compra to place calls for bids, neither post the bidding results online. That is, information on price, quantity, and sellers to the school feeding program and Hambre Cero is not easily accessible to non-winning bidders.
  4. The school and text book program implemented by MINED this year was part of a project financed by the World Bank, and, as such, followed Bank procurement standards. The bidding was done internationally, and the books were designed and printed in Colombia.
  5. Most vaccines used for the MINSA's PAI program are purchased by PAHO through a rotating fund to which the GoN contributes. PAI places its immunization requests a year in advance, with confirmations every 3 months, and PAHO purchases the vaccines and sends them to the international airport in Managua. Quality tests are made by PAHO as part of its mandate, and the information on the vaccines being delivered (e.g., packaging number, expiration date) is sent to PAI roughly 10 weeks before the products are delivered. Other vaccines are secured through donations from pharmaceutical companies and also delivered to the airport. Recently, some vaccines received co-financing from GAVI, but even in this case, the purchase was made through PAHO's rotating fund mechanism. MINSA personnel (not from PAI) are charged with clearing customs from the airport and pick up the vaccine containers from the international airport within 2 days from arrival to maintain the cold chain system.
  1. Distribution
  1. Except for the distribution of animals for Hambre Cero, the distribution systems for all other programs covered in this report start at the central level and spread out to the local levels. Several of them are controlled through administrative systems, using receipts to confirm delivery.
  2. The distribution systems are also not the same within each Ministry. Both the books and the foodstuffs purchased by MINED are delivered to the Ministry's facilities in Managua after purchase, but different groups are responsible for their distribution. Books are distributed by MINED offices in Managua to offices in each department, and municipal delegates are responsible for picking up those books and distributing them to the schools. Each municipal delegation is given a coupon for gas by MINED's departmental offices to finance the collection and distribution of books.
  3. Administrative records certify that the books were distributed through a series of receipts that have to be signed and counter-signed: when the books are picked up by the municipal delegations, the number of books is verified and a form is signed acknowledging the receipt of the books and the gas coupon; at the school-level, the principal is responsible for verifying the number of books and signing a form acknowledging their receipt. The municipal delegation aggregates the information from these receipts, noting the name of the schools that received the books, and transmits the information to the departmental office in paper or electronic form along with the gas coupon receipts. Some departments transfer this information to MINED's central office by email, but most information is aggregated again by the departmental offices, so that that MINED's central office only has information on the number of schools in municipalities that received books, but not the name of the schools or date of delivery. This aggregate information is kept as a record by MINED's administrative department. MINED officials estimate it will take as much as 6 months to have complete information on how many schools in each municipality across the country received the books.
  4. Staff from the MINED office in Managua also visits schools throughout the country to check whether the books have been received, but schools are not selected systematically. Similarly, municipal officials visit schools in "greatest need" to ensure that they have received the books. The World Bank has requested a survey of 19 municipalities (not randomly selected) from MINED to see if textbooks have been received, but is still waiting for the report.
  5. With the school feeding program, on the other hand, municipal delegations - generally comprised of local government members - are responsible for collecting the food from those locations and delivering them either to schools or to municipal-level distribution centers, depending on the size of the municipality and the remoteness of the schools. For schools that do not receive the food directly from the municipal delegation, a volunteer "school feeding committee" helps by picking up the food at the centers and bringing them to the schools. "School feeding committees" are mostly comprised of parents and other volunteers in the neighborhood and are not limited only to schools that do not receive food directly from the municipal delegations. It is assumed that these volunteers also help prepare the food and sometimes contribute to the feeding program at their local school with other foodstuffs besides the five non-perishable items provided by the GoN.
  6. The cold-chain run by the Ministry of Health for the PAI program follows a similar distribution system.When the vaccine containers are brought to MINSA facilities in Managua, PAI personnel responsible for maintaining the cold chain inspect each vaccine and notify PAHO on the temperature, packaging conditions (i.e., whether the box was open or not), and any other observations within a day.These reports are also kept by the PAI office for future reference if necessary.