Impact Evaluation of Performance Based Contracting Schemes for
General Health and HIV/AIDS Services in Rwanda
Research Support Grant Proposal
April 2006
Submitted by: Dr. Paul Gertler (Chief Economist HDNVP)
1. Specific Aims
The global shortage of human resources for health care delivery is reaching crisis conditions in the poorest countries, adversely affecting the lives of millions and preventing achievement of the Millennium Development Goals. (Chen 2004, Joint Learning Initiative 2004, World Bank 2005). The deficiency of well trained and highly motivated health care workers in developing countries is reflected by the high levels of absenteeism (Chaudhury et al forthcoming) and worker emigration to richer countries (Stilwell et al 2004). The human resource crisis has received particular attention in recent years with the calls for massive increases in ODA to combat the ongoing HIV/AIDS crisis, make faster progress towards achieving the child and maternal health MDGs, and address the persistence of treatable infectious diseases such as malaria and tuberculosis.
The global community is searching for solutions to this crisis in order to improve both the delivery of health care services to poor communities and the health status of vulnerable populations. One promising method is to use pay for performance or performance based contracting (PBC) as an incentive to motivate health care providers to see more patients and provide better quality of care. Typically PBC mechanisms are two-part tariffs that pay providers a fixed amount plus an addition that depends on some performance or outcome indicator. Many times the indicator is a measure of utilization such as number of mothers who start prenatal care in the first trimester or number of child growth monitoring visits. By conditioning part of the payment on utilization, there is a financial incentive to show up and increase quality to attract more patients. However, PBC may provide an incentive to decease quality in excess demand situations. In this case, the PBC needs to also condition on quality indicators or at least verify a minimum level of quality.
Performance based contracting is also a mechanism to implement the WDR (2004) framework of empowering the client. Since payment is conditioned on patient utilization decisions, money follows their feet. In this light, PBC interventions also serve as a means for governments and aid agencies to allocate limited resources more efficiently to services used more by vulnerable groups such as the treatment of infectious and parasitic diseases.
We propose to evaluate the impact of a new PBC scheme for health care being implemented in Rwanda for basic preventive and curative health care services and for treatment of HIV/AIDS. Specifically, we intend to test the hypotheses that PBC for both general and HIV/AIDS health care providers increases the quantity and quality of health services delivered, as well as improves the health status of the population. In addition, the HIV/AIDS component will use the PBC as an instrument to ask whether improved health status from access to ART improves the socio-economic status HIV/AIDS patients and their families.
We are requesting $75,000 to supplement an existing budget of over $1.,14 000,000. million. The current budget supports the experimental design and monitoring of the intervention, household and facility survey design, and data collection. We seek additional funds to support supervision of the second round survey and the analysis of the data.
2. Background and Significance
The 2004 WDR highlighted improving access and quality of services for the poor as the critical next step towards improving human development and alleviating poverty. A critical theme presented in the 2004 WDR and echoed in the 2004 Joint Learning Initiative is the central role of human resources in the delivery of health services. Medical personnel are the software that combines health care inputs such as diagnostic tests, pharmaceuticals and therapies into the clinical content (i.e. quality) of care. Investment in health centers, access to affordable and effective drug therapy, will not be enough to combat the persistence of HIV/AIDS, malaria, tuberculosis, child and maternal mortality rates without an efficient and motivated workforce. Hence, massive increases of donor aid into a country without sufficient human resources or a poorly motivated workforce is unlikely to have a sustainable impact on health outcomes.
A number of studies have demonstrated a direct link between a strong health care workforce and specific health outcomes. Cross-sectional data analysis presents significant relationships between maternal, infant and under-five mortality rates and the density of health care human resources (Anand and Baminghuasen 2004; Robinson and Wharrad 2000 & 2001). Additionally, panel data over the course of 30 years in 21 OECD countries suggests that physician density predicts child mortality (Or et al 2005). In a recent study, Barber et al (forthcoming) use a change in civil service hiring to identify the effect of changes in health care personnel and health outcomes in Indonesia. The authors find that the changes in the number of medical doctors and midwives in a community are significant predictors for the height of children less than 12 months.
This research suggests a need to investigate the effects of various incentive schemes for the health care workforce such as PBC. Since PBC is fairly new as a means for providing incentives for health care workers, there is limited evidence as to its effectiveness. In Cambodia, HealthNet International applied a contracting approach based on performance within 8 districts, serving 1 million people. Preliminary indicators show an increase in health care utilization, as well as a decrease in total family health expenditure resulting from the strict monitoring of the performance based scheme (Soeters and Griffiths 2003). Performance based contracting in Haiti demonstrated significant increases in the number of immunizations for children, as well as accessibility of birth control to mothers. (Eichler et al 2001).
Our research will expand greatly on these studies with an evaluation of performance based contracting interventions in Rwanda. Rwanda serves as a prime setting for evaluating the impacts of performance based contracting in the health sector for several reasons. First, it is imperative to decrease infant and child mortality rates, with 10% of children not surviving to their first birthday, 20% of children not surviving past their fifth birthday, and 43% stunted. Treatable infectious diseases, such as malaria, are the leading causes for infant and child mortality. In addition, maternal mortality in Rwanda is one of the highest in the world, with nearly 1 out of 6 deaths of women between the ages of 15 and 49 resulting from maternal causes.[1] Finally, the delivery of HIV/AIDS treatments could overwhelm the health care delivery system as the infection rate is over 3 percent. Improvement and expansion in the delivery of health care services is a key factor in addressing these issues.
There is reason to believe that the PBC will have positive impacts in Rwanda based on small scale pilot data from two districts, Butare and Cyangugu. Through PBC, facility budgets increased by close to 45 percent. The largest single use of the performance based funds was for staff compensation, with between 40% and 95% of the subsidies channeled to staff. Comparison of health worker income between PBC health facilities and non-PBC health facilities show that workers under the PBC scheme have a 22.7% higher compensation. Qualitative assessment revealed that approach resulted in a shift in the mentality of health care managers from passive to pro-active entrepreneurship. In addition, facilities reported a large increase in utilization. Several facilities indicated that in meetings to allocate the funds, providers who were regularly absent were explicitly excluded from any increase in compensation. The facilities noted that these providers started showing up subsequent to the meetings.
This research will provide some of the first rigorous empirical evidence as to whether PBC is a feasible method for increasing access to quality health care services, improving quality of care, and significantly increasing health outcomes. It will also be the first study of PBC in the African context. This work is especially timely since the human resources crisis, lack of progress towards the MDGs, and the health needs of vulnerable populations (e.g. HIV/AIDS and Malaria) are the greatest in Africa. This research will enable us to expand our knowledge base on the link between programs designed to improve worker motivation and the quality of health care provided. The anticipated results from our evaluation of PBC programs for health service delivery in Rwanda will provide valuable information regarding the effects of results based program design on specific stakeholders’ behaviors, as well as how this design may impact specific health outcome goals These results will not only fundamentally serve the Rwandan government and donor agencies as they prepare for expansion of performance based contracting programs within Rwanda, but also the international community as it searches for more effective means for addressing the human resource crisis in health care.
3. The Intervention
The PBC scheme is being implemented as part of the Health Sector Strategic Plan under the umbrella of the Poverty Reduction Strategy Credit (PRSC). The rules of operation have been developed and the funds available for implementation have been included in this year’s Ministry of Health budget.
The PBC scheme involves the transfer of conditional funds to public health care providers to supply a package of basic health services to the population. The payments are two part tariffs, where the first part is a fixed budget to finance a minimum set of inputs that is independent of performance, while the second part is a payment whose size depends on a set of performance indicators. Funds from the second performance-based part can be used for any purpose, including topping up staff salaries, at the discretion of the facility. The performance-part of the PBC can increase provider budgets as much as 50 percent if they meet all performance targets.
The indicators consist of a set of essential services in the areas of prenatal care, neonatal care, postnatal and new born care, immunization, delivery by skilled attendants, family planning, VCT, as well as ART when available. In addition, providers together with local NGOs would also be able to obtain additional resources through the success of public health promotions for bed net use, and increased use of safe water and sanitation. At the hospital level, the indicators additionally include inpatient days, major surgeries, and complicated deliveries including caesarean sections.
The Districts contract health centres and hospitals, with the terms of the contract uniformly laid out by the central government. Districts are responsible for distributing the funds after verifying that indicators which trigger payment have been met and that providers are supplying at least a minimum quality of service. The Ministry of Finance and Donors will provide the funds to the District authorities.
In order to accurately track each facility’s performance, the Government has improved routine data collection by standardizing health facility registers, strengthening the computerized health information system, as well as improving data analysis and feed back mechanisms. In addition, in order to prevent data falsification, community based organizations verify accuracy of records by following a random sample of patients from the health facility in order to match subsidy payment with actual service delivery. Any instance of falsification results in a 10% reduction in subsidy payment the following month, with a second offence resulting in cancellation of the contract.
4 Experimental Design
The evaluation will take advantage of a prospective quasi-experimental design. There are approximately 300 health care facilities in Rwanda. The Government intends to eventually expand PBC to all facilities. The evaluation sample consists of 168 facilities, with approximately half the facilities receiving PBC in 2006, and half in the next phase of the rollout. The facilities who are incorporated into PBC first will be the treatment groups and those incorporated 24 months later will be the controls. While the treatment facilities will receive extra resources through the PBC, the control facilities will also received extra resources of an equal amount through an increase in their input based budgets. The purpose of this is to test the effect of the PBC incentives and not confound the identification with an increase in resources. If we did not also increase the input budget of the control facilities, then we would not be able to tell if any increase in outcomes in the treatment facilities relative to the controls facilities was due to the PBC incentives or due to the increase in resources through PBC.
Pre-intervention baseline data and post-intervention follow-up data will be collected in both treatment and control areas, and used to construct difference in difference estimates of the impact of PBC on facility performance and individual outcomes. The identifying assumption for difference in difference analysis is that the change in the controls is a consistent estimate of the counterfactual, i.e. the change in treatments if the treatments had not been treated. The Government designed the rollout of PBC to take place at the District level. Hence, all facilities in a district must be in either the first or the second phase of the rollout. Since we are unable to use random assignment to balance the treatment and control samples, we need to explicitly balance on observed characteristics.
Balancing the districts in Phase I and Phase II required several steps. First, different regions were mapped and overlaid with information on relief, rainfall, and population density for the Rwanda demographic atlas (census 2002). The areas of the country without current performance based contracting in health centers were then paired in groups, where the members of the pairs had similar characteristics for relief rainfall and population density. In addition, we verified from Rwandan collaborators that within each of the pairs, there were no important differences in livelihoods. For example, areas where agriculture is the dominant livelihood were not paired with areas where livestock raising was dominant. Once districts were grouped into similar couples, each district was then assigned to either Phase I or Phase II of the program. Districts in Phase I are considered “treatment” localities, whereas districts in Phase II are considered “comparison” localities. For reasons of equity and fairness, assignment into Phase I or Phase II was determined by random assignment for each of the couples.