Getting Research Into Practice

GETTING RESEARCH INTO PRACTICE:

THE EXPERIENCE OF THE ‘SAZA’ PROJECT

An opinion piece

by

Jane Doherty,1 Lucy Gilson1,2 and Di McIntyre3

1. Centre for Health Policy, University of the Witwatersrand, South Africa.
2. Health Economics and Financing Programme, LondonSchool of Hygiene and Tropical Medicine, United Kingdom.
3. Health Economics Unit, University of Cape Town, South Africa.

June 2002

Prepared for:

“Establishing and Reinforcing Links between EC-financed Health Systems Research and Technical Assistance/Development Projects”

24–26 June

Department of Tropical Hygiene and Public Health, University of Heidelberg, Germany

1.Introduction

Between November 1997 and April 2001 the European Union funded a research project which evaluated health financing reform in South Africa (SA) and Zambia (ZA). The ‘SAZA’ Project, as it became known, was funded through the European Union’s DG XII International Co-operation Programme,[1] and involved five research partners, three from Africa[2] and two from European countries.[3] The South African component of the project examined the evolution of three high-profile policies during the first term of the first democratic government (1994-1999). These were the removal of fees for certain health care services (including primary health care), social health insurance, and the geographic re-allocation of budgets according to a population-based formula.

This paper recounts the experience – from the South African research team’s point of view - of trying to ensure that the project’s findings influenced policy-making. In doing so, the paper draws heavily on the findings of the project which itself scrutinised the relationship between technical analysts and decision-making in government.[4] The aim of the paper is to contribute to ideas on how to maximise the appropriateness and impact of research relating to government policy, without stifling the independence and creativity that allows academic institutions to initiate innovative projects. The paper concludes by reflecting on how these ideas might be applied to the activities of donor-funded research and development projects.

2.Background

In 1996, when the proposal for the SAZA Project was drafted, South Africa’s first democracy was only two years old. Government was still transforming itself into a non-racial, quasi-federal system with nine provinces. The new health ministry (known as the Department of Health or DOH) was restructuring along these lines, whilst at the same time introducing major reforms that included the development of a district health system based on the primary health care approach. In this climate of intense policy development and implementation, a long-term, academic evaluation of a subject as impenetrable as health financing reform may have seemed somewhat of a quirky choice of project, especially as the project’s emphasis was less on the economic analysis of the design of reforms than on understanding the process of change.

Nonetheless, by the time results began to emerge in 1999, the new government was reflecting more generally on the successes and limitations of its first term in office, and strategic difficulties in pushing through controversial reforms had sensitised many in government to the problems of change management. The project was able to provide a comprehensive analysis of the actual or likely impact of health financing policies, and explain why certain reforms had succeeded whilst others had failed, pointing sometimes to problems in the design of a reform and sometimes to problems in the way a policy was formulated or implemented. Feedback from the eleven reviewers of the project’s draft report, and from a much wider range of government officials and independent analysts who accessed the final report, has been that the findings and recommendations were credible and useful. In this sense, at least, the project could be labelled ‘a success story’. The following section assesses whether the research results have had a demonstrable impact on the approach to policy-making in South Africa.

3.The impact of the project

To date there has been no independent evaluation of the project.[5] The points that follow have simply been generated by members of the research project through a process of reflection. The discussion divides impacts into two parts, the first relating to impacts on policy, the second to impacts on capacity development. It is by no means claimed that the project has been alone in generating these impacts. Rather, the project has clearly been one amongst a number of mutually reinforcing influences on a changing environment.

3.1The impact of the projecton policy

It is relatively easy to point to the direct impact of research, that is, when it is instrumental in producing ‘changes in behaviour and practice.’[6] However, research may also have a more conceptual use, contributing indirectly to policy by producing ‘changes in levels of knowledge, understanding or attitude.’6 This sort of impact is harder to prove, because the role of research is only one amongst a number of influences. The SAZA Project has made both direct and indirect contributions, but the conceptual contributions may possibly have been the most important. As a whole, the project produced the first comprehensive analysis of financing reform. Apart from critiquing the content and impact of reforms, it demonstrated the key role played by contexts, actors and processes in shaping the particularities of health financing reform in South Africa. These features helped to capture the history of reform efforts, creating a rare form of ‘institutional memory’ in an environment where the mobility of skilled staff, both between sectors and within government, means that the lessons of the past are often forgotten. These features also provided frameworks for thinking about future reforms. If the SAZA Project had never happened, it is conceivable that the priority areas and mechanisms for intervention in health financing matters may have been far more difficult to identify.

In more specific terms, the project highlighted the fact that the actual or likely impact of certain reforms went contrary to stated government policy. Neither the resource allocation formula used by National Treasury to divide the government budget between provinces, nor the first official policy proposal on social health insurance (produced in 1997), did much for the reduction of inequity. In fact, the formula aggravated differences in government health care expenditure per capita between some groups. This information was presented directly to the Director General (Permanent Secretary) of the national Department of Health (DOH) and to his management team, clearly causing considerable concern and strengthening their resolve to review these areas of reform. The research team also made a submission on social health insurance to a commission of enquiry investigating issues related to social security. Indeed, the DOH has subsequently revised an aspect of the resource allocation mechanism (specifically the conditional grants that are top-sliced from the government budget before the application of the formula, which formerly led to the over-compensation of better-off provinces for their provision of highly specialised services and training). The social health insurance policy proposal has also emerged from the commission of enquiry in a different format. This may reflect some consideration of the critique developed by the SAZA Project, although certain important aspects of the critique have not been taken into account.

The SAZA Project also seems to have contributed to a growing appreciation by government of the need to prepare strategically for reform processes. This is reflected particularly in the current process to develop a comprehensive approach to interactions between the public and private sectors. The SAZA Project had identified the lack of a comprehensive policy on the private sector as a significant problem. It had also emphasised the need to prepare clear objectives for any reform process, to involve key stakeholders and to take a fairly long-term approach where large-scale reform is involved. These principles are certainly being applied in the case of the aforementioned process, and may also have influenced another current process reviewing the configuration and financing of highly specialised services. The commissioning of three stakeholder analyses relating to social health insurance and the re-regulation of the private health insurance industry is another example of how government has become more sensitive to the management of interest groups during the reform process.

Equally importantly, the lessons of the SAZA Project have had an enormous impact on the way members of the research team support current policy processes (such as the aforementioned policy framework guiding public-private interactions, which has seen direct involvement by SAZA researchers). Some of these researchers had been involved in past policy processes that were the subject of the SAZA critique. The project showed that the poor strategic and tactical skills of analysts working both inside and outside government had contributed to their inability to ensure that sophisticated technical analyses influenced policy makers.

A last, and smaller, intervention that the project has led to is the probable commissioning by government of new research into the levels of cross-subsidisation that exist within the health care system.

3.2The contribution of the project to capacity-building

As will be discussed in more detail later, the SAZA Project was one activity amongst several that helped to improve the networking between the research units and the Directorate of Health Financing and Economics (DHFE) in the DOH. This had the end result of strengthening the capabilities of all three partners. Hopefully, the lessons – and even the methodologies used by the project (such as stakeholder analyses) – have improved the skill with which all three partners deal with new challenges. One of the researchers now heads the DHFE, while others of the original research team have begun to apply the lessons of the project to the development of a policy framework for interactions with the private sector. The SAZA Project has also proved a rich source of material for training materials for courses for post-graduates and senior managers. These are impacts on the local front, but materials summarising the findings of the SAZA Project as a whole have also been circulated quite extensively internationally with the aid of one of the co-funders of the project, the Partnerships for Health Reform initiative of USAID.

4.Factors facilitating the project’s positive impact

The factors that facilitated the project’s success, both in terms of the credibility of its findings and in terms of the diffuse impact that these findings seem to have had, can be grouped into those relating to the way the project was designed and staffed, and those relating to the evolving receptiveness of government to the information generated by the research institutions involved in the project. Some of these factors are presented below, together with recommendations for applying these lessons to other projects.

4.1Features of the project itself

4.1.1Timeliness of the project

Government’s attention is often focused on dealing with day-to-day management of services, or developing policies in response to immediately critical issues. Government often finds it difficult to engage in processes with a longer-term agenda, or to commission other groups to do so on its behalf. As an independently initiated effort, the SAZA Project was able, by being proactive, to ensure that highly relevant information could be generated by the time it was needed by government. This included an explanation of why the resource re-allocation process was not having its desired effects, and the critique of the 1997 social health insurance policy just as it was coming up for review. This feature of the project was largely dependent upon it’s location within academic institutions which are more generally able to conduct long-term analysis than government or, for that matter, consultancy agencies. In addition, these institutions had a long history of applied research and, indeed, involvement in policy development. This allowed them to ‘read’ the policy environment and identify key gaps in information appropriately. Lastly, substantial funding was made available to these institutions by an international funding community alert to the potential local and international contribution of this sort of research.

4.1.2The creation of a sustainable and successful mechanism for consultation with government

The progress of the project was communicated to government throughout its lifespan, from the initial conceptualisation of the project, through to its closing stages. The initial channel of communication was through ad hoc meetings or telephonic and e-mail discussions with the leader of the DHFE.[7] A second channel, which was eventually discontinued, was the creation of a ‘reference group’ involving the leader of the DHFE as well as another key person in government. It proved difficult to get these people to meetings because of constraints on their time.

A third channel that proved very successful, and has continued beyond the lifespan of the project, was the holding of regular meetings between the two research units involved in the project and several members of the DHFE, which grew steadily over the years. The first of these meetings was held to discuss the project formally once the methods had been developed in some detail (hence, there were concrete issues to review, such as the details of what to raise with key informants). The discussion proved fruitful and it was decided to continue with these formal meetings on a quarterly basis. Importantly, it was decided to expand the scope of the meeting to include a more general discussion of the work of the three units, as the need for this sort of liaison had frequently been raised in the past. The purpose of these more broadly defined discussions was to exchange information on what research was being done. It was hoped this would encourage research findings to be used more widely, avoid the duplication of research, exploit the synergies between the work of different units, and identify important gaps in the current portfolio of research.

This new format for the meetings improved communication between the three units dramatically. It created a sense that the range of health economics research in the country was becoming better understood[8] and it allowed participants to get to know one another. Importantly, it promoted the exchange of ideas in a collegial atmosphere. Hitherto, researchers had tended to come into contact with the DHFE in contexts either of negotiating the terms of commissioned work, or of lobbying for certain policy options. The quarterly meetings allowed participants to put aside their respective ‘positions’ and institutional affiliations, at least to some extent.

Soon, additional meetings were being placed back-to-back to the more general meeting. As one of the research units was geographically distant from the other two units, and as generally participants had very full diaries, this improved the logistics and reduced the cost of getting together. One of the additional meetings became the quarterly review of what was known as the ‘capacity-building project’. This important project was commissioned by the DHFE and funded by the EU through a direct grant to government. The project required the two research units to conduct research that had the dual purpose of training young researchers in health economics and providing expert support to the DHFE. Work undertaken under this project had to respond to priorities identified jointly by the DHFE and the units, but specifically excluded brief and hurried pieces of work responding to crises in government. The project provided a direct mechanism for the research units to support government policy needs, and allowed the immediate implementation of some of the recommendations of the SAZA Project with respect to strengthening the working relationships between technical analysts working inside and outside government. A number of other directorates and senior officials in the Department have pointed to the emerging relationship between the three units as a ‘success story’ and have sought to establish their own links with academic institutions in order to reap similar benefits. Thus, the SAZA project acted as a spur to a more ‘institutionalised’ relationship between research units and government, although it was not the only factor responsible for sustaining this relationship over time.

4.1.3The creation of a trusted team of researchers

One of the reasons why the quarterly meetings worked successfully was because the research team was trusted by government. The researchers had a long history of working towards reform, either in South Africa (in the apartheid era and subsequently) or in other developing countries. Their track record combined a tradition of good quality, applied research with trustworthiness in dealing with confidential issues, and a fairness in reflecting different points of view. This facilitated not only the quarterly meetings but also meant that easy access was granted to senior key informants, whether inside or outside government, even when their policy positions were known to be at odds with those of the researchers. The manner in which the interviews were conducted was important in maintaining the relationship of trust. Interviewers were courteous, and made it clear that they were genuinely interested in hearing ‘the story’ that the interviewee had to tell about his or her part in the policy process. Interviewees seemed interested in the project and appeared to enjoy the interview experience. The key informant interviews were a vital component of the research, as was the review of the first draft of the project’s report by a number of key informants and other individuals. Whilst the researchers reserved the right to make a final judgement on the interpretation of information, these inclusive techniques allowed the triangulation of ideas and undoubtedly added to the depth and specificity of the findings. The use of important policy-makers as reviewers was also a successful mode of disseminating findings ahead of the project’s completion.