GHM201_S0615 November 2018Page 1 of 60
Before you start working on updating this session, please turn on tracked changes.
It is also advisable that you familiarise yourself with the interactive version of the session as this editable version doesn’t have all the formatting and interactions.
This session was written by Loveday Penn-Kekana, Ellen Nolte, Martin McKee and Dina Balabanova
GHM201 - Session 6: Assessing health systems
Session Table of Contents
1 Aims and objectives
2 Planning your study
3 Introduction
4 Why assess health systems?
5 Developing the field of health system research
6 Selected approaches for health systems assessment
7 Challenges in assessing health systems
8 Doing health systems research
9 Integrating activity
10 Summary
11 References
1 Aims and objectives
Page 1
Aims
The aims of this session are to introduce key concepts and practical methods used in assessing and comparing health systems in high-, middle- and low-income countries.
Objectives
By the end of this session you should be able to:
- Discuss why you might want to assess and compare health systems and the challenges that you may face in doing so.
- Describe some of the main approaches you might use to assess and compare health systems.
- Critically appraise a range of these approaches.
- Explain some of the key conceptual and methodological challenges to assessing and comparing health systems and choose approaches that are appropriate to the settings you are working in and the issues you are addressing.
2 Planning your study
Page 1
In this session you should first work through the different screens and spend time on the various activities and exercises. You will also be required to read any essential reading, as indicated. This should take you about two to three hours.
You should then complete the integrating activity, referring to the essential reading provided and the recommended reading within the session and drawing on your own personal experience. This should take you about two hours.
Finally, you should spend a further two hours on self-study covering the recommended reading and two or three texts from the further reading (optional).
3 Introduction
Page 1
In your reading for the module so far, you will have noted the wide range of different approaches and methodologies that have been used when analysing the key features of health systems. There is a growing interest in assessing how heath systems perform and sharing the lessons that can be learned. Consequently, this session will first discuss why there is now such a great interest in evaluating health systems.
If you reflect on what you have learned in the previous sessions, you will recall that assessing health systems is not a simple task.
- Health systems have multiple parts (illustrated in the ‘building blocks’ framework, see Session 1).
- Moreover, when you bring them together, they interact in ways that are often difficult to predict, in part because they are shaped by their context and environment, but also because they involve actions of real people, who do not always behave in the ways that you think they will, as they have their own interests, agendas and preconceptions (not all of which they may be willing to share).
- Finally, health systems are often pursuing several different goals that may conflict with each other (e.g. improving health outcomes while reducing costs, or trying to respond to patients’ expectations, even when they are asking for treatment that is ineffective), and those working in them will have their own goals, which may range from serving the poor to making a quick profit.
Before we even begin to assess or compare health systems, we have to be sure what we think constitutes a ‘health system’. Session 1 demonstrated the range of definitions of health systems that are used, and the difficulties in establishing the boundaries of a health system.
Then we need to decide what we want the system to do. Most (although not all: Remember that health care is a major sector in the economy in many countries, so some people will view it as being no different from any other sector, such as leisure services or manufacturing.) of us can agree that it should improve the health of the population.
Yet it can be very difficult to attribute health improvements to health care policies, given how many factors, both inside and outside the health system, influence health. All of these considerations are influenced by the broader context – wealth, history, values – of the country that is so important in understanding the way that health systems operate.
Page 2
This session will introduce you to the principles of health systems research and to several approaches that have been proposed to help undertake systematic health systems research.
Many people have tried to simplify this complexity. One example is that adopted by the World Health Organization (WHO), in its World Health Report (WHO 2000), which treated health systems as a sort of ‘black box’ [link – show definition on click – see box below] (although it did identify certain core functions), ranking countries on their ability to achieve a set of goals, health improvement, responsiveness to patients’ needs, and fairness of financing, but without going into great detail about how it did so.
[definition]
With a Black Box you can see what goes in and what comes out, but you will not know how the transformation takes place
[/end definition]
We will then look at another way in which we can begin to simplify the complexity of health systems, by focusing on the different levels of decision-making within them: the micro-level (the process and outcomes of patient care); the meso-level (the organisational framework within which this care occurs); and the macro-level, which includes governance, financing and design of the delivery system.
You will then work through a series of commonly used health systems research methods, discussing their advantages and disadvantages, and considering when they might be appropriate to answer particular research questions. The approaches and methods chosen will depend on the question being asked and the data available. Crucially, they will also depend on the values and objectives of those conducting the assessment. This session will not cover in detail the methods, which are often generic to social sciences. The specific challenges facing low-income countries will be outlined, including what research capacity exists for carrying out rigorous assessment. The concepts and methods covered will then be practised in an exercise in which you will design a health system assessment.
4 Why assess health systems?
Page 1
Reflection point
Drawing on your knowledge and personal experience, why do you think it would be useful to assess the functioning or achievement of a health system?
What might be some of the likely challenges in doing so?
Let us take the example of international targets such as the Millennium Development Goals (MDGs), which have prioritised improvements in maternal and child health, and which can therefore be thought of as indicators of health system performance.
Can you think of at least three ways in which maternal and child health outcomes (MDGs 4 and 5) can be problematic as measures of health system achievement?
Please post some of your reflections on Moodle[link] and respond to other posts
Question feedback
- Why study health systems? Because medical advances (new drugs, innovative technologies, increased knowledge of what works) mean that health systems are increasingly important in preventing death; because they cost money that could be spent in other ways; because they raise fundamental political questions about the obligations linking the state and the individual; and because they can attract media attention when things go wrong etc. (see Session 1).
- What are the challenges? The boundaries of a health system can be difficult to define; the elements within it are complex and interact in ways that can be difficult to predict; the system is embedded in and influenced by a range of contextual factors (geographical, economic, political, historical and cultural), which make comparing and assessing health systems difficult.
- The example of MDGs 4 and 5: the first problem in assessing the ability of health systems to achieve progress towards these goals, measured by maternal and child health (MCH) outcomes, is that these are strongly influenced by factors outside the conventional boundaries of a health system. Thus, access to clean water, sanitation, primary education for girls, nutrition and women’s employment may have as big an impact on MCH (particularly for under-5 mortality) as anything the health system does. Second, MCH outcomes are influenced by many factors within the health system, making it difficult to attribute progress, or lack of it, to specific policies (thereby identifying those that might be helpful elsewhere). On the other hand, it is argued that MCH is a good global indicator of overall health system performance as it reflects so many parts of the health system – from clinics, to hospitals, to referral routes, to labs, to availability of blood, to skilled health professionals – all of which must work together to achieve improvements in maternal mortality. Disentangling the role of the different factors and their interaction will influence the way evaluation is designed and conducted. For a more detailed discussion, see Harmer, in Balabanovaet al. 2011.
5 Developing the field of health system research
Page 1
Health systems performance assessment
Interest in understanding a functioning health system and its impact on health is not new. Much of the early interest of health systems research focused on assessing the performance of the various elements of health services.
Nolte and McKee (2012 ) give examples from the mid eighteenth century, with Florence Nightingale, an English nurse and, although less well known, pioneering statistician, describing variations in outcomes among different hospitals, and subsequently arguing for the systematic collection of mortality data to inform policies that might improve the quality of service delivery (quoting Spiegelhalter 1999). Others followed, most notably Emory Codman, a Boston surgeon working in the first decade of the twentieth century, who founded the ‘End Results Hospital’, one of the first attempts to monitor outcomes routinely.
However, despite the efforts of these early pioneers, there was little progress until the mid- 1980s, which saw what Relman (1988 ) described as the ‘third revolution in health care’. This stimulated the first attempts to undertake international comparisons, most notably in work by the Organisation for Economic Cooperation and Development (OECD), which in turn led to the benchmarking of countries’ performance (OECD 1985).
Interest in comparisons of health system performance made a quantum leap in 2000, with the publication of the landmark World Health Report (WHO 2000). In an extremely controversial move, the authors ranked 192 countries in terms of the performance of their health systems. This was both ambitious, because of the significant advances in methodology and the scale of data collection involved (and in many cases, estimation of data where they were lacking), and courageous, given that those countries ranked low would be unhappy, a particular challenge for the WHO as it was, and is, dependent on support (financial and political) from its member states.
The report redefined and expanded the concept of ‘performance’, setting out three goals for a health system: improving health, delivering services that are responsive to legitimate patient needs, and which generate revenue fairly. It considered both the level and distribution (and therefore equity) of health outcomes and responsiveness, making five goals in all (Nolte and McKee 2012). As expected, the report generated widespread debate among the academic and policy communities and, although the WHO has never repeated it, it made major advances to the field of health system performance and stimulated much greater efforts to collect data that previously had been unavailable.
The rankings in the 2000 World Health Report largely sidestepped the debate about what aspects of health systems were important in achieving the outcomes being studied, although these were discussed in some detail in the accompanying text, even if not linked to outcomes. Since then, however, researchers have sought to understand what elements of health systems are most important and, in particular, the scale and nature of barriers to delivering effective care and how these can be overcome. This work has drawn attention to the importance of context – those characteristics of countries that arise from their history, culture, politics and much else, all of which impact on the delivery of health care.
In summary, much of the early work focused on examining the functioning of particular health services, in terms of structures, processes and outcomes. However, more recent work has begun to acknowledge the crucial role of context. Research has also started to demonstrate the complex pathways between the way services are organised and the outcomes and outputs, and suggests the need for a broader (or what we call ‘systems’) perspective to explain reality.
Page 2
From services to systems: towards a new understanding of health systems research
The landmark World Health Report (WHO 2000) set out a comprehensive framework within which health systems can be studied and managed. It identified certain key functions (stewardship, financing, delivery) and goals (improved health, fairness and responsiveness) (see Session 1). These were further disaggregated in subsequent work, demonstrating the complex relations among inputs that are required to achieve health systems goals. It was established that any ‘system-level’ assessment of a health system should involve examining the roles of multiple ‘building blocks’ in enabling particular outcomes, even if the question of interest referred to only one element of a health system.
Click here for an example[link – on click show hidden text]
[hidden text]
For example, understanding human resources shortages from a system perspective will require understanding of broader governance and institutional arrangements to recruit, train and retain health workers, understanding of the financial resources required, and knowledge of delivery models which affect the supply of health workers.
[/end hidden text]
Stimulated by this report and the heated debate following it, interest in assessing, comparing and evaluating health systems increased rapidly, with a proliferation of frameworks, toolkits, studies and manuals. Some of these will be considered later in this session. However, funding for empirical work has been scarce, and many of the approaches remain untested.
The publication of the report coincided with other developments leading to increased interest in health systems research (see Session 1, topic 4). Countries at all levels of development took greater interest in how their systems were performing and whether this represented value for money. Health systems are increasingly being recognised as essential to improving population health. A massive transformation in information technology also offers the prospect, although rarely fulfilled, of collecting, storing and processing vast amounts of information that had not previously been possible. There is also interest in learning lessons from countries that are considered to have achieved success in tackling some of these issues.
The end of the 1990s marked the beginning of new global health initiatives. For example, the creation of the Global Fund to fight AIDS, Tuberculosis and Malaria (GFATM) and the Global Alliance for Vaccines and Immunisation (GAVI). Such initiatives recognised the need for what was called ‘health system strengthening’. This was seen as a departure from vertical disease-specific programmes, and the principles of ‘comprehensive and inclusive health care’, requiring ‘all-round system building’, were seen as the way forward.
As a result, the critical role of health systems is now well-recognised and there have been efforts to advance the field of health systems research both conceptually and practically. Organisations such as the European Observatory on Health Systems and Policies and the Alliance for Health Policy and Systems Research (see below) have been instrumental in promoting such research and remain important repositories of analytical work.
Reference links
European Observatory on Health Systems and Policies:
(accessed October 2013) Alliance for Health Policy and Systems Research:
(accessed October 2013)
Page 3
Contemporary interest: ‘systems thinking’
The early phase of health systems research (from mid 1990s to late-2000) created considerable enthusiasm and generated demand for health systems research. Many of the research questions reflected practical concerns of policy-makers and implementers.
In 2007 the WHO published a revised version of its framework and sought to operationalise key dimensions of health systems, seeking better to inform policy and research agendas. The First Symposium for Health Systems Research, held in Montreux, Switzerland, in 2010, sought to create a platform for developing shared concepts and definitions in health systems research. It, and subsequent developments, emphasised the importance of strengthening the methodological and theoretical basis of such research (Sheikh et al. 2011).