MR outline 04/13

May 2013

UK STATISTICS AUTHORITY

Monitoring Review Outline

Patient Outcome Statistics in England

'Outlines' set out the 'concern' to which a Monitoring Review relates, describe the issues to be explored in the Review, the geographic coverage of the Review, a broad timetable, and the broad methodology that we expect to follow in conducting the Review. Outlines are prepared before the main work on a Review is undertaken. They serve both as a guide for the team doing the research and as an indication of work in hand for others who might wish to make a contribution to the Review.

The concern

The recent Inquiry by Robert Francis QC into the care provided by Mid Staffordshire NHS Foundation Trust[1]concluded, inter alia[2]:

  • that official statistics about patient outcomes may not be readily useable by the public; and
  • that there may be some patient outcome statistics that the Department of Health (or the Health and Social Care Information Centre) do not currently treat as official statistics, but which should be treated as such (and, by extension, should be assessed against the Code of Practice for Official Statistics with a view to designation as National Statistics).

In response, the Authority stated that it will undertake an independent review of those patient outcome statistics that are currently recognised as official statistics (and thus within the Authority's statutory remit) with a particular focus on any ways in which the published form of these statistics may be made more readily useable by the public. This review will also consider whether there are any patient outcome statistics, including hospital-level mortality indicators, that the Department of Health or Information Centre do not currently regard as being official statistics, but which the Statistics Authority would recommend in future be treated as official statistics and therefore compiled and published in accordance with the Code of Practice for Official Statistics. The Statistics Authority also stated that in undertaking this review, it would work closely with, but independently of, the Department of Health and the Information Centre.

Background

The Francis Report made several recommendations relating to official statistics; most relevant to this Monitoring Review are:

  1. There is a need for a review by the Department of Health, the Information Centre and the UK Statistics Authority of the patient outcome statistics, including hospital mortality and other outcome indicators. In particular, there could be benefit from consideration of the extent to which these statistics can be published in a form more readily useable by the public (Recommendation 270).
  2. To the extent that summary hospital-level mortality indicators are not already recognised as national or official statistics, the Department of Health and the Health and Social Care Information Centre should work towards establishing such status for them or any successor hospital mortality figures, and other patient outcomes statistics, including reports showing provider-level detail (Recommendation 271).

For the purposes of this Review, ‘patient outcome statistics’ can be interpreted as the suite of indicators that illustrate, in aggregate terms, the condition of patients following contact with the National Health Service. The UK Government’s NHS Outcomes Framework[3], published in 2011, sets out 60 indicators defining how the NHS will be accountable for outcomes – in other words, the indicators which the Government believes should be the basis of evaluating the effectiveness of the NHS from patients’ perspectives. These are arranged according to five ‘domains’:

  1. Preventing people from dying prematurely
  2. Enhancing quality of life for people with long-term conditions
  3. Ensuring that people have a positive experience of care
  4. Helping people to recover from episodes of ill health or following injury
  5. Treating and caring for people in a safe environment and protecting them from avoidable harm.

Hospital mortality indicators are a subset of the fifth domain.

This Review focuses on the statistical position in England. Different interpretations of ‘patient outcomes’ have been adopted elsewhere in the UK, in line with the devolved nature of health policy. The Statistics Authority has reported elsewhere[4] on the desirability of greater coherence in the suite of official health statistics produced across the UK; it expects that the focus of the current review on England will nevertheless be helpful to producers of patient outcome statistics elsewhere in the UK.

Structure of the Monitoring Review Report

  1. Statement of the concerns being addressed as above.
  1. Findings and conclusions
  1. The range of patient outcome statistics:
  1. Identifying the relevant patient outcomes statistics (including hospital-level mortality indicators) in use in England, including their levels of disaggregation.
  2. Determining their status (not official statistics, or official statistics – and if so, whether National Statistics).
  3. Identifying whether any ‘non-official’ statistics might appropriately be treated in future as official statistics.
  4. Identifying priorities for the assessment (against the Code of Practice) of any official statistics that are not currently National Statistics.
  1. The accessibility and usability of official patient outcome statistics:
  • the presentation and accessibility, including guidance in relation to use, of those patient outcome statistics that are covered by the definition of official statistics
  • whether information about users’ experiences of accessing and using the statistics is sufficient and whether it is used to inform ongoing development and presentation
  • whether there is transparency of information about patient outcomes on relevant websites.

Relevant aspects of the Code of Practice for Official Statistics

Principle 1 Meeting user needs

P1.1 Engage effectively with users of statistics to promote trust and maximise public value, in accordance with Protocol 1.

P1.2 Investigate and document the needs of users of official statistics, the use made of existing statistics and the types of decision they inform.

Principle 4 Sound methods and assured quality

4.1 Ensure that official statistics are produced according to scientific principles. Publish details of the methods adopted, including explanations of why particular choices were made.

4.2 Ensure that official statistics are produced to a level of quality that meets users’ needs, and that users are informed about the quality of statistical outputs, including estimates of the main sources of bias and other errors, and other aspects of the European Statistical System definition of quality.

4.3 Adopt quality assurance procedures, including the consideration of each statistical product against users’ requirements and of their coherence with other statistical products.

Principle 8 Frankness and accessibility

8.2 Prepare and disseminate commentary and analysis that aid interpretation, and provide information about the policy or operational context of official statistics. Adopt formats for the presentation of statistics in graphs, tables and maps that enhance clarity, interpretability and consistency.

Protocol 1 User engagement

1.3 Take account of users’ views on the presentation of the statistics, and associated commentary, datasets and metadata.

Protocol 3 The use of administrative sources for statistical purposes

3.2 Only base statistics on administrative data where the definitions and concepts are good approximations to those appropriate for statistical purposes.

If aspects of compliance with other elements of the Code arise based upon the evidence gathered, these will be identified within the Review.

Proposed Methods of evidence gathering

Desk research:

  • review of statistical releases and metadata
  • review of publications by public and parliamentary bodies, other expert groups and media reports (including Kings Fund, Nuffield Trust, Health Foundation)
  • review of published hospital inspection reports about the collection of patient outcome statistics (Care Quality Commission), audits of morbidity data and reports of the Parliamentary and Health Services Ombudsman
  • review of the relevant evidence presented to the Francis Inquiry

Collation of documentation from producers:

  • service level agreements with data providers
  • most recent Statement of Administrative Sources
  • latest user engagement action plans and outcomes

Seek views of experts (via email and in person):

  • statisticians responsible for producing statistics and statisticians producing similar statistics in Devolved Administrations
  • key users and stakeholders such as Health Statistics User Group, Monitor, the NHS Confederation, Healthwatch, the Nuffield Trust, the King’s Fund, and relevant Patients’ Associations (such as patientopinion.org.uk, the Patients Advisory Service), providers of private sector standardised mortality data (such as CHKS and Dr Foster) and some independent healthcare providers (such as BUPA and WPA).

Timetable

We are currently gathering and reviewing evidence, and plan to report to the Authority Board in early October 2013.

Your views

Your views on any of the issues to be covered in this Review are welcomed. Please send any comments to: by 31 July 2013 if possible but late responses would still be welcome.

[1]

[2]ibid Ch 26 pp1673

[3]

[4]