The Appropriateness Evaluation Protocol is a poor predictor of in-hospital mortality
O’Regan NA, Healy L, O Cathail M, Law TW, O’Carroll G, Clare J, Timmons S, O’Connor KA
Niamh Annmarie O’Regan, Centre for Gerontology and Rehabilitation, School of Medicine, University College Cork, Ireland
Liam Healy, Department of Geriatric Medicine, South Infirmary Victoria University Hospital, Cork, Ireland
Micheál Ó Cathail, Department of Geriatric Medicine, Mercy University Hospital, Cork, Ireland
Tze Wen Law, Department of Geriatric Medicine, Mallow General Hospital, Co. Cork, Ireland
Grace O’Carroll, Department of Geriatric Medicine, Mercy University Hospital, Cork, Ireland
Josie Clare, Department of Geriatric Medicine, Waterford Regional Hospital, Waterford, Ireland
Suzanne Timmons, Centre for Gerontology and Rehabilitation, School of Medicine, University College Cork, Cork, Ireland
Kieran A O’Connor, Department of Geriatric Medicine, Mercy University Hospital, Cork, Ireland
Corresponding Author: Dr. Niamh O’Regan, email:
Abstract word count:239
Manuscript word count: 1,741
References: 16
Author contributions
ICJME criteria for authorship:
Authors have 1) substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published.
O’Regan NA 1,2,3; Healy L 1,2,3; O Cathail M 1,2,3; Law TW 1,2,3; O’Carroll G 1,2,3; Clare J 1,2,3; O’Connor KA 1,2,3; Timmons S 1,2,3.
O’Regan, Clare, O’Connor and Timmons: all substantially contributed to study conception and design, data collection, statistical analysis and interpretation, drafting the initial manuscript, and subsequent critical revision of the manuscript and final approval.
Healy, O Cathail, Law, and O’Carroll: all majorly involved in data acquisition, critical revisions and final approval of the manuscript.
Introduction
Appropriate allocation of resources is fiscally important in delivering healthcare. Reducing acute hospital admissions is viewed as a key method to curb healthcare costs. Hospital utilisation review (UR) programmes have been shown to reduce utilisation and expenditures[1]. However, there is no evidence that UR improves overall efficiency of resource use. Furthermore, its impact on care delivery and patient outcomes has not been described[2].
Many tools have been developed to measure acute hospital bed utilisation. The Appropriateness Evaluation Protocol (AEP), developed in 1981 by Gertman and Restuccia[3] is the most widely used and is regarded as superior to other tools[4,5]. It was developed as an objective, diagnosis-independent, criteria-based system to prospectively determine appropriateness of hospital days (see figure 1). It is divided into Admission Criteria and Day of Care Criteria, which are used to define respectively whether a patient’s admission to an acute bed was appropriate; and whether or not continuing care in that setting was required. The Admission Criteria consists of Severity of Illness and Intensity of Service Required Criteria: a patient must meet one of these criteria to be deemed appropriate for admission. The Severity of Illness criteria are based on objective measures of disease severity, divided into ten categories with six descriptive parameters (e.g. acute loss of sight or hearing), scored as present or absent; and seven numerical physiological parameters (e.g. diastolic blood pressure) with cut-off values beyond which results are regarded as sufficiently abnormal to warrant admission (see figure 1). Its authors conceded that such a concise list of criteria could never capture all circumstances requiring admission, but would facilitate usability of the tool. Hence, they included an ‘override facility’, where if necessary, clinical judgment could be used to overrule the decision based on the AEP’s parameters. During its development and validation (against a gold standard of senior clinical opinion), there was good overall agreement between raters, but poor specific agreement.
The AEP was initially developed in the U.S.A., and its original and modified forms have since been studied and utilised in many countries, most commonly as a tool to retrospectively ascertain levels of admission appropriateness. In Ireland, the AEP was recently used to inform a National Acute Hospital Bed Review[6] which stated that on average 12% of acute hospital admissions are inappropriate.
A systematic review of methods to ascertain appropriateness of acute hospital use, performed by McDonagh et al[4] in 2000, declared the AEP the most valid and reliable instrument available, but there are well-documented concerns with the use of any tool such as this to identify inappropriate admissions[4,2,7]. Sikka et al showed that severity of illness tool are poorly predictive of outcomes in older patients[8]. Hence, the primary objective of this study was to assess the clinical utility of the AEP in assessing appropriateness of admission, in older patients compared to younger patients.
Methods
We used analytical retrospective cohort study design to assess admission parameters of adult patients who died within ten days of emergency admission to four hospitals in Cork and Waterford: Waterford Regional Hospital (WRH); Mercy University Hospital (MUH); South Infirmary Victoria University Hospital (SIVUH); and Mallow General Hospital (MGH). All patients had been admitted under medical, surgical or other specialty services. Ethical approval was granted by the local research ethics committee. The average study period was 17 months from 1st January 2009 to 31st December 2010. We used proximate death as a robust measure of illness severity. The Hospital Inpatient Enquiry System (HIPE) identified patients who were eligible for inclusion. We used available ED charts, paramedic documentation, and computerised laboratory records to assign AEP criteria. Given that, in general, it is more likely that abnormal rather than normal physiology is documented with most consistency, missing datapoints were taken as not meeting the AEP criteria. We also collected demographic data, details of clinical presentation and measured length of stay before death. Cause of death was not available. Data were analysed using Statistical Product and Service Solutions (SPSS version 18).
Results
There were 80,970 admissions in the study period and 1,416 in-hospital deaths. Of these, 803 occurred within ten days of admission. Of the 803 requested, 585 unselected (72.9%)charts were available for review and 490 (83.8%) of these patients were aged 65 years or older. The median age was 79 years and 50.8% were female. The mediantime from admission to death was approximately four days in both groups (see table 1). The primary reason for ED attendance is illustrated in table 2. Vital signs and serum electrolytes were available in 437 / 585 patients (74.7%). ECGs were either reported or available for review in 349 / 585 (59.7%) patients. Venous bicarbonate and / or pH was documented in 184 / 585 (31.5%).Of 803 requested charts,218 (27.1%) were unavailable for multiple reasons (e.g. were stored off-site; damaged during the Cork floods). The proportion of unavailable charts was spread evenly across the four hospitals, ranging from 20.5% in SIVUH to 30.4% in WRH.
When we applied the AEP criteria, 179 of 585 patients (30.6%)were AEP non-compliant on admission: 31 / 95 (32.6%) younger patients and 148 / 490 (30.2%) of older. Seventy-two of 179 patients(40.2%) who did not meet the admission criteria had been coded as severely unwell on ED arrival by triage staff: 13/31 (41.9%) younger; 59 / 148 (39.8%) older. In those who were AEP compliant, the most common parameter reached in both age groups was abnormal diastolic blood pressure, followed by abnormal GCS (Glasgow Coma Scale), figure 2. Derangements in serum sodium and potassium were also very common across all age groups, but only a small proportion of these were severe enough to meet admission criteria, figure 2. Very few patients (42 / 585, 7.2%) met the criteria for temperature (>37.78°C), but other temperature abnormalities, including hypothermia, were common in both age groups.
Discussion
This study highlights problems with using the AEP retrospectively to assess admission appropriateness in a severely unwell cohort. Our study hypothesis was that the AEP may be less effective at identifying severe illness in older patients. Worryingly, our study has illustrated its deficiency in recognising moribund younger patients also.
Our study has some limitations. Firstly, we only reviewed ED charts and paramedic documentation and not the complete medical charts from all admissions, hence may have missed some important data that was documented at a later point in admission.Additionally, due to difficulty obtaining medical records, we were unable to review 27% of the total eligible patient charts, and given that demographic informationpertaining to these missed cases is unavailable to us, this may be a source of selection bias. Nonetheless, chart availability was dependent on circumstances out of our control and all available charts were reviewed. Being retrospective and dependent on ED charts, we were unable to discern if individual missing data points had been measured or simply not documented. We resolved to treat missing data as not meeting the criteria, as when documentation is concise, we concluded that it is probably more likely to reflect abnormalities rather than normal values. Of note, bed utilisation reviews are also performed retrospectively, and hence, encounter similar difficulties. Importantly, in such reviews, missing datapoints are similarly considered AEP non-compliant[6]. Furthermore, we cannot be sure that some patients did not succumb to nosocomial illness, as cause of death was unavailable. Given the median time to death of four days, it is likely that most patients were severely unwell on arrival.
To our knowledge, the only other study which describes the prognostic significance of the AEPwas conducted by Braband and colleagues[9]. This prospective study of patients admitted to a Danish medical admission unit found that 38% of admissions did not meet the AEP criteria. Additionally, of 84 in-hospital deaths, 12.4% were AEP non-compliant on admission. Of note, all of these patients died within 10 days of admission with a median time to death 4.75 days (similar to our study). Although a similar proportion of the Danish cohort had ECG data available (61.3%) as had in our study (59.7%), we had more missing datapoints (25.3% versus <10%), which may account for the difference in AEP non-compliance rates. Multiple other scoring systems based on patient physiology have been used for decades to predict illness severity, particularly in the intensive care setting, such as APACHE (Acute Physiology and Chronic Health Evaluation)[10] and SAPS (Simplified Acute Physiologic Score)[11], and can provide accurate predictions of inpatient mortality risk. More recently, track and trigger systems such as the MEWS (Modified Early Warning Score)[12]have been shown to have utility in identifying general medical and surgical patients at risk of rapid deterioration and critical illness[13,14]. These systems, however, are used to assess patients who have already been admitted and hence are not suitable nor designed for ascertaining admission appropriateness.
Importantly, bed utilisation review programmes were originally designed to concurrently screen cases requiring further clinical review[3]. Instead they are being used retrospectively to assess performance in acute hospitals. To us, this is a major concern, given that almost one-third of our moribund cohort were missed by the AEP when used retrospectively.
When we scrutinise the severity of illness criteria of the AEP, many parameter cut-offs are startlingly exclusive. Taking sodium level for example, a patient meets the AEP criteria with a sodium level of <123 mmol/L or >156 mmol/L, however it is well-recognised that sodium levels between these two values can be associated with severe illness. Moreover, it is the acuity of the sodium level changes, and not necessarily one isolated result, that generates clinical concern. To illustrate further, pyrexial patients are considered appropriate for admission, but the criteria neglects hypothermia as an important parameter. The authors allowed for the instrument’s exclusivity by applying a clinical override facility, but this is not often used in practice.
Patients and referring doctors favour alternatives to acute medical care as shown by Campbell in 2001[15]. Unfortunately, adequate alternatives are rarely available. Our fragmented health service in Ireland means that options vary based on geographic location and other factors. Critics argue that if there is no available alternative, then an admission should not be labelled “avoidable” [7,16].
Obviously we cannot ignore the concept of appropriateness of healthcare resource use. It is intuitive that we should have the ‘right’ services for the ‘right’ people in the ‘right’ place. However, this paper highlights significant concerns with the current methods employed to ascertain admission appropriateness, as well as the lack of focus on safe alternatives which is central to the issue.
Funding
This study received no external funding.
1
Tables and figures
Table 1: Patient demographics
Total / Younger(<65 years) / Older
(≥65 years) / Sig.
n= 585 / Age (years), median (range) / 79 (21-103) / 57 (21- 64) / 82 (65- 103)
Gender, n (%) male
female / 288 (49.2)
297 (50.8) / 63 (66.3)
32 (33.7) / 225 (45.9)
265 (54.1)
n= 432 / LOS (days), mean (SD) / 4.52 (3) / 4.14 (3) / 4.59 (3) / p=0.2531
1Independent samples t-test
Primary reason for ED attendance / Number (% total)Respiratory / 166 (28.4)
CVS / 83 (14.2)
GIT / Hepatobiliary / 64 (10.9)
Sepsis / 56 (9.6)
Oncology / Haematology / 45 (7.7)
Stroke / SAH / 45 (7.7)
Trauma / Falls / 23 (3.9)
Vascular / Skin / Soft Tissue / Joint / 21 (3.6)
GU / Renal / 18 (3.1)
LOC / Confusion / Neurology / 17 (2.9)
Other / 11 (1.9)
Not documented / 36 (6.2)
Table 2: Reason for attendance grouped by category
ED= Emergency Department; CVS= Cardiovascular; GIT= Gastrointestinal; SAH= subarachnoid haemorrhage; GU= genitourinary; LOC= loss of consciousness
Figure 1: The Appropriateness Evaluation Protocol: Admission Criteria, A. Severity of Illness[3]
Figure 2:Graph showing the percentage of patients (in two age groups) who (1) met each physiological AEP criterium for admission, and (2) who had abnormal values outside those specified by the AEP criteria (intemperature, serum sodium and serum potassium) (HR = heart rate, SBP = systolic blood pressure, DBP = diastolic blood pressure, Temp = temperature, Na = serum sodium, K = serum potassium, GCS = Glasgow Coma Scale (i.e. criterium A1), ECG = electrocardiogram (evidence of new ischaemia), pH = = arterial pH, Bicarb venous bicarbonate)
References:
1. Wickizer TM, Wheeler JR, Feldstein PJ (1989) Does utilization review reduce unnecessary hospital care and contain costs? Med Care 27 (6):632-647
2. Coast J (1996) Appropriateness versus efficiency: the economics of utilisation review. Health Policy 36 (1):69-81. doi:0168851095007970 [pii]
3. Gertman PM, Restuccia JD (1981) The appropriateness evaluation protocol: a technique for assessing unnecessary days of hospital care. Med Care 19 (8):855-871
4. McDonagh MS, Smith DH, Goddard M (2000) Measuring appropriate use of acute beds. A systematic review of methods and results. Health Policy 53 (3):157-184. doi:S0168851000000920 [pii]
5. Strumwasser I, Paranjpe NV, Ronis DL, Share D, Sell LJ (1990) Reliability and validity of utilization review criteria. Appropriateness Evaluation Protocol, Standardized Medreview Instrument, and Intensity-Severity-Discharge criteria. Med Care 28 (2):95-111
6. Feely E (2008) Health Service Executive Second National Acute Hospital Bed Review.
7. Vetter N (2003) Inappropriately delayed discharge from hospital: what do we know? BMJ 326 (7395):927-928. doi:10.1136/bmj.326.7395.927 326/7395/927 [pii]
8. Sikka P, Jaafar WM, Bozkanat E, El-Solh AA (2000) A comparison of severity of illness scoring systems for elderly patients with severe pneumonia. Intensive care medicine 26 (12):1803-1810
9. Brabrand M, Knudsen T, Hallas J (2011) The characteristics and prognosis of patients fulfilling the Appropriateness Evaluation Protocol in a medical admission unit; a prospective observational study. BMC health services research 11:152. doi:10.1186/1472-6963-11-152
10. Knaus WA, Draper EA, Wagner DP, Zimmerman JE (1985) APACHE II: a severity of disease classification system. Critical care medicine 13 (10):818-829
11. Le Gall JR, Loirat P, Alperovitch A, Glaser P, Granthil C, Mathieu D, Mercier P, Thomas R, Villers D (1984) A simplified acute physiology score for ICU patients. Critical care medicine 12 (11):975-977
12. Subbe CP, Kruger M, Rutherford P, Gemmel L (2001) Validation of a modified Early Warning Score in medical admissions. QJM : monthly journal of the Association of Physicians 94 (10):521-526
13. Wright MM, Stenhouse CW, Morgan RJ (2000) Early detection of patients at risk (PART). Anaesthesia 55 (4):391-392
14. Subbe CP, Davies RG, Williams E, Rutherford P, Gemmell L (2003) Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. Anaesthesia 58 (8):797-802
15. Campbell J (2001) Inappropriate admissions: thoughts of patients and referring doctors. Journal of the Royal Society of Medicine 94 (12):628-631
16. Beech R (2008) Identifying the Need and Scope for Intermediate Care. In: Intermediate and Continuing Care: Policy and Practice. Blackwell Publishing Ltd, pp 19-32. doi:10.1002/9780470774960.ch2
1