And We Will Now Go to Today's Presenter, Dr. Nembhard

Benchmarking the Emergency Department – Which “Best Practices” Really Drive Efficiency
Presenter: Dr. Harriet Nembhard, Penn State University
Recorded on: June 25, 2014

And we will now go to today's presenter, Dr. Nembhard.

> Thank you very much, Sherri, I appreciate this opportunity to present our work to the group. Let's just dive right in. This is a deep dive webinar. And start to put on the table some of the ideas that we are dealing with.

Number one, according to the National Center for Health Statistics, in the past two decades, the number of ED visits has skyrocketed in the US, from 105 million to 136 million. As a result of this increase, as well as other factors, many ED's and hospitals have faced an overcrowding issue.

A recent survey by the American Hospital Association said that 35% of hospitals indicated that their EDs were at or over capacity. This ED overcrowding has caused long wait times, sometimes ambulance diversions. These things compromise patient safety, and, of course, as we know, lower patient satisfaction. In a CHOT project conducted over the past year, our team has taken a deeper look at some of the available data and we have tried to integrate our domain expertise to understand more about what is really driving efficiency.

The data master on our team is Miss Hyojung Kang, who is pursuing her PHD in Industrial Engineering with me as her adviser at Penn State. Our ED guru, is Dr. Chris DeFlitch, who brings much expertise, experience, strategy, and perspective to this work. So, what I would like to do during this webinar today is to discuss ways we can go about improving the ED and benchmarking the ED.

We will be using systems engineering methods, specifically, data envelopment analysis, or DEA, to analyze efficiency. This will be a very high level and brief presentation. I expect to just use about 15 or 20 minutes. I have tried to keep the number of slides here to a minimum. I think I've got 10 more or 12 more after this one.

Because the main thing I would like to do in this webinar is to leave the bulk of our time for discussion. This review is a preliminary study and we would like to benefit from the perspective of the participants in helping us to think about some of the next steps and implications of this work.

So, there are three questions that we have at the end, that you can be mulling over as we are proceeding through the talk. What would be other appropriate inputs and outputs to use? I will show the ones that we have used in evaluations of the operational efficiency, but surely there may be others and people may have some suggestions on what they may be.

We also found that smaller hospitals had fewer hospitals that were efficient by the method and standards that I will describe in a little bit. But what is it about smaller hospitals that leaves them perhaps to be less efficient? We can discuss this. And then lastly, what are some of the effective strategies for adapting best practices as we are able to uncover them by our data detective work.

So, with that, let me turn to giving a little bit of brief background. We said that we're looking at ED crowding. We know that there are many factors in ED crowding. And in response to these crowding challenges, many EDs have tried to implement various improvement initiatives. Some of them have tried to expand their physical plants.

Some have pursued IT solutions, such as computerized technician order entry and electronic medical record implementation. In some cases, the processes may have been re-engineered to include things like point of care testing, triage protocols, and separate care packs for low acuity patients. However, only a limited number of EDs have successfully reduced overcrowding by these measures, as indicated in the statistics that I cited at the outset.

In the meantime, healthcare organizations have pushed EDs to tackle the issues that result from overcrowding and to improve the efficiency of care. For example, insurance providers have requested hospitals to report a set of process metrics. They have also, many insurance providers, have offered targeted based financial incentives for the level of efficiency.

The centers for Medicare and Medicaid services has developed performance measure sets for timely care in the ED and has made the information publicly available, perhaps to some controversy in the recent months. But what we want to do is to look at benchmarking as a tool, a tool that we can use to learn from and hopefully to improve the performance of emergency departments.

There are a couple of ideas within benchmarking. Problem-based benchmarking efforts will talk about specific concerns, perhaps such as the desire to improve an error rate or specific cycle times. But we want to focus here on operational efficiency. So we are concerned with process-based benchmarking, which is oftentimes tied to continuous policy improvement efforts.

There are four types of benchmarking. I will just say that functional, generic and internal benchmarking all have a specific definition and appropriate uses, but here we want to focus on competitive benchmarking. Competitive benchmarking, as the name implies, entails looking at your competitors, and comparing your work processes to those of your best competitors.

Now, even though organizations can be competitive, hopefully they can still be collegial, especially in healthcare where the overriding desire is for better health. And to that point, the Emergency Department Benchmarking Alliance is a group of such competitors. Let me pause here for just a moment to take a message from one of the participants.

Okay, I think everybody is able to join. So Was there another question? Okay, I'm looking at perhaps one of my co-presenters. Might need to add a remark. Sherry, can you allow her audio?

> She's on mute. Yes. Can you allow her audio?

> Yes. She is on mute.

> Oh. Okay. So. Go ahead.

> Hello?

> I have a text on my screen that says that you would like to speak. Do you have something to add there on that slide or the previous one?

> No.

> Okay. I'm not sure if one of the other presenters, as I said has been working on the data analysis part and Christoph Lych has been our emergency department guru.

We may have a little bit of technical difficulties to get them to add in at the moment. So what I will do is proceed through these slides as I said I just have a few. And then hopefully at the end Sherry can un-mute us all for a general discussion and I can even go back to some of the slides that they might want to embellish.

Okay. So moving along as I said, the emergency department benchmarking alliance is a group of colleagues. It's a not for profit organization which purports the people who manage emergency departments across the country. It's to completely balance your organization. The organization is created by the membership. It is for the use of the membership.

There's no commercial interest attached to it or government regulation attached to it whatsoever. They said is there just to support the membership of those who manage emergency departments. And it is through this database that has almost 1000 hospitals, that we were able to get a good clean set of data to start looking at some of these bench marking ideas.

Here is the overall structure of the project. The objective is to develop a data-driven framework for benchmarking the efficient emergency department. We're doing this in a specific way. To look at what the emergency department efficiency frontier is. And the efficiency frontier is a part of the terminology that in those systems engineering message.

We want to contrast those with the inefficient emergency departments in order to think about, of course, how an inefficient emergency department may become a more efficient one. We are looking at three areas in this study. We are looking at structural characteristics such as what kind of hospital it is.

We are looking at operational characteristics such as the volume of the emergency department, and we're looking at advanced features, such as how do they index patients, how do they use documentation in the emergency department? And perhaps whether they use a fast prep or the triage of patients? So, in using data envelopment analysis.

As I said, this is a systems engineering tool that's used to evaluate the efficiency of each emergency department, among a set of peer groups and compared their performance. Now again I'm not going into a lot of the technical details of DEA. We do have a technical proceedings paper that we have submitted for this work if people are interested in more of the details.

Let us say here that we're focusing on using DEA because it will show us the relative performance scores of the decision making units, which is another term of DEA and it will incorporate multiple inputs and outputs. While other commonly used metrics may provide absolute outputs and values that based on just only the output of the system.

So I have here just one example, a little bit of a technical example, to show how it works. The idea is that in a group of hospitals, so in this diagram labeled as H1 through H10, you want to find among some factors, what is the efficient frontier. And on that efficient front here, the score that the hospital receives for it's level of efficiency would be one.

So one would be the perfect score, that is again the hospital that is on the efficient frontier. So here in this group, we see that hospitals one, eight, and seven are along the efficient frontier and the other hospitals are not. The beauty of DEA is that it allows us to understand when a hospital that is not on the efficient frontier by the factors and variables studied, what that hospital may do as a set of alternatives to improve.

But for example, if we take a look at hospital H10 here, we can say that there are a couple of ways that hospital H10 might become more efficient. One would be to increase admissions in other words to become a larger hospital. The other route would be to decrease the nursing hours needed to take care of the patients that it has.

Those two alternatives and the range in between are things that could, by this DEA approach, help the hospital to become more efficient. In a DEA procedure, what is important is to establish what the decision making unit should be or would be for the group of hospitals in this case, whose performance is being benchmarked.

Again, we use the EDBA, the ED benchmarking alliance 2012 database. A small image is there to give some idea of the types of questions that the participants in the alliance will answer in regard to their ED. And as I said there are almost 1000 EDs involved in the database.

By looking at the data, and making sure that we had the needed data filled in in all of the cells for the hospitals that would be Involved in this study, cleaning up that database left us with 449 EDs that had enough information in the database for us to analyze.

So our analysis refers to about half of those hospitals. Then, the next critical step is to determine the inputs and outputs that will be used in this study. Again, to look at a manageable scope in this first pass of this study, our inputs are the number of And the number of hours the physicians and nurses involved in the hospital.

Of course there are other care providers but our focus is these. And then with the output what is the throughputs per day? What is the adjusted length of stay in the ED? So that is taking in account the patient who might be transferred to other units. As well as the left without being seen rate, those patients who essentially walked because there is too long of a wait in the waiting room.

Using a linear programming approach as a part of the DEA procedure on the 449 EDs, we are able to solve the LP and I spared you the various equations and mathematical notations, but we solve the linear program, a system of equations. To find out what are the optimal waits for the inputs and outputs that we are using in this study and assign a score between zero and one according to the performance of that particular hospital.

So doing that this slide then shows the initial result for all 449 EDs. It shows their efficiency score by their volume. What you can see in this diagram is that there is a linear relationship between ED volume and efficiency score. Or as my colleague Doctor Deflitch would say that's you know only obvious that the volume will certainly have something to do with how well an ED can operate.

So with that, what we did next was to cluster the hospitals, or group the hospitals, by size and then proceed to analyze their efficiency. So the six groups here, I'll just point out that as an artifact of how the linear program is setup as a minimization problem this goes in reverse order.

So the volume of hospitals, this is up to 20,000, it should be units of thousands of patients. From up to 20,000 patients on an annual basis is our group six. Then those hospitals that see up to 40,000 patients is our group five, up to 60,000 patients is our group four, and so on on through group one.

The largest hospitals that have in excess of 100,000 patients per year. So separating that out by group, this slide shows their efficiency scores. So I'll start here at the bottom, since they started with group six, the smallest hospitals before. You can see the efficiency scores for group six are from the volume, from zero up to 20,000 here.

And the efficiency scores for those hospitals are from zero to one, as I indicated before. We can see a certain segment of them are able to reach a fairly high efficiency in that case. But many many hospitals are not operating at their maximal DEA efficiency. And so on is the story through the other groups, up through the largest group of hospitals, with volumes from 100,000 patients and onwards.

We see here, by quick comparison. That many more of the large hospitals are able to achieve DEA efficiency. I'll translate these results into a more simpler bar graph here. Again, by group of hospital. We can see that there is a significant difference between the larger patient volume EDs.

Where about 45% to 55% of the hospitals are able to achieve EE, or I'm sorry, DEA efficiency as compared to the smaller hospitals where only about 15% to 35% of them are able to achieve efficiency. Certainly we would have to do a further investigation to determine the cause of these results.

We're certainly not saying that it's only because they are small that they are inefficient but we can say there's a large efficiency gap between the small hospitals and the large hospitals that perhaps suggest that a more rapid proliferation of such practices is needed among these small hospitals in order to improve performance.

So last slide here before we move into discussion, I want to just highlight some of the further analysis that we were able to do. So based on the efficiency score, what we did was to perform a logistic regression to analyze which factors were associated with the efficient emergency departments.

So in this study we looked at three factors. Certainly there are other factors or advanced features, but we looked at three here, and focused on these three for discussion purposes today. In the first set of slides here, looking again across those 500 or so, 449 EDs. We do have from the EDBA information on their intake model.

We can see from our study that those hospitals that have performance that is along the efficient frontier use always nurses to do the intake. Whereas the less efficient groups, use nurses about half the time or 53% of the time and then the remaining time you position a mid level provider in LP for the intake of patients.

We can also look at technology IT to improve the workflow. In our efficient hospitals, we found that 75% of them use some type of computerized physician order entry system. With 25% of them did not. In contrast, in the less efficient groups 88% of the hospitals used C.P.O.E and only 12% did not.

And then lastly, many ED's have adopted new patient flow models such as new triage protocols, a fast track system where patients who have a low acuity might be tracked off differently than patients who have a high acuity or high problem. And some of the implementation of these features as we looked back retrospectively through observational studies don't necessarily contribute to a greater efficiency.

What we found in this among this set of hospitals is that 75% of them had a fast track, 25% of them did not. Whereas among the less efficient groups, more of them had a fast track system and only 10% of them Did not. So, with that, that lays out some of the ideas that we wanted to open for discussion here today in looking at this data.

So I'll ask Sherry if she could perhaps unmute everybody. Maybe before we go to the specific questions that I have here we might have general remarks of people or from the other people on my team.

> Okay, I have now muted everyone who joined the webinar using their phone.

And for those who joined using the microphone from their computer. Please let me know if you have any question because if I mute you we will hear a lot of noise. Dr. Dusilich, can you hear us? I can hear you. Can you hear me?