ROI Project

Bob Page

EDAE 590

Dr. Jeff Foley

12-15-14

Introduction:

In 2010, The American Heart Association published evidence-based guidelines for the management of Acute Myocardial Infarction. This biggest recommendation was that of Emergency Medical Services (EMS) providers (Paramedics and EMT’s) start acquiring a 12 Lead Electrocardiogram to recognize the heart attack early and get the patient to definitive care. They suggest a standard of time when the first 12 lead is acquired. This is called first medical contact to first 12 lead time. This time needs to be as soon as possible, and the results transmitted or told to the emergency department at the hospital. The benefit of early recognition and management of acute MI has been shown to save lives and lifestyles. In fact one leading cardiologist opines in his seminar talks that“every second delay in acquiring a 12 lead ECG, 500 heart cells die.” Many EMS organizations are plunging into this new skill set with a hodgepodge of training programs designed to introduce this new program. I have found these efforts to be costly and unsuccessful in achieving their goals. I believe the program I offer in 12 lead ECG interpretation can produce better results at less costs than sending a few people to a class out of town and having them come back and teach the others. Furthermore without the motivation and understanding the value of doing a 12 lead quickly, the learner may not be moved to change practice.

This Project

I looked this over carefully especially the Kirkpatrick/Phillips model for evaluating this and I found the first four levels are fairly easy to do and were very telling. However, I could not find a monetary value to place on “cardiac cells” so it was extremely difficult if not impossible for me calculate a ROI based on their model. However, I did read about the intangibles and in my profession, this is what I have here. These results are qualitative and can easily be debated. A patient’s condition does not always respond logically the way the data would suggest they would. Outcome studies, no matter how well done, have their question marks and are not always reliable or have stated reliability. That has lead some in EMS Leadership to question whether EMS earns the money that it is paid. In other words, “Why don’t we just take “everybody” to hospital like a taxi would and not go through the extra training and effort to put skilled clinicians on the ambulance if it does not improve outcome?” Another comment I had on a level one pre-evaluation was “The 12 Lead does not change what I do for the patient so it is a waste of time.”

For this reason I plan to show that the training can make a difference in the “intangible” metrics we can measure such as time to read a 12 lead, the paramedics desire to run the 12 leads sooner, and understanding the value or getting a 12 lead early.

The Chain of Impact:

Level 1 Reaction, Satisfaction and Planned action.

This in effect measures participants the learner’s satisfaction with the program and their plans to use what they had learned. For this one I had before and after questionnaires. The results are below: The first table reflects a brief pre-coursesurvey provided participants on screen and learner responses were collected with audience response devices. The average Likert scores are reflected in the columns.

Question / Likert scale 1-5 AVG
1-low confidence
5 – high confidence
I feel confident in interpreting 12 leads for STEMI / 2.2
I feel confident in running 12 leads on patients / 2.3
I feel I run 12 leads in a timely manner / 3.5
I got 12 lead training in my paramedic class / 1.2
I got 12 lead training from our training officer / 3.8
Rapid 12 lead acquisition is emphasized in our service. / 2.0

After the course was over and an assessment was performed with feedback, the post course survey was administered. N=73

Question / Likert scale 1-5 AVG
1-low confidence
5 – high confidence
Pre-course / Likert scale 1-5 AVG
1-low confidence
5 – high confidence
Post-course
I feel confident in interpreting 12 leads for STEMI / 2.2 / 4.6
I feel confident in running 12 leads on patients / 2.3 / 4.9
I feel I run 12 leads in a timely manner / 3.5 / 1.5
I got 12 lead training in my paramedic class / 1.2 / 1.2
I got 12 lead training from our training officer / 3.8 / 3.8
Rapid 12 lead acquisition is emphasized in our service. / 2.0 / 1.0
How likely are you to change your practice based on what
You have learned in this class / 4.8
This class was useful and worth my time to take it. / 5.0

This level one evaluation revealed that this service does not emphasize the running of 12 lead ECG’s. There was a 2-point major difference in running a 12 lead in a timely manner. The learner’s confidence in running and reading the 12 lead were increased.All of the participants were satisfied with the class, many made comments that this class “needs to be required of everybody.”It should be pointed out that these results alone do not guarantee the overall success of this program.

Level 2 Learning

A level 2 assessment actually rest the individual to see if the learning has occurred. There are many ways to do these; scenarios, tests, group evaluations, etc. For this level 2, I had a unique opportunity to isolate one shift of a three-shift service but only had two shifts that could attend the (two day) class I was teaching at a Fire Department in Florida. I did group evaluation with “A” shift being my control Group. They took the same a pre and post-test that the other two shifts had, only they did not have the class. They have never had the class before and did not have the class this trip. The other 2 shifts, B and C shifts both attended the class and were evaluated by pre and post test to assess their absorption of the material. I constructed a table with the results.

Assessment / A Shift
Pre / A Shift
Post / B Shift
Pre / B Shift
Post / C Shift
Pre / C Shift
Post / NET
Gain
STEMI recognition / 34% / 34% / 30% / 98% / 38% / 100% / 62-70%
15 Lead interpretation / 12% / 14% / 10% / 100% / 16% / 100% / 84-90%
Wide Complex tachycardia / 20% / 22% / 18% / 96% / 24% / 94% / 74-78%
IntraventricularBlocks / 2% / 2% / 2% / 98% / 4% / 96% / 96-98%

The raw scores indicate that the knowledge was absorbed and they were able to interpret the 12 leads correctly after having the class. Incredible net gains in critical skills performance. Several in A shift wanted to know why I was testing them again after only a day or two with no extra knowledge or feedback on how they did. I told them that they were my control group for comparison. I’ll return after the first of the year and provide“A” shift the same training the other teams had. This demonstrated a key point: Without feedback to know if they are correct, then the leaner in most cases, is not motivated to make an effort to learn. When I looked at their score versus their confidence level it was apparent that as a group, some “did not know what they did not know,” and other did know as their confidence was not high in that skill. This measure is also a selling point for getting all shifts in one trip rather than spacing it out as this group did. Even though we can show a significant increase in knowledge through assessment and evaluation, this alone is not enough to prove ROI to the financial minded. Still I think the chance to isolate one shift is very telling in the effectiveness of the program based on program objectives.

Level 3: Behavior, Application, and Implementation

This was interesting. I had two out of three services ready to implement their new skills. I had my own question to answer, does the learning gained in this class relate to an increased confidence of their skills to affect their practice? The chief wanted to rewrite the policy immediately to make all shifts do the new procedures they had learned, the training officer convinced him to wait until A shift got the same training. I glad they listened to the training officer. I instructed him to collect data from all three shifts on their FMC (first medical contact) to first 12 lead time. This is in keeping with the AHA criteria discussed in the introduction. Their ECG machines actually record the time for us so this was easy to measure. The next table reflects the results for 30 days after the class.

FMC to 12 Lead time / System before
class / A shift control
After class / B shift
after class / C shift
after class
< 10 min / 3% / 2% / 100% / 100%
< 5min / 0% / 0% / 90% / 96%

Based on this level 3 look it is obvious that behaviors were changed in the practice of the paramedics that took the class compared to their baseline. The Chief based his wanting to implement this policy of acquiring 12 leads in less than 5 and no more than 10 minutes immediately. But without A shift knowing want to do the results would be hard to achieve for one shift and morale would suffer. This is another example of the“intangible”impact of incomplete or inconsistent training.

The training officer was given a training packet and instructions on how to implement a competency program where the paramedics can be tested every 90 days to reestablish their confidence through feedback. The data will be able to locate weak areas of interpretation that can be used to individualize remediation. The competency program is proactive and had been done without extra expenses to the company. Having a program like this in place only adds to the value of a high quality, consistent initial program. Even those these results are very convincing, there is no monetary that can be drawn for a bottom line comparison.

Level 4 Business Impact: Sales versus Cells

This is where I encountered my problems, I could not quantify these results in monetary value. If I were using heart cells as money, I could assign them a value like one dollar per cell. That way we can label the business cost in cells saved not sales made. Below I have an example from the system that I trained.

FMC is the time marked at Power On. FMC is 12 lead #1

Shift A Summary FMC to 12 Lead 1 Summary B Shift Example FMC to 12 Lead 1

12 min, 24 seconds 1 min 16 seconds

744 sec x 500 cells = 372,000 lost 76 sec x 500 cells = 38,000 lost

A cell savings of 334,000!

Now cells are not dollars and no one can put a price on a human life. This is the biggest intangible of all. This class changed the practice of the medics that attended

and this particular comparison, it showed a cost (cell) savings of 334,000 cells simply by running the 12 lead sooner that they did before the class. This is a total paradigm shift from current paramedic practice. Some ill informed physicians will say “time is muscle” only to turn around and tell the paramedics not to run a 12 lead immediately and to hurry and get them to the hospital. This delays the First 12 lead time as much as 20 to 30 minutes.

Level 5: ROI

Ok, I’m going to call cells, “dollars” for the purpose of using the formulas. Based on annual number of STEMI’s (people actually having a heart attack) in their system to be1000 per year, that would be a net cell savings of $344000 x 1000 = $344,000,000 cells annually. The cost of three seminars would be a total expense of

Costs: $9480 daily cost of seminar x 3 days (shifts) = $28,440, which includes the speaker fees, personnel overtime to attend.

BCR is Total Program Benefits$334,000,000

______= 11,744:1

Total Program Costs $28,440

For every dollar spent, we save 11,744 heart cells of citizens in our community.

ROI is Benefit - Cost x 100% $334,000,000 – $28,440

Cost $28,440

$333,943,120 x 100%

$28,440

= 1,174,202%

If a heart cell were worth a dollar that would be a huge return on investment. Obviously this analysis has many intangibles and many areas that could affect the bottom line. Other intangibles may include barrier times, uncooperative patients, a new paramedic and/or others who are uninformed. The Heart Association focuses on the process of recognizing are heart attack early, even suggesting time benchmarks of 5 minutes FMC to first 12 lead time. It has evidence to suggest that time is muscle and we save muscle cells by understanding this important metric. However, whether you call it a dollar or a cell, the return on investment appears to be well worth it.

Summary:

This was a great exercise in the calculation and the process of determining ROI. In synthesis, I will start to use this process on all of my classes and when I move into doing more online sales to show the actual ROI in these areas. What great scaffolding this was throughout this course, from something as simple (and powerful) as a CAT to full tilt industrial ROI Model. At first I thought I should just work with a group and answer the questions as my option, but now, I am so much better to have done this. Speaking of intangibles, I wonder if my classmates caught the example you set in this class on how to do the class, perfect examples of instructional design. I had on my teacher hat and instructor hat at times and I’m glad I did! I had to experience doing this to appreciate it. I lived it, stressed over it, read other examples, now I walk away from it with the knowledge in my backpack for future use.