Modeling safety outcomes on patient care units 1
Modeling Safety Outcomes on Patient Care Units
Anita Patil
Department of Electrical and Computer Engineering
University of Arizona
Judith Effken
College of Nursing
University of Arizona
Kathleen Carley
Department of Social and Decision Sciences
Carnegie-Mellon University
Ju-Sung Lee
Department of Social and Decision Sciences
Carnegie-Mellon University
1.1 Introduction
In its groundbreaking report, “To Err is Human,” the Institute of Medicine reported that as many as 98,000 hospitalized patients die each year due to medical errors (IOM, 2001). Although not all errors are attributable to nurses, nursing staff (registered nurses, licensed practical nurses, and technicians) comprise 54% of the caregivers. Therefore, it is not surprising, that AHRQ commissioned the Institute of Medicine to do a follow-up study on nursing, particularly focusing on the context in which care is provided. The intent was to identify characteristics of the workplace, such as staff per patient ratios, hours on duty, education, and other environmental characteristics. That report, “Keeping Patients Safe: Transforming the Work Environment of Nurses” was published this spring (IOM, 2004).
In the last five years, AHRQ also funded a number of research studies directed at patient safety issues. The Impact Studydescribed in this paper is one of those efforts. In the Impact Study, we chose to focus on studying workplace characteristics (organization and patient care unit) that can affect patient safety and quality outcomes. To do so, we collected data from 37 patient care units in 12 Arizona hospitals. Hospitals that participated in the research included teaching and non-teaching hospitals, as well as public and privately funded hospitals ranging in size from 60 to over 400 beds. To ensure a more comparable analysis across units, only adult medical or surgical units were included in the sample. Data were collected in two “waves;” patient care units from half the hospitals were assigned to each wave. Each wave of data collection required six months to complete. Data related to each of the model components were collected through surveys of patients, staff, managers, quality improvement (QI) departments, and information services (IS). In all, 1179 patients and 867 staff were surveyed.
Figure 1. The Conceptual Model Used to Guide the Impact Study showing only the variables used for computational modeling
The framework for the research was based on the 1996 American Academy of Nursing Health Outcomes Model (Fig. 1) (Mitchell, Ferketich & Jennings, 1998). Patient safety outcomes studied included medication errors and patient falls, with and without injury. Quality outcomes investigated included symptom management, simple and complex self care, and perception of being well cared for. In this paper, we will focus only on the safety outcomes. The data were subjected to descriptive analysis, as well as causal modeling prior to their being used to develop the computational model. Our emphasis was on identifying changes that nurse managers could make at the patient unit level, because organizational changes are not under their control. Similarly, nurse managers cannot change the characteristics of the patients they see on their units; instead patient characteristics were treated as risk adjusters in the analysis.
1.2 Using OrgAhead to Create Virtual Patient Care Units
1.2.1 Orgahead
We used OrgAhead, a computational modeling program, to create virtual units that functionally matched our actual units and then evaluated the patient safety outcomes for the virtual units under various conditions. OrgAhead is a theoretically based computational modeling program developed by Dr. Kathleen Carley and her team at Carnegie Mellon University. The theoretical perspectives grounding OrgAhead derive from social network, complexity, and organizational theories(Carley & Hill, 2001) and are consistent with the theoretical basis of the conceptual model used for the Impact Study. OrgAhead had been used in a number of military and business settings, but this provided its first application in healthcare.
1.2.2 The modeling process
The computational modeling process (Fig. 2) included four distinct steps: First, variables in our research model were matched to the variables in OrgAhead. Variables that were found to be significant in our model that were not initially in OrgAhead were created for OrgAhead in collaboration with Dr. Carley and her team at Carnegie Mellon University. Next, we determined the range for each independent variable in OrgAhead. In the third step, values were set for all other variables used in OrgAhead. Finally, experiments were run to validate the virtual model with actual data and then to generate hypotheses about the kinds of changes that might be made to improve patient outcomes (Effken et al., 2003, Effken et al., in preparation) using a static version of the model.
One of the strengths of computational modeling is its ability move beyond static, snap-shot analyses to examine organizational performance over time. To explore how patient safety outcomes might change under various changes in structure and unit characteristics, we utilized the annealing characteristics of the model. In Orgahead, individual learning occurs through a standard stochastic learning model for boundedly rational agents (Carley, 1996). Organizational adaptation or learning occurs as a simulated annealing process. The annealing model was developed originally to solve complex combinatorial optimization problems (Kirkpatrick, 1983; Rutenbar,1989). Simulated annealing is a heuristic for optimization and a computational analog of the physical process of annealing (heating and cooling) a solid, in which the goal of the process is to find the atomic configuration that minimizes energy costs. In organizations, this is analogous to a design problem in which the organization is trying to optimize organizational performance under various constraints (Carley, 1997). For our purposes, we assume that the patient care unit endeavors to optimize performance (e.g., achieving desired quality and patient safety outcomes) while reducing or maintaining costs.
Organizational adaptation, for our modeling purposes, has two components: executive decisions about particular restructuring goals and strategies and individual employees’ experiential learning (Carley, 1996). Executive decisions are commonly assumed to be “satisficing,” rather than optimizing. That is, the manager doesn’t consider all possible strategies are compared; instead the first one that seems likely to move the organization toward the goal is selected (March, 1958; Simon, 1954). Similarly, nurse managers do not consider every possible intervention, but select the first one that seems likely to work, given their current constraints.
Figure 2. The Computational Modeling Process
1.2.3 Modeling the patient care unit
The patient care unit is modeled as two interlocking networks: an authority structure (who reports to whom?) and a resource management structure (who has access to which resources?). We assume a 3-layered structure with RNs at the top level, LPNs and patient care technicians (PCTs) at the second level, and unit clerks at the bottom level. Individuals may have one or more subordinates and report to one or more managers. This allows us to model teams, hierarchies, matrix structures, etc.
Both the organization and individual employees operate in a “task” environment where a “task” equals a patient. In Orgahead, patients are modeled as 9-bit binary choice tasks. For each “patient,” which is modeled as a 9-bit series of a’s and b’s, the organization (patient care unit, in our research) has to determine whether there are more a’s or b’s in the string. Different levels of staff see different numbers of bits. RNs see 4 bits, LPNs and PCTs see 2 and Unit Clerks see 3. No individual can make a patient unit decision alone (assuming bounded rationality and distributed decision making); instead the unit decision is created as a majority vote of the individual decisions. For more details, see Carley, 1998.
1.3 Results and Discussion
For the static experiments with the 32 virtual units, the correlation co-efficient (r) of the rank order for accuracy (virtual units) and total errors (actual units) was found to be0.83. Thisexceeded our target of acceptable level of correspondence of 0.80 (Effken et al, 2003). The correlation co-efficient for the actual values for accuracy and total errors was found to be -0.55, which is acceptableat the value level.
The results of our static experiments to improve performance by varying the initial values of various independent variables (task complexity, workload, turbulence, standard operating procedures (SOP), training and memory) aresummarized in Table 1for 6 pilot units. These 6 pilot units are different in their key characteristics, as well as in their safety outcomes. Improvement in accuracy was obtained by varying the independent variables in the static experiments is given. A 5% improvement in accuracy in the virtual world corresponds to a decrease in 5 errors in the real world. Managers could potentially select a number of strategies to decrease task complexity or increase training, for example, to achieve this level of improvement in their real units.
Dynamic simulations involved setting up annealing parameters and allowing the virtual units to adapt over time. Figure 3 shows a snapshot of how the 6 pilot units adapt over time by changing the authority and resource management structures resulting in the performance varying over time.
Unit / A / B / C / D / E / FTask
Complexity / 15
(17) / 12
(14) / 8
(11) / 9
(12) / 8
(9) / 11
(13)
Training / 834 (234) / 1000
(365) / 903 (303) / 943
(343) / 404 (404) / 903
(403)
Memory / 500 (100) / 1000 (100) / 900
(100) / 800
(100) / 100
(100) / 800
(100)
SOP
(a) RN / 0.4
(0.0) / 0.3
(0.28) / 0.1
(0.30) / 0.3
(0.33) / 0.1 (0.24) / 0.3
(0.51)
(b) LPN/PCT / 0.8
(0.6) / 0.6
(0.33) / 0.5
(0.29) / 0.8
(0.23) / 0.5 (0.35) / 0.6
(0.71)
(c)Unit Clerk / 0.9
(0.62) / 0.99
(0.46) / 0.9
(0.39) / 0.9
(0.42) / 0.9 (0.18) / 0.8
(0.67)
% Accuracy / 80.16
(77.72) / 83.78
(82.22) / 86.07
(78.16) / 85.16 (76.55) / 86.39
(79.98) / 78.19
(74.95)
%Improvement / 2.44 / 1.56 / 7.91 / 8.61 / 6.41 / 3.24
Table 1. PercentageImprovement in Accuracy (based on static modeling) achieved for 6 Pilot Units modeled. The numbers shown in parentheses are the values for the corresponding actual unit; all other numbers are those values used in OrgAhead.
Figure 3. Accuracy (safety outcomes) by Time for 6 pilot units
The average accuracy over time for the dynamic run for each unit was consistent with the total errors reported on the actual units. For example,Figure 3 shows that Units C and D initially had similar levels of accuracy, but over time performed differently, reflecting their actual performance values. The actual Unit C produced 4.73 errors, but its corresponding virtual unit showed better performance (avg. accuracy = 79.99%) as compared to Unit D, for which the actual unit reported total errors of 26.18, and the virtual unit showed an average accuracy of 70.04%.
After validating that the dynamic average accuracy values for the 6modeled units reflected their actual outcomes, we selected some of the best and the worst accuracy values during the adaptation process and studied the respective authority and resource management matrices to see how they influenced the outcomes. This gave us some insight into the structures that lead to both successful and unsuccessful performance. Of these values, we picked only those structures that are most likely to occur in the real scenario. The result of one such successful structure for Unit E along with its initial structure is as shown in Figure 4.
Interestingly, the better performing structures were flatter. This generally resulted in eliminating the unit clerk position, which is probably not realistic because of their pivotal roles on the unit. This may represent a limitation of the modeling, which assumes that all members of the team are working on the same problem. In fact, the unit clerk has a rather different role. These results suggest that, in the future, we may need to omit the clerks from the models and perhaps include the nurse manager instead. Further modeling will be needed to inform that decision.
Figure 4. The initial and high performance structures for Unit E.
1.4 Conclusion
Using OrgAhead, we created 32 virtual patient care units that were functionally similar to the actual units in their key characteristics and safety outcomes with the outcomes (accuracy with total errors) having an acceptable level of correspondence. Using these virtual units, for the static model we varied the various independent variables for 6 pilot units, and studied how each would improve the safety outcomes. We then used OrgAhead dynamically, and verified that the average dynamic accuracy is consistent with the safety outcomes. Using the dynamic runs, we were able to pick the authority and resource management structures that produced the best and worst safety outcomes. Our computational model could be a very effective tool for nurse managers to visualize various structures and strategiesthat they could use to improve patient safety outcomes on their units, before extending them to the real world.
References
Carley, K.M. (1996). Adaptive organizations: A comparison of strategies for achieving optimal performance, in Proceedings of the 1996 International Symposium on Command and Control Research and Technology (pp. 322-330), June, Monterey, CA.
Carley, K.M. (1998). Organizational adaptation. Annals of Operations Research, 75, 25-47.
Carley, K.M. (Unpublished manuscript). Validating computational models.
Carley, K.M., & Hill, V.,2001, Structural Change and Learning Within Organizations, inDynamics of organizations: Computational modeling and organization theories, edited by A. Lomi and E. R. Larsen, MIT Press/AAAI (Menlo Park, CA).
Effken, J.A., Brewer, B.B., Patil, A., Lamb, G.S., Verran, J.A., & Carley, K.M., 2003, Using computational modeling to transform nursing data into actionable information,Journal of Biomedical Informatics, 36, 351-361.
Effken, J., Verran, J., Lamb, G., Patil, A., Garcia-Smith, D., Shea, K., Brewer, B., & Carley, K., in preparation, What does computational modeling add to traditional data analysis?
Institute of Medicine, 2001, To err is human: Building a safer healthcare system, National Academy Press (Washington, DC).
Institute of Medicine, 2004, Keeping patient safe: Transforming the work environment of nurses, National Academy Press (WashingtonDC).
Kirkpatrick, S., Gelatt, C.D., & Vecchi, M.P., 1993, Optimization by simulated annealing, Science, 220(4598), 671-680.
March, J.G., 1958. Organizations, Wiley (New York).
Mitchell, P., Ferketich, S. & Jennings, B.M., 1998. Quality Health Outcomes Model, Image: Journal of Nursing Scholarship, 30(1), 43-46.
Rutenbar, R.A., 1989, Simulated Annealing Algorithms - an Overview. IEEE Circuit Devices Magazine,5(1):19-26.
Simon, H.A., 1954, Decision-making and administrative organizations, Public Administration Review,4, 316-331.