Appendix 1: Summary of Medline OVID Search Strategy
1: safety.ti,ab.,
2: safe.ti,ab.,
3: 1 or 2,
4: exp Organizational Culture/ or exp Culture/,
5: culture*.ti,ab.,
7: 4 or 5
8: (patient* adj2 safe*).ti,ab.
9: (safe* adj2 culture*).ti,ab.
10: (safe* adj2 climate*).ti,ab.
11: (safe* adj2 attitude*).ti,ab.
12: (safe* adj2 behavior*).ti,ab.
13: (safe* adj2 behaviour*).ti,ab.
14: 8 or 9 or 10 or 11 or 12 or 13,
15: hospital*.ti,ab.,
16: exp hospitals, teaching/ or exp hospital units/ or exp hospitals/,
17: 15 or 16,
18:14 and 17
19: 3 and 7 and 17
20: 18 or 19
21: limit 20 to (english language and humans and yr="1995 -Current" and (addresses or bibliography or biography or comment or congresses or dictionary or directory or duplicate publication or editorial or in vitro or interview or lectures or legal cases or legislation or letter or news or newspaper article or patient education handout or periodical index or portraits)),
22: 20 not 21,
Exp = explode, ti = title, ab=abstract, yr = year
Appendix 2: Summary of Included Studies
Study reference / Study design / Population / setting / Intervention / Outcome measure / Key findings and implementation lessons / Critical AppraisalBenning et al (2011a and b) UK / 2 phase controlled before and after study / 22 National Health Service (NHS) Trust hospitals.
Phase 1: 4 intervention / 18 control hospitals. Phase 2: 9 intervention / 9 control hospitals
Random sample of 850 staff members from each organisation were recruited for completion of the staff survey. / Multi-component organisational intervention: Safer Patient Initiative (SPI), involving learning sessions, site visits, resource materials, evidence based standards, evaluation and monitoring. / Safety culture using 11 of 28 survey questions from the NHS national staff survey.[16] Other outcome measures: Rates of adverse events and in-hospital mortality
Phase 1: 18 month follow up
Phase 2: 2 year follow up
Qualitative analysis including focus groups, interviews and ward observations / Phase 1: A small time and intervention effect was found for one of the 11 survey questions (organisational climate) (mean difference: 0.08; 99% CI: 0.02-0.13, P<0.01). There was no effect found for other questions on the staff survey.
Phase 2: A small negative effect was found over time for the intervention for one of the 11 survey questions (organisational climate) (mean difference: -0.07; 99% CI: -0.14-0.00, P=0.009).
No significant difference between groups for rates of adverse events or in-hospital mortality in both phase 1 and 2 of the study.
Qualitative analysis reported gaps between executive level view and what was happening at the ward level. / Non random allocation to control and intervention groups.
No sample size or power calculation.
Descriptive characteristics of participants not reported and no assessment of statistical difference between groups.
Range between hospital climate scores at baseline were 0.55, no p value reported.
Fair response rates reported in phase 1 (49-50%). Response rates not able to be determined for phase 2.
Analysis was adjusted for age, sex, ethnic background, occupation group, length of service and management status.
No reported validation of the tool used for assessment of safety climate.
Large scale organisational intervention implemented with limited capacity to adapt to contextual issues.
Study findings potentially generalisable to National Health Service Trust hospitals.
Berg et al (2008) USA / Historically controlled study, using historical hospital safety climate survey results for baseline. / One rural hospital, 10 clinical teams / Simulation-based training program:A standardized simulation based Crisis Team Training (CTT) program consisting of on-line pre-course didactic material and face to face scenario based training, conducted in a one day workshop. / Safety Climate Survey (SCSu).[27]
12 month post intervention commencement / There was a significant increase in mean hospital Safety Climate between 2007 and 2008 (2007: mean safety climate = 3.7; 2008 mean safety climate = 3.87) P=0.006.
There was a relative % increase in the safety climate score of 21%, post- intervention which was statistically significant (P=0.02). / Uncontrolled pre and post intervention study. Authors noted similar year to year variability in safety climate scores prior to the introduction of the intervention. Therefore improvements may be coincidental and confounding factors cannot be ruled out.
A cause and effect association between the introduction of simulation based training and education and patient safety climate cannot be determined.
Fair overall response rate (46%).
Potential for response bias.
Poor generalisability of findings as study was conducted in only one organisation.
Sustainability of results is unclear.
Bleakley et al (2006) UK / Controlled before and after study / An acute rural multi-site hospital, 2 operating theatre complexes. / Structured educational program:4 single day seminars over 6 months, followed by 6 months of iterative feedback and monitoring cycles.
Focused on: 1. Self-review and reporting; 2. Briefing and debriefing; 3. Human factors. / Teamwork Climate; a component of the Safety Attitudes Questionnaire (SAQ).[6]
12 month follow up / Regression analysis found a time and intervention effect for the intervention, showing a positive effect on the teamwork component of the SAQ (P=0.039). / Non random allocation to control and intervention groups.
Convenience sampling of staff participants.
No sample size or power calculation.
Good response rate reported at baseline and follow up (68-73%)
No assessment of statistical difference between groups.
Unspecified whether there were differences in intervention and control group participant characteristics.
Risk of contamination between groups due to anaesthetist staff movement between theatre complexes.
Regression analysis reported only 1.6% of the variance could be accounted for by the statistical model (R2=0.016).
No adjustment for potential confounders (eg. age, sex, length of service, provider group and management status).
Intervention effect size is unclear.
Limited generalisability of findings as study was conducted within a single organisation.
Blegen et al (2010) USA / Historically controlled study / One inpatient medical unit from 3 medium sized hospitals in the San Francisco Bay Area / Multi-faceted unit-based program:Triad for Optimal Patient Safety (TOPS) project, a multi-disciplinary teamwork and communication intervention comprising of: teamwork training; Safety Teams (TrUSTs) and daily patient goal cards. / Hospital Survey on Patient Safety Culture (HSOPS).[28]
12 month follow up / There were significant increases in safety climate scores for 5 of the 10 safety culture dimensions measured (P<0.05).
Safety climate scores varied across hospital sites. With one clinical unit showing no improvement in safety climate scores post intervention.
The unit-based safety teams (TrUSTs) differed across clinical units as they worked with the different staff configurations and clinical concerns at each site. / Uncontrolled pre and post intervention study, making it difficult to rule out other potential confounders, such as changes to staffing levels and mix, workload and case-mix, equipment and environment. Study design is at risk from time related effects on the outcomes of interest.
Good response rates at baseline (96%) and follow up (81%).
A structured intervention providing scope to adapt to contextual surroundings.
Sustainability of effect is unclear, although two units showed improvement at 12 months post intervention.
One clinical unit showed no change in safety climate scores.
Poor generalisability of study findings.
Cooper et al (2008) USA / Controlled before and after study / Six hospitals,6 anaesthesia departments (4 intervention groups / 2 controls). / Simulation-based training program:Crisis resource management (CRM).
A one day training session: 6-7 hours based on 5 set principles (role clarity, communication, resources, support and global assessment) / Survey tool based on the Patient Safety Cultures in Healthcare Organisations (PSCHO). [13]
3 year follow up / No time or intervention effect was found for any of the 8 constructs of the safety climate score or the overall score.
There were significant differences between hospitals on each of the 8 scales of the safety climate survey and the overall safety climate score reported (P<0.001). / Non random allocation to control and intervention groups.
Convenient sampling of clinical units, leading to a potential risk of selection bias.
No sample size or power calculation.
Control group was added 1 year following commencement of the study.
Response rates were poor to fair and differed across the 6 hospitals (20.8-63.2%).
Factor analysis of modified survey tool conducted with acceptable internal consistency and content reliability (α: 0.66-0.84).
Baseline assessments of the two groups were conducted at different time points.
Significant statistical difference for hospitals in baseline and follow up safety climate scores.
Unspecified whether there were differences in intervention and control group participant characteristics.
Risk of contamination between groups due staff prior exposure to training.
Intervention was limited to only a 1 day training session and not specifically focused on patient safety culture. Sensitivity of the safety climate tool may have been insufficient to detect an effect following the CRM training. No adjustment for potential confounders (eg. age, sex, length of service, provider group and management status)
Limited generalisability of study findings.
Frankel et al (2008) USA / Historically controlled study / Two hospitals (out of 7) participated
Hospital A: 9 of 20 clinical areas
Hospital B: 12 of 24 clinical areas / Leadership walk round:Weekly Leadership WalkRounds following a seven step structured guide to identify patient safety concerns / Safety Attitude Questionnaire (SAQ). [6]
18 months follow up / There was a relative increase in % of positive safety climate scores 18 months following intervention implementation for hospital A and B. Hospital A: safety climate scores: 62% pre & 77 % post ( P=0.03); Hospital B: safety climate scores: 46% pre & 56% post (P=0.06).
Patient safety concerns commonly identified during WalkRounds included: equipment or environment and communication.
WalkRounds provide a formal structure and ongoing mechanism for providers to discuss and address patient safety defects. / Uncontrolled pre and post intervention study making it difficult to rule out confounding factors.
Good response rate (>60%), however only units with response rates > 50% were included in the study, leading to potential response/volunteer bias.
Small sample size, potentially limiting statistical power of results.
Differences between baseline and post intervention participants may have resulted in potential bias.
Risk of selection bias.
Poor generalisability of findings as study was conducted in only 2 hospitals.
Ginsburg et al (2005) Canada / Controlled before and after study / Nurse clinical leaders at 2 multi-site teaching hospitals / Structured educational program:2 voluntary patient safety workshops provided over a 6 month period, aimed at improving perception and attitudes of the patient safety culture. / A survey tool adapted from the Patient Safety Cultures in Healthcare Organizations (PSCHO) survey, involving 3 dimensions [81]
10 month follow up / Regression analysis found a time and intervention effect for one dimension of the PSCHO survey valuing safety (P<0.001) for those that attended the workshop compared with those that did not. A positive effect from the intervention for this dimension of safety climate.
An association between the leadership for improvement variable and valuing safety scores was also found (P<0.001).
Limited implementation of safety initiatives were reported at both participating hospitals, due to competing priorities and resource constraints. / Non random allocation to control and intervention groups.
Risk of selection bias.
No sample size or power calculation.
Good reported response rate (69%).
Factor analysis of modified survey tool conducted with acceptable internal consistency and content reliability (α: 0.68-0.81).
Differences between group baseline measures reported for 2 of 3 survey dimensions (valuing safety: P<0.001; perceived state of safety: P<0.05)
Assessment of differences in intervention and control group participant characteristics were reported.
Analysis not conducted on an intention to treat basis.
Intervention effect size is unclear.
Analysis was adjusted for job status and age. Secondary analysis also reported leadership to be a likely significant confounding factor to intervention effectiveness.
Limited generalisability of findings as study was conducted across only two organisations.
Gore et al (2010) USA / Historically controlled study / One hospital, 1 Operating room, all personnel / Simulation-based training program:An 8-hour training session in crew resource management and team training (using pre-procedural briefings) delivered to all operating room personnel by a company of aviation pilots / A survey of 45 questions made available by the AHRQ was described but not named by the authors
6 months post completion of training session / There was a significant improvement (P< 0.05) for:
(i) 2 of 13 questions on the survey relating to reporting errors; and
(ii) 2 of 11 questions relating to safety climate.
Sub group analysis found no significant changes in faculty physician survey results. / Uncontrolled pre and post intervention study making it difficult to rule out confounding factors.
Poor response rates at baseline (34.5%) and follow up (27.6%).
Differences between baseline and post intervention participants may have resulted in potential bias.
The identity of the measurement tool is unclear; hence validity of the instrument cannot be determined.
Short observation and intervention period may have led to limited effectiveness. Sustainability of change is unclear.
Poor generalisability of findings as study was conducted in only one organisation and one clinical unit.
Haynes et al (2011) USA / Historically controlled study / Eight hospitals participating in a pilot study of the WHO surgical safety checklist.
1 site failed to distribute questionnaires / Other patient safety culture strategy:WHO surgical safety checklist. / Modified operating room version of Safety Attitudes Questionnaire (SAQ). [6]
Questionnaire distributed to operating theatre staff 2 weeks pre and 2 weeks post intervention. / Clinicians from 7 of the 8 hospitals completed the survey during the 2 phases and were included in the analysis.
Aggregate mean safety attitude score rose from 3.91 to 4.01 following implementation of the check list (P=0.0127).
Improvements in postoperative outcomes were associated with improved perception of teamwork and safety climate suggesting these changes may be partially responsible for the effect of the checklist.
A majority of staff agreed that the checklist improved operating theatre safety and communication (80%) and would want the checklist used if they were having an operation (93%). / Uncontrolled pre and post intervention study.
Factor analysis of the six items taken from the SAQ found acceptable levels of internal consistency (α:0.60).
Short intervention and observational period (2 weeks).
Survey administered in the context of a larger quality and safety improvement project potentially contributing to response bias.
Limited generalisability since hospitals were diverse and not chosen to be representative of hospitals providing surgical care.
Inability to determine if a causal association between improvements in safety attitudes score and reduction in post-operative complications exists.
Hellings et al (2010) Belgium / Historically controlled study / Five hospitals (3 private and 2 public) / Other patient safety culture strategy:Interventions varied by hospital and were poorly described. They were labelled "the improvement approach" / Hospital Survey on Patient Safety Culture (HSOPS). [28]
Pre and post intervention.
Timing of intervention was unclear (18-26 month follow up) / After an 18 to 26 month period only one survey dimension “hospital management support for patient safety” was found to show a significant difference across hospitals (P<0.05).
Survey results varied across hospitals.
Regression analysis suggests there is a significant difference between professional subgroups. / Uncontrolled pre and post intervention study.
Good overall response rate (77% pre intervention to 68% post intervention).
After translation into Dutch, a psychometric (re)validation based on pilot hospital data was completed using the original questionnaire validation strategy.
The interventions used by hospitals were poorly described and therefore not reproducible.
Limited generalisability as participating hospitals are potentially non representative.
McCulloch et al (2009) UK / Historically controlled study / One UK teaching hospital, 1 theatre suite / Simulation-based training program:9 hour classroom non-technical skills course based on aviation "Crew Resource Management" (CRM) followed by 3 months of coaching from CRM experts. / Teamwork Climate; a component of the Safety Attitudes Questionnaire (SAQ). [6]
Pre and post intervention (3 month follow up) / An increase in the mean score of the teamwork component of the SAQ was found (t=-2.95, P=0.007).
There was found to be no significant effect on other SAQ components.
Training in non-technical skills resulted in improvement in attitudes to safety, team non-technical performance and technical error rates both in the operative field and outside it (P=0.021)
Operative technical error rate declined significantly from 1.73 to 0.98 per operation (P=0.009). / Uncontrolled pre and post intervention study, making it difficult to rule out other potential confounders, such as changes to staffing levels and mix, workload and case-mix, equipment and environment. Before and after study design is at risk from time related effects on the outcomes of interest.
Only one component of the SAQ, Teamwork Climate was reported.
Potential contribution of "Hawthorne effect".
Poor generalisability of findings as study was conducted in only one organisation.
O'Leary et al (2010) USA / Controlled before and after study / One tertiary teaching hospital (897 beds), 2 general medical units (30 beds) / Team based strategy:Daily structured 30-40 min interdisciplinary ward rounds (SIDR), using a communication tool. / Teamwork Climate and Patient Safety Climate; components of the Safety Attitudes Questionnaire (SAQ). [6] Other outcome measures: Patient length of stay (LOS) and costs.
6 month follow up / No intervention effect was found for one dimension of the SAQ Safety climate between the two groups (P=0.9).
All providers on the intervention ward rated Teamwork climate (one dimension of the SAQ) higher when compared to the control ward (mean score: 82.4+11.7 vs 77.3+12.3, P=0.01).
There was no intervention effects found for patient LOS and costs. / Non random allocation to control and intervention groups.
No sample size or power calculation.
Good reported response rate (84-92%).
Assessment of differences in intervention and control group participant and patient characteristics were reported.
Timing of survey for different provider groups not consistent.
Intervention was focused on teamwork rather than patient safety culture.
No adjustment for potential confounders in the analysis of SAQ (eg. age, sex, length of service, provider group and management status).
Short observational and intervention period (6 months) may have led to limited effectiveness.
Limited generalisability of findings as study was conducted across only one organisation.
Paine et al (2010) USA / Prospective cohort study / One urban academic medical centre, 144 clinical units in / Multi-faceted unit-based program:Comprehensive Unit-Based Safety Program (CUSP), an 8 step safety program.
CUSP steps: (1) culture of safety assessment; (2) education; (3) staff identification of safety concerns; (4) executive sponsorship ; (5) improvements implemented; (6) evaluation; (7) results shared; and (8) culture reassessment. / Safety Attitudes Questionnaire (SAQ). [6]