Ornstein: Presentation Summary

Expanding Research and Evaluation Designs and Methods…QII in Health Care and Public Health

September 14, 2005

Section / Issue / TRIP-II (2000-2003)[1] / A-TRIP (2002-2006)[2]
Background / What is the quality problem being addressed? / Translation of guidelines for CVD care into primary care practice / Translation of guidelines for
CVD, DM, cancer screening, immunizations, respiratory disease/infectious disease, mental health and substance abuse, nutrition and obesity, and drug prescribing in the elderly into primary care practice
What is the nature of the evidence base for both the underlying change that is sought and the QI strategy to be employed? / Guidelines from JNCVI, NCEP ATP-III, ACC/AHA, ADA.
Prior evidence that multifaceted interventions are more effective than single interventions. / Guidelines from JNCVI, NCEP ATP-III, ACC/AHA, ADA, USPSTF, Beer’s criteria.
PPRNet-TRIP QI model developed from process evaluation of TRIP-II study.
Objectives / What was the aim of the evaluation? / Assess whether multimethod QII was more effective than a less intensive intervention for improving adherence with 21 CVD care indicators. / 1) Recruit 100 primary care practices for QII, 2) Develop data analyses/reporting approach for ~80 indicators of care, 3) Disseminate PPRNet-TRIP QI model to participating practices and assess impact on changes over time in adherence with 80 indicators of care.
Intervention / What theoretical framework(s) guided your approach? / Audit and feedback, academic detailing, quality improvement facilitation with participatory planning, use of EMR tools, complexity science, best practice approaches. / More formalized PPRNet-TRIP QI model developed in TRIP-II. Multi-method intervention: practice performance reports, practice site visits, and network meeting (based on personal and organizational theories for motivation/change). Multi-faceted improvement model: Prioritize Performance, Involve All Staff, Redesign Delivery System, Activate Patients, Use EMR Tools.
What was actually done? / Quarterly practice-level performance reports on 21 indicators of care for all practices. 6-7 practice site visits for academic detailing & discussion/planning of practice improvement approaches; two network meetings to share best practices for practices randomized to intervention group. / Quarterly practice-level performance reports using SPC methodology on ~80 indicators of care; optional twice yearly practice site visits for academic detailing & discussion/planning of practice improvement approaches; optional annual network meetings to share best practices.
Section / Issue / TRIP-II (2000-2003) / A-TRIP (2002-2006)
Intervention “targets” / Who or what was expected to improve as a result of the QII? / Health care providers (physicians, mid-level providers, staff who care for primary care patients) / Health care providers (physicians, mid-level providers, staff who care for primary care patients)
Settings / Where did the QII take place? / 20 community-based primary care practices in 14 states that participate in a practice-based research network(PPRNet) whose members use common EMR and share data for QI and research. Total of 61 HCP and 87,291 patients. / 100 community-based primary care practices in 37 States that participate in a practice-based research network (PPRNet) whose members use common EMR and share data for QI and research. Current total of 502 HCP; ~500K adult pts.
Measures / 21 CVD care measures, both process (test done, Rx prescribed or Dx made) and outcome (treatment target reached). / ~80 care measures, both process (test done, Rx prescribed or Dx made) and outcome (treatment target reached). 3 summary measures: # measures > performance target, # measures > PPRNet median, Summary Quality Indicator Measure (SQUID™)
Outcome measures / How was improvement measured?
What data sources were used? / Practice-level outcome: % of performance targets (90th percentile among practices at baseline) achieved. Patient-level outcome: % of patients for whom process measures had occurred or treatment outcome reached. / Objectives 1) and 2): Descriptive outcomes.
Objective 3): Descriptive + practice-level outcome: % of performance targets (90th percentile among practices at baseline) achieved, and patient-level outcome: % of patients for whom process measures had occurred or treatment outcome reached.
Study Design / Basic research design/adaptations. / Group randomized trial, intervention 2 years in duration. / Demonstration project, 3.5 years in duration.
Control group / Selection criteria for the control group. / Randomization, intervention and control group HCP and patients similar. / None.
Analytic approach / How were the data analyzed?
Was there a sub-analysis to examine whether disparities in care were improved? / Practice-level analyses: randomization test comparing improvement in targets reached between groups, Wilcoxon signed-rank tests to assess improvement in each group. Patient- level analyses: generalized mixed-regression models with random practice effects. Sub-analysis on race-ethnic disparities among 3 intervention group practices with > 10% minority population. / Repeated ANOVA models, repeated measures analyses, including generalized mixed-regression models with random practice effects.
Results / Practice-level analyses: improvement in both intervention and control group practices in % of targets reached; intervention group had greater improvement in 18 of 21 indicators.Patient-level analyses: improvement in intervention group > control group for 2 of 21 measures. / Objectives 1 and 2 have been met.
Objective 3 will be assessed at conclusion of project on June 30, 2006.
Section / Issue / TRIP-II (2000-2003) / A-TRIP (2002-2006)
Conclusions re study strengths, limitations, and lessons learned or to be learned
********** / What went well in this study and why? / No drop-outs after study initiation in either group.
Data extraction/analyses process worked smoothly.
Intervention-group practices found site-visits and network meetings motivating and catalyzed changes in practice. / 1) Recruitment targets achieved without difficulty.
2) Reporting tools developed on schedule and enhanced beyond project plans.
3) Uptake of practice-site visits greater than expected. Evidence of improvement in adherence with indicators.
What intervention and/or study design changes or innovations occurred?
How would these breakthroughs be helpful to other QII evaluators? / Modified frequency and duration of practice site visits (from 2 day visits q 6 months to 1 day visit q3 months). In some practices conducted site visits before or after office hours to limit practice productivity loss. / Began conducting some practice-site visits using internet-based conferencing tools.
Where did the study face challenges in intervention implementation or study design and methods? How were these handled? What changes were made?
Did any tradeoffs have to be made because of the setting, the sponsors or other reasons?
Are there any lessons yet to be learned?
Was the QII strategy used after or before the study was completed? If so, where, for whom, and with what modifications, if any? / 1) Some providers in larger practices did not “buy” in to study. Response: focused on more amenable members.
2) Initial focus on academic detailing to influence providers to follow practice guidelines and enhanced use of EMR tools may have been over-emphasized; most clinicians were aware of guidelines and preferred/had to because of technical issues to use EMR in idiosyncratic fashion. Response: site visits changed focus to QI approach at microsystem level. Academic detailer replaced by QI expert.
3) Importance of non-provider staff in implementing practice-guideline recommendations insufficiently recognized at study initiation. These staff encouraged to play greater role in participatory planning, implementation at later visits. In addition, non-provider staff from each practice participated in 2nd network meeting.
4) All practices did not record data in consistent fashion for all study variables. Response: eliminated these practices from analyses of these variables.
5) HCP and their staff were unsophisticated about statistical analyses. Without good coaching and easily interpretable analyses they may attribute causality to interventions that produce statistically insignificant changes and not appreciate the impact of interventions shown to be effective by “proper” statistical models. / Important study team member (site visitor) limited participation in project. Response: replaced.
Request by participants for patient-level analyses. Response: development of these analyses completed in manner safe-guarding pt anonymity.
Turnover in practice clinical leadership, or administrative management, retirement of or relocation of physician in solo practice, change in EMR product used or loss of interest leads to project attrition ~10%/year. Also, practices joined project at different times—all create analyses challenges.
Section / Issue / TRIP-II (2000-2003) / A-TRIP (2002-2006)
Conclusions re improving the science base for QII evaluation research / What practice, policy, and/or research designs and methods lessons can be learned from this evaluation?
Would you have done anything differently in terms of design and methods knowing what you know now:
What advice would you give to your peers and to funders, study section members, journal reviewers, research training directors, and frontline health care settings and policymakers? / Many primary care practices that volunteer to participate in QII and QII research generally enjoy the work and will participate in an intervention that they deem beneficial to their patients.
Given the constraints of practices, intervention approaches and emphases have to be customized at the microsystem-level. Study sections accustomed to specific protocols that require rigid adherence to assure this requirement may need to appreciate this reality.
The importance of non-provider staff in QII may be insufficiently recognized. In primary care, these individuals span the spectrum from RNs to individuals whose only training is provided by the practice. These individuals need proper supervision, focused training, and inclusion as equally respected team members in QI planning activities in the practice setting. Practice leaders need to be developed so as to better provide these steps. Research projects on how to properly develop and employ these individuals and develop practice leaders in QII should be a priority.
Although the literature is inconsistent on the issue of whether multiple interventions are more effective than single interventions, this project, plus complexity and microsystem science suggest that the same QI approach will not be effective in all settings and that a “menu” of approaches that can be locally adapted may be a preferred approach. However, in such designs, it may be difficult to assess whether a particular approach is more effective. PIs that adopt this approach need to develop/include methods to ascertain if there are more effective approaches among the “menu” and interactions among “menu” items. Study sections should look for these methods in applications. / A great number of primary care practices will participate in QII, particularly when they receive a tangible benefit (e.g., free practice reports, CME, support to attend meetings), believe that the project is in the best interests of their patients, andcan titrate their level of involvement to suit their particular needs/level of interest.
Methods to conduct analyses of practices with different levels of exposure to the intervention (time in project, uptake of different components) need to be developed and/or standard approaches used in project analyses.
In addition to overall QI improvements (outcome measures), studies need to look for the behavioral/organizational changes that take place as a result of different intervention strategies so approaches can be tailored efficiently to needs
Funding source / AHRQ (grant no. 1 U18 HS11132-01). / AHRQ (grant no. 5 U18 HS013716).
Was the evaluation plan subject to peer review and/or IRB approval? / IRB approval from Medical University of SC / IRB approval from Medical University of SC

[1]Translating Research into Practice (TRIP)-II, Manuscript summarizing project published in the Annals of Internal Medicine, 2004; 141(7):523-532.

[2] Accelerating the Translation of Research into Practice (A-TRIP).