IHI Evaluation Workbook
L16: Rapid-cycle Evaluation for Improvement Leaders
Developed by G. Parry, S. Cohen, and A. Reid
1) Basic InformationProject Name
Start Date
End Date
2) Project Overview
Please provide a brief description (3-5 sentences) of the project (elevator speech). Include the purpose of the project/program, your organization’s role in it, timeline, and reach (i.e., number of teams, cohorts, practices).
3) What are you trying to accomplish?a. Describe the high-level improvement goal for your project (how much by when).
b. Overall, do you believe your project is at the innovation, testing or spread phase?
4) Change Concepts: What are the primary change concepts (e.g. clinical, behavioral, procedural, cultural) included in this project that will lead to improved outcomes?a. Describe the change concepts (i.e., what changes will you encourage improvement teams to adopt and adapt?). Please describe how the change concepts will impact long-term outcomes. If there is a working driver diagram available, please include it as an appendix.
b. What is your degree of belief that the change concepts described above can be amended by participating sites to work in their setting? (Low, Medium, High)
5) Execution Theory: How do you predict that participation in the improvement initiative and learning acquired will lead to desired behavior or process changes?a. Outline key activities planned for this initiative (e.g. coaching calls, face to face meetings, improvement methods training, milestone calls, etc.) and connect them to predicted results. If there is a logic model, please include it as an appendix.
b. What is the rationale for choosing this execution method?
6) Measurement: How will you know that a change is an improvement?a. What goals are you considering and what measures can be used to evaluate your progress? At which Kirkpatrick levels are these goals? (See Appendix A)
Kirkpatrick Level / Goals / Measures1. Experience
2. Learning
3. Process /
Behavior
4. Outcomes
7) Rapid-Cycle Learning: How will you use data to update your approach?
a. What quantitative data can you collect that will indicate whether your project is producing the intended results and estimate the ultimate impact of this project?
b. What qualitative data can you collect to better understand how and why the project is experiencing certain results in the applied context(s)? How can these data be used to update your program theory? What questions might you ask various individuals or partner organizations in this project?
c. Summarize your overall plan to apply rapid-cycle, formative evaluation over the life of your project. For instance, how often and by what means will you:
· Convene key participants and collaborators to review results, reflect on your program theory, and strategically prioritize next steps?
· Feedback results, learning, and course corrections to the participant(s)? Broader organization(s)?
· Disseminate summative findings and revised program theory to the field (Appendix D includes a guide to assist your dissemination plan)?
Page 4
IHI Evaluation Workbook
L16: Rapid-cycle Evaluation for Improvement Leaders
Developed by G. Parry, S. Cohen, and A. Reid
Page 4
IHI Evaluation Workbook
L16: Rapid-cycle Evaluation for Improvement Leaders
Developed by G. Parry, S. Cohen, and A. Reid
Page 4
IHI Evaluation Workbook
L16: Rapid-cycle Evaluation for Improvement Leaders
Developed by G. Parry, S. Cohen, and A. Reid
APPENDIX B. Driver Diagram
APPENDIX C. Logic Model
Page 4
IHI Evaluation Workbook
L16: Rapid-cycle Evaluation for Improvement Leaders
Developed by G. Parry, S. Cohen, and A. Reid
APPENDIX D. Dissemination GuideGuiding Questions:
· What is the message of this project?
· Who needs to know the impact and findings of this work (internally, externally)?
· What might be the most effective, feasible mechanism(s) for sharing our experience and findings with the right people and at the right scale?
Target Audience: Who are you trying to reach with the results of this project? How large of audience do you intend to reach? Does this information need to be delivered to an internal audience, an external audience, or both?
Resources: What time, personnel, and budget can be devoted to creating and executing a dissemination plan?
Timeline: When should this information be shared? What checkpoints, if any, exist to meet this deadline?
Degree of Belief: What is the level of confidence regarding the integrity of these results? Are the results unbiased? Do they report meaningful data?
Outlets Selected: Describe what potential or planned mechanisms to use for dissemination (e.g. peer-review publications, non-reviewed publications, scientific conferences, marketing, new programming).
Page 4