Evaluation Framework Worksheet:
< SOS Pentiction/summerland residential care program >
Prepared by: Oluseyi oyedele Approved By: SOS Divsion Representative & andrew neuner
Evaluation Working Group Members: Terrie crawford, cindy regier, lynda casey, arlene herman, oluseyi oyedele
Last Date Framework was Updated: 9th November 2012 Target Date for Evaluation: May 2013 Evaluation Budget (if applicable): n/a
- Program Overview -
1. Project/program PurposeWhat is the purpose of the project/program?
[List project/program goals/objectives here if applicable]
2. Target Population
Who is/are the target population(s) of this Project/Program? / To develop a comprehensive model of physician support and care to complex residents in residential care. The service delivery model proposes delivery of quality medical care in residential care for Interior Health Authority owned and contracted residential care facilities in the Penticton and Summerland health service areas.
The service delivery model is designed to meet the following specific outcomes:
· To provide proactive quality medical care for individuals in residential care setting in 7 facilities in Penticton and Summerland
· To establish longitudinal and relational attachment of family physician to patient in this facilities
· To ensure provision of the right care at the right time in the right setting for the clients served
The service delivery model covers the Interior Health owned and operated contracted residential facilities in the Summerland and Penticton health service areas
· Penticton has 5 Facilities with 464 beds & approximately 2.3 admissions per week (120 per year)
· Summerland has 2 facilities with 155 beds & approximately 1.2 admissions per week (62 per year)
· Total of 619 beds in 7 facilities
The Service Delivery Model
The proposed model of care will support longitudinal, relational care by South Okanagan primary care physicians for individuals living in seven residential care facilities in Summerland and Penticton communities.
A two phased approach. Was recommended:
Phase 1: This will be a three pronged approach in which the most responsible physician (MRP) is the key to providing the continuity of care but will be supported by the Doctor of the Day Program for urgent care when required after hours. This approach allows family physicians the ability to provide comprehensive/high quality care to patients in residential care. It also includes a Division Residential Care Medical Coordinator role that will be established to work on developing linkages and network with local systems and processes as well as to identify local gaps in care and services, leading practices and quality improvement opportunities.
Phase 2: The second phase will focus on expanding either the Residential Care Medical Coordinator Position or consider a Health Authority Regional Role if the program is to be expanded throughout the Health Authority. The focus would be to promote best practice and standards, implement clinical quality improvement initiatives that will improve the quality of care for all residents as well as promote educational and continuing professional development opportunities. Additional funding would be required.
· All current residents of the participating 7 residential care homes
- Context & Situational Analysis -
3. Potential Influences on the Program / EvaluationAre there any contextual factors or risks that might influence the program and/or evaluation? / · The success of the initiative is highly dependent on the active participation of family physicians and residential care staff in the residential and urgent care aspects of the program
· With 7 facilities participating in the program, the capacity for the program coordinator to get timely data to monitor progress is very crucial
· The capacity to measure some of the key indicators will rely on the capacity of the evaluation working group’s ability to access current data
· Any change in bed capacity not aligned with the current program compensation structure
· Changes in bed capacity between communities which may result in transition of residents between communities
· Change in residential home management
- Evaluation Stakeholders -
4. KEY EVALUATION STAKEHOLDERSWho will use the evaluation? How will they use it?
Stakeholder
(or Stakeholder Group) / Interest / Perspective / Represented By / Role (in the Evaluation) / How and When to Engage
SOS Division of Family Practice
SOS CSC
IHA Representatives
MOH Representative / Decision making / Terrie/Dr Mack
Cindy
Dr Vanketamaran
Claire Ann Brodie
MOH Rep / Endorse/oversee evaluation; Review findings as appropriate / Obtain approval for evaluation before finalizing evaluation planning
Ongoing updates as appropriate
SOS Residential Program Evaluation Working Group / Plan and implement the evaluation and produce reports / Terrie Crawford
Arlene Hernam
Cindy Reiger
Lynda Casey
Seyi Oyedele / Act as members of the Evaluation Working Group: They will inform evaluation planning and activities; Participate in data collection and analysis, Write and Share findings as appropriate / Ongoing involvement in Evaluation Working Group that meets regularly to plan and implement evaluation, review findings e.t.c
BC Division / As appropriate
Additional Notes:
- Evaluation Focus & Key Questions -
5. EVALUTION PURPOSEWhat are the main reasons for the Evaluation?
6. Focus of the Evaluation
What will we evaluate (which project/program or aspect/domain of a project/program)?
7. Questions
What do you specifically want to know? / · To respond to MOH funding expectations regarding project outcomes
· To provide information on project progress to CSC, Division& Residential facilities to inform decision making for quality improvement and expansion
· To document program learning’s for continuous quality improvement throughout IH and Division
· To demonstrate program impact
Accessibility
· Number of GP’s providing MRP care at Residential Care facilities at start of program& bi-yearly
· Number of GP’s providing urgent care to residential care facilities at start of program & bi-yearly
Appropriateness
· Reduced number of residential care visits to ER
Coordination
· Number of residents with current documented completed advance care plan
· Number of residential care facilities with an active medication reconciliation
Patient and Provider satisfaction/experience
· Improved Physician satisfaction
· Improved residential facility manager/staff satisfaction
· Improved resident / family satisfaction
What were the lessons learned?
- What worked?
- What didn’t work?
- What needs to be improved?
· To what extent has the program been implemented as planned?
· What impact has the program had on acute care (hospitals)?
· What impact has the initiative had on client/care giver and provider experience?
· What were the lessons learned?
- Data Collection Plan -
(7.) QuestionsWhat do you want to know? / 8. Evidence / Indicators
How will we know it? / 9. Data collection
Sources
Who will have this information?
What will our information sources be? / Methods
How will we gather the information?
What tools shall we use? / Timing
When should we collect data? / Responsibility
Who will collect the data?
Accessibility
To what extent has the program been implemented as planned? / · # of GP’s providing MRP care at Residential Care facilities at start of program& bi-yearly
· # of GP’s providing urgent care to residential care facilities at start of program & bi-yearly / Project documents and Project staff
Sites Self reported - / Document review
Bi-annual MRP payment
Site report / Ongoing / Arlene & Cindy
Appropriateness
What impact has the program had on acute care (hospitals)? / · Reduced number of ER visits by residents / Meditech
ER Physicians / IMIT
Physicians report / ongoing / Rita Zaffino
Patient and Provider Experience
What impact has the initiative had on client/care giver and provider experience? / · Improved Physician experience
· Improved residential facility manager/staff experience
· Improved resident / family experience / Surveys (Seyi to get tool for family survey) / Physician experience survey
Provider experience Survey
Client/family member experience survey / Annually / Arlene & EWG
Care Coordination / · Medication Review
· Number of residents with current documented completed advance care plan
· / Audit
Chart Audit / Audit tracking tool / Annually
(Baseline data for 2012) / Site Staff
What were the lessons learned?
- What worked?
- What didn’t work?
What needs to be improved? / Emerging themes / Various sources
Surveys
Document review and capturing process issues using meeting notes etc / Ongoing / EWG
- Plan for Data Analysis & Interpretation -
[and who will be responsible for this?] / 11. How will the data be interpreted?
[and who will be responsible for this?]
Quantitative Data:
· # and type of clients
· # of GP’s providing MRP care at Residential Care facilities at start of program& bi-yearly
· # of GP’s providing urgent care to residential care facilities at start of program & bi-yearly
· Reduced number of ER visits by residents
· Reduced number of hospital admissions
Qualitative Data:
· Description of program implementation – summarized for key themes
· Summary of key themes emerging from client and care giver survey
· Summary of key themes emerging from provider surveys, focus groups or interviews
Other Notes: / The Evaluation working group members will collaborate to interpret data, summarize key findings and make appropriate recommendations.
Recommendations will be made based on key findings from project data and analyses.
- Communication & Use of Evaluation Findings -
To Whom? (audience) / What? (information) / How / When / Where? (format) / For what purpose? (use)
· SOS Division of Family Practice
· IHA Leadership
· SOS CSC
· MOH
· BCMA
· Other IH programs
· Clients and other external partners / · Key findings and recommendations
· Key findings and recommendations
· Key findings and recommendations
· Progress reports
· Key findings and recommendations
· Key findings and recommendations
· Key findings and recommendations / · Regular meeting updates; final evaluation report and/or presentation
· Regular meeting updates; final evaluation report and/or presentation
· Regular meeting updates; final evaluation report and/or presentation
· Ongoing Reporting requirements
· Final evaluation report and/or presentation
· Final evaluation report and/or presentation
· Final evaluation report and/or presentation / · Oversight, feedback, decision-making for future planning
· Oversight, feedback, decision-making for future planning
· Oversight, feedback, decision-making for future planning
· Oversight, feedback
· For information, feedback
· For information, feedback
· For information, feedback
- ETHICAL CONSIDERATIONS -
13. Ethical Risks Identified by ARECCI Screening Tool· / Plans to mitigate identified ethical risks
·
14. Key Documentation to review in preparation for this evaluation?
· Service Delivery Model proposal
· SOS_Measureables document
· Email correspondences from Terrie Crawford (Executive Lead SOS Division of Family Practice)
15. Evaluation Workplan: Evaluation Schedule:
Week / Dates / Stage / Activities1 / 20th Sept 2012 / Pre-planning / First Meeting of Evaluation Working Group
2 / 19th Oct 2012 / Planning / Development of Evaluation Framework
3 / 9th Nov 2012 / Planning / Development of Evaluation Framework – Focus on Indicators
4 / 3rd Dec 2012 / Planning / Development of Evaluation Framework - Focus on Measurements – Methods and data collection
5
6
7…
16. Next Steps to support this evaluation (as of ):
An Example of Typical Steps to Conducting Evaluations –(UW Extension Program) :
Program Logic Model
*worksheet modified from the UW Extension Program Page 3 Version Date: Nov 9th 2012