Standardized Program Evaluation ProtocolPA Task Matrix

Steps for SPEP in a New Pilot County Juvenile Probation Department

Training

  • 2-day training on research and implementation of SPEP
  • Key Leader orientation and planning [A, B, D, I]
  • Community stakeholder orientation and information session(s) (Kickoff meeting) [C, E, F, G, H]

Identification of Services

  • Develop continuum of services used by probation dept.
  • Prioritize services/programs for going through SPEP process [J]
  • Coordinate with EPIS liaison
  • Prepare provider for SPEP process [K, L, U]

SPEPing of Services (ongoing)

  • Service categorization & quality of service interviews [N, O]
  • Duration, dosage, and risk data collection and analysis [M]
  • Develop and deliver feedback report, discuss improvement implications [P, Q, R]

Improvement of Services

  • Provider and probation collectively determine improvement priorities
  • Develop performance improvement plan [S]
  • Identify needed resources, obtain approvals, implement plan [T]
  • Quarterly contact for updates and technical assistance

Task: A / Develop a Rollout Communication Plan
The plan should be a guide for SPEP roll out, including the dissemination of written materials. (who, what, when, where, why and how)
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Meet with county juvenile probation personnel / x / Plan for 2 – 4 weeks after a commitment is made to participate in SPEP / Copies of documents off of the EPISCenter website related to SPEP
Sample Power Points from past key leader and kickoff presentations
Refer to SPEP Webinars
Provide a copy of Task Matrix
Brainstorm goals/ideas JPO
personnel have about process for rollout / x / x
Establish timeframe for kickoff meeting / x / x
Identify and schedule training for JPO and other staff to be involved in the SPEP process / x / x
Identify and schedule SPEP shadowing for JPO and other staff / x / x
Task: B / Key Leader Orientation (KLO) Meeting
This is the initial meeting of local stakeholders that explains what all involved (including key leadership) need to know and understand about the SPEP process.
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Contact point person in county juvenile probation office / x / Within 10 business days of initial communication plan meeting with juvenile probation office personnel / Meet with staff to review kickoff process, including KLO meeting
Discuss stakeholders; provide copies of sample agendas for KLO meetings, PowerPoint examples for KLO meetings, and other relevant documents*:
  • SPEP User's Guide
  • Lipsey’s research, SPEP Score Template,
  • SPEP Service Type Fact Sheets, Quality of Service Measures;
  • EPISCenter Website, Performance Improvement Life Cycle,
  • Key leader meeting evaluation form
/ *EPIS staff will develop standard packet
Schedule a date to meet at a time and location suitable to stakeholders / x / x
Establish meeting structure (agenda, PowerPoint, meeting objectives) / x / x
Identify which presenters will be at the KLO meeting / x / x
Task: C / Determine Desired Structure for Kickoff Meeting
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Decide upon format:
Two half day meetings (same information, different attendees)
or one larger meeting for all attendees with a break for lunch / x / x / Should occur prior to KLO meeting / Present agendas on both options from previously held kickoff meetings and discuss pros and cons to both
Identify invitees (KLO participants, juvenile court personnel, county human service personnel, service providers)
Task: D / Key Leader Orientation Meeting
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Local stakeholders will have a better understanding of the SPEP process; most notably the performance improvement component / x / x / 4-6 weeks following invitation / Agenda and copies of:
  • Lipsey’s research, SPEP Score Template, SPEP Service Type Fact Sheets,
  • Quality of Service Measures,
  • EPISCenter Website, Performance Improvement Life Cycle, PowerPoint w/notes pages)

Identify who Key SPEP Leadership will be (JPO SPEPper, Quality Assurance, Director, Deputy Chief JPO, Supervisors, JJSES Coordinator, etc.) / x
Participant questions have been answered and discussed. / x / x
Identify role of those involved in Kickoff Meeting (who will present, what will be presented, time frames) / x / x
Prioritization of providers selected / x
Task: E / Establish Date(s) and Location(s) for SPEP Kickoff Meeting
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials
Locate venue to accommodate number of invitees andpresenters / x / 4-6 weeks prior to meeting / Reserve space to accommodate participants
Task: F / Announce Kickoff Meeting
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Send invitation to all stakeholders: county juvenile court personnel, human service administrators (if deemed appropriate), service providers, etc. / x / x / 4-6 weeks prior to kickoff meeting / EPIS staff will provide assistance with agenda items and recommended participants
Establish RSVP date for invited participants
Task: G / Compile Agenda Items and Supporting Materials
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Generate list of handouts and compile master copies of all handouts / x / x / 2 weeks prior to meeting / SPEP process documents, Pre-SPEP checklist, background readings, etc.,
Sign in sheet/attendee list, and kickoff evaluation
Task: H / Kickoff Meeting To Disseminate Knowledge on the Process of SPEP
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Provide opportunity for participants to have questions answered / x / x / x / Within 90 days from initial communication meetingAgenda / Handouts (include PowerPoint w/notes pages)
Refreshments?
Task: I / Collect County Specific Data
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Obtain JCMS Data Reports from CJJT&RCheck on this with Becky Anderson / x / Prior to kickoff Meeting / Request the following from CJJT&R Help Desk Contact Person: (will differ based on county):
  • YLS by Disposition
  • YLS by Supervision Type
  • YLS by Placement
  • YLS by Out of Home Service??
  • SVCO data

Provides context within the JJSES as to why SPEP is valuable and creates snapshot of serious, violent, chronic offender (SVCO) data (Can be included in kickoff meeting if available/desired) / x
Task: J / Identify First Program for SPEP
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Focus on most utilized providers that have a strong relationship with Juvenile probation, and that can be considered an “easy” program (ie: low-hanging fruit) / x / 4 weeks prior to SPEP Interview / JPO administrators should speak with program administrators to determine if provider is willing and able to be SPEPped (does documentation and data necessary exist?)
Task: K / Establish Date(s) with Provider for SPEP
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Schedule SPEP interview (for residential services, several interviews may be necessary, depending on number of services identified. At least one Interview for “unpacking” services and several interviews for quality assessment); EPIS full time staff or consultant should also be available to assist to ensure consistency of process / x / x / Minimum of 2 weeks prior to meeting / Send out letter introducing SPEP, confirming meeting date and time as well as documentation required at interview (see confirmation letter template).
Include Pre-Visit Checklist
Task: L / Compile Documentation Materials and Identify and Confirm Staff to be Interviewed
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Upon receipt of confirmation letter, provider will compile required materials in preparation for SPEP interview(s); / x / x / To be completed by interview date / Refer to letter for list of items to have available for review during the interview; also review Pre-Visit Checklist
Identify staff to be present for SPEP interview(s) and discuss interview process with them / x / x / x
Task: M / Collect Cohort Data
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Youth in cohort should have received the service preferably no more than 6 months to 1 year prior to SPEP interview / x / x / Preferably prior to interview for community based services and following the “unpacking” of residential services (refer to Task N) / Data to be obtained internally through local county generated reports or requested from JCMS; service provider may also provide data
Youth Level of Service (YLS) results for youth in cohort, identifying the risk level as low, moderate, high or very high
Data should include the number of hours and weeks a youth received the service
Obtain names, start dates and end dates as well as the number of hours youth were receiving service preferably broken out by each day/session they attended *note: do not send identifiers (youth names) via email / x / x
Cohort should include 10 or more youth / x / x
Cohort includes youth no longer receiving the service / x / x
Task: N / Conduct Interview(s) with Providers
Part 1: Service Categorization (known as “unpacking” in a residential SPEP) &
Part 2: Quality of Service, which may require several meetings
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Obtain as much information as possible on the service: what it is, how it’s delivered, daily schedule; information on staff hiring, education and training requirements; how are staff are supervised and evaluated / x / x / x / Occurs during interview / Review of policies, procedures, protocols, manuals etc. provided by agency personnel
SPEP Interview Worksheet (SPEP Score sheet, SPEP Quality Measures, SPEP Fact Sheets), SPEP User’s Guide
Evaluation
Consider creating an agenda to guide the interview / x / x
Task: O / SPEP Team Debrief
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Consensus on service type and quality measures / x / x / Should be held as soon as possible following the SPEP interview (for community based services, preferably same day as interview) / Notes from interview; SPEP score sheet; quality measures checklist; program profile
(included in SPEP packet)
Task: P / Obtain SPEP Score
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Identify service within SPEP service categories; completion of PA Quality Measures document; calculate point total on quality measures, review data related to duration and contact hours; review risk level data of all youth in cohort; complete the SPEP score sheet, and include Program Optimization Percentage / x / x / Within 7 days post interview / SPEP Users Guide and other training materials; review of information gathered during interview; gather additional information (post interview) as necessary
*If service is identified as a Blueprint Model Program,
duration and dosage is calculated at 80% of developer’s recommendation for both
Task: Q / Prepare Initial Feedback Report(s)
(residential programs in particular may have multiple feedback reports and one comprehensive report that aligns with the performance improvement process outlined in the individual feedback reports).
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Prepare narrative report that encompasses information collected during interview, quality measures review, and data reviewed / x / x / Within 2 weeks post interview / Use feedback report template
SPEP score sheet; notes from interviews; review of documentation; review of PA quality measures checklist
Task: R / Discuss Feedback Report Findings and Begin Discussion on Performance Improvement Planning
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Schedule meeting to include SPEP interviewers, probation and provider to review Feedback Report, answer questions, and discuss performance improvement. / x / x / x / For community based services, within 30 days from date of SPEP interview / SPEP Users Guide; notes from interviews; SPEP score sheet and Feedback Report & supporting material.
SPEP Performance Improvement Plan & Performance Improvement Template.
Provide information and available reference materials, applicable for performance improvement.
x / x / x / For residential based services, the timeframe is approximately 4-6 months
Task: S / Development of Performance Improvement Plan
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Probation and provider collaborate on ways to improve services informed by Feedback Report findings. / x / x / For community based services, 1-30 days from the date of the feedback review meeting.
For residential based services, the timeframe will vary. / Performance Improvement Guide for Juvenile Justice Stakeholders to include Concepts of Performance Improvement, Performance Improvement Template, Continuous Quality Improvement (CQI- from Qii initiative) Guide, SPEP Service Type Fact Sheets, SMART Plan Goals, Questionnaire and Template
*The agency may use an existing performance improvement template or create their own. See Page 11 and Appendix A in the Performance Improvement Guide for assistance in incorporating all of the essential information.
Performance Improvement Plan goals are created. / x / x
Performance Improvement Plan is reviewed by all stakeholders (this might include program administrators, board members, juvenile court, etc.). / x / x / x
Plan includes timeframes to achieve each goal and a schedule for progress calls. / x / x / x
Task: T / Implementation of Performance Improvement Goals
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Service provider and juvenile probation staff work to implement the goals in the plan through the objectives identified. / x / x / Performance Improvement Plan should be implemented beginning 30 days through 12 months following the review of the feedback report. / Resources will vary based upon the recommendations in the Feedback Report and established goals for the Performance Improvement Plan.
Technical assistance will be provided as needed through the EPIS Center or others with expertise in continuous quality improvement.
Progress review conference calls or in person meetings will occur with probation, provider and the EPISCenter assigned staff. / x / x / x / Phone calls or meetings will occur on a quarterly basis and as necessary
Completion of Performance Improvement Goals / x / x / Substantial completion of the Performance Improvement Goals should occur within 12 months. Data can then be collected to measure improvements.
Task: U / Data Collection
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Preparing for SPEP re-assessment of service(s). / x / x / 12-24 months following implementation of performance improvement goals. Goals in Performance Improvement Plan completed or near completion. / Data collection mechanism to track new cohort (duration, dosage and risk level). Youth in the new cohort should no longer be receiving the improved service.
Task: V / Readiness for SPEP Re-assessment
Objective / EPIS / JPO / Provider / Targeted Timeframe / Resources/Materials / Status/Notes
Determine readiness for SPEP Re-assessment / x / x / x / 6 to 24 months- to allow a sufficient cohort of youth to receive the improved service(s).
Task: W / Refer to “K” and proceed with SPEP process
Last Revised 10.09.2014 / 1