Problem Solving/
Response to Intervention Project
2014-2015 Evaluation Report
Stockslager, K.M., Castillo, J.M., Brundage, A.H., and Batsche, G.M.
July 2015
This project is a collaboration between the University of South Florida and
the Florida Department of Education.
Table of Contents
Contact Information 3
Acknowledgements 4
Executive Summary 5
Introduction 7
Evaluation Design 9
Findings 10
Summary of Findings and Future Directions 26
Appendix A: Training Evaluation Survey 27
Appendix B: Self-Assessment of MTSS 29
Contact Information
July 2015
Project Director
George Batsche, EdD, NCSP
Project Leader
Clark Dorman, Ed.S
Project Evaluators
José Castillo, PhD, NCSP
Kevin Stockslager, PhD, NCSP
Amber Humm Brundage, PhD, NCSP
For specific staff contact information, go to http://floridarti.usf.edu/contact/contactinfo.html
For an electronic copy of this document, go to http://floridarti.usf.edu/resources/program_evaluation/index.html
Please contact Kevin Stockslager at for additional information on this report.
Acknowledgements
The Florida Problem Solving/Response to Intervention (PS/RtI) Project would like to thank all Project staff who were so instrumental in collecting the data reviewed in this report.
Another special thanks is extended to the following Project Graduate Research Assistants for their efforts in entering, managing, and analyzing the data presented:
Leah Bonilla
Mercedes Cambric
Meagan Cherry
Justine Connolly
Tara Delach
Olivia Gonzalez
Natalie Hoffman
Nycole Kauk
Kimberly Knapp
Kayla Lawler
Amira Mattison
Dalton Maynard
Stacey Mulrenin
Renee Ornduff
Camille Randle
Casey Schick
Joy Wang
Finally, the Project would like to thank Judi Hyde for her assistance formatting the charts, graphs, and other components of this report needed to clearly communicate findings to readers.
Executive Summary
This report presents information on the Florida Problem-Solving/Response to Intervention (PS/RtI) Project from the 2014-15 school year. The Florida PS/RtI Project was created by the Florida Department of Education in 2006 to help facilitate implementation of PS/RtI. The Project’s mission is to build the capacity of Florida’s schools and districts to implement an integrated multi-tiered system of supports[1] (MTSS). This report focuses on the activities that Project staff engaged in to assist districts in building capacity to implement MTSS as well as outcomes associated with those supports.
Findings
Findings are organized around the evaluation questions that the Project asked during the 2014-15 school year. The evaluation questions focused on several topic areas including the goals and needs of Florida schools and districts, the number and types of services provided by Project staff, the knowledge and skills of stakeholders, and implementation of MTSS components.
Project staff reported engaging in the dissemination of products and delivery of training and service delivery (e.g., ongoing support and technical assistance) during 2014-15. Project staff reported training 10,951 stakeholders, providing ongoing service delivery to 19,322 stakeholders, and disseminating 34,894 copies of products. Examples of training topics included the eight-step small group planning and problem-solving process, intensive interventions for struggling learners, integrating technology into instruction/intervention, and evaluation of MTSS. Service delivery foci included general MTSS implementation support, school/district improvement plans, implementation of MTSS in early childhood and secondary settings, and evaluation of MTSS. Finally, product examples included the Project website, technical assistance papers, professional development modules, and a school-level instrument designed to measure MTSS implementation.
Increases in stakeholders’ perceived knowledge and skills were evident. During each of the four quarters of the school year, over 80% of stakeholders indicated an increase in knowledge following training or service delivery provided by Project staff. While lower during the first three quarters (ranging from 69% to 71%), about 96% of stakeholders indicated increases in skills during quarter four.
Stakeholders were satisfied with the services provided by Project staff. During all four quarters, stakeholders indicated high levels of satisfaction with training and service delivery. The percentage of respondents reporting being “satisfied” or “very satisfied” with services provided by Project staff during the four quarters ranged from 78% to 96%.
School leadership teams reported engaging in activities to develop consensus, build infrastructure, and implement components of an MTSS model. Data were collected through the use of a school-level MTSS instrument, which focused on 6 domains that corresponded to the change model adapted by the Project: Leadership and Communication/Collaboration (consensus), Building Capacity/Infrastructure and Data/Evaluation (infrastructure), and Data-Based Problem Solving and Three-Tiered Instruction/Intervention (implementation). Items were rated on a 4-point scale ranging from 0 to 3 (0=Not Started, 1=Emerging/Developing, 2=Operationalizing, and 3=Optimizing). Average scores across all schools ranged between 1.64 and 2.47. These scores indicate that schools, on average, ranged from slightly below “Operationalizing” for certain components to in between “Operationalizing and “Optimizing” for higher rated components. On average, schools reported higher scores for the items addressing implementation (e.g., engaging in data-based problem-solving, operating within a three-tiered model of instruction and intervention) when compared to average scores for items that addressed consensus (e.g., school leadership team structures, communicating an collaborating with staff and stakeholders) and infrastructure (e.g., providing MTSS professional development, utilizing data systems).
Introduction
Florida’s Problem Solving/Response to Intervention (PS/RtI) Project
The Florida PS/RtI Project was created by the Florida Department of Education in 2006 to help facilitate implementation of PS/RtI. The Project’s mission is to build the capacity of Florida’s schools and districts to implement an integrated multi-tiered system of supports (MTSS[2]). In an effort to create and sustain a model for implementing an integrated system of supports, Project leadership and staff from the Florida PS/RtI Project and Florida’s Positive Behavior Support:MTSS Project have been engaged in collaborative efforts since 2011. The following vision was collaboratively constructed by the Florida PS/RtI and Florida's Positive Behavior Support:MTSS Projects.
“The collaborative vision of the Florida Problem-Solving/Response to Intervention (PS/RtI) Project and Florida’s Positive Behavior Support:MTSS Project is to:
· Enhance the capacity of all Florida school districts to successfully implement and sustain a multi-tiered system of student supports with fidelity in every school;
· Accelerate and maximize student academic and social-emotional outcomes through the application of data-based problem solving utilized by effective leadership at all levels of the educational system;
· Inform the development, implementation, and ongoing evaluation of an integrated, aligned, and sustainable system of service delivery that prepares all students for post-secondary education and/or successful employment within our global society.”
During the 2014-15 school year, Project staff worked toward building the capacity of districts through the five initiatives listed below:
· Statewide Professional Development and Support for MTSS Implementation. One of the main goals of the Project is to provide professional development and support to schools and districts implementing an MTSS. While most Project staff deliver some professional development and support, this initiative is primarily supported by three Regional Coordinators (RCs) and two Regional Facilitators (RFs). The RCs and RFs provide support to districts through training, technical assistance, and additional support. Common activities engaged in to support MTSS implementation include regional PLC calls and needs-based regional and district technical assistance and support.
· PreK-12 Alignment. This initiative focuses on aligning the implementation of data-based problem-solving within an MTSS across all levels of education. Project staff who support this initiative have been responsible for training and technical assistance efforts to increase implementation of intensive interventions for struggling students. Data-based problem-solving, the development and delivery of instruction/intervention, and the use of Early Warning Systems at the district and K-12 levels are important aspects of this initiative.
· Differentiated Accountability (DA) Support for MTSS. The DA Model was adopted by the FDOE to improve student performance in the State’s most struggling schools. Differentiated Accountability MTSS Specialists support the work of the State Differentiated Accountability (DA) teams. The focus of the MTSS Specialists’ work is to provide training, technical assistance, and evaluation assistance to ensure that data-based problem-solving within a multi-tiered system is a critical component of the school improvement efforts in targeted schools. MTSS Specialists provide internal professional development to their regional teams on MTSS and collaborate with other content (e.g., reading, math) specialists to address the performance of schools in need of improvement.
· Technology to Support Improved Outcomes for All Students. The Regional Technology Network team supports the use of technology to improve the outcomes of all students. The team maintains five regional technology support centers throughout the state and ensures that students with disabilities have the assistive technology and materials they need to support their learning and performance. Staff from the Technology team also collaborate with Project staff from other initiatives to support the integration of technology and universal learning design principles into statewide MTSS implementation efforts.
· Integrated MTSS Model for Academic and Behavior Supports. This initiative focused on developing and supporting the statewide implementation of an integrated MTSS model of academic and behavior supports in Florida. The primary goal was to build district capacity to implement an MTSS by systematically collaborating with Florida’s Positive Behavior Support:MTSS Project to support MTSS implementation.
Evaluation Design
Purpose and Design
The Project’s program evaluation design focuses on evaluating progress towards increasing the capacity of Florida districts to implement MTSS. Prior to the start of the 2014-15 school year, training was provided to Project staff regarding evaluation plan processes and procedures designed to address critical evaluation questions. The Project’s evaluation questions focus on several areas including stakeholder goals for professional development, activities that Project staff engage in, knowledge and skills of stakeholders, and implementation of MTSS components. Process (e.g., activity logs), survey data, self-report implementation measures, and interviews were used to answer the evaluation questions listed below.
Evaluation Questions
1. To what extent are Project staff providing training, service delivery, and products to stakeholders?
2. To what extent are the trainings, service delivery, and products being provided increasing the knowledge of stakeholders?
3. To what extent are the trainings, service delivery, and products being provided increasing the skills of stakeholders?
4. To what extent are stakeholders satisfied with the trainings, service delivery, and products being provided?
5. To what extent do trainings, service delivery, and products assist stakeholders to build consensus regarding MTSS implementation?
6. To what extent do trainings, service delivery, and products assist stakeholders to develop the infrastructure to implement an MTSS model with fidelity?
7. To what extent are trainings, service delivery, and products contributing to implementation of practices associated with an MTSS model?
8. What are the needs and characteristics of stakeholders supported by Project staff?
9. How do stakeholder goals and contextual and external factors relate to professional development and support provided by Project staff?
Findings
The Findings section is organized around the nine evaluation questions presented above. For each evaluation question, a description of the data used to answer the question is provided, followed by a summary display of the data (i.e., bulleted list of stakeholder responses, table display, graphical depiction of survey or implementation data) and a brief analysis or explanation of the data. Analyses and explanations of the data are primarily descriptive in this report. We encourage Project stakeholders to further examine the data and to engage in discussions regarding implications for the services they provide. See the Appendix for surveys that were used to collect information from stakeholders in an effort to answer the Project evaluation questions.
To what extent are Project staff providing training, service delivery, and products to stakeholders?
Description of Data. The table below depicts the number of stakeholders trained, the number of stakeholders who were provided support (service delivery), and the number of products disseminated by Project staff during the 2014-15 school year. For the purpose of reporting activities, trainings were defined as any activity in which structured professional development was provided (e.g., training at a conference, institute, or school/district professional development event). Service delivery was defined as any activity in which Project staff provided one-time or ongoing technical assistance to a school, district, or other stakeholder. Products were defined as any tangible resources created by Project staff and disseminated to stakeholders. Data were gathered from the Project Tracking System (PTS), a web-based system for reporting the delivery of training/service delivery and dissemination of products to stakeholders.
Training / Service Delivery / ProductProject Totals / 10,951 / 19,322 / 34,894
Explanation of Data. The vast majority of services provided by Project staff during the 2014-15 school year were categorized as products. Project staff members reported disseminating 34,894 copies of products during the year. Product examples included the Project website, technical assistance papers, professional development modules, and a school-level instrument designed to measure MTSS implementation. Project staff reported providing service delivery (e.g., ongoing support and technical assistance) to 19,322 stakeholders during the year. Example service delivery foci included general MTSS implementation support, school/district improvement plans, implementation of MTSS in early childhood and secondary settings, and evaluation of MTSS. In addition to the products and service delivery, staff reported delivering trainings to 10,951 stakeholders. Example training topics included the eight-step small group planning and problem-solving process, intensive interventions for struggling learners, integrating technology into instruction/intervention, and evaluation of MTSS.
To what extent are the trainings, service delivery, and products being provided increasing the knowledge of stakeholders?
Description of Data. The graph below depicts stakeholders’ self-reported increases in knowledge as a result of training or service delivery provided by Project staff. These data were collected from stakeholders through the use of training or technical assistance surveys (See Appendix A: Training Evaluation Survey) following the delivery of services. Responses are disaggregated according to which quarter the services were delivered. The scale for this item indicates that a rating of 1=Not at all and a rating of 6=Greatly.
*Note. Quarter 1=August 2014-October 2014, Quarter 2=November 2014-January 2015, Quarter 3=February 2015-April 2015, Quarter 4=May 2015-July 2015.
Explanation of Data. Overall, stakeholders attending training and service delivery sessions conducted by Project staff reported increases in knowledge (as indicated by a survey item response of 4, 5, or 6). During all four quarters, at least 80% of stakeholders indicated an increase in skills following support provided by Project staff. Additionally, the percentage of stakeholders that indicated increases in knowledge increased from Quarter 1 (82%) to Quarter 4 (95%). (Note. Total responses for Q1=176, Q2=984, Q3=409, Q4=102).