Executive Summary
Timothy J. Runge, Co-Principal Investigator
Mark J. Staszkiewicz, Co-Principal Investigator
Kevin H. O'Donnell, Research Assistant
Indiana University of Pennsylvania
January 2012
1 / Error! No text of specified style in document. | Hewlett-Packard
Acknowledgements
The following agencies and organizations are acknowledged for their collaboration with the authors on this project: Pennsylvania Department of Education (PDE); Pennsylvania Bureau of Special Education (BSE); Pennsylvania Training and Technical Assistance Network (PaTTAN); Pennsylvania Positive Behavior Support Network (PAPBS Network); Office of Special Education Programs (OSEP) Technical Assistance Center on Positive Behavioral Interventions and Supports (PBIS); Educational and Community Supports at the University of Oregon; the Educational and School Psychology Department at Indiana University of Pennsylvania (IUP); the IUP Research Institute, and the IUP School of Graduate Studies and Research.
Specific recognition is extended to Dr. James Palmiero, Director of PaTTAN Pittsburgh; Ronald Sudano, Statewide SWPBIS Coordinator; Dr. Tina Lawson, Eastern Regional SWPBIS Coordinator; Lisa Brunschwyler, PaTTAN Consultant; and Teresa Stoudt, Central Regional SWPBIS Coordinator. We acknowledge past and present IUP Educational and School Psychology Research Assistants Rebecca Tagg, Aleksey Aleskeev, Melissa Gilroy, Cong Xu, Kevin O'Donnell, and Stephen McFall. Thanks are also offered to Celeste Dickey and the staff of the University of Oregon's Educational and Community Supports.
Most importantly, our deepest admiration is bestowed on PAPBS Network schools, supportive local communities, and collaborating mental health agencies that work tirelessly to develop, implement, and improve the SWPBIS framework in their educational buildings.
Data analysis and summation of results which formed the basis of this Executive Summary were supported in part by a contract from the PaTTAN / Intermediate Unit 1, PDE, and BSE. Opinions expressed within are solely those of the authors and do not necessarily reflect the position of the funding agencies or the Pennsylvania Department of Education, and such endorsements should not be inferred.
Preface
The purpose of this Executive Summary is to present outcome data related to the implementation of School-Wide Positive Behavioral Interventions and Supports (SWPBIS) in schools that are members of the Pennsylvania Positive Behavior Support Network (PAPBS Network). The general framework for the 2010-2011 evaluation of SWPBIS in PAPBS Network schools is based on recommendations of leaders in the field of large-scale SWPBIS evaluations (i.e., Algozinne et al., 2010).
Methodology
Data for this program evaluation come from a combination of direct and indirect sources.
When publicly available, data were collected independently by the researchers. Confidential data (e.g., referrals to special education, office discipline referrals) were voluntarily submitted by participating PAPBS Network schools. Consistent with the IUP Institutional Review Board approval (Log No. 08-251) approving this research, schools were assured that these sensitive data would be aggregated and publicly reported in a manner that insured their anonymity. In no situation will a school be identified individually or as a group with specific outcome results.
Analyses of outcomes were performed using inferential statistics when complete longitudinal data were available or in cases where cross sectional analyses were appropriate. The interpretation of statistically significant findings is that there is a high probability that the changes over time are real changes, not a statistical artifact due to random sampling error. Subsequently, the conclusion is that SWPBIS is related to the observed changes over time.
Caveats
Readers should consider results and interpretations contained within this Executive Summary with some caution due to the following limitations.
· SWPBIS was implemented in a small number of PAPBS Network Schools beginning in fall 2007. A considerable number of schools were trained in 2009-2010 with implementation occurring thereafter. As such, data are aggregated into two distinct cohorts given the different contexts in which SWPBIS was implemented across cohorts (R. Horner, personal communication, December 9, 2011). Throughout this report, the 33 schools that began implementation in fall 2007 are referred to as "cohort 1" and the schools that began implementing as early as 2009-2010 are referred to as "cohort 2."
· Given that significant changes in outcomes related to school reform efforts such as SWPBIS are not expected for a few years (Curtis, Castillo, & Cohen, 2008; McGlinchey & Goodman, 2008), present results should reveal changes in outcomes within cohort 1 schools but not cohort 2 schools.
· Absence of complete longitudinal data for certain outcome variables limited the extent to which pre- and post-SWPBIS implementation changes could be analyzed. A longitudinal methodology is preferred as it provides more confidence that the observed changes in the data are associated with SWPBIS implementation and therefore generalizable to other schools. Cross sectional approaches were used when robust longitudinal data were unavailable from schools. Within cross sectional designs, the number of schools for which data are reported will change from one year to the next as a consequence of which and how many schools submitted data. Cross sectional analyses require a much more cautious and conservative interpretation of SWPBIS effects on outcome variables.
· An inherent bias may be present in these results as a product of a school's willingness to voluntarily share its data with the researchers. Readers should, therefore, be mindful that most schools complying with data submission requests were schools that were successfully implementing SWPBIS.
· The method by which the implementation status of SWPBIS was measured presents a threat to the validity of findings. From fall 2007 to fall 2008, fidelity of SWPBIS implementation was documented via the Team Implementation Checklist (TIC; Sugai, Horner, & Lewis-Palmer, 2002, 2009). Many schools began using the more psychometrically sound Benchmarks of Quality (BoQ; Kincaid, Childs, & George, 2005) in spring 2009. In spring and fall 2010 many schools were independently audited using the Schoolwide Evaluation Tool (SET; Sugai, Lewis-Palmer, Todd, & Horner, 2005). When multiple sources of fidelity data were available for a school during the same general period of time, a greater reliance was placed on SET and BoQ data over TIC data.
· This evaluation utilized an ex post facto design in which schools were not randomly assigned to treatment and control groups. It is possible that PAPBS Network schools implementing SWPBIS are different from the typical, non-SWPBIS school given any number of reasons. In the absence of a true experimental design, with random assignment of SWPBIS and non-SWPBIS schools, cause and effect relationships cannot be concluded.
· Release of this Executive Summary is at the discretion of PDE, BSE, and PaTTAN. Analyses and interpretations contained within are the expressed opinion of the authors and do not necessarily reflect the views of the sponsoring agencies. Additionally, the authors are not responsible for any misrepresentations of these results.
Introduction
School-Wide Positive Behavioral Interventions and Supports (SWPBIS) is a three-tiered system that "establish[es] the social culture and individualized behavior supports needed for a school to be a safe and effective learning environment for all students" (Sugai & Horner, 2009, p. 309). All students and staff in a building are exposed to the Tier 1, or universal school-wide, practices which are intended to prevent problematic and disruptive behavior from occurring. These school-wide practices include careful consideration of all school environments to increase adult supervision and minimize inappropriate behavior; systematic and explicit instruction of behavioral rules and expectations in all school settings; reinforcement of desirable behavior through a token economy system and educative verbal recognition; and frequent review of multiple data to evaluate efficacy of these school-wide practices. Considerable empirical evidence (Bradshaw, Mitchell, & Leaf, 2010; Luiselli, Putnam, Handler, & Feinberg, 2005; Muscott, Mann, & LeBrun, 2008; Spaulding et al., 2010; Sugai & Horner, 1999) documents that for elementary school populations, over 80% of the student population responds well to this universal, school-wide level of prevention. That is, these students receive one or no disciplinary referrals for inappropriate behavior in an entire school year. The percentages of students for which this is the case in middle and high schools drops to approximately 73% and 67%, respectively. Despite this relative decline in the upper grades, school-wide prevention efforts work for large majorities of students.
Approximately 15-30% of all students (depending on elementary or secondary grade status) do not respond favorably to school-wide universal prevention efforts as evidenced by cumulative office discipline referrals of two to five per academic year (Spaulding et al., 2010). These students require supplemental behavioral intervention and supports in addition to the school-wide prevention techniques (Crone, Horner, & Hawken, 2004; Lewis & Sugai, 1999; March & Horner, 2002). Tier 2, or strategic, interventions typically include small group counseling or therapy, implementation of commercially-available standard protocol interventions, or interventions tailored from brief functional behavioral assessments (Walker et al., 1996). The goal of these strategic interventions is to provide students with academic, behavior, social, and emotional skills to minimize the barriers they face, thus augmenting the effectiveness of tier 1 school-wide techniques.
Even with high fidelity tier 1 school-wide behavioral supports and strategic tier 2 interventions fully in place, a small percentage of students still fail to respond appropriately. That is, these students exhibit chronic externalizing behaviors in schools and typically receive six or more office discipline referrals in an academic year. For these students, highly individualized and intensive supports are needed in conjunction with the tier 1 and 2 supports. This tertiary level of intervention is student-centered and family-oriented in that supports are implemented not only for the student, but also for the family given that there are often significant needs that extend across all the student's ecologies. Positive behavior support plans (PBSP) and intensive wrap-around services are typically implemented across multiple life domains (Eber, Sugai, Smith, & Scott, 2002). Research suggests that 3-8% of students require this level of support, with higher percentages occurring in secondary grades (Spaulding et al., 2010).
Purpose of PA SWPBIS Evaluation
PDE, BSE, and PaTTAN selected an initial cohort of 33 schools to implement a SWPBIS framework beginning in fall 2007. Participating schools received training, onsite and on-going technical assistance, and other resources from PaTTAN in exchange for their long-term commitment to this project and willingness to submit data on key outcome variables. A second group of schools was trained by PaTTAN, Intermediate Units (IU), and PAPBS Network Facilitators beginning in fall 2009. Some of these second cohort schools began implementing in 2009-2010, although most did not commence implementation until 2010-2011.
No Child Left Behind (NCLB; 2002) and Individuals with Disabilities Education Act (IDEA; 2004) firmly place the onus on educational systems to document the effects of practices implemented in public schools - and SWPBIS receives no special exemption from this requirement. Annual reports of the PAPBS Network SWPBIS efforts were synthesized for two consecutive years (Runge & Staszkiewicz, 2009, 2010). An executive summary report (Runge & Staszkiewicz, 2011) was released to the public in January 2011 via the PAPBS Network website www.papbs.org. The present Executive Summary highlights results from the 2006-2011 evaluation report (Runge, Staszkiewicz, & O'Donnell, 2011).
Framework of PAPBS Network SWPBIS Evaluation
The general framework offered by technical assistants at the OSEP PBIS Network (Algozzine et al., 2010) served as the organizational structure for the present Executive Summary. This structure is based on five broad domains: Context; Input; Fidelity; Impact; and Replication, Sustainability, and Improvement. The data reported within are very similar to data presented in previous reports; however, the organization of this program evaluation is different. Each domain is summarized below and has a corresponding section within this Executive Summary:
1. Context - explicitly stated goals of SWPBIS implementation; documentation of what training and support were provided for implementation; documentation of who provided this training and support to schools; which school staff attended the team training; and which schools received support to implement
2. Input - documentation of professional development content and activities; participants' satisfaction with team training and on-going support; and the depth and breadth of technical support provided to participating schools
3. Fidelity - the quality with which SWPBIS framework was implemented as prescribed
4. Impact - effect of SWPBIS on outcomes including office discipline referrals (ODRs), out-of-school suspensions, in-school suspensions, instructional time regained, school safety, school organizational health, staff retention rates, and academic achievement
5. Replication, Sustainability, and Improvement - the sustainability of implementation in schools; the capacity to replicate, or scale-up, SWPBIS in other schools and districts; and documentation of economic, political, and legislative efforts to establish SWPBIS as a fundamentally core mechanism by which schools operate
58 / 2010-2011 PAPBS Network SWPBIS Executive SummaryContext of PA SWPBIS
Stated Goals
Implementation of SWPBIS in Pennsylvania is under the leadership of a diverse set of stakeholders including PDE, BSE, PaTTAN, Pennsylvania Governor's Commission on Children and Families, private providers, Pennsylvania Departments of Health and Public Welfare, advocacy groups, and higher education. As indicated on the PAPBS Network (n.d.) website (http://www.papbs.org), the goals are:
· Develop and implement a school-wide cross-system approach for supporting the academic and emotional well-being of all students using research-based positive behavioral supports and strategies of varying intensity: 1) universal or preventative strategies for the benefit of all students; 2) secondary strategies for those who will achieve with enhanced supports; and 3) tertiary or intensive services for those who will achieve with intensive and coordinated supports.
· Achieve sustainability by seeking funding and legislative support for demonstration models, providing training and technical assistance, and encouraging the facilitation of collaborative partnerships among schools, families, youth and agencies.
· Foster a consistent application of best practice standards among schools, families and agencies.
· Promote shared values that are consistently demonstrated through practice and partnerships of schools, agencies and families.
· Develop and embed opportunities for collaboration between systems partners and families.
· Establish a dialogue that will inform ongoing training needs.
· Reduce fragmentation of training resources.