Where Have We Been, and What Have We Done?
Timothy J. Runge, PhD, NCSP Indiana University of Pennsylvania May 2017 Implementers’ Forum
Note: Opinions expressed within are solely those of the authors and do not necessarily reflect the position of the funding agencies or the Pennsylvania Department of Education, and such endorsements should not be inferred.
1
SWPBIS
Response to Intervention
MTSS Is Integrated Continuum
Evaluation Framework
Algozzine et al. (2010) described five domains of large‐scale SWPBIS evaluation
1. 2. 3. 4.
Context Input Fidelity Sustainability, Replication, Improvement 5. Impact
2
PAPBS History: Where Have We Been?
Context of PAPBS Network’s SWPBIS
3
PAPBS Network Membership • Cohort 1 in summer 2007 through winter 2008 – 33 schools (1 withdrew) • At least 570 more since fall 2009 – Cohort 2 – Not a true cohort, in traditional sense • Cannot independently verify precise training dates or baseline years
• As of spring 2016, at least 604 PAPBS Network sites • At least 183 collaborating community mental health agencies (increase from 165 last year)
Participating Buildings / LEA or Charter School / IUs by Region and Grade Level
4
Geographic Location of Participating PAPBS Network Schools – Combined Cohorts
Note. Blue circles represent cohort 1 LEAs; red circles represent cohort 2 LEAs
Input of PAPBS Network’s SWPBIS
5
Written Commitments to Participate From
Superintendent
Building Principals
Building‐level Teams
Network Facilitator
6
Funding Support for SWPBIS • Majority – IDEA Part B (School‐age) and Part C (Early Childhood) • Personnel and resources from CoP on SBBH Co‐Directors • LEAs • Federal grants – Safe Schools / Healthy Students – School Climate Transformation • SBBH Grants totaling $361,965 + $40,000 in IF vouchers
2015‐2016 SBBH Grantees
• Culturally‐Responsive PBIS 1. 2. 3. 4. 5. 6. 7.
Abington School District Harrisburg School District Pittsburgh Public Schools Propel Schools Saucon Valley School District Shaler Area School District Steel Valley School District
$70,000
• Model Sites 1. 2. 3. 4. 5. 6.
Bellefonte Area High School‐ Bellefonte Area School District Upper Grandview Elementary School‐ Highlands School District Juniata Valley Elementary School‐Juniata Valley School District Laurel Elementary & Junior/Senior High Schools‐ Laurel School District Lehigh Career & Technical Institute Donald Eichhorn Middle School‐ Lewisburg School District
$7,500
7
2015‐2016 SBBH Grantees • Establish SWPBIS
• Expand SWPBIS
1. 2. 3. 4. 5. 6. 7. 8.
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.
BLaST Intermediate Unit #17 Butler Area School District Galeton Area School District Hatboro‐Harsham School District Homer Center School District Keystone Oaks School District Lackawanna Trail School District West Chester Area School District
$79,965
Abington School District Bald Eagle Area School District Chartiers Valley School District East Stroudsburg School District Highlands School District Lehigh Career & Technical Institute Moniteau School District Norwin School District Perkiomen Valley School District Pocono Mountain School District Scranton School District Williamsport School District
$110,000
2015‐2016 Tier 3 SBBH Grantees Project RENEW 1. 2. 3. 4. 5. 6. 7. 8. 9.
Altoona Area School District Bangor Area School District Clairton City School District CLASS Academy East Stroudsburg School District Freedom Area School District Greater Latrobe Area School District Highlands School District Lincoln Intermediate Unit #12
10. McGuffey School District 11. Mount Lebanon School District 12. Northeastern Intermediate Unit #19 13. PENNCREST School District 14. Pittsburgh Public Schools 15. Schuylkill Have School District 16. Seneca Highlands Intermediate Unit #9 17. Tyrone Area School District 18. West Greene School District 19. Williamsport Area School District
$94,500
8
Training and Technical Assistance • Cadre of 127 Independent Network Facilitators (increase from 113 the previous year) • Rigorous application and training process while in provisional status
Attendance Figures at the Annual Pennsylvania PBIS Implementers’ Forums 2011‐2016
9
Implementers’ Forum Attendees by Role
2016 Implementers’ Forum Attendees’ Satisfaction
10
Fidelity of PAPBS Network’s SWPBIS
Primary Tier SWPBIS Implementation Status
11
How Many Schools Are Implementing by Grade Span in Spring 2016?
Summary of Fidelity Data • Since 2007, there has been an increase in the number of: – PAPBS Network schools – Schools implementing with integrity
• Full implementation of primary‐tiered SWPBIS in 217 schools • 6 schools indicate sustained implementation since training 9 years ago
12
Replication, Sustainability, and Improvement of PAPBS Network’s SWPBIS
Sustained Implementation is Possible!
Note. These data reflect any school that ever submitted primary‐tiered SWPBIS fidelity data since 2007‐2008. Many schools can achieve full implementation within 1 year of initial training; however, a small percentage require 2‐4 years to fully implement SWPBIS.
13
PAPBS History: What Have We Done?
Impact of PAPBS Network’s SWPBIS: Attitudes / Beliefs
14
Staff Perceptions Regarding Level of PBIS Implementation • Perceived PBIS Implementation measured by EBS: SAS (Sugai et al., 2003) • Participants select one of three choices: • full (fully implemented) • partial (partially implemented) • none (not implemented) • May not be an objective measure of SWPBIS
2015‐16 EBS: SAS Measures of Staff Perceptions of Primary‐Tiered SWPBIS Implementation
]
15
Staff Perceptions Regarding Level School Safety • School Safety Survey (Sprague et al., 2005) – Staff survey of perceptions of Risk and Protective Factors associated with school violence and safety • Risk = drug / gang activity; vandalism; truancy; community poverty and crime; child abuse • Protective = extracurricular opportunities; parental involvement in school; school‐community collaboration; acceptance of diversity; high expectations for all students
2015‐16 Descriptive Statistics for Protective and Risk Factors N = 501
Protective Factors (P)
M 73.35
Risk Factors (R)
Ratio of P / R
Difference (P‐R)
SD
M
SD
M
SD
M
SD
9.37
42.47
13.25
1.98
0.94
30.88
20.85
• SWPBIS schools have more positive sense of their protections against school violence • Protective Factors M = 73.35 • Risk Factors M = 42.47
16
Risk / Protective Factors Associated with SWPBIS Implementation
Relationships Between Perceived Implementation and Perceived Safety 1. Statistically significant correlations between staff perceptions of primary‐tiered SWPBIS implementation and: – Protective factors (r = .781) – Protective to risk factor ratio (r = .466) – Protective minus risk factor score (r = .589) – Risk factors (r = ‐.413) 2. All relationships are in the expected direction
17
Impact of PAPBS Network’s SWPBIS: Behaviors
Office Discipline Referrals • ODRs / Day / 100 Students – Data reported as Median not Mean / Average
18
ODR Rates Differ By Grade Level
Sample sizes: Pre‐implementation = 121 Year 1 = 157
Year 2 = 125
Year 3 = 76
Year 4 = 64
Sample sizes: Pre‐implementation = 32 Year 1 = 34 Year 2 = 23 Year 3 = 12
Year 4 = 8
ODR Rates Differ By Grade Level
Sample sizes: Pre‐implementation = 8 Year 1 = 8 Year 2 = 6 Year 3 = 5
Year 4 = 5
Sample sizes: Pre‐implementation = 23 Year 1 = 10 Year 2 = 10 Year 3 = 4
Year 4 = 3
19
ODR Rates Differ By Grade Level
Sample sizes: Pre‐implementation = 15 Year 1 = 15 Year 2 = 5
Year 3 = 4
Year 4 = 4
Sample sizes: Pre‐implementation = 11 Year 1 = 12 Year 2 = 17 Year 3 = 3
Longitudinal ODR Change – Elementary Schools
Note. ODR = office discipline referrals; solid red line represents the national median; dashed red lines represent the 25th and 75th percentiles (SWIS, 2015). No significant change over time.
20
Longitudinal ODR Change – Secondary Schools
Note. ODR = office discipline referrals; solid red line represents the national median; dashed red lines represent the 25th and 75th percentiles (SWIS, 2015). No significant change over time.
Longitudinal ODR Changes – Alternative Schools
*Note: No national data to compare PA Alternative Education settings
21
Office Discipline Referrals • ODR Triangle Data (Kaufman et al. 2010; Spaulding et al., 2010) – Approximately 80% of all students = 0 or 1 ODR per year – Approximately 10‐15% of all students = 2‐5 ODRs per year – Approximately 5‐10% of all students = 6 or more ODRs per year
Proportion of Students Receiving ODRs
• Statistically significant differences between elementary compared to middle and at times junior/senior high schools • K‐8 comparable to all building levels across each year
22
Elementary Schools’ ODR Triangle Data
• Primary‐tiered SWPBIS has desired, sustained effect on behaviors • 90‐92% of all elementary students receive one or no ODR in academic year • Under 3% of all elementary students exhibit substantial behavior challenges across multiple years of SWPBIS
Secondary Schools’ ODR Triangle Data
• ODR Triangle Data did not statistically change across multiple years • Approximately 85‐97% of all students respond favorably to primary‐tiered SWPBIS
23
Longitudinal Changes in OSS – Elementary Schools
Note. Mean Out-of-School Suspensions (OSS) rates were significantly lower in Year 2 compared to pre-implementation rates.
Longitudinal Changes in OSS – Secondary Schools
Note. Mean OSS rates were significantly lower in Year 2 compared to preimplementation rates.
24
Expulsions – Secondary Schools
Note. Mean expulsion rates did not statistically change after two years of SWPBIS implementation.
Number of Schools Implementing Tier 2 Standard Protocol (CICO)
Note. CICO = Check-In / Check-Out. Other indicates schools with non-traditional grade configurations, including PreK-8 and PreK-12.
25
Sustained CICO Implementation Across All Levels
Note. CICO = Check-In / Check-Out.
Efficacy of CICO 2008 – present
Note. CICO = Check-In / Check-Out. Schools implementing CICO over multiple years initially experience very high levels of success with a marked decline, then a return to very high levels
26
Impact of PAPBS Network’s SWPBIS: Educational Placements
LRE Placements
Note. The percentage of students in the ≥80% LRE index statistically increased by Year 3; the percentage of students in the 40-79% LRE index statistically decreased by Year 3; the percentage of students in the ≤39% LRE index did not statistically change by Year 3.
27
Out‐of‐School (OSS) Placements
Note. OOS = out-of-school; ED = emotional disturbance. Data are from all building types except alternative education settings. OOS placements for all students and for students with emotional disturbance were statistically lower at the third year of SWPBIS implementation compared to pre-implementation rates.
Academic Outcomes • PSSA results were compared among the participating schools based upon the schools’ level of perceived implementation using the EBS: SAS (Sugai et al., 2003) • EBS: SAS data and PSSA data were available for 369 schools
28
Quartile Ranges for PAPBS Network Schools’ Perceived Level of SWPBIS Implementation Quartile of Implementation First Second Third Fourth TOTAL
N 92 97 87 93 369
Range of Scores 5.42% 44.44% 63.00% 74.56% 5.42%
43.33% 62.73% 74.42% 96.67% 96.7%
M 30.29% 54.14% 69.04% 81.76% 58.67 %
Note. Scores represent the percentage of items endorsed by staff on the schoolwide composite of the EBS: SAS survey Implementation ranged from a perceived low of 5.42% to a perceived high of 96.7% .
PSSA Performance by Perceived Level of Implementation
Note. PSSA = Pennsylvania System of School Assessment; ELA = English Language Arts; Successful performance is defined as the aggregate percentage of students performing in the Proficient / Advanced ranges on the PSSA.
29
PSSA Findings • As perceived level of SWPBIS implementation increased: – Increase in percentage of students who passed (proficient or advanced) PSSA Math and English Language Arts • Schools that perceive they are fully implementing SWPBIS performed significantly better on PSSA Math and English Language Arts than schools who perceive they are at a lower level of implementation
SUMMARY
30
Context and Input • 604 schools from 203 LEAs and Charters in PAPBS Network (increase from last year)
• Multiple funding sources • Standardized training and technical assistance tailored to fit PA context • Cadre of 127 Network Facilitators (increase from last year) • Annual Implementers’ Forum – Increasing attendance; positive feedback
Fidelity • By spring 2016, 217 schools implementing primary‐tiered SWPBIS – Integrity unknown for large minority (37.5%) – Most at elementary level (73%)
• Long‐term sustainability is possible – 39 (10%) schools implementing for >5 years (increase from last year)
• 183 collaborating mental health agencies
31
SWPBIS Associated with…. • Staff recognition of high fidelity SWPBIS • Staff perceptions of fidelity significantly associated with more protective factors and fewer risk factors for school violence • Sustained ODR rates at rates at or below the 50th PR nationally • ODR rates better at elementary and middle schools
SWPBIS Associated with…. • Large majorities of students with