Where Have We Been, and What Have We Done? - Amazon Web ...

Report 1 Downloads 95 Views
Where Have We Been, and What   Have We Done?

Timothy J. Runge, PhD, NCSP Indiana University of Pennsylvania May 2017 Implementers’ Forum

Note: Opinions expressed within are solely  those of the authors and do not necessarily  reflect the position of the funding agencies or  the Pennsylvania Department of Education, and  such endorsements should not be inferred.

1

SWPBIS

Response to  Intervention

MTSS  Is Integrated  Continuum

Evaluation Framework

Algozzine et al. (2010) described five domains of  large‐scale SWPBIS evaluation

1. 2. 3. 4.

Context Input Fidelity Sustainability, Replication,  Improvement 5. Impact

2

PAPBS History: Where Have We Been?

Context of PAPBS Network’s   SWPBIS

3

PAPBS Network Membership • Cohort 1 in summer 2007 through winter 2008 – 33  schools (1 withdrew) • At least 570 more since fall 2009 – Cohort 2 – Not a true cohort, in traditional sense • Cannot independently verify precise training dates or baseline years

• As of spring 2016, at least 604 PAPBS Network sites • At least 183 collaborating community mental health  agencies (increase from 165 last year)

Participating Buildings / LEA or Charter School / IUs by Region and Grade Level

4

Geographic Location of Participating PAPBS Network Schools – Combined Cohorts

Note. Blue circles represent cohort 1 LEAs; red circles represent cohort 2 LEAs

Input of PAPBS Network’s   SWPBIS

5

Written Commitments   to Participate From

Superintendent

Building Principals

Building‐level Teams

Network Facilitator

6

Funding Support for SWPBIS • Majority – IDEA Part B (School‐age) and Part C (Early  Childhood) • Personnel and resources from CoP on SBBH Co‐Directors  • LEAs • Federal grants – Safe Schools / Healthy Students – School Climate Transformation • SBBH Grants totaling $361,965 + $40,000 in IF vouchers

2015‐2016 SBBH Grantees

• Culturally‐Responsive PBIS 1. 2. 3. 4. 5. 6. 7.

Abington School District Harrisburg School District Pittsburgh Public Schools Propel Schools Saucon Valley School District Shaler Area School District Steel Valley School District

$70,000

• Model Sites 1. 2. 3. 4. 5. 6.

Bellefonte Area High School‐ Bellefonte Area School District Upper Grandview Elementary  School‐ Highlands School  District Juniata Valley Elementary  School‐Juniata Valley School  District Laurel Elementary &  Junior/Senior High Schools‐ Laurel School District Lehigh Career & Technical  Institute  Donald Eichhorn Middle School‐ Lewisburg School District

$7,500

7

2015‐2016 SBBH Grantees • Establish SWPBIS

• Expand SWPBIS

1. 2. 3. 4. 5. 6. 7. 8.

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

BLaST Intermediate Unit #17 Butler Area School District Galeton Area School District Hatboro‐Harsham School District Homer Center School District Keystone Oaks School District Lackawanna Trail School District West Chester Area School District 

$79,965

Abington School District Bald Eagle Area School District Chartiers Valley School District East Stroudsburg School District Highlands School District Lehigh Career & Technical Institute  Moniteau School District Norwin School District Perkiomen Valley School District Pocono Mountain School District Scranton School District Williamsport School District

$110,000

2015‐2016 Tier 3 SBBH Grantees Project RENEW 1. 2. 3. 4. 5. 6. 7. 8. 9.

Altoona Area School District Bangor Area School District Clairton City School District CLASS Academy East Stroudsburg School District Freedom Area School District Greater Latrobe Area School  District Highlands School District Lincoln Intermediate Unit #12

10. McGuffey School District 11. Mount Lebanon School District 12. Northeastern Intermediate Unit  #19 13. PENNCREST School District 14. Pittsburgh Public Schools 15. Schuylkill Have School District 16. Seneca Highlands Intermediate  Unit #9 17. Tyrone Area School District 18. West Greene School District 19. Williamsport Area School  District

$94,500

8

Training and Technical Assistance • Cadre of 127 Independent Network  Facilitators (increase from 113 the previous year) • Rigorous application and training process  while in provisional status

Attendance Figures at the Annual Pennsylvania  PBIS Implementers’ Forums 2011‐2016

9

Implementers’ Forum Attendees by Role

2016 Implementers’ Forum Attendees’  Satisfaction

10

Fidelity of PAPBS Network’s   SWPBIS

Primary Tier SWPBIS Implementation Status

11

How Many Schools Are Implementing by  Grade Span in Spring 2016?

Summary of Fidelity Data • Since 2007, there has been an increase in the  number of: – PAPBS Network schools  – Schools implementing with integrity

• Full implementation of primary‐tiered SWPBIS  in 217 schools • 6 schools indicate sustained implementation  since training 9 years ago

12

Replication, Sustainability,   and Improvement of PAPBS   Network’s SWPBIS

Sustained Implementation is Possible!

Note. These data reflect any school that ever submitted primary‐tiered SWPBIS fidelity  data since 2007‐2008.  Many schools can achieve full implementation within 1 year of  initial training; however, a small percentage require 2‐4 years to fully implement  SWPBIS.

13

PAPBS History: What Have We Done?

Impact of PAPBS Network’s   SWPBIS: Attitudes / Beliefs

14

Staff Perceptions Regarding Level of   PBIS Implementation • Perceived PBIS Implementation measured by EBS:  SAS (Sugai et al., 2003) • Participants select one of three choices: • full (fully implemented) • partial (partially implemented) • none (not implemented) • May not be an objective measure of SWPBIS

2015‐16 EBS: SAS Measures of Staff Perceptions of  Primary‐Tiered SWPBIS Implementation

]

15

Staff Perceptions Regarding Level   School Safety  • School Safety Survey (Sprague et al., 2005) – Staff survey of perceptions of Risk and Protective  Factors associated with school violence and safety • Risk = drug / gang activity; vandalism; truancy; community  poverty and crime; child abuse • Protective = extracurricular opportunities; parental  involvement in school; school‐community collaboration;  acceptance of diversity; high expectations for all students

2015‐16 Descriptive Statistics for Protective and  Risk Factors  N = 501

Protective Factors (P)

M 73.35

Risk Factors (R)

Ratio of P / R

Difference (P‐R)

SD

M

SD

M

SD

M

SD

9.37

42.47

13.25

1.98

0.94

30.88

20.85

• SWPBIS schools have more positive sense of their protections against school  violence • Protective Factors M = 73.35 • Risk Factors M = 42.47

16

Risk / Protective Factors Associated  with SWPBIS Implementation

Relationships Between Perceived  Implementation and Perceived Safety 1. Statistically significant correlations between staff  perceptions of primary‐tiered SWPBIS implementation  and: – Protective factors (r = .781) – Protective to risk factor ratio (r = .466) – Protective minus risk factor score (r = .589) – Risk factors (r = ‐.413) 2. All relationships are in the expected direction

17

Impact of PAPBS Network’s   SWPBIS: Behaviors

Office Discipline Referrals • ODRs / Day / 100 Students – Data reported as Median not Mean / Average

18

ODR Rates Differ By Grade Level

Sample sizes: Pre‐implementation = 121 Year 1 = 157

Year 2 = 125

Year 3 = 76

Year 4 = 64

Sample sizes: Pre‐implementation = 32 Year 1 = 34 Year 2 = 23 Year 3 = 12

Year 4 = 8

ODR Rates Differ By Grade Level

Sample sizes: Pre‐implementation = 8 Year 1 = 8 Year 2 = 6 Year 3 = 5

Year 4 = 5

Sample sizes: Pre‐implementation = 23 Year 1 = 10 Year 2 = 10 Year 3 = 4

Year 4 = 3

19

ODR Rates Differ By Grade Level

Sample sizes: Pre‐implementation = 15 Year 1 = 15 Year 2 = 5

Year 3 = 4

Year 4 = 4

Sample sizes: Pre‐implementation = 11 Year 1 = 12 Year 2 = 17 Year 3 = 3

Longitudinal ODR Change – Elementary Schools

Note. ODR = office discipline referrals; solid red line represents the national median; dashed red lines represent the 25th and 75th percentiles (SWIS, 2015). No significant change over time.

20

Longitudinal ODR Change – Secondary  Schools

Note. ODR = office discipline referrals; solid red line represents the national median; dashed red lines represent the 25th and 75th percentiles (SWIS, 2015). No significant change over time.

Longitudinal ODR Changes – Alternative Schools

*Note: No national data to compare PA Alternative Education settings

21

Office Discipline Referrals • ODR Triangle Data (Kaufman et al. 2010; Spaulding et al., 2010) – Approximately 80% of all students = 0 or 1 ODR  per year – Approximately 10‐15% of all students = 2‐5 ODRs  per year – Approximately 5‐10% of all students = 6 or more  ODRs per year

Proportion of Students Receiving ODRs

• Statistically significant differences between elementary compared to middle and at  times junior/senior high schools • K‐8 comparable to all building levels across each year 

22

Elementary Schools’ ODR Triangle Data

• Primary‐tiered SWPBIS has desired, sustained effect on behaviors • 90‐92% of all elementary students receive one or no ODR in academic year • Under 3% of all elementary students exhibit substantial behavior challenges across  multiple years of SWPBIS

Secondary Schools’ ODR Triangle Data

• ODR Triangle Data did not statistically change across multiple years • Approximately 85‐97% of all students respond favorably to primary‐tiered SWPBIS

23

Longitudinal Changes in OSS – Elementary Schools

Note. Mean Out-of-School Suspensions (OSS) rates were significantly lower in Year 2 compared to pre-implementation rates.

Longitudinal Changes in OSS – Secondary Schools

Note. Mean OSS rates were significantly lower in Year 2 compared to preimplementation rates.

24

Expulsions – Secondary Schools

Note. Mean expulsion rates did not statistically change after two years of SWPBIS implementation.

Number of Schools Implementing Tier  2 Standard Protocol (CICO)

Note. CICO = Check-In / Check-Out. Other indicates schools with non-traditional grade configurations, including PreK-8 and PreK-12.

25

Sustained CICO Implementation Across  All Levels

Note. CICO = Check-In / Check-Out.

Efficacy of CICO 2008 – present

Note. CICO = Check-In / Check-Out. Schools implementing CICO over multiple years initially experience very high levels of  success with a marked decline, then a return to very high levels

26

Impact of PAPBS Network’s   SWPBIS: Educational   Placements

LRE Placements

Note. The percentage of students in the ≥80% LRE index statistically increased by Year 3; the percentage of students in the 40-79% LRE index statistically decreased by Year 3; the percentage of students in the ≤39% LRE index did not statistically change by Year 3.

27

Out‐of‐School (OSS) Placements

Note. OOS = out-of-school; ED = emotional disturbance. Data are from all building types except alternative education settings. OOS placements for all students and for students with emotional disturbance were statistically lower at the third year of SWPBIS implementation compared to pre-implementation rates.

Academic Outcomes • PSSA results were compared among the  participating schools based upon the schools’  level of perceived implementation using the  EBS: SAS (Sugai et al., 2003) • EBS: SAS data and PSSA data were available for  369 schools

28

Quartile Ranges for PAPBS Network Schools’ Perceived   Level of SWPBIS Implementation   Quartile of  Implementation First Second Third Fourth TOTAL

N 92 97 87 93 369

Range of Scores 5.42% 44.44% 63.00% 74.56% 5.42%

43.33% 62.73% 74.42% 96.67% 96.7%

M 30.29% 54.14% 69.04% 81.76% 58.67 %

Note. Scores represent the percentage of items endorsed by staff on the schoolwide composite of the EBS: SAS survey Implementation ranged from a perceived low of 5.42% to a perceived high of 96.7% .

PSSA Performance by Perceived Level   of Implementation

Note. PSSA = Pennsylvania System of School Assessment; ELA = English Language Arts; Successful performance is defined as the aggregate percentage of students performing in the Proficient / Advanced ranges on the PSSA.

29

PSSA Findings • As perceived level of SWPBIS implementation  increased: – Increase in percentage of students who passed  (proficient or advanced) PSSA Math and English  Language Arts • Schools that perceive they are fully implementing  SWPBIS performed significantly better on PSSA Math  and English Language Arts than schools who perceive  they are at a lower level of implementation 

SUMMARY

30

Context and Input • 604 schools from 203 LEAs and Charters in PAPBS Network  (increase from last year)

• Multiple funding sources • Standardized training and technical assistance tailored to fit  PA context • Cadre of 127 Network Facilitators (increase from last year) • Annual Implementers’ Forum – Increasing attendance; positive feedback

Fidelity • By spring 2016, 217 schools implementing  primary‐tiered SWPBIS – Integrity unknown for large minority (37.5%) – Most at elementary level (73%)

• Long‐term sustainability is possible – 39 (10%) schools implementing for >5 years (increase  from last year)

• 183 collaborating mental health agencies

31

SWPBIS Associated with…. • Staff recognition of high fidelity SWPBIS • Staff perceptions of fidelity significantly associated  with more protective factors and fewer risk factors for  school violence • Sustained ODR rates at rates at or below the 50th PR  nationally • ODR rates better at elementary and middle schools

SWPBIS Associated with…. • Large majorities of students with