Teachers
Effects of Implementation on Intervention Impact Connecting Outcomes to Practice
Presented at the
Carole Gallagher, PhD Joseph Van Matre Sharon Twitty
STEM Learning Opportunities Providing Equity (SLOPE) Investing in Innovation (i3) Study Partners Grantee California Educational Round Table Intersegmental Coordinating Committee (ICC) Implementation Team Alliance for Regional Collaboration to Heighten Educational Success (ARCHES)
Evaluation Team WestEd National i3 Evaluators Abt Associates Inc.
Evaluation Overview
2010 Investing in Innovation (i3) Developmental Grant Intervention •
Three project-based, drop-in instructional units incorporated into the existing grade 8 Algebra I curriculum
•
Responsive coaching
•
A college awareness curriculum
•
Pre-Algebra summer program
Goal Improve student achievement in Algebra I by reinforcing key math concepts with hands-on learning
Primary Outcome Students’ performance on state’s end-of-year assessment in Algebra I
Research Context
Transition to the Common Core State Standards (CCSS) •
Major shifts for mathematics teaching practices
•
Districts’ desire for research on “what works”
Teachers’ Needs •
Community of practice focused on student outcomes
•
Linking mathematics standards directly to college and career readiness
•
Making the project relevant and applicable to the classroom
Evaluation Participants
• Teachers and students from 28 middle schools in 8 California districts (urban, suburban, and rural) • Students from all classrooms assigned to participating Treatment (T) and Control (C) teachers
• Informed consent for all teachers and students • Final sample (in the impact analyses): Treatment
Control
28 teachers 1,384 students
24 teachers 1,088 students
Findings from Impact Analyses
Research Question
Analysis
On average, does performance on the state’s grade 8 test in Algebra I differ between students who are exposed to any intervention components and students who are not exposed to any intervention components?
A 2-level hierarchical linear model (HLM) with students nested in classrooms that are randomized to treatment status
Research Question
n
Adjusted Mean
T/C Mean Difference (Effect Size)
All Treatment vs. All Control
T = 1,384 C = 1,088
T = 319.57 C = 324.95
-5.38 (-0.09)
p-value
.174
Implementation Matrix Abbreviated Example
Component Y1 Algebra I: Implementation of Algebra Drop-in Unit 1
Operational Definition
Possible Implementation Range
a. ≥ 80% of implementation range (i.e., 4 lessons) / No or Yes: 0-1 b. 0–150 minutes b. ≥ 70% of (varies for each lesson) implementation range for each taught lesson / No or Yes: 0-1 for each lesson c. 0–8 steps (varies for b. ≥ 70% of each lesson) implementation range for each taught lesson / No or Yes: 0-1 for each lesson d. 1–3 (1 = gave only a d. ≥ 2 for each taught little attention; 3 = lesson / No or Yes: 0-1 gave a lot of attention) for each lesson
a. Number of lessons a. 0–5 lessons used
b. Time spent on each lesson
c. Number of steps used within each lesson
d. Average level of attention given to used steps within each lesson
Expected Teacherlevel Implementation “with Fidelity”
Teacher-level Criterion for implementation “with fidelity”
Program-level Criterion
0-16 80% of teachers Adequate: ≥ 75% of meet criterion possible highest composite score; i.e., the score for implementation with fidelity is at least 12 (this indicates that the minimum number of lessons to be taught is 4; no way to reach a score of 12 or above if teachers cover 3 or less lessons)
Implementation Study Year 1 Findings by Component
Component
Description
Number of Teachers Meeting Fidelity Threshold
Percentage of Teachers Meeting Fidelity Threshold
Programlevel Fidelity
Coaching
Average of one or more hours of professional coaching for each unit.
16/28
57%
No
Unit 1: Puzzle Cube
Teacher implements Unit 1 of intervention curriculum.
25/28
89%
Yes
Unit 2: Air Traffic Control
Teacher implements Unit 2 of intervention curriculum.
18/28
64%
No
Unit 3: Catapult Game
Teacher implements Unit 3 of intervention curriculum.
21/28
75%
No
180%
20
160%
18
140%
16 14
120%
12
100%
10 80%
8
60%
6
40%
4
20%
2
0%
0
Teachers Not Implementing a Lesson
Percentage of Anticipated Time
Percentages of Expected Time Spent on Each Lesson
Zeros
Unit Lesson
Mean
Comparison of High vs. Low Fidelity Teachers
• “High-fidelity” Teachers: •
Met implementation criterion for each unit (3 total units).
•
57% were “high-implementing” (n=16).
• “Low-fidelity” Teachers: •
Met implementation criterion for 1, 2, or none of the units.
•
43% were “low-implementing” (n=12).
Percentage of Anticipated Time Used by Lesson High vs. Low Implementation Teachers 200%
Percentage of Anticipated Time
180% 160% 140% 120% 100%
Ideal
80% 60%
High Implementation
40%
Low Implementation
20%
0%
Unit Lesson
Intervention Impact on Student Test Scores, by Teachers’ Fidelity Level
Group Membership
n
Adjusted Mean
Mean Difference
p-value
Effect Size
Treatment (low fidelity)
581
311.62
Treatment (high fidelity)
803
324.65
Control
1,088
324.62
Diff: T-low and C
-13.00
0.039*
-0.21
Diff: T-high and C
0.03
0.996
0.00
Diff: T-low and T-high
-13.03
0.048*
-0.21
*significant at .05 level
Interpretation of Findings
• Students with high-fidelity treatment teachers did not perform differently from control students on the Algebra I assessment. • Students with low-fidelity treatment teachers, however, scored significantly lower than control students. • Similarly, students with low-fidelity treatment teachers also scored significantly lower than students with high-fidelity treatment teachers.
Research Implications
• Intervention Context Matters •
Educational interventions are mediated through teachers.
•
Impact is based on treatment received, not treatment intended.
•
Finding “no effect” does not imply the ideal treatment is ineffective.
• Communication Is Key •
Teachers need clear expectations and ongoing communication about the intervention.
• Monitoring Implementation Is Best Practice •
Any claims beyond intent-to-treat require implementation data.
Carole Gallagher, PhD [email protected]
Joseph Van Matre [email protected]
Sharon Twitty [email protected]
Presented at the
Carole Gallagher, PhD Joseph Van Matre Sharon Twitty
STEM Learning Opportunities Providing Equity (SLOPE) Investing in Innovation (i3) Study Partners Grantee California Educational Round Table Intersegmental Coordinating Committee (ICC) Implementation Team Alliance for Regional Collaboration to Heighten Educational Success (ARCHES)
Evaluation Team WestEd National i3 Evaluators Abt Associates Inc.
Evaluation Overview
2010 Investing in Innovation (i3) Developmental Grant Intervention •
Three project-based, drop-in instructional units incorporated into the existing grade 8 Algebra I curriculum
•
Responsive coaching
•
A college awareness curriculum
•
Pre-Algebra summer program
Goal Improve student achievement in Algebra I by reinforcing key math concepts with hands-on learning
Primary Outcome Students’ performance on state’s end-of-year assessment in Algebra I
Research Context
Transition to the Common Core State Standards (CCSS) •
Major shifts for mathematics teaching practices
•
Districts’ desire for research on “what works”
Teachers’ Needs •
Community of practice focused on student outcomes
•
Linking mathematics standards directly to college and career readiness
•
Making the project relevant and applicable to the classroom
Evaluation Participants
• Teachers and students from 28 middle schools in 8 California districts (urban, suburban, and rural) • Students from all classrooms assigned to participating Treatment (T) and Control (C) teachers
• Informed consent for all teachers and students • Final sample (in the impact analyses): Treatment
Control
28 teachers 1,384 students
24 teachers 1,088 students
Findings from Impact Analyses
Research Question
Analysis
On average, does performance on the state’s grade 8 test in Algebra I differ between students who are exposed to any intervention components and students who are not exposed to any intervention components?
A 2-level hierarchical linear model (HLM) with students nested in classrooms that are randomized to treatment status
Research Question
n
Adjusted Mean
T/C Mean Difference (Effect Size)
All Treatment vs. All Control
T = 1,384 C = 1,088
T = 319.57 C = 324.95
-5.38 (-0.09)
p-value
.174
Implementation Matrix Abbreviated Example
Component Y1 Algebra I: Implementation of Algebra Drop-in Unit 1
Operational Definition
Possible Implementation Range
a. ≥ 80% of implementation range (i.e., 4 lessons) / No or Yes: 0-1 b. 0–150 minutes b. ≥ 70% of (varies for each lesson) implementation range for each taught lesson / No or Yes: 0-1 for each lesson c. 0–8 steps (varies for b. ≥ 70% of each lesson) implementation range for each taught lesson / No or Yes: 0-1 for each lesson d. 1–3 (1 = gave only a d. ≥ 2 for each taught little attention; 3 = lesson / No or Yes: 0-1 gave a lot of attention) for each lesson
a. Number of lessons a. 0–5 lessons used
b. Time spent on each lesson
c. Number of steps used within each lesson
d. Average level of attention given to used steps within each lesson
Expected Teacherlevel Implementation “with Fidelity”
Teacher-level Criterion for implementation “with fidelity”
Program-level Criterion
0-16 80% of teachers Adequate: ≥ 75% of meet criterion possible highest composite score; i.e., the score for implementation with fidelity is at least 12 (this indicates that the minimum number of lessons to be taught is 4; no way to reach a score of 12 or above if teachers cover 3 or less lessons)
Implementation Study Year 1 Findings by Component
Component
Description
Number of Teachers Meeting Fidelity Threshold
Percentage of Teachers Meeting Fidelity Threshold
Programlevel Fidelity
Coaching
Average of one or more hours of professional coaching for each unit.
16/28
57%
No
Unit 1: Puzzle Cube
Teacher implements Unit 1 of intervention curriculum.
25/28
89%
Yes
Unit 2: Air Traffic Control
Teacher implements Unit 2 of intervention curriculum.
18/28
64%
No
Unit 3: Catapult Game
Teacher implements Unit 3 of intervention curriculum.
21/28
75%
No
180%
20
160%
18
140%
16 14
120%
12
100%
10 80%
8
60%
6
40%
4
20%
2
0%
0
Teachers Not Implementing a Lesson
Percentage of Anticipated Time
Percentages of Expected Time Spent on Each Lesson
Zeros
Unit Lesson
Mean
Comparison of High vs. Low Fidelity Teachers
• “High-fidelity” Teachers: •
Met implementation criterion for each unit (3 total units).
•
57% were “high-implementing” (n=16).
• “Low-fidelity” Teachers: •
Met implementation criterion for 1, 2, or none of the units.
•
43% were “low-implementing” (n=12).
Percentage of Anticipated Time Used by Lesson High vs. Low Implementation Teachers 200%
Percentage of Anticipated Time
180% 160% 140% 120% 100%
Ideal
80% 60%
High Implementation
40%
Low Implementation
20%
0%
Unit Lesson
Intervention Impact on Student Test Scores, by Teachers’ Fidelity Level
Group Membership
n
Adjusted Mean
Mean Difference
p-value
Effect Size
Treatment (low fidelity)
581
311.62
Treatment (high fidelity)
803
324.65
Control
1,088
324.62
Diff: T-low and C
-13.00
0.039*
-0.21
Diff: T-high and C
0.03
0.996
0.00
Diff: T-low and T-high
-13.03
0.048*
-0.21
*significant at .05 level
Interpretation of Findings
• Students with high-fidelity treatment teachers did not perform differently from control students on the Algebra I assessment. • Students with low-fidelity treatment teachers, however, scored significantly lower than control students. • Similarly, students with low-fidelity treatment teachers also scored significantly lower than students with high-fidelity treatment teachers.
Research Implications
• Intervention Context Matters •
Educational interventions are mediated through teachers.
•
Impact is based on treatment received, not treatment intended.
•
Finding “no effect” does not imply the ideal treatment is ineffective.
• Communication Is Key •
Teachers need clear expectations and ongoing communication about the intervention.
• Monitoring Implementation Is Best Practice •
Any claims beyond intent-to-treat require implementation data.
Carole Gallagher, PhD [email protected]
Joseph Van Matre [email protected]
Sharon Twitty [email protected]
