Developing a Model Pre-K Program: Lessons Learned
Do Later Teachers and Schools Provide Sustaining Environments for Pre-K? Mark Lachowicz, F. Alvin Pearman, II, & Matthew Springer Peabody Research Institute Vanderbilt University
March 2nd, 2017 Preliminary results: Do not cite at this time
Motivation • Benefits of pre-K on students’ test scores tend to fade out at least by third grade if not much sooner (Magnuson et al., 2007; Deming, 2009; Puma et al., 2010; Lipsey et al., 2013)
• Research has focused on post-pre-K environments to understand the contexts in which pre-K benefits are potentially sustained (Currie & Thomas, 2000; Lee & Loeb, 1995; Mashburn et al., 2008; Swain et al., 2015)
• Identifying these contexts is critical for the development of effective early childhood interventions and buffering against fade out.
Motivation • Factors underlying fade out are not well understood – Evidence from recent studies suggest a potential mechanism may be the lack of subsequent learning environments to sustain the early effects (Bailey et al., 2017) • Teacher effectiveness – Teacher effectiveness augments the persistence of pre-K benefits (Swain, Springer, & Hofer, 2015)
• School quality – School quality post pre-K augments the persistence of benefits (Currie & Thomas, 2000; Lee & Loeb, 1995)
Research Question: Do later good teachers and schools provide sustaining environments for pre-k effects?
Data Description: Tennessee Teacher Evaluation • First implemented statewide and across all grade levels in the 2010-2011 school year (the year TNVPK Cohort 2 children entered kindergarten) • Purpose – Improve teacher and organizational effectiveness – Performance categories have factored into decision making for tenure, and salary increases and promotions (TN-SB #1528)
• Evaluation – Composite of observation score and school-level growth score for grades in which state level tests are administered (3-8)
TN Teacher Evaluation (Observation score) • Adaptation of Charlotte-Danielson rubric by National Institute for Excellent Teaching, formerly called the Teacher Advancement Program (TAP) – Instruction (12 items); Planning (3 items); Environment (4 items) – Assessed multiple times per year
All TN Teachers 2012 • Teachers assigned overall effectiveness score (0 – 500) •
Effectiveness score translated into 5 performance categories 1 = Significantly below expectation 2 = Below expectation 3 = At expectation 4 = Above expectation 5 = Significantly above expectation
Use of Teacher Evaluations for This Study
• For each school year, teacher effectiveness ratings were averaged across observations during the year • Most elementary students have one primary teacher. Where more than one rating existed, they were averaged • Highly effective teachers were considered those with average teacher ratings of 4 and above
School Quality in This Study • School quality was defined as the percentage of students considered ‘proficient’ within a given school. – Proficiency based on available TCAP reading and math scores from 3rd, 4th, and 5th grades. – Math and ELA scores were averaged to yield overall proficiency for school
• School quality was averaged across school years. – The vast majority of students remained in the same school
Data Sources and Data Elements • Tennessee Education Research Alliance Data Warehouse • Teacher records, including evaluation results, course assignments, years of experience, race/ethnicity, salary, and age • School records, including demographic and performance information as well as information from NCES’ Common Core of Data • Student records, including age, race/ethnicity, course assignments, grade progression, etc. • Peabody Research Institute TNVPK Experimental Records • Peabody Research Institute collected data on two cohorts of over 3,000 children in TN that were randomly assigned to pre-k treatment or control conditions
Sample - CONSORT Diagram Cohort 2 n=1246
Excluded n=464 - No teacher data n=8 - Missing teacher assessment data n=345 - Missing TCAP scores n=111
Included in analytic sample n=782
Assigned VPK n=493
VPK Attendee n=416
No Assigned VPK n=289
No Show n=77
Did Not Attend VPK n=221
Crossover n=68
Key Indicators – Teacher Effectiveness
50%
Exposure to Highly Effective Teachers (TE > = 4) *
43%
% HE Teachers
40%
41% 35%
36%
33% 34%
36% 37%
30% VPK No VPK
20% 10% 0%
K
* p < 0.05
Grade 1
Grade 2
Grade 3
Key Indicators – Teacher Effectiveness Exposure to Highly Effective Teachers (TE >= 4)
40%
% HE Teachers
34% 29%
30%
26% 27%
23%22%
20%
17%
10% 0%
VPK No VPK
14%
5%
0
1
2
Number of HE Teachers
3
3%
4
Key Indicators – Teacher Effectiveness Exposure to Highly Effective Teachers (TR >= 4) 40% 34% 27% 26%
29%
30%
33% 30%
23% 22%
20%
16%
10%
5%
3%
0%
0
18%
17% 14%
1 2 3 Number of HE Teachers
3%
4
VPK No VPK State
Key Indicators – Teacher Effectiveness All TN Teachers 2012
Exposure to Highly Effective Teachers (TR >= 4) 40%
VPK No VPK
30%
State
20% 10% 0%
0
1 2 3 Number of HE Teachers
4
Key Indicators – School Quality 50%
% Proficient
40% 30% 20% 10% 0%
* 45% 41%
VPK No VPK
Analytic Strategy – Main Effects 𝒀𝒀𝒊𝒊𝒊𝒊 = 𝜷𝜷𝟏𝟏 𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒊𝒊𝒊𝒊 + 𝒁𝒁′𝒋𝒋 𝜸𝜸 + 𝑿𝑿′𝒊𝒊 𝜹𝜹 + 𝒖𝒖𝒋𝒋 + 𝒆𝒆𝟏𝟏𝒊𝒊𝒊𝒊 𝒀𝒀𝒊𝒊𝒊𝒊 = 𝜷𝜷𝟐𝟐 𝑺𝑺𝑺𝑺𝒋𝒋 + 𝒁𝒁′𝒋𝒋 𝜸𝜸 + 𝑿𝑿′𝒊𝒊 𝜹𝜹 + 𝒖𝒖𝒋𝒋 + 𝒆𝒆𝟐𝟐𝒊𝒊𝒊𝒊 𝒀𝒀𝒊𝒊𝒊𝒊 = ∑𝒌𝒌 𝜷𝜷𝟑𝟑𝟑𝟑 𝑻𝑻𝑻𝑻𝒊𝒊𝒊𝒊𝒊𝒊 + 𝒁𝒁′𝒋𝒋 𝜸𝜸 + 𝑿𝑿′𝒊𝒊 𝜹𝜹 + 𝒖𝒖𝒋𝒋 + 𝒆𝒆𝟑𝟑𝒊𝒊𝒊𝒊
•
Yij is TCAP math or reading scores in third grade for student i in school j.
•
preKij is dummy variable coding 1 if student participated in any TN-VPK.
•
𝑺𝑺𝑺𝑺𝒋𝒋 is average school quality variable for school j.
•
•
•
TEijk is teacher effectiveness dummy variable that takes value of 1 if student had k highly effective (HE) teachers (k = 1, 2, or 3). •
Reference group had no HE teachers
•
Few students had HE teachers all 4 years, combined with 3 HE teachers group
𝑿𝑿𝒊𝒊 is a vector of student baseline characteristics, 𝒁𝒁𝒋𝒋 is a vector of school characteristics, 𝒖𝒖𝒋𝒋 is the estimated school-specific residual, and 𝒆𝒆𝒊𝒊𝒊𝒊 is individual residual. What are the overall effects of the key indicators 𝜷𝜷𝟏𝟏 , 𝜷𝜷𝟐𝟐 , and 𝜷𝜷𝟑𝟑𝟑𝟑 .
Results – Main Effects Estimated main effects of pre-k, school quality, and number of highly effective teachers on TCAP math and reading scores Math
Reading
-0.05 (0.08)
-0.02 (0.07)
0.014 (0.005)**
0.009 (0.004)*
HE teacher 1 of 4 yrs (β31)
-0.04 (0.10)
-0.005 (0.10)
HE teacher 2 of 4 yrs (β32)
0.09 (0.10)
0.13 (0.10)
HE teacher 3 or more yrs (β33)
0.16 (0.11)
0.08 (0.11)
Pre-K (β1) School Quality (β2)
+ p
March 2nd, 2017 Preliminary results: Do not cite at this time
Motivation • Benefits of pre-K on students’ test scores tend to fade out at least by third grade if not much sooner (Magnuson et al., 2007; Deming, 2009; Puma et al., 2010; Lipsey et al., 2013)
• Research has focused on post-pre-K environments to understand the contexts in which pre-K benefits are potentially sustained (Currie & Thomas, 2000; Lee & Loeb, 1995; Mashburn et al., 2008; Swain et al., 2015)
• Identifying these contexts is critical for the development of effective early childhood interventions and buffering against fade out.
Motivation • Factors underlying fade out are not well understood – Evidence from recent studies suggest a potential mechanism may be the lack of subsequent learning environments to sustain the early effects (Bailey et al., 2017) • Teacher effectiveness – Teacher effectiveness augments the persistence of pre-K benefits (Swain, Springer, & Hofer, 2015)
• School quality – School quality post pre-K augments the persistence of benefits (Currie & Thomas, 2000; Lee & Loeb, 1995)
Research Question: Do later good teachers and schools provide sustaining environments for pre-k effects?
Data Description: Tennessee Teacher Evaluation • First implemented statewide and across all grade levels in the 2010-2011 school year (the year TNVPK Cohort 2 children entered kindergarten) • Purpose – Improve teacher and organizational effectiveness – Performance categories have factored into decision making for tenure, and salary increases and promotions (TN-SB #1528)
• Evaluation – Composite of observation score and school-level growth score for grades in which state level tests are administered (3-8)
TN Teacher Evaluation (Observation score) • Adaptation of Charlotte-Danielson rubric by National Institute for Excellent Teaching, formerly called the Teacher Advancement Program (TAP) – Instruction (12 items); Planning (3 items); Environment (4 items) – Assessed multiple times per year
All TN Teachers 2012 • Teachers assigned overall effectiveness score (0 – 500) •
Effectiveness score translated into 5 performance categories 1 = Significantly below expectation 2 = Below expectation 3 = At expectation 4 = Above expectation 5 = Significantly above expectation
Use of Teacher Evaluations for This Study
• For each school year, teacher effectiveness ratings were averaged across observations during the year • Most elementary students have one primary teacher. Where more than one rating existed, they were averaged • Highly effective teachers were considered those with average teacher ratings of 4 and above
School Quality in This Study • School quality was defined as the percentage of students considered ‘proficient’ within a given school. – Proficiency based on available TCAP reading and math scores from 3rd, 4th, and 5th grades. – Math and ELA scores were averaged to yield overall proficiency for school
• School quality was averaged across school years. – The vast majority of students remained in the same school
Data Sources and Data Elements • Tennessee Education Research Alliance Data Warehouse • Teacher records, including evaluation results, course assignments, years of experience, race/ethnicity, salary, and age • School records, including demographic and performance information as well as information from NCES’ Common Core of Data • Student records, including age, race/ethnicity, course assignments, grade progression, etc. • Peabody Research Institute TNVPK Experimental Records • Peabody Research Institute collected data on two cohorts of over 3,000 children in TN that were randomly assigned to pre-k treatment or control conditions
Sample - CONSORT Diagram Cohort 2 n=1246
Excluded n=464 - No teacher data n=8 - Missing teacher assessment data n=345 - Missing TCAP scores n=111
Included in analytic sample n=782
Assigned VPK n=493
VPK Attendee n=416
No Assigned VPK n=289
No Show n=77
Did Not Attend VPK n=221
Crossover n=68
Key Indicators – Teacher Effectiveness
50%
Exposure to Highly Effective Teachers (TE > = 4) *
43%
% HE Teachers
40%
41% 35%
36%
33% 34%
36% 37%
30% VPK No VPK
20% 10% 0%
K
* p < 0.05
Grade 1
Grade 2
Grade 3
Key Indicators – Teacher Effectiveness Exposure to Highly Effective Teachers (TE >= 4)
40%
% HE Teachers
34% 29%
30%
26% 27%
23%22%
20%
17%
10% 0%
VPK No VPK
14%
5%
0
1
2
Number of HE Teachers
3
3%
4
Key Indicators – Teacher Effectiveness Exposure to Highly Effective Teachers (TR >= 4) 40% 34% 27% 26%
29%
30%
33% 30%
23% 22%
20%
16%
10%
5%
3%
0%
0
18%
17% 14%
1 2 3 Number of HE Teachers
3%
4
VPK No VPK State
Key Indicators – Teacher Effectiveness All TN Teachers 2012
Exposure to Highly Effective Teachers (TR >= 4) 40%
VPK No VPK
30%
State
20% 10% 0%
0
1 2 3 Number of HE Teachers
4
Key Indicators – School Quality 50%
% Proficient
40% 30% 20% 10% 0%
* 45% 41%
VPK No VPK
Analytic Strategy – Main Effects 𝒀𝒀𝒊𝒊𝒊𝒊 = 𝜷𝜷𝟏𝟏 𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒊𝒊𝒊𝒊 + 𝒁𝒁′𝒋𝒋 𝜸𝜸 + 𝑿𝑿′𝒊𝒊 𝜹𝜹 + 𝒖𝒖𝒋𝒋 + 𝒆𝒆𝟏𝟏𝒊𝒊𝒊𝒊 𝒀𝒀𝒊𝒊𝒊𝒊 = 𝜷𝜷𝟐𝟐 𝑺𝑺𝑺𝑺𝒋𝒋 + 𝒁𝒁′𝒋𝒋 𝜸𝜸 + 𝑿𝑿′𝒊𝒊 𝜹𝜹 + 𝒖𝒖𝒋𝒋 + 𝒆𝒆𝟐𝟐𝒊𝒊𝒊𝒊 𝒀𝒀𝒊𝒊𝒊𝒊 = ∑𝒌𝒌 𝜷𝜷𝟑𝟑𝟑𝟑 𝑻𝑻𝑻𝑻𝒊𝒊𝒊𝒊𝒊𝒊 + 𝒁𝒁′𝒋𝒋 𝜸𝜸 + 𝑿𝑿′𝒊𝒊 𝜹𝜹 + 𝒖𝒖𝒋𝒋 + 𝒆𝒆𝟑𝟑𝒊𝒊𝒊𝒊
•
Yij is TCAP math or reading scores in third grade for student i in school j.
•
preKij is dummy variable coding 1 if student participated in any TN-VPK.
•
𝑺𝑺𝑺𝑺𝒋𝒋 is average school quality variable for school j.
•
•
•
TEijk is teacher effectiveness dummy variable that takes value of 1 if student had k highly effective (HE) teachers (k = 1, 2, or 3). •
Reference group had no HE teachers
•
Few students had HE teachers all 4 years, combined with 3 HE teachers group
𝑿𝑿𝒊𝒊 is a vector of student baseline characteristics, 𝒁𝒁𝒋𝒋 is a vector of school characteristics, 𝒖𝒖𝒋𝒋 is the estimated school-specific residual, and 𝒆𝒆𝒊𝒊𝒊𝒊 is individual residual. What are the overall effects of the key indicators 𝜷𝜷𝟏𝟏 , 𝜷𝜷𝟐𝟐 , and 𝜷𝜷𝟑𝟑𝟑𝟑 .
Results – Main Effects Estimated main effects of pre-k, school quality, and number of highly effective teachers on TCAP math and reading scores Math
Reading
-0.05 (0.08)
-0.02 (0.07)
0.014 (0.005)**
0.009 (0.004)*
HE teacher 1 of 4 yrs (β31)
-0.04 (0.10)
-0.005 (0.10)
HE teacher 2 of 4 yrs (β32)
0.09 (0.10)
0.13 (0.10)
HE teacher 3 or more yrs (β33)
0.16 (0.11)
0.08 (0.11)
Pre-K (β1) School Quality (β2)
+ p
