Device Comparability Study Update
Device Comparability Study Update Joe Fitzpatrick University of Kansas Presented at the CCSSO Fall 2016 TILSA SCASS Meeting
Overview • Updates on progress of device comparability studies • Phases 1 & 2 – Item flagging • Phase 3 (future work) – Follow-ups with selected items
• Final report will be submitted for inclusion in Center for Assessment compendium
Data Samples • Two state assessment programs • State 1 – Data from 2014 through 2016 • State 2 – 2015 only
• ELA and Math general assessments • Grades 3, 5, 8, and HS • Two-stage test design
Research Questions 1. Are there differences in item performance or overall results by device used? If differences are found in the first year iPads and Chromebooks are introduced, do those same differences still hold in Year 3? 2. Focusing specifically on students using the text-tospeech option in mathematics, are there differences in how students perform on items by device?
Methods • Sample grouped by assessment device • PCs, iMacs, iPads, & Chromebooks
• Sub-samples drawn and matched on provided overall scale scores • Linear equating of State 2 scores onto State 1 scale
Device Comparability • Grade 3 - Total Sample Descriptives N
Math
SD
27,919
298.1
24.77
Mac
9,911
298.1
24.20
iPad
5,806
299.4
24.94
Chromebook
2,487
300.8
23.81
Overall
46,123
298.4
24.63
PC
27,624
302.8
24.10
Mac
9,989
303.6
24.08
iPad
6,000
305.5
24.27
Chromebook
2,560
306.2
21.73
46,173
303.5
23.60
PC ELA
Mean
Overall
Device Comparability • Grade 3 – Matched Sample Descriptives N
ELA
Math
Mean
SD
PC
2,487
300.8
23.81
Mac
2,487
300.8
23.81
iPad
2,487
300.8
23.82
Chromebook
2,487
300.8
23.81
Overall
9,948
300.8
23.81
PC
2,560
306.2
21.73
Mac
2,560
306.2
21.72
iPad
2,560
306.2
21.74
Chromebook
2,560
306.2
21.73
10,240
306.2
21.73
Overall
Device Comparability • Grade 5 - Total Sample Descriptives N
Math
SD
25,649
298.7
25.03
Mac
9,438
298.1
24.91
iPad
6,247
298.6
24.99
Chromebook
3,536
300.4
23.80
Overall
44,870
298.7
24.91
PC
25,830
292.4
24.44
Mac
9,273
292.1
21.86
iPad
6,362
292.3
23.42
Chromebook
3,421
294.5
22.27
44,886
292.5
23.63
PC ELA
Mean
Overall
Device Comparability • Grade 5 - Matched Sample Descriptives N
ELA
SD
PC
3,536
300.4
23.80
Mac
3,536
300.4
23.81
iPad
3,536
300.4
23.78
Chromebook
3,536
300.4
23.80
14,144
300.4
23.79
PC
3,421
294.5
22.27
Mac
3,421
294.5
22.25
iPad
3,421
294.5
22.24
Chromebook
3,421
294.5
22.27
13,684
294.5
22.26
Overall
Math
Mean
Overall
Device Comparability • Grade 8 - Total Sample Descriptives N
Math
SD
25,893
286.5
24.69
Mac
7,005
282.8
24.69
iPad
5,311
285.9
24.38
Chromebook
6,409
287.1
23.90
Overall
44,618
285.9
24.58
PC
26,052
287.0
23.97
Mac
7,240
284.6
19.50
iPad
5,130
282.6
21.95
Chromebook
6,222
286.4
20.64
44,644
286.0
22.67
PC ELA
Mean
Overall
Device Comparability • Grade 8 - Matched Sample Descriptives N
ELA
SD
PC
5,311
285.9
24.38
Mac
5,311
285.8
24.28
iPad
5,311
285.9
24.38
Chromebook
5,311
286.0
24.25
21,244
285.9
24.32
PC
5,130
282.6
21.95
Mac
5,130
283.2
21.37
iPad
5,130
282.6
21.95
Chromebook
5,130
284.0
20.69
20,520
283.12
21.50
Overall
Math
Mean
Overall
Device Comparability • Grade 10 - Total Sample Descriptives N
ELA
Math
Mean
SD
PC
24,797
287.7
24.37
Mac
10,221
285.9
25.54
iPad
2,464
282.5
23.88
Chromebook
5,593
286.5
24.35
Overall
43,075
286.8
24.66
PC
25,037
286.7
23.85
Mac
10,371
285.5
22.58
iPad
2,449
281.2
19.78
Chromebook
5,250
285.2
21.29
43,107
285.9
23.07
Overall
Device Comparability • Grade 10 - Matched Sample Descriptives N
ELA
Math
Mean
SD
PC
2,464
282.5
23.88
Mac
2,464
282.5
23.88
iPad
2,464
282.5
23.88
Chromebook
2,464
282.5
23.86
Overall
9,856
282.5
23.87
PC
2,449
281.2
19.78
Mac
2,449
281.2
19.78
iPad
2,449
281.2
19.78
Chromebook
2,449
281.2
19.76
Overall
9,796
281.2
19.77
Device Comparability - Methods • Question 1 (Overall device comparability) • DIF Methods • PCs compared with iMacs, iPads, and Chromebooks individually • So far, all others as anchors (increases Type I error rate) • IRT-based approach with Wald tests (Langer, 2008) • All others as anchors • Flagging criterion: • Significant total chi-square & Significant parameter chi-square • Alpha = 0.001
Device Comparability - Methods • Question 2 (Device comparability with TTS) • DIF Methods • Same as Q1, ensuring sufficient responses per item • N > 1000 for Grades 3, 5, and 8 • N > 400 for Grade 10
• More liberal alpha threshold (0.05)
Flagged Items – Question 1 (Overall) Grade 3
iMac ELA iPad Chrome iMac Math iPad Chrome
Grade 5
Grade 8
Grade 10
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
0 3 0 1
0 1 0 0
0 2 0 0
0 0 0 0
4 6 6 2
1 1 1 3
0 2 3 0
1 2 2 3
4 3
3 1
5 1
1 1
6 8
2 2
3 1
2 2
Uniform DIF Direction– Question 1 (Overall) iPad vs. PC Comparison
ELA
Math
Favors iPads Favors PCs Overall Favors iPads Favors PCs Overall
Grade 3 2 1 3 2 2 4
Grade 5 1 1 2 2 3 5
Grade 8 2 4 6 1 3 4
Grade 10 0 2 2 0 3 3
Flagged Items – Question 2 (TTS) Grade 3
iMac ELA iPad Chrome iMac Math iPad Chrome
Grade 5
Grade 8
Grade 10
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
6 5 7 2
1 0 2 4
3 2 1 1
1 0 3 1
2 1 0 1
1 1 3 4
1 2 3 0
3 0 2 7
6 5
1 8
6 2
0 1
3 2
3 4
2 2
2 4
Uniform DIF Direction– Question 2 (TTS) iPad vs. PC Comparison
ELA
Math
Favors iPads Favors PCs Overall Favors iPads Favors PCs Overall
Grade 3 2 3 5 3 3 6
Grade 5 0 2 2 3 3 6
Grade 8 0 1 1 1 2 3
Grade 10 2 0 1 1 1 2
Device Comparability – Q1 Item Features Items favoring… iMacs
PCs
Testlet-based items reading passages that require scrolling (2 items)
Items with exponents or small symbols (3 items)
Item with options in 2 columns rather than one (1 item)
ELA
Math
Both
Device Comparability – Q1 Item Features Items favoring… iPads
ELA
Drop-down menu embedded within text passages (3 items)
PCs
Math
Match boxes across columns by clicking each box (3 items)
Constructed response (2 items) Items with exponents or relatively small symbols (4 items)
Both
Testlet-based items reading passages that require scrolling (2 favored iPad; 2 favored PC)
Select text within a reading passage (1 favored iPad; 1 favored PC)
Options are arranged in two columns rather than one (1 favored iPad; 2 favored PC)
Device Comparability – Q1 Item Features Items favoring… Chromebooks
PCs Testlet-based long reading passages with scrolling and MC-MS item (2 items)
ELA
Matrix interaction (1 item)
Constructed response (1 item)
Math
Match boxes across columns by clicking each box (1 item)
Items with exponents or small symbols (4 items)
Items with options in 2 columns rather than 1 (2 items)
Both
Testlet-based long reading passages that require scrolling (1 favored Chromebook; 2 favored PC)
Device Comparability – Q2 Item Features (TTS Sample) Items favoring… PCs
iPads
Both
Testlet-based items with reading passages that require scrolling (4 favored iPad; 3 favored PC)
Matrix interactions (1 favored iPad; 1 favored PC)
ELA
Math
Items with shapes (2 items)
Items with graphs (3 items)
Device Comparability – Next Steps • Cognitive Labs on selected items
• Years 1 and 3 analyses for State 1 • More robust DIF analyses? • Different sampling (differentiate at state level)? • Wald-2 – Purification and anchor selection • Generalized CMH
Overview • Updates on progress of device comparability studies • Phases 1 & 2 – Item flagging • Phase 3 (future work) – Follow-ups with selected items
• Final report will be submitted for inclusion in Center for Assessment compendium
Data Samples • Two state assessment programs • State 1 – Data from 2014 through 2016 • State 2 – 2015 only
• ELA and Math general assessments • Grades 3, 5, 8, and HS • Two-stage test design
Research Questions 1. Are there differences in item performance or overall results by device used? If differences are found in the first year iPads and Chromebooks are introduced, do those same differences still hold in Year 3? 2. Focusing specifically on students using the text-tospeech option in mathematics, are there differences in how students perform on items by device?
Methods • Sample grouped by assessment device • PCs, iMacs, iPads, & Chromebooks
• Sub-samples drawn and matched on provided overall scale scores • Linear equating of State 2 scores onto State 1 scale
Device Comparability • Grade 3 - Total Sample Descriptives N
Math
SD
27,919
298.1
24.77
Mac
9,911
298.1
24.20
iPad
5,806
299.4
24.94
Chromebook
2,487
300.8
23.81
Overall
46,123
298.4
24.63
PC
27,624
302.8
24.10
Mac
9,989
303.6
24.08
iPad
6,000
305.5
24.27
Chromebook
2,560
306.2
21.73
46,173
303.5
23.60
PC ELA
Mean
Overall
Device Comparability • Grade 3 – Matched Sample Descriptives N
ELA
Math
Mean
SD
PC
2,487
300.8
23.81
Mac
2,487
300.8
23.81
iPad
2,487
300.8
23.82
Chromebook
2,487
300.8
23.81
Overall
9,948
300.8
23.81
PC
2,560
306.2
21.73
Mac
2,560
306.2
21.72
iPad
2,560
306.2
21.74
Chromebook
2,560
306.2
21.73
10,240
306.2
21.73
Overall
Device Comparability • Grade 5 - Total Sample Descriptives N
Math
SD
25,649
298.7
25.03
Mac
9,438
298.1
24.91
iPad
6,247
298.6
24.99
Chromebook
3,536
300.4
23.80
Overall
44,870
298.7
24.91
PC
25,830
292.4
24.44
Mac
9,273
292.1
21.86
iPad
6,362
292.3
23.42
Chromebook
3,421
294.5
22.27
44,886
292.5
23.63
PC ELA
Mean
Overall
Device Comparability • Grade 5 - Matched Sample Descriptives N
ELA
SD
PC
3,536
300.4
23.80
Mac
3,536
300.4
23.81
iPad
3,536
300.4
23.78
Chromebook
3,536
300.4
23.80
14,144
300.4
23.79
PC
3,421
294.5
22.27
Mac
3,421
294.5
22.25
iPad
3,421
294.5
22.24
Chromebook
3,421
294.5
22.27
13,684
294.5
22.26
Overall
Math
Mean
Overall
Device Comparability • Grade 8 - Total Sample Descriptives N
Math
SD
25,893
286.5
24.69
Mac
7,005
282.8
24.69
iPad
5,311
285.9
24.38
Chromebook
6,409
287.1
23.90
Overall
44,618
285.9
24.58
PC
26,052
287.0
23.97
Mac
7,240
284.6
19.50
iPad
5,130
282.6
21.95
Chromebook
6,222
286.4
20.64
44,644
286.0
22.67
PC ELA
Mean
Overall
Device Comparability • Grade 8 - Matched Sample Descriptives N
ELA
SD
PC
5,311
285.9
24.38
Mac
5,311
285.8
24.28
iPad
5,311
285.9
24.38
Chromebook
5,311
286.0
24.25
21,244
285.9
24.32
PC
5,130
282.6
21.95
Mac
5,130
283.2
21.37
iPad
5,130
282.6
21.95
Chromebook
5,130
284.0
20.69
20,520
283.12
21.50
Overall
Math
Mean
Overall
Device Comparability • Grade 10 - Total Sample Descriptives N
ELA
Math
Mean
SD
PC
24,797
287.7
24.37
Mac
10,221
285.9
25.54
iPad
2,464
282.5
23.88
Chromebook
5,593
286.5
24.35
Overall
43,075
286.8
24.66
PC
25,037
286.7
23.85
Mac
10,371
285.5
22.58
iPad
2,449
281.2
19.78
Chromebook
5,250
285.2
21.29
43,107
285.9
23.07
Overall
Device Comparability • Grade 10 - Matched Sample Descriptives N
ELA
Math
Mean
SD
PC
2,464
282.5
23.88
Mac
2,464
282.5
23.88
iPad
2,464
282.5
23.88
Chromebook
2,464
282.5
23.86
Overall
9,856
282.5
23.87
PC
2,449
281.2
19.78
Mac
2,449
281.2
19.78
iPad
2,449
281.2
19.78
Chromebook
2,449
281.2
19.76
Overall
9,796
281.2
19.77
Device Comparability - Methods • Question 1 (Overall device comparability) • DIF Methods • PCs compared with iMacs, iPads, and Chromebooks individually • So far, all others as anchors (increases Type I error rate) • IRT-based approach with Wald tests (Langer, 2008) • All others as anchors • Flagging criterion: • Significant total chi-square & Significant parameter chi-square • Alpha = 0.001
Device Comparability - Methods • Question 2 (Device comparability with TTS) • DIF Methods • Same as Q1, ensuring sufficient responses per item • N > 1000 for Grades 3, 5, and 8 • N > 400 for Grade 10
• More liberal alpha threshold (0.05)
Flagged Items – Question 1 (Overall) Grade 3
iMac ELA iPad Chrome iMac Math iPad Chrome
Grade 5
Grade 8
Grade 10
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
0 3 0 1
0 1 0 0
0 2 0 0
0 0 0 0
4 6 6 2
1 1 1 3
0 2 3 0
1 2 2 3
4 3
3 1
5 1
1 1
6 8
2 2
3 1
2 2
Uniform DIF Direction– Question 1 (Overall) iPad vs. PC Comparison
ELA
Math
Favors iPads Favors PCs Overall Favors iPads Favors PCs Overall
Grade 3 2 1 3 2 2 4
Grade 5 1 1 2 2 3 5
Grade 8 2 4 6 1 3 4
Grade 10 0 2 2 0 3 3
Flagged Items – Question 2 (TTS) Grade 3
iMac ELA iPad Chrome iMac Math iPad Chrome
Grade 5
Grade 8
Grade 10
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
Uni
Nonuni
6 5 7 2
1 0 2 4
3 2 1 1
1 0 3 1
2 1 0 1
1 1 3 4
1 2 3 0
3 0 2 7
6 5
1 8
6 2
0 1
3 2
3 4
2 2
2 4
Uniform DIF Direction– Question 2 (TTS) iPad vs. PC Comparison
ELA
Math
Favors iPads Favors PCs Overall Favors iPads Favors PCs Overall
Grade 3 2 3 5 3 3 6
Grade 5 0 2 2 3 3 6
Grade 8 0 1 1 1 2 3
Grade 10 2 0 1 1 1 2
Device Comparability – Q1 Item Features Items favoring… iMacs
PCs
Testlet-based items reading passages that require scrolling (2 items)
Items with exponents or small symbols (3 items)
Item with options in 2 columns rather than one (1 item)
ELA
Math
Both
Device Comparability – Q1 Item Features Items favoring… iPads
ELA
Drop-down menu embedded within text passages (3 items)
PCs
Math
Match boxes across columns by clicking each box (3 items)
Constructed response (2 items) Items with exponents or relatively small symbols (4 items)
Both
Testlet-based items reading passages that require scrolling (2 favored iPad; 2 favored PC)
Select text within a reading passage (1 favored iPad; 1 favored PC)
Options are arranged in two columns rather than one (1 favored iPad; 2 favored PC)
Device Comparability – Q1 Item Features Items favoring… Chromebooks
PCs Testlet-based long reading passages with scrolling and MC-MS item (2 items)
ELA
Matrix interaction (1 item)
Constructed response (1 item)
Math
Match boxes across columns by clicking each box (1 item)
Items with exponents or small symbols (4 items)
Items with options in 2 columns rather than 1 (2 items)
Both
Testlet-based long reading passages that require scrolling (1 favored Chromebook; 2 favored PC)
Device Comparability – Q2 Item Features (TTS Sample) Items favoring… PCs
iPads
Both
Testlet-based items with reading passages that require scrolling (4 favored iPad; 3 favored PC)
Matrix interactions (1 favored iPad; 1 favored PC)
ELA
Math
Items with shapes (2 items)
Items with graphs (3 items)
Device Comparability – Next Steps • Cognitive Labs on selected items
• Years 1 and 3 analyses for State 1 • More robust DIF analyses? • Different sampling (differentiate at state level)? • Wald-2 – Purification and anchor selection • Generalized CMH
