Items
Data Analyses to Support SCASS work on Device Comparability and TE Item Validity Cognitive Labs
Gail Tiemann, PhD University of Kansas
Purpose To explore how student interactions with computer-based assessment items differ by device
Framework Technology-Enhanced Item Utility Framework – (Russell, 2016) o Interaction space – Area where a student marks an answer o Usability – Intuitiveness - easy to manipulate – Functionality – minimize number of clicks required to answer – Layout – minimize space between objects, yet not cause confusion
Items 2014-2015 data from two states Items flagged using device DIF procedure Math and ELA items at 3rd, 5th, 8th, 10th grades Secure passages and secure, standard MC items removed from item pool Text-to-speech items not used
Forms by Grade Grade
Content Area
Item Count
Item Types
3
ELA
2
Dropdown, Multiple dropdown
3
Math
8
MC, MCMS, Matrix
5
ELA
2
Passage + MC, MCMS
5
Math
6
Matrix, MC, Short constructed response, MCMS
Forms by Grade Grade
Content Area
Item Count
Item Types
8
ELA
10
2 Passages + MC & MCMS Select text, MC
8
Math
8
MC, MCMS, Matrix
10
ELA
1
MC
10
Math
2
MC, *MCMS (Item dropped due to difficulty)
Item-Type Example
3RD Grade ELA – Multiple Dropdowns
Students To reduce exposure, 8th & 10th grade items presented to 9th grade students 12 students at each grade Volunteer principals at two buildings, one elementary and one high school; 3 home visits
Students selected by school staff to represent a variety of proficiency levels
Devices Chromebook o ~11.5 inch screen iPad o ~9.5 inch screen Laptop o 13 inch screen, running Mac OS 10.11 4 students observed per device
Procedures Student completed items on a single device Device was rotated with each new student Each student practiced thinking aloud Interviews were tape-recorded Screen activity was noted by observer on data collection forms Each student was interviewed for further discussion of any difficulties
Analysis All written observations were compiled and grouped by device type Observations were then reviewed within and across device types, with patterns noted based on the research framework
Results – Interaction Spaces Amount of viewable screen space accessible without scrolling varied by device Longer MCMS, 3x2, or 2x2 layouts – Laptop typically displayed more of the answer choices in default viewing area (before scrolling) Interaction space features consistent across devices (navigation, tools, buttons, etc.)
Results – Interaction Spaces Scrolling o iPad - No difficulties finger-scrolling o Chromebook - No difficulties scrolling – Primarily click and drag of scrollbars or use of up/down arrows o Laptop (Mac) - Some students needed help scrolling, but scrolled easily after one prompt –Primarily touchpad, 2-finger scrolling
Results – Intuitive Interactions Vast majority of students easily manipulated all item types; did not vary by device o Matrix item – three Grade 3 students attempted to drag and drop or needed help to begin – One Grade 5 student became stuck but reasoned how to proceed independently
o Matching item – One 9th grade student needed help – One 9th grade student became stuck but reasoned how to proceed independently
Results – Intuitive Interactions Zero students clicked the onscreen help even though help was mentioned in the directions
Results - Functionality Item functionality was equivalent among devices o Touches on an iPad mirrored mouse clicks on the Chromebook or Laptop o Matching and dropdown items required two clicks per answer choice –Minimal difficulties with matching –No difficulties with dropdowns
Results – Layout
3x2 MCMS Answer Choice Layout With Image in Each AC
3x2 MCMS Layout Simulation Interaction Space
Item stem here. Item stem here. Item stem here. Choose all that apply. Image
Image
Image
Image
Default Viewing Area Boundary
Image Not Visible without Scrolling
Results - Layout 3x2 Answer Choices with Images 3rd Grade Item o Chromebook and iPad – only 2x2 in view without scrolling –Zero students scrolled to see lower left, correct answer choice o Laptop displayed 3x2 choices o Data showed uniform DIF favoring PC over iPad and Chromebook
Results - Layout 3x2 Answer Choices with Images 10th Grade Item o All devices displayed 2x2 in default viewing area. o 10/12 students scrolled down to lower left, correct answer choice. o Data showed uniform DIF favoring PC over iPad
Results – Layout MCMS with Long List of Answer Choices
3rd Grade Item o No device pattern to scrolling or not scrolling. o Six students saw all of the ACs but chose to select only one answer o Data showed uniform DIF favoring PC over Chromebook
Results – Layout MCMS with Long List of Answer Choices
5th Grade Item o No device pattern to scrolling or not scrolling o Data showed uniform DIF favoring PC over iPad
Results – Layout o No differences among devices were noted on any of the remaining items
Results – Follow-up Interviews Did you have any difficulties getting around in the test? o Vast majority of students said no. o Two students mentioned being less familiar with scrolling on a mac (3rd) o One student mentioned trying to drag and drop on a matrix item (3rd) o Another student mentioned having to redo a problem if their answer wasn’t on the screen (3rd) o Another student mentioned being irritated by long reading passages (9th)
Results – Follow-up Interviews Once you had the answer in your head, how easy or hard was it to get your answer into the computer / to show on the computer screen?
o Vast majority of students said it was easy. o “Some of the questions were hard but using a computer was not hard.” (3rd) o “No it seemed just fine. Just click it or type it in.” (5th)
Results – Follow-up Interviews Three students mentioned gaining experience with computer-based testing over time. o “On matching, in the past I tried dragging. …Other then that, I know what to do.” (9th) o “Not really. In the past it has been like that but I think it has changed. Maybe b/c I'm older.” (9th) o “I like to play with it first before I put in answer so I know what I need to do” (on the screen). (9th)
Results – Follow-up Interviews What item types did you like answering the most? (Observer mentioned different types for reference.) o 3rd graders liked math o 3rd graders didn’t distinguish from the content of the question and the action required to answer
Results – Follow-up Interviews What item types did you like answering the most? (Observer mentioned different types for reference.) o 5th grade –More than one favorite – 4 –MC – 4 –MCMS – 1 –Short constructed response – 2 –Don’t know - 1
Results – Follow-up Interviews What item types did you like answering the most? (Observer mentioned different types for reference.) o 9th grade –They liked matching! –Matching and something else - 4 –Matching - 3 –Select text - 2 –MC - 3
Results – Follow-up Interviews A few students had suggestions: o “I wish the codes (passwords) could be the same every year.” (5th) o "When I used the guideline, and went to scroll down, the guideline followed my finger and I lost my place. Maybe I could click it and it could stay there.” (5th) o “The only thing was the striker was really complicated to understand.” (5th)
Limitations Volunteer sample Third graders had little pre-exposure to the test delivery engine Students had considerable technology exposure overall o 5th graders 1:1 iPads for school use o 9th graders 1:1 iPads for school & home use o All but one student had computers / devices at home. This student used the computer at “his mom’s shop.”
Discussion 9th Graders were very proficient with scrolling and the testing interaction space 3rd graders were more likely to select only one answer on an MCMS item Students found entering answers to be easy. 3x2 MCMS layouts containing images were problematic at 3rd grade
Further Research 3x2 MCMS layouts containing images and long MCMS items should be further explored through device-based distractor analysis, especially at lower grades 3x2 MCMS layout analysis could include a look at nesting in schools/classrooms; detection of which students received training / have higher levels of testwiseness
Gail Tiemann, PhD University of Kansas
Purpose To explore how student interactions with computer-based assessment items differ by device
Framework Technology-Enhanced Item Utility Framework – (Russell, 2016) o Interaction space – Area where a student marks an answer o Usability – Intuitiveness - easy to manipulate – Functionality – minimize number of clicks required to answer – Layout – minimize space between objects, yet not cause confusion
Items 2014-2015 data from two states Items flagged using device DIF procedure Math and ELA items at 3rd, 5th, 8th, 10th grades Secure passages and secure, standard MC items removed from item pool Text-to-speech items not used
Forms by Grade Grade
Content Area
Item Count
Item Types
3
ELA
2
Dropdown, Multiple dropdown
3
Math
8
MC, MCMS, Matrix
5
ELA
2
Passage + MC, MCMS
5
Math
6
Matrix, MC, Short constructed response, MCMS
Forms by Grade Grade
Content Area
Item Count
Item Types
8
ELA
10
2 Passages + MC & MCMS Select text, MC
8
Math
8
MC, MCMS, Matrix
10
ELA
1
MC
10
Math
2
MC, *MCMS (Item dropped due to difficulty)
Item-Type Example
3RD Grade ELA – Multiple Dropdowns
Students To reduce exposure, 8th & 10th grade items presented to 9th grade students 12 students at each grade Volunteer principals at two buildings, one elementary and one high school; 3 home visits
Students selected by school staff to represent a variety of proficiency levels
Devices Chromebook o ~11.5 inch screen iPad o ~9.5 inch screen Laptop o 13 inch screen, running Mac OS 10.11 4 students observed per device
Procedures Student completed items on a single device Device was rotated with each new student Each student practiced thinking aloud Interviews were tape-recorded Screen activity was noted by observer on data collection forms Each student was interviewed for further discussion of any difficulties
Analysis All written observations were compiled and grouped by device type Observations were then reviewed within and across device types, with patterns noted based on the research framework
Results – Interaction Spaces Amount of viewable screen space accessible without scrolling varied by device Longer MCMS, 3x2, or 2x2 layouts – Laptop typically displayed more of the answer choices in default viewing area (before scrolling) Interaction space features consistent across devices (navigation, tools, buttons, etc.)
Results – Interaction Spaces Scrolling o iPad - No difficulties finger-scrolling o Chromebook - No difficulties scrolling – Primarily click and drag of scrollbars or use of up/down arrows o Laptop (Mac) - Some students needed help scrolling, but scrolled easily after one prompt –Primarily touchpad, 2-finger scrolling
Results – Intuitive Interactions Vast majority of students easily manipulated all item types; did not vary by device o Matrix item – three Grade 3 students attempted to drag and drop or needed help to begin – One Grade 5 student became stuck but reasoned how to proceed independently
o Matching item – One 9th grade student needed help – One 9th grade student became stuck but reasoned how to proceed independently
Results – Intuitive Interactions Zero students clicked the onscreen help even though help was mentioned in the directions
Results - Functionality Item functionality was equivalent among devices o Touches on an iPad mirrored mouse clicks on the Chromebook or Laptop o Matching and dropdown items required two clicks per answer choice –Minimal difficulties with matching –No difficulties with dropdowns
Results – Layout
3x2 MCMS Answer Choice Layout With Image in Each AC
3x2 MCMS Layout Simulation Interaction Space
Item stem here. Item stem here. Item stem here. Choose all that apply. Image
Image
Image
Image
Default Viewing Area Boundary
Image Not Visible without Scrolling
Results - Layout 3x2 Answer Choices with Images 3rd Grade Item o Chromebook and iPad – only 2x2 in view without scrolling –Zero students scrolled to see lower left, correct answer choice o Laptop displayed 3x2 choices o Data showed uniform DIF favoring PC over iPad and Chromebook
Results - Layout 3x2 Answer Choices with Images 10th Grade Item o All devices displayed 2x2 in default viewing area. o 10/12 students scrolled down to lower left, correct answer choice. o Data showed uniform DIF favoring PC over iPad
Results – Layout MCMS with Long List of Answer Choices
3rd Grade Item o No device pattern to scrolling or not scrolling. o Six students saw all of the ACs but chose to select only one answer o Data showed uniform DIF favoring PC over Chromebook
Results – Layout MCMS with Long List of Answer Choices
5th Grade Item o No device pattern to scrolling or not scrolling o Data showed uniform DIF favoring PC over iPad
Results – Layout o No differences among devices were noted on any of the remaining items
Results – Follow-up Interviews Did you have any difficulties getting around in the test? o Vast majority of students said no. o Two students mentioned being less familiar with scrolling on a mac (3rd) o One student mentioned trying to drag and drop on a matrix item (3rd) o Another student mentioned having to redo a problem if their answer wasn’t on the screen (3rd) o Another student mentioned being irritated by long reading passages (9th)
Results – Follow-up Interviews Once you had the answer in your head, how easy or hard was it to get your answer into the computer / to show on the computer screen?
o Vast majority of students said it was easy. o “Some of the questions were hard but using a computer was not hard.” (3rd) o “No it seemed just fine. Just click it or type it in.” (5th)
Results – Follow-up Interviews Three students mentioned gaining experience with computer-based testing over time. o “On matching, in the past I tried dragging. …Other then that, I know what to do.” (9th) o “Not really. In the past it has been like that but I think it has changed. Maybe b/c I'm older.” (9th) o “I like to play with it first before I put in answer so I know what I need to do” (on the screen). (9th)
Results – Follow-up Interviews What item types did you like answering the most? (Observer mentioned different types for reference.) o 3rd graders liked math o 3rd graders didn’t distinguish from the content of the question and the action required to answer
Results – Follow-up Interviews What item types did you like answering the most? (Observer mentioned different types for reference.) o 5th grade –More than one favorite – 4 –MC – 4 –MCMS – 1 –Short constructed response – 2 –Don’t know - 1
Results – Follow-up Interviews What item types did you like answering the most? (Observer mentioned different types for reference.) o 9th grade –They liked matching! –Matching and something else - 4 –Matching - 3 –Select text - 2 –MC - 3
Results – Follow-up Interviews A few students had suggestions: o “I wish the codes (passwords) could be the same every year.” (5th) o "When I used the guideline, and went to scroll down, the guideline followed my finger and I lost my place. Maybe I could click it and it could stay there.” (5th) o “The only thing was the striker was really complicated to understand.” (5th)
Limitations Volunteer sample Third graders had little pre-exposure to the test delivery engine Students had considerable technology exposure overall o 5th graders 1:1 iPads for school use o 9th graders 1:1 iPads for school & home use o All but one student had computers / devices at home. This student used the computer at “his mom’s shop.”
Discussion 9th Graders were very proficient with scrolling and the testing interaction space 3rd graders were more likely to select only one answer on an MCMS item Students found entering answers to be easy. 3x2 MCMS layouts containing images were problematic at 3rd grade
Further Research 3x2 MCMS layouts containing images and long MCMS items should be further explored through device-based distractor analysis, especially at lower grades 3x2 MCMS layout analysis could include a look at nesting in schools/classrooms; detection of which students received training / have higher levels of testwiseness
