Writing Webinar Sept 2016
Arkansas Department of Education
ASPIRE WRITING DATA – REPORTING CATEGORIES RESOURCES Student Assessment JJ Morley Walker [email protected]
September 2016
Today’s Focus • Writing Overview • Modes per grade • Writing prompts – development • Reporting categories • Domains vs. reporting categories • Rubrics • Score Goal • Scoring • Scores • National score average per grade vs. state score average per grade • State reporting category per grade • Score Reports • Skill Proficiency by Subject – School Report • Skill Proficiency by Teacher – Educator Report • Student Response File – (SPF)
• Resources • What do I do now? • Up next in the series!
ACT Aspire Writing: Overview Reporting Categories and Tasks
Writing Modes
Reflective Narrative
Writing Tasks
Grades per Mode 3 and 6
Analytical Expository
4, 7, 9/10
Persuasive Argumentative
5 and 8
3
Writing Prompt Development • 3 parts of a writing prompt • Rhetorical situation • Task/idea is introduced within a rhetorical situation
• Scaffolding • Designed to align with rubric (IAA), (DAS), (ORG), • Helps students generate ideas/brainstorm • Not a checklist – only suggestions • Decreases as the grade band increases
• Task • Exact task is revealed • Key words within task should help writer to develop the writing task
(describes) (learned) (explained) • At this time, for this year the testing time will remain at 30 minutes.
Grade 3 Prompt: Reflective Narrative Exemplar (Released to Public)
Rhetorical situation is clearly defined at the start
Scaffolding bullets align with the rubric, enable students to: • generate ideas for a story, reflect on an experience. (IAA) • Move from specific to generalized ideas in the last two bullets. (IAA) • develop the story actions, details, thoughts, and feelings. (DAS) • Order of bullets suggests organization of ideas. (ORG) Failure to address each bulleted question will not negatively impact score.
The task: • Story conveys organizational expectation (ORG) – as opposed to, e.g., response, or list. • Emphasis on describe/learned enables the student to develop the story. (DAS) • Student is encouraged to move from specific to generalized 5 ideas while also reflecting on the meaning of the experience. (IAA)
Grade 4 Prompt: Expository Analytical Exemplar (Released to Public)
Rhetorical situation is clearly defined at the start.
Scaffolding bullets align with the rubric, enable students to: • generate ideas and move from specific (your favorite movie) to generalized ideas (a good movie). (IAA) • develop ideas with details or reasons. (DAS) • organize ideas by moving through the bullets in order. (ORG) Failure to address each bulleted question will not negatively impact score.
The task: • Emphasis on describe and explain, enable student to develop the essay, as does a reminder to include reasons and details (DAS) • Student is encouraged to move from specific to 6 generalized ideas (IAA).
Grade 5 Prompt: Persuasive Argumentative Exemplar (Released to Public)
Rhetorical situation is clearly defined first The rhetorical situation is concrete: a local library, a choice between two resources ScaffoldingAligned with rubric: to learn about the past enables students to address implications and complications of their choice (IAA trait) The task gives the student a specific directive aligned to rubric: • reasons for the choice • Explain why … encourages student to address counterargument 7 • Include details … IAA and DAS traits in the rubric
Grade 8 Prompt: Persuasive Argumentative Exemplar (Released to Public)
The rhetorical situation invites the student to resolve a dilemma in light of various considerations Bullets offer scaffolding to help the student demonstrate competency in writing through addressing implications/ complications, or arguments/ counterarguments
The task provides the specific writing directive: • Words like take a position, persuasive essay, and argue for your position clearly convey the writing purpose and mode • Reasons for your position and examples to support your reasons line 8 up with the rubric (IAA and DAS)
Reporting Categories per Mode Reporting Categories
Ideas and Analysis 4 Reporting Categories for all Writing Modes
Development and Support Organization Language Use and Conventions
ACT Aspire Writing Domains – Reporting Categories Domains – Rubrics • Reflective Narrative/
Analytical/Persuasive
Reporting Categories –Reports • Ideas and Analysis Reflective Narrative Analytical Expository Persuasive Argumentative
• Development
• Development and Support
• Organization
• Organization
• Language Use
• Language Use/Conventions
Good things to know !! • Rubrics – Analytically vs. Holistically • Each domain is scored separately (analytic), but each point in each domain is not a checklist (holistic). • Remember – A score point of 4 across the rubric domains is the
goal. • Use the new Performance Level Descriptors to gain a deeper
understanding of the rubrics. • Scored sample papers/essays at all score points can be found in
the exemplar Test Question Booklets (pdf) for each grade. Go to the exemplar page for writing and scroll to the bottom.
Scoring • ACT Aspire follows standard industry practices of: • Hiring qualified people to score • Training scorers • 60 training sets • Blind validity papers included
• All questionable papers (off topic, etc.) are sent to the scoring
leader for a final decision. • All papers are rescored if a scorer is dismissed. • No 3rd grade responses were AI scored – the AI engine could
not be trained • Humans are involved in AI scoring – 10% read behind
AI Scoring AI Scoring - ACT Aspire uses a machine learning approach in which the engine is trained to score based on the collective wisdom of trained human scorers… the engine learns the different features that human scorers evaluate. (Technical Manual 4.4)
• AI Scoring Research: • Automated vs. Human Essay Scoring • Grading Essays: Humans vs. Machine • Contrasted Automated and Human Scoring of Essays
Arkansas Writing Scores National vs. State - per grade Reporting Categories per grade
Writing Scores Percentage of Students Scoring Ready and Exceeding Grade
National
State
3rd
16%
20%
+4
4th
19%
13.5%
- 5.5
5th
26%
20%
-6
6th
42%
44%
+2
7th
26%
28.5%
+2.5
8th
26%
23%
-3
9th
35%
44%
+9
10th
45%
52%
+7
Blue scores indicate Arkansas scores that are higher than the national average.
Percentages by Mode Ready and Exceeding Mode Reflective Narrative
Analytical Expository
Persuasive Argumentative
National
Arkansas
3rd
16%
3rd
20%
6th
42%
6th
44%
4th
19%
4th
13.5%
7th
26%
7th
28.5%
9th
35%
9th
44%
10th
44%
10th
52%
5th
26%
5th
20%
8th
26%
8th
23%
Blue scores indicate Arkansas scores that are higher than the national average.
Writing Reporting Categories/Grade/Mode Percentage of Students Scoring 4 or Better
Writing Reporting Categories/Grade/Mode Percentage of Students Scoring 4 or Better
Writing Reporting Categories Percentage of Students Scoring 4 or Better
Writing Reporting Categories Percentage of Students Scoring 4 or Better
Score Reports School Report • Skill Proficiency by Subject (by grade)
Educator Report • Skill Proficiency by Teacher (by grade) These reports will rank the reporting categories for writing highest to lowest.
Skill Proficiency by Subject – School Report
Reporting Categories – highest to lowest Students are ranked by scale score – look at the scores of 427/426 Below Readiness Range – BUT WHAT DID THEY SCORE? 1? 2? 3?
Skill Proficiency by Subject –
Dynamic Educator Report - Will not be available after 9/22 COB
Reporting Categories – highest to lowest Students are ranked by scale score – look at the scores of 427/426 Below Readiness Range – BUT WHAT DID THEY SCORE? 1? 2? 3?
Questions to ask yourself ????? • How can I know what each of my students made on each
reporting category? • How many students in my class scored 3s? 4s? 5s? 6s? • Did any of my students score 1s? 2s? • How many of my students were at the very top of the
Close category in the scaled score column ?
Student Performance File –
Administrator permissions only !
1. Filter by grade level – Column N
2. Filter by Writing Scale Score – Largest to Smallest – Column HR
3. Filter each of the reporting categories (one at a time) to analyze specific data - Largest to Smallest - Columns IC, IG, IK, IO
30
ACT Aspire Writing Resources • ACT Aspire Landing Page • Exemplars
• TestNav and Test Question Booklets (pdf format) – scored sample papers/grade • Avocet and Training Management System > Visit TMS
• Reporting and Data Usage
• ADE - Student Assessment LiveBinder >ACT Aspire
LiveBinder • > Technical Manual – Writing Test – sections 3.11- 3.13 • > Technical Manual – Scoring – sections 4.2, 4.4 • >Technical Manual – Rubrics – Appendices F.16-F.28
• ACT Aspire Score Reporting LiveBinder >Types of Reports • Performance Level Descriptors – see next slide
Performance Level Descriptors - Writing
Ready Level Goal: Scale Score – 428 and above Per Reporting Category – Score point 4 and above
What do I do now? • First and foremost • Continue teaching the Arkansas English Language Arts Standards ! • Teach all modes of writing every single year. • Write across the curriculum. • Writing is not just an ELA teacher’s responsibility. • Students should be writing in every content area. • DATA – • Look at the SPF file and note the students that were just below the 428 benchmark. • Look at each grade’s individual reporting categories and analyze the data. What is it telling you? • Share this webinar with colleagues – will be posted on the
Aspire LiveBinder under Writing Webinar tab.
Coming Soon!!!! Up next in our webinar series! • Watch for a Commissioner’s Memo ! • ADE Literacy Content Team will be developing
instructional strategies and resources for writing. • Please note – these will be added to the ADE site and the Aspire LiveBinder > Writing Webinar tab. Who can I call with questions? Data Analysis – Student Assessment office – 501-682-4558 Curriculum/Standards – 501-682-7442
ASPIRE WRITING DATA – REPORTING CATEGORIES RESOURCES Student Assessment JJ Morley Walker [email protected]
September 2016
Today’s Focus • Writing Overview • Modes per grade • Writing prompts – development • Reporting categories • Domains vs. reporting categories • Rubrics • Score Goal • Scoring • Scores • National score average per grade vs. state score average per grade • State reporting category per grade • Score Reports • Skill Proficiency by Subject – School Report • Skill Proficiency by Teacher – Educator Report • Student Response File – (SPF)
• Resources • What do I do now? • Up next in the series!
ACT Aspire Writing: Overview Reporting Categories and Tasks
Writing Modes
Reflective Narrative
Writing Tasks
Grades per Mode 3 and 6
Analytical Expository
4, 7, 9/10
Persuasive Argumentative
5 and 8
3
Writing Prompt Development • 3 parts of a writing prompt • Rhetorical situation • Task/idea is introduced within a rhetorical situation
• Scaffolding • Designed to align with rubric (IAA), (DAS), (ORG), • Helps students generate ideas/brainstorm • Not a checklist – only suggestions • Decreases as the grade band increases
• Task • Exact task is revealed • Key words within task should help writer to develop the writing task
(describes) (learned) (explained) • At this time, for this year the testing time will remain at 30 minutes.
Grade 3 Prompt: Reflective Narrative Exemplar (Released to Public)
Rhetorical situation is clearly defined at the start
Scaffolding bullets align with the rubric, enable students to: • generate ideas for a story, reflect on an experience. (IAA) • Move from specific to generalized ideas in the last two bullets. (IAA) • develop the story actions, details, thoughts, and feelings. (DAS) • Order of bullets suggests organization of ideas. (ORG) Failure to address each bulleted question will not negatively impact score.
The task: • Story conveys organizational expectation (ORG) – as opposed to, e.g., response, or list. • Emphasis on describe/learned enables the student to develop the story. (DAS) • Student is encouraged to move from specific to generalized 5 ideas while also reflecting on the meaning of the experience. (IAA)
Grade 4 Prompt: Expository Analytical Exemplar (Released to Public)
Rhetorical situation is clearly defined at the start.
Scaffolding bullets align with the rubric, enable students to: • generate ideas and move from specific (your favorite movie) to generalized ideas (a good movie). (IAA) • develop ideas with details or reasons. (DAS) • organize ideas by moving through the bullets in order. (ORG) Failure to address each bulleted question will not negatively impact score.
The task: • Emphasis on describe and explain, enable student to develop the essay, as does a reminder to include reasons and details (DAS) • Student is encouraged to move from specific to 6 generalized ideas (IAA).
Grade 5 Prompt: Persuasive Argumentative Exemplar (Released to Public)
Rhetorical situation is clearly defined first The rhetorical situation is concrete: a local library, a choice between two resources ScaffoldingAligned with rubric: to learn about the past enables students to address implications and complications of their choice (IAA trait) The task gives the student a specific directive aligned to rubric: • reasons for the choice • Explain why … encourages student to address counterargument 7 • Include details … IAA and DAS traits in the rubric
Grade 8 Prompt: Persuasive Argumentative Exemplar (Released to Public)
The rhetorical situation invites the student to resolve a dilemma in light of various considerations Bullets offer scaffolding to help the student demonstrate competency in writing through addressing implications/ complications, or arguments/ counterarguments
The task provides the specific writing directive: • Words like take a position, persuasive essay, and argue for your position clearly convey the writing purpose and mode • Reasons for your position and examples to support your reasons line 8 up with the rubric (IAA and DAS)
Reporting Categories per Mode Reporting Categories
Ideas and Analysis 4 Reporting Categories for all Writing Modes
Development and Support Organization Language Use and Conventions
ACT Aspire Writing Domains – Reporting Categories Domains – Rubrics • Reflective Narrative/
Analytical/Persuasive
Reporting Categories –Reports • Ideas and Analysis Reflective Narrative Analytical Expository Persuasive Argumentative
• Development
• Development and Support
• Organization
• Organization
• Language Use
• Language Use/Conventions
Good things to know !! • Rubrics – Analytically vs. Holistically • Each domain is scored separately (analytic), but each point in each domain is not a checklist (holistic). • Remember – A score point of 4 across the rubric domains is the
goal. • Use the new Performance Level Descriptors to gain a deeper
understanding of the rubrics. • Scored sample papers/essays at all score points can be found in
the exemplar Test Question Booklets (pdf) for each grade. Go to the exemplar page for writing and scroll to the bottom.
Scoring • ACT Aspire follows standard industry practices of: • Hiring qualified people to score • Training scorers • 60 training sets • Blind validity papers included
• All questionable papers (off topic, etc.) are sent to the scoring
leader for a final decision. • All papers are rescored if a scorer is dismissed. • No 3rd grade responses were AI scored – the AI engine could
not be trained • Humans are involved in AI scoring – 10% read behind
AI Scoring AI Scoring - ACT Aspire uses a machine learning approach in which the engine is trained to score based on the collective wisdom of trained human scorers… the engine learns the different features that human scorers evaluate. (Technical Manual 4.4)
• AI Scoring Research: • Automated vs. Human Essay Scoring • Grading Essays: Humans vs. Machine • Contrasted Automated and Human Scoring of Essays
Arkansas Writing Scores National vs. State - per grade Reporting Categories per grade
Writing Scores Percentage of Students Scoring Ready and Exceeding Grade
National
State
3rd
16%
20%
+4
4th
19%
13.5%
- 5.5
5th
26%
20%
-6
6th
42%
44%
+2
7th
26%
28.5%
+2.5
8th
26%
23%
-3
9th
35%
44%
+9
10th
45%
52%
+7
Blue scores indicate Arkansas scores that are higher than the national average.
Percentages by Mode Ready and Exceeding Mode Reflective Narrative
Analytical Expository
Persuasive Argumentative
National
Arkansas
3rd
16%
3rd
20%
6th
42%
6th
44%
4th
19%
4th
13.5%
7th
26%
7th
28.5%
9th
35%
9th
44%
10th
44%
10th
52%
5th
26%
5th
20%
8th
26%
8th
23%
Blue scores indicate Arkansas scores that are higher than the national average.
Writing Reporting Categories/Grade/Mode Percentage of Students Scoring 4 or Better
Writing Reporting Categories/Grade/Mode Percentage of Students Scoring 4 or Better
Writing Reporting Categories Percentage of Students Scoring 4 or Better
Writing Reporting Categories Percentage of Students Scoring 4 or Better
Score Reports School Report • Skill Proficiency by Subject (by grade)
Educator Report • Skill Proficiency by Teacher (by grade) These reports will rank the reporting categories for writing highest to lowest.
Skill Proficiency by Subject – School Report
Reporting Categories – highest to lowest Students are ranked by scale score – look at the scores of 427/426 Below Readiness Range – BUT WHAT DID THEY SCORE? 1? 2? 3?
Skill Proficiency by Subject –
Dynamic Educator Report - Will not be available after 9/22 COB
Reporting Categories – highest to lowest Students are ranked by scale score – look at the scores of 427/426 Below Readiness Range – BUT WHAT DID THEY SCORE? 1? 2? 3?
Questions to ask yourself ????? • How can I know what each of my students made on each
reporting category? • How many students in my class scored 3s? 4s? 5s? 6s? • Did any of my students score 1s? 2s? • How many of my students were at the very top of the
Close category in the scaled score column ?
Student Performance File –
Administrator permissions only !
1. Filter by grade level – Column N
2. Filter by Writing Scale Score – Largest to Smallest – Column HR
3. Filter each of the reporting categories (one at a time) to analyze specific data - Largest to Smallest - Columns IC, IG, IK, IO
30
ACT Aspire Writing Resources • ACT Aspire Landing Page • Exemplars
• TestNav and Test Question Booklets (pdf format) – scored sample papers/grade • Avocet and Training Management System > Visit TMS
• Reporting and Data Usage
• ADE - Student Assessment LiveBinder >ACT Aspire
LiveBinder • > Technical Manual – Writing Test – sections 3.11- 3.13 • > Technical Manual – Scoring – sections 4.2, 4.4 • >Technical Manual – Rubrics – Appendices F.16-F.28
• ACT Aspire Score Reporting LiveBinder >Types of Reports • Performance Level Descriptors – see next slide
Performance Level Descriptors - Writing
Ready Level Goal: Scale Score – 428 and above Per Reporting Category – Score point 4 and above
What do I do now? • First and foremost • Continue teaching the Arkansas English Language Arts Standards ! • Teach all modes of writing every single year. • Write across the curriculum. • Writing is not just an ELA teacher’s responsibility. • Students should be writing in every content area. • DATA – • Look at the SPF file and note the students that were just below the 428 benchmark. • Look at each grade’s individual reporting categories and analyze the data. What is it telling you? • Share this webinar with colleagues – will be posted on the
Aspire LiveBinder under Writing Webinar tab.
Coming Soon!!!! Up next in our webinar series! • Watch for a Commissioner’s Memo ! • ADE Literacy Content Team will be developing
instructional strategies and resources for writing. • Please note – these will be added to the ADE site and the Aspire LiveBinder > Writing Webinar tab. Who can I call with questions? Data Analysis – Student Assessment office – 501-682-4558 Curriculum/Standards – 501-682-7442
