Innovation in Formative Assessment Development
Innovation in Formative Assessment Development Nicolle Romero and Eric W. Crane WestEd
92nd Annual CERA Conference Anaheim, CA December 5, 2013
WestEd.org
Context and History Begins with College Board’s Advanced Placement exams (updated frameworks) But does not have to: any curricular material can be subject to this development process
WestEd.org
AP Biology Course Revision Reduced breadth for improved depth of understanding Specific learning objectives Focus on application of knowledge/science practices
WestEd.org
Components of AP Insight Content Standards Resources for Teachers Resources for Students Feedback, even for a correct response Reporting
WestEd.org
New Approach Content standards alone are not sufficient to do this assessment development. An awareness of students’ frequent misunderstandings helps to drive the assessment development.
WestEd.org
WestEd.org
WestEd.org
Parallels to the NGSS Next Generation Science Standards released April 2013
WestEd.org
Parallels to the NGSS Both the AP redesign and the NGSS: • Increase the focus on the application of knowledge. • Identify learning goals that require practice and content to be in context of one another. • Identify overlapping science practices.
WestEd.org
Input Development College Board worked with AP Biology teachers to identify: • Challenge Areas • Misunderstandings
WestEd.org
Input Development Challenge Area
Success Criteria + Misunderstandings
Assessment
WestEd.org
Input Development Challenges during the development of the Inputs: • Targeting misunderstandings • Overlap of success criteria • Appropriate level of rigor
WestEd.org
Input Development From Appendix C of the NGSS: “One often overlooked aspect of combining demanding practices with strong content in standards is the effect on rigor. It is also well to keep in mind that calling for application of mathematics in a performance generally raises the level of rigor.” WestEd.org
Item Development Two types of items developed: • Multiple Choice • Constructed Response
WestEd.org
Item Development Functional goals: 1. Mirror the summative exam, yet be designed for formative assessment. 2. Identify students struggling to overcome specific misunderstandings and provide targeted feedback. 3. Assess the application of content knowledge (Science Practices). WestEd.org
Item Development Typical approach to developing MC items: Stem Correct response Distractors and rationales WestEd.org
Item Development Feedback-focused approach to MC items: Stem Correct response and distractors Rationales WestEd.org
Item Development Stem Choice (A) Choice (B) Choice (C) Choice (D)
Aligns with Misunderstanding 1 Aligns with Misunderstanding 2 Aligns with Misunderstanding 3 Correct response: Evidence of Success Criteria
WestEd.org
Item Development Each rationale includes: • A best estimate of what the student knows and doesn’t know • The specific misunderstanding held • What the student needs to work on in order to close the gap (actionable feedback)
WestEd.org
Item Development Feedback-focused approach to CR items: Stem Scaffolded parts Rationales for different point values WestEd.org
Summary of Innovative Features • Item types mirror the summative exam (MC and CR), but AP Insight items provide substantially more information due to their feedback-focused architecture. • Components align tightly to common misunderstandings and provide rationales that break down how the student’s selection provides insight into his or her understanding. • Rationales provide clear next steps for students to help close the learning gap. WestEd.org
Closing Thoughts A brief word about Professional Development Questions?
WestEd.org
Presenters • Nicolle Romero, [email protected] • 831.464.8120
• Eric Crane, [email protected] • 916.492.4080
WestEd.org
92nd Annual CERA Conference Anaheim, CA December 5, 2013
WestEd.org
Context and History Begins with College Board’s Advanced Placement exams (updated frameworks) But does not have to: any curricular material can be subject to this development process
WestEd.org
AP Biology Course Revision Reduced breadth for improved depth of understanding Specific learning objectives Focus on application of knowledge/science practices
WestEd.org
Components of AP Insight Content Standards Resources for Teachers Resources for Students Feedback, even for a correct response Reporting
WestEd.org
New Approach Content standards alone are not sufficient to do this assessment development. An awareness of students’ frequent misunderstandings helps to drive the assessment development.
WestEd.org
WestEd.org
WestEd.org
Parallels to the NGSS Next Generation Science Standards released April 2013
WestEd.org
Parallels to the NGSS Both the AP redesign and the NGSS: • Increase the focus on the application of knowledge. • Identify learning goals that require practice and content to be in context of one another. • Identify overlapping science practices.
WestEd.org
Input Development College Board worked with AP Biology teachers to identify: • Challenge Areas • Misunderstandings
WestEd.org
Input Development Challenge Area
Success Criteria + Misunderstandings
Assessment
WestEd.org
Input Development Challenges during the development of the Inputs: • Targeting misunderstandings • Overlap of success criteria • Appropriate level of rigor
WestEd.org
Input Development From Appendix C of the NGSS: “One often overlooked aspect of combining demanding practices with strong content in standards is the effect on rigor. It is also well to keep in mind that calling for application of mathematics in a performance generally raises the level of rigor.” WestEd.org
Item Development Two types of items developed: • Multiple Choice • Constructed Response
WestEd.org
Item Development Functional goals: 1. Mirror the summative exam, yet be designed for formative assessment. 2. Identify students struggling to overcome specific misunderstandings and provide targeted feedback. 3. Assess the application of content knowledge (Science Practices). WestEd.org
Item Development Typical approach to developing MC items: Stem Correct response Distractors and rationales WestEd.org
Item Development Feedback-focused approach to MC items: Stem Correct response and distractors Rationales WestEd.org
Item Development Stem Choice (A) Choice (B) Choice (C) Choice (D)
Aligns with Misunderstanding 1 Aligns with Misunderstanding 2 Aligns with Misunderstanding 3 Correct response: Evidence of Success Criteria
WestEd.org
Item Development Each rationale includes: • A best estimate of what the student knows and doesn’t know • The specific misunderstanding held • What the student needs to work on in order to close the gap (actionable feedback)
WestEd.org
Item Development Feedback-focused approach to CR items: Stem Scaffolded parts Rationales for different point values WestEd.org
Summary of Innovative Features • Item types mirror the summative exam (MC and CR), but AP Insight items provide substantially more information due to their feedback-focused architecture. • Components align tightly to common misunderstandings and provide rationales that break down how the student’s selection provides insight into his or her understanding. • Rationales provide clear next steps for students to help close the learning gap. WestEd.org
Closing Thoughts A brief word about Professional Development Questions?
WestEd.org
Presenters • Nicolle Romero, [email protected] • 831.464.8120
• Eric Crane, [email protected] • 916.492.4080
WestEd.org
