Placement webinar slides
Placement Webinar How placing students at the right level raises standards
Assessment needs in the ELT sector? Barrier tests Achievement tests Diagnostic tests Progress tests Placement tests Proficiency tests High Stakes, Low Stakes, Medium Stakes?
● Tests need to be designed specifically to meet the intended purpose. ● Tests should be used for the purpose they were built for.
Spend your first week of class with your students, not marking their test papers The start of a course can be one of the busiest times of year, especially for teachers. Using up teacher’s valuable time and resource. Placement removes the hassle of creating, administering and marking tests, freeing up teachers so they can get to know their students.
Placement is an essential core component of raising standards If a student isn’t correctly placed at the start of their learner journey, then everything that follows is predicated on incorrect assumptions about the learner. Poor placement can lead to: • Mismatched expectations • Decrease in learners engagement and motivation • Difficulty assessing progress
Our research told us that customers wanted… A Placement test that is: • Short (30 – 40 mins) • Accurate & reliable • Covers all four skills • Affordable • Easy to Administer • Provide actionable information • Be available online
The shorter the test, the less accurate and reliable the test becomes. There is therefore a trade off between test length and test accuracy.
To resolve these challenges we have: • Built adaptive algorithms for test efficiency • Calibrated all items in our item banks for test reliability and accuracy • Created different options for Placement to cater for different customer needs: Skills covered, Diagnostic / reporting information • Used automated scoring technologies for marking speaking
What is Pearson English Placement? Pearson English Placement is an online, adaptive placement test delivered in up to 40 minutes. Instant, accurate results place students in the right level of English class faster, getting them off to the best possible start of their English language journey
● ● ● ● ● ● ● ● ●
Online test International English Up to 40 minutes Delivered through MyEnglishLab Tests speaking, writing, reading, grammar, vocabulary Beginner to advanced Scores available instantly Adaptive test Scored on the CEFR and the Global Scale of English
How was the test developed? Content Creation / Item writing • Teams of item writers in the UK, US & Australia Field Testing • Tested on 10,000 learners in 18 countries & 26 L1s • Automated scoring & Human rating Analysis • Classical Stats, IRT (Rasch) modelling Calibration • Item difficulty, Distractor analysis Engine Training • Acoustic models and latent semantic analysis Piloting • Live piloting with several hundred students Iterations • Improvements and refinements following pilots Release • Released in China followed by Global release
Outcomes: A large item bank of fully calibrated items across the ability spectrum. Tests are unique and efficient using adaptive algorithms.
Automated Scoring FAQs: https://www.youtube.com/user/PearsonPTE/playlists?sort=dd&shelf_id=6&view=50
The Adaptive Algorithms
High Level View Item bank X
X
X
X
X
X
X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X
Course GSE Level C2
X
X X X X XX X X X X X X X X X XX X X X X
Course GSE Level C1
Difficulty 10 - 90
X
Placement Test
Course of Study
Course GSE Level B2 Test result 10 - 90 Course GSE Level B1
Course GSE Level A2
X X
X
X X X
X
Course GSE Level A1
Test Design: Part 1 – Adaptive algorithm Placement Essential & Placement Profile Hardest
Bank of items – ordered by difficulty
Correct – take ✔the harder item
Correct – take the harder item
✔ Starting with the middle item
Easiest
– take ✖ Incorrect the easier item
As the Ability estimate stabilises the Items ‘home in’ on testee's level
Test Design: Adaptive & Linear Parts 1 & 2 together Placement + Speaking
Part 1 - Adaptive
Part 2 - Linear
Hardest
Bank of items – ordered by difficulty
Correct – take ✔ the harder item
Correct – take ✔ the harder item
Incorrect – take the easier item
Linear Starting with the middle item
Ability estimate stabilises Items ‘home in’ on testee's level
Easiest
*
✖
Test Design: Parts 1 & 2 together Part 1 - Adaptive
Hardest
Part 2 - Linear
Bank of items – ordered by difficulty
Correct – take ✔ the harder item
✖
Correct – take ✔ the harder item
Incorrect – take the easier item
Linear Starting with the middle item
Ability estimate stabilises Items ‘home in’ on testee's level
Easiest
Reading
Grammar
Vocabulary
Listening
Overall Score GSE / CEFR
Speaking
Writing
Raising Standards – The Global Scale of English The Global Scale of English (GSE), the world’s first truly global English language standard, will allow you to measure progress accurately and easily. Our ultimate ambition is to provide learners – wherever they are in the world – with the answers to these questions: How good is my English? Am I progressing? What do I need to do next? The GSE comprises: - The scale itself - The GSE Learning Objectives - Course materials - Assessment https://www.english.com/gse To help Teachers and Syllabus designers the GSE Teacher toolkit is available free for use on: https://www.english.com/gse/teacher-toolkit
The Test
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
The Results
Intake creation, management, student data and reporting
Placement Profile – Skills profile per student
Placement Profile – Can Do statements per student identifying weak areas
Matt Saunders Senior Tenured Lecturer Ritsumeikan Asia Pacific University, Oita, Japan
Ritsumeikan APU University: Placement Profile Pilot •
• • •
•
Previously used an online 2 skills adaptive placement test to assess reading & listening skills Created an in-house test to assess speaking & writing skills Piloted Pearson English Placement Test with 60 students in September 2016 Pearson English Placement provided the same results as the 2 skills online test and the inhouse test combined Scores were provided on the CEFR and the GSE meaning they had a more granular view of their students level of English, and could more closely align them to the correct levels
34
Assessment needs in the ELT sector? Barrier tests Achievement tests Diagnostic tests Progress tests Placement tests Proficiency tests High Stakes, Low Stakes, Medium Stakes?
● Tests need to be designed specifically to meet the intended purpose. ● Tests should be used for the purpose they were built for.
Spend your first week of class with your students, not marking their test papers The start of a course can be one of the busiest times of year, especially for teachers. Using up teacher’s valuable time and resource. Placement removes the hassle of creating, administering and marking tests, freeing up teachers so they can get to know their students.
Placement is an essential core component of raising standards If a student isn’t correctly placed at the start of their learner journey, then everything that follows is predicated on incorrect assumptions about the learner. Poor placement can lead to: • Mismatched expectations • Decrease in learners engagement and motivation • Difficulty assessing progress
Our research told us that customers wanted… A Placement test that is: • Short (30 – 40 mins) • Accurate & reliable • Covers all four skills • Affordable • Easy to Administer • Provide actionable information • Be available online
The shorter the test, the less accurate and reliable the test becomes. There is therefore a trade off between test length and test accuracy.
To resolve these challenges we have: • Built adaptive algorithms for test efficiency • Calibrated all items in our item banks for test reliability and accuracy • Created different options for Placement to cater for different customer needs: Skills covered, Diagnostic / reporting information • Used automated scoring technologies for marking speaking
What is Pearson English Placement? Pearson English Placement is an online, adaptive placement test delivered in up to 40 minutes. Instant, accurate results place students in the right level of English class faster, getting them off to the best possible start of their English language journey
● ● ● ● ● ● ● ● ●
Online test International English Up to 40 minutes Delivered through MyEnglishLab Tests speaking, writing, reading, grammar, vocabulary Beginner to advanced Scores available instantly Adaptive test Scored on the CEFR and the Global Scale of English
How was the test developed? Content Creation / Item writing • Teams of item writers in the UK, US & Australia Field Testing • Tested on 10,000 learners in 18 countries & 26 L1s • Automated scoring & Human rating Analysis • Classical Stats, IRT (Rasch) modelling Calibration • Item difficulty, Distractor analysis Engine Training • Acoustic models and latent semantic analysis Piloting • Live piloting with several hundred students Iterations • Improvements and refinements following pilots Release • Released in China followed by Global release
Outcomes: A large item bank of fully calibrated items across the ability spectrum. Tests are unique and efficient using adaptive algorithms.
Automated Scoring FAQs: https://www.youtube.com/user/PearsonPTE/playlists?sort=dd&shelf_id=6&view=50
The Adaptive Algorithms
High Level View Item bank X
X
X
X
X
X
X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X X X X X X XX X X X X X
Course GSE Level C2
X
X X X X XX X X X X X X X X X XX X X X X
Course GSE Level C1
Difficulty 10 - 90
X
Placement Test
Course of Study
Course GSE Level B2 Test result 10 - 90 Course GSE Level B1
Course GSE Level A2
X X
X
X X X
X
Course GSE Level A1
Test Design: Part 1 – Adaptive algorithm Placement Essential & Placement Profile Hardest
Bank of items – ordered by difficulty
Correct – take ✔the harder item
Correct – take the harder item
✔ Starting with the middle item
Easiest
– take ✖ Incorrect the easier item
As the Ability estimate stabilises the Items ‘home in’ on testee's level
Test Design: Adaptive & Linear Parts 1 & 2 together Placement + Speaking
Part 1 - Adaptive
Part 2 - Linear
Hardest
Bank of items – ordered by difficulty
Correct – take ✔ the harder item
Correct – take ✔ the harder item
Incorrect – take the easier item
Linear Starting with the middle item
Ability estimate stabilises Items ‘home in’ on testee's level
Easiest
*
✖
Test Design: Parts 1 & 2 together Part 1 - Adaptive
Hardest
Part 2 - Linear
Bank of items – ordered by difficulty
Correct – take ✔ the harder item
✖
Correct – take ✔ the harder item
Incorrect – take the easier item
Linear Starting with the middle item
Ability estimate stabilises Items ‘home in’ on testee's level
Easiest
Reading
Grammar
Vocabulary
Listening
Overall Score GSE / CEFR
Speaking
Writing
Raising Standards – The Global Scale of English The Global Scale of English (GSE), the world’s first truly global English language standard, will allow you to measure progress accurately and easily. Our ultimate ambition is to provide learners – wherever they are in the world – with the answers to these questions: How good is my English? Am I progressing? What do I need to do next? The GSE comprises: - The scale itself - The GSE Learning Objectives - Course materials - Assessment https://www.english.com/gse To help Teachers and Syllabus designers the GSE Teacher toolkit is available free for use on: https://www.english.com/gse/teacher-toolkit
The Test
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
Question types in Placement
The Results
Intake creation, management, student data and reporting
Placement Profile – Skills profile per student
Placement Profile – Can Do statements per student identifying weak areas
Matt Saunders Senior Tenured Lecturer Ritsumeikan Asia Pacific University, Oita, Japan
Ritsumeikan APU University: Placement Profile Pilot •
• • •
•
Previously used an online 2 skills adaptive placement test to assess reading & listening skills Created an in-house test to assess speaking & writing skills Piloted Pearson English Placement Test with 60 students in September 2016 Pearson English Placement provided the same results as the 2 skills online test and the inhouse test combined Scores were provided on the CEFR and the GSE meaning they had a more granular view of their students level of English, and could more closely align them to the correct levels
34
