Subjects
Undergraduate & Graduate Category: Social Science, Business & Law 1 2 3 4 5 Degree Level: BA /BS /MS /JD /Faculty Abstract ID# 747
TesBng Juror Comprehension: evolving methodologies
5, Andrea Medrano Leger3, Alana Dore1, Katherine Fiallo1,4, Aaron McPherson2, and John Tan2 Janet R andall
Abstract What’s the best way to test juror comprehension? This comparison of 3 methodologies shows the relaUve advantages for linguisUc research of a new online crowd-‐sourcing resource, Amazon MTurk. With MTurk, we can test a range of subjects that mirrors the actual juror populaUon, to help us idenUfy specific problems that jury instrucUons pose and how best to fix them.
Background
One study of jurors found that > 25% couldn’t define:
To address this problem, some states have rewricen their jury instrucUons. Should we? A serious problem faces society: though all ciUzens over 18 are eligible to be jurors, not all jurors are equally and fairly engaged in the jury process Many are confused by “jury instrucUons” -‐-‐ the complicated direcUves that the judge reads to them before they deliberate a case (Charrow & Charrow, 1979; Diamond et.al. 2012). This has led to unrepresentaUve juries and even misinformed verdicts (Benson, 1984; Marder, 2006).
Methodologies Previous Methodology 1 (M1): NU-‐student subjects
“
… burden of proof
… admissible evidence … inference
”
(Randall, J. & L. Graf, 2014)
Subjects 214 unpaid subjects, all Northeastern University undergraduates. Materials & Design • 4 x 2 design: (CL) Current instrucUons, subjects Listened only (CR) Current instrucUons, subjects Read the text while listening (PL) Plain English instrucUons, subjects Listened only (PR) Plain English instrucUons, subjects Lead the text while listening • Each subject got a booklet containing T/F quesUons about 6 instrucUons • Audio recordings of all instrucUons, read by the same actor Procedure • Experimenters tested groups of subjects in controlled sessions in NU classrooms • Subjects listened to 6 instrucUons one at a Ume, beginning with a pracUce instrucUon, then answered T/ F quesUons about them. Subjects heard the instrucUons in one of 3 orders, to minimize order effects. • Each test session lasted 30-‐40 minutes Pros & Cons • pros recruiBng easy: teachers volunteered their classes for the study cost $0: subjects were not paid yield high: very few subjects had to be disqualified logisBcs easy: we ran the experiment during class periods; subjects did not have to travel • cons moBvaBon low: subjects were not paid to parUcipate educaBon too high: compared to the typical Massachusecs juror’s (see Fig. 1) Student subjects likely performed becer than jurors would have. Why? CompleUng some college courses improves comprehension (see Fig. 2). Comprehension of Jury InstrucUons by EducaUonal Level
EducaUonal Level in Massachusecs 18+ years
90%
70% 60% 50% 40% 30% 20%
0% 1
2.21
2.4
3.01
3.11
3.5
3.71
3.75
4.3
15.3
InstrucUon Number 0-‐8th grade
College (12-‐16 years)
High School (≤12 years)
Graduate School (16 years)
Figure 1
46% of Massachusecs ciUzens (age 18 and over) have completed 12 or fewer years of schooling (2014 Census Data)
High School (12 years or fewer) College (12 -‐16 years) Graduate School (16 years or more)
Figure 2 Subjects with high-‐school educaUon or less perform much worse than those with college or graduate school (Charrow & Charrow, 1979)
Methodology 2 (M2): Subjects recruited via Craigslist & fliers Subjects
63 paid subjects, high school educaUon or less
Materials, Design & Procedure
Comprehension Rates Methodologies 1 vs. 2
• Same as Methodology 1
100%
Pros & Cons • pros moBvaBon high: subjects were paid
• cons
educaBon recruiBng cost yield logisBcs
90% 80%
low: closer to the typical juror difficult: subjects were not readily accessible high: $20 per subject + Craigslist charges low: many no-‐shows; some unqualified subjects tried to parUcipate difficult: had to coordinate many schedules
60% 50% 40%
Methodology 1 vs. 2: Take-‐Aways
20%
OL
PL Methodology 1
Because of their higher-‐than-‐typical-‐juror educaUon level, M1 subjects consistently outperformed M2 subjects (see Fig. 3). But: can we find a single methodology level that becer approximates jurors’ educaUon levels and uses a larger, cheaper, more accessible subject pool?
What is MTurk? Amazon’s Mechanical Turk crowd-‐sourcing website provides a market for tasks that require human intelligence. “Requesters” put up HITs – Human Intelligence Tasks – that “Workers” complete in exchange for payment. For research projects, MTurk provides a diverse populaUon of subjects for tesUng (Aker et.al. (2012). Subjects • 360 paid subjects, drawn from a mix of educaUon levels from across Massachusecs • Paid a minimal amount ($2 per subject) Materials & Design • Same 4 x 2 design as Methodology 1 & 2 Procedure • Subjects sign on individually to the MTurk website and link to one of our tests • MTurk solware records and codes subject informaUon, demographics and responses Pros & Cons • pros recruiBng easy: MTurk doubles as both recruiter and tester cost low: $2 per person, 1/10 of Methodology 2 yield high: (expected) 360 subjects moBvaBon high: subjects are paid educaBon wide-‐ranging: closer to the typical juror • cons logisBcs risky: subjects may not follow direcUons or may get distracted
How does Methodology 3 Compare?
M3 can provide a lot of data (surpassing M2) and matches juror responses (surpassing M1). As Figure 4 shows, M3 is the best. But there is one risk. All the advantages of M1 + M2, • recruiBng: easy to recruit large groups of subjects • cost: low • yield: high, because MTurk pre-‐screens parUcipants • moBvaBon: high, since subjects are paid only aler successful compleUon • logisBcs: easy, since Mturk is available on personal computers, at any Ume. But there is one risk: • logisBcs: an uncontrolled tesUng environment, which could negaUvely affect subject performance
Methodology 1-‐3 Pros & Cons
1
recruiUng
cost
yield
moUvaUon
educaUon
logisUcs
2
3
x x x x x x
?
Figure 4 Methodology 3 combines the advantages of M1 & M2.
Conclusion
Research methodologies are constantly subject to review and revision. While our ulUmate goal is to determine whether or not Massachusecs jurors understand the jury instrucUons that judges read to them, and to idenUfy the factors that impede understanding, our first task is to find a methodology that will allow us to answer these quesUons in a pracUcal, cost-‐effecUve, and revealing way. Our new MTurk methodology is promising. It combines the advantages of two earlier methodologies with fewer drawbacks. But MTurk’s most posiUve acribute is that its subjects will closely approximate Massachusecs jurors, allowing us to validly generalize from their responses to actual juror responses. Based on this, we will be able to determine which instrucUons are poorly understood, pinpoint which linguisUc factors are involved (syntacUc construcUons, lexical choices, ambiguiUes, …) and determine how best to eliminate them to make the instrucUons clearer. Clearer instrucUons will lead to becer-‐informed juries, whose verdicts are based on an understanding of how to apply the law, and whose verdicts are, therefore, truly just.
30%
0%
New Methodology Methodology 3 (M3): MTurk subjects
70%
10%
80%
10%
100%
Comprehension Rate
? ? ?
… impeach
Are MassachuseLs jurors confused by our current instrucBons? Our Plain English Jury InstrucUon Project is trying to answer this quesUon. We are studying how well Massachusecs instrucUons are understood, and examining which linguisUc factors affect understanding, to guide us in rewriUng difficult instrucUons. However, we cannot access actual jurors, so we have been looking for a methodology using other subject pools to try to approximate actual juror performance in a pracUcal research design.
OR
PR
Methodology 2
Figure 3 M1 (NU student) subjects performed becer than M2 (no college) subjects, in all 4 condiUons
Acknowledgements We are grateful to the Massachusecs Bar AssociaUon for providing grant support and for sponsoring Professor Janet Randall as a 2012-‐15 VisiUng Research Fellow. The NU CSSH Research Development Fund and Undergraduate Research IniUaUve provided addiUonal funding. Thanks also goes to CSSH Associate Dean for Research Jack McDevic and the members of the MBA Plain English Jury InstrucUon Task Force for their comments and suggesUons.
References Aker, A., M-‐D. Albakour & U. Kruschwitz (2012) Assessing Crow-‐sourcing Quality through ObjecUve Tasks. In the 8th Language Resources and EvaluaUon (LREC 2012), Istanbul, Turkey. Charrow, Robert P., & Charrow, Veda R. (1979) Making Legal Language Understandable: A PsycholinguisUc Study of Jury InstrucUons. Columbia Law Review 79:1306-‐1347. Marder, Nancy S. (2006) Bringing Jury InstrucUons Into the Twenty-‐First Century. Notre Dame Law Review 81:449-‐512. Benson, Robert W. 1985. The End of Legalese: The Game is Over.Review of Law & Social Change 8:519-‐573. Diamond, S.S., B. Murphy, & M.R. Rose (2012) The “Kecleful of Law” in Real Jury DeliberaUons: Successes, Failures and Next Steps, 106 Northwestern U. Law Rev. 1537-‐1608. EducaUonal Acainment of the PopulaUon 18 Years and Over, by Age, Sex, Race, and Hispanic Origin: 2013, All Races. U.S. Census Bureau, Current PopulaAon Survey, 2013 Annual Social and Economic Supplement. Accessed electronically on March 21, 2014. hcps://www.census.gov/hhes/socdemo/educaUon/data/cps/2013/tables.html Randall, Janet & Lucas Graf (2014) LinguisUcs meets "legalese": syntax, semanUcs, and jury instrucUon reform. Presented at the LinguisUcs Society of America Annual MeeUng, Boston, MA.
TesBng Juror Comprehension: evolving methodologies
5, Andrea Medrano Leger3, Alana Dore1, Katherine Fiallo1,4, Aaron McPherson2, and John Tan2 Janet R andall
Abstract What’s the best way to test juror comprehension? This comparison of 3 methodologies shows the relaUve advantages for linguisUc research of a new online crowd-‐sourcing resource, Amazon MTurk. With MTurk, we can test a range of subjects that mirrors the actual juror populaUon, to help us idenUfy specific problems that jury instrucUons pose and how best to fix them.
Background
One study of jurors found that > 25% couldn’t define:
To address this problem, some states have rewricen their jury instrucUons. Should we? A serious problem faces society: though all ciUzens over 18 are eligible to be jurors, not all jurors are equally and fairly engaged in the jury process Many are confused by “jury instrucUons” -‐-‐ the complicated direcUves that the judge reads to them before they deliberate a case (Charrow & Charrow, 1979; Diamond et.al. 2012). This has led to unrepresentaUve juries and even misinformed verdicts (Benson, 1984; Marder, 2006).
Methodologies Previous Methodology 1 (M1): NU-‐student subjects
“
… burden of proof
… admissible evidence … inference
”
(Randall, J. & L. Graf, 2014)
Subjects 214 unpaid subjects, all Northeastern University undergraduates. Materials & Design • 4 x 2 design: (CL) Current instrucUons, subjects Listened only (CR) Current instrucUons, subjects Read the text while listening (PL) Plain English instrucUons, subjects Listened only (PR) Plain English instrucUons, subjects Lead the text while listening • Each subject got a booklet containing T/F quesUons about 6 instrucUons • Audio recordings of all instrucUons, read by the same actor Procedure • Experimenters tested groups of subjects in controlled sessions in NU classrooms • Subjects listened to 6 instrucUons one at a Ume, beginning with a pracUce instrucUon, then answered T/ F quesUons about them. Subjects heard the instrucUons in one of 3 orders, to minimize order effects. • Each test session lasted 30-‐40 minutes Pros & Cons • pros recruiBng easy: teachers volunteered their classes for the study cost $0: subjects were not paid yield high: very few subjects had to be disqualified logisBcs easy: we ran the experiment during class periods; subjects did not have to travel • cons moBvaBon low: subjects were not paid to parUcipate educaBon too high: compared to the typical Massachusecs juror’s (see Fig. 1) Student subjects likely performed becer than jurors would have. Why? CompleUng some college courses improves comprehension (see Fig. 2). Comprehension of Jury InstrucUons by EducaUonal Level
EducaUonal Level in Massachusecs 18+ years
90%
70% 60% 50% 40% 30% 20%
0% 1
2.21
2.4
3.01
3.11
3.5
3.71
3.75
4.3
15.3
InstrucUon Number 0-‐8th grade
College (12-‐16 years)
High School (≤12 years)
Graduate School (16 years)
Figure 1
46% of Massachusecs ciUzens (age 18 and over) have completed 12 or fewer years of schooling (2014 Census Data)
High School (12 years or fewer) College (12 -‐16 years) Graduate School (16 years or more)
Figure 2 Subjects with high-‐school educaUon or less perform much worse than those with college or graduate school (Charrow & Charrow, 1979)
Methodology 2 (M2): Subjects recruited via Craigslist & fliers Subjects
63 paid subjects, high school educaUon or less
Materials, Design & Procedure
Comprehension Rates Methodologies 1 vs. 2
• Same as Methodology 1
100%
Pros & Cons • pros moBvaBon high: subjects were paid
• cons
educaBon recruiBng cost yield logisBcs
90% 80%
low: closer to the typical juror difficult: subjects were not readily accessible high: $20 per subject + Craigslist charges low: many no-‐shows; some unqualified subjects tried to parUcipate difficult: had to coordinate many schedules
60% 50% 40%
Methodology 1 vs. 2: Take-‐Aways
20%
OL
PL Methodology 1
Because of their higher-‐than-‐typical-‐juror educaUon level, M1 subjects consistently outperformed M2 subjects (see Fig. 3). But: can we find a single methodology level that becer approximates jurors’ educaUon levels and uses a larger, cheaper, more accessible subject pool?
What is MTurk? Amazon’s Mechanical Turk crowd-‐sourcing website provides a market for tasks that require human intelligence. “Requesters” put up HITs – Human Intelligence Tasks – that “Workers” complete in exchange for payment. For research projects, MTurk provides a diverse populaUon of subjects for tesUng (Aker et.al. (2012). Subjects • 360 paid subjects, drawn from a mix of educaUon levels from across Massachusecs • Paid a minimal amount ($2 per subject) Materials & Design • Same 4 x 2 design as Methodology 1 & 2 Procedure • Subjects sign on individually to the MTurk website and link to one of our tests • MTurk solware records and codes subject informaUon, demographics and responses Pros & Cons • pros recruiBng easy: MTurk doubles as both recruiter and tester cost low: $2 per person, 1/10 of Methodology 2 yield high: (expected) 360 subjects moBvaBon high: subjects are paid educaBon wide-‐ranging: closer to the typical juror • cons logisBcs risky: subjects may not follow direcUons or may get distracted
How does Methodology 3 Compare?
M3 can provide a lot of data (surpassing M2) and matches juror responses (surpassing M1). As Figure 4 shows, M3 is the best. But there is one risk. All the advantages of M1 + M2, • recruiBng: easy to recruit large groups of subjects • cost: low • yield: high, because MTurk pre-‐screens parUcipants • moBvaBon: high, since subjects are paid only aler successful compleUon • logisBcs: easy, since Mturk is available on personal computers, at any Ume. But there is one risk: • logisBcs: an uncontrolled tesUng environment, which could negaUvely affect subject performance
Methodology 1-‐3 Pros & Cons
1
recruiUng
cost
yield
moUvaUon
educaUon
logisUcs
2
3
x x x x x x
?
Figure 4 Methodology 3 combines the advantages of M1 & M2.
Conclusion
Research methodologies are constantly subject to review and revision. While our ulUmate goal is to determine whether or not Massachusecs jurors understand the jury instrucUons that judges read to them, and to idenUfy the factors that impede understanding, our first task is to find a methodology that will allow us to answer these quesUons in a pracUcal, cost-‐effecUve, and revealing way. Our new MTurk methodology is promising. It combines the advantages of two earlier methodologies with fewer drawbacks. But MTurk’s most posiUve acribute is that its subjects will closely approximate Massachusecs jurors, allowing us to validly generalize from their responses to actual juror responses. Based on this, we will be able to determine which instrucUons are poorly understood, pinpoint which linguisUc factors are involved (syntacUc construcUons, lexical choices, ambiguiUes, …) and determine how best to eliminate them to make the instrucUons clearer. Clearer instrucUons will lead to becer-‐informed juries, whose verdicts are based on an understanding of how to apply the law, and whose verdicts are, therefore, truly just.
30%
0%
New Methodology Methodology 3 (M3): MTurk subjects
70%
10%
80%
10%
100%
Comprehension Rate
? ? ?
… impeach
Are MassachuseLs jurors confused by our current instrucBons? Our Plain English Jury InstrucUon Project is trying to answer this quesUon. We are studying how well Massachusecs instrucUons are understood, and examining which linguisUc factors affect understanding, to guide us in rewriUng difficult instrucUons. However, we cannot access actual jurors, so we have been looking for a methodology using other subject pools to try to approximate actual juror performance in a pracUcal research design.
OR
PR
Methodology 2
Figure 3 M1 (NU student) subjects performed becer than M2 (no college) subjects, in all 4 condiUons
Acknowledgements We are grateful to the Massachusecs Bar AssociaUon for providing grant support and for sponsoring Professor Janet Randall as a 2012-‐15 VisiUng Research Fellow. The NU CSSH Research Development Fund and Undergraduate Research IniUaUve provided addiUonal funding. Thanks also goes to CSSH Associate Dean for Research Jack McDevic and the members of the MBA Plain English Jury InstrucUon Task Force for their comments and suggesUons.
References Aker, A., M-‐D. Albakour & U. Kruschwitz (2012) Assessing Crow-‐sourcing Quality through ObjecUve Tasks. In the 8th Language Resources and EvaluaUon (LREC 2012), Istanbul, Turkey. Charrow, Robert P., & Charrow, Veda R. (1979) Making Legal Language Understandable: A PsycholinguisUc Study of Jury InstrucUons. Columbia Law Review 79:1306-‐1347. Marder, Nancy S. (2006) Bringing Jury InstrucUons Into the Twenty-‐First Century. Notre Dame Law Review 81:449-‐512. Benson, Robert W. 1985. The End of Legalese: The Game is Over.Review of Law & Social Change 8:519-‐573. Diamond, S.S., B. Murphy, & M.R. Rose (2012) The “Kecleful of Law” in Real Jury DeliberaUons: Successes, Failures and Next Steps, 106 Northwestern U. Law Rev. 1537-‐1608. EducaUonal Acainment of the PopulaUon 18 Years and Over, by Age, Sex, Race, and Hispanic Origin: 2013, All Races. U.S. Census Bureau, Current PopulaAon Survey, 2013 Annual Social and Economic Supplement. Accessed electronically on March 21, 2014. hcps://www.census.gov/hhes/socdemo/educaUon/data/cps/2013/tables.html Randall, Janet & Lucas Graf (2014) LinguisUcs meets "legalese": syntax, semanUcs, and jury instrucUon reform. Presented at the LinguisUcs Society of America Annual MeeUng, Boston, MA.
