SHRP2 L14 AWS
SHRP2 L14 EFFECTIVENESS OF DIFFERENT APPROACHES TO DISSEMINATING TRAVELER INFORMATION ON TRAVEL TIME RELIABILITY
SHRP2 L14 TETG Briefing – July 18, 2012
Meeting Agenda
Welcome / Technology Check Introductions Meeting Objectives Project Objectives & Progress to Date Progress Report on Enhanced Experiment Remaining Project Timeline Final Remarks
2
Introductions
3
Meeting Objectives
INFORM TETG of the progress on the enhanced experiment. UPDATE the TETG on progress in the development of final deliverables. REVIEW the remaining project timeline.
4
Project Objectives & Progress to Date Beverly Kuhn, TTI
Phase 1: Foundational Research
SHRP2 L14: CONTRIBUTIONS OF EXPANDED EXPERIMENTATION
Innovation Scan, State of Practice Report, Expert Interviews
Definition of travel time reliability information Current forms and media for reliability information Focus Groups/ Surveys Preferred terms Alternative media
Precursor Experiment Evaluation methodology Serenity-sensitive utility function
Phase 2: Focused Experiments
Expanded Experimentation
8
Evaluated forms and terms Valuation of reliability info Guidebook • • • • •
What is reliability information? How does it differ from real-time traveler information? How can reliability information be conveyed? How can the impacts of reliability information be valued? What forms of reliability information are most effective?
Phase 1: Foundational Research
SHRP2 L14: RELIABILITY INFORMAITON GUIDEBOOK DEVELOPMENT
Innovation Scan, State of Practice Report, Expert Interviews
Definition of travel time reliability information Current forms and media for reliability information Focus Groups/ Surveys Preferred terms Alternative media
Precursor Experiment Evaluation methodology Serenity-sensitive utility function
Phase 2: Focused Experiments
Expanded Experimentation
Evaluated forms and terms Valuation of reliability info Guidebook • • • • •
What is reliability information? How does it differ from real-time traveler information? How can reliability information be conveyed? How can the impacts of reliability information be valued? What forms of reliability information are most effective?
9
Lexicon
8
Guidebook Outline
Introduction Key Concepts Travel
Time Reliability Reliability Information
Relevant Travel Time Reliability Terms Methods of Communication (Lexicon) Measuring Information Effectiveness Final Remarks References 9
Progress Report on Enhanced Experiment Karl Wunderlich, Noblis Vaishali Shah, Noblis
Experiment Hypotheses
Provision of reliability information improves on-time performance versus a control group that receives no reliability information Some
forms will be more effective than others Perceived benefit will trail actual benefit
Benefits of reliability information will decrease over time as participants build a larger experience base The provision of reliability signposts in real-time information will improve on-time performance
Two Distinct Experiments
Experiment X –Compare different forms and content of data to unfamiliar travelers
5 sets of reliability information tested by each participant through 5 weeks of simulated commuting, each week in a different city.
Experiment Y – Compare the learning curve for unfamiliar travelers with or without reliability information Half of participants receive simulated reliability information over a 4 week period in one city Other half do not receive any reliability information.
Experiment X: Reliability Terms to be Tested (based on terminology document)
F.
Baseline radio information presented textually Text-based 95th percentile Text-based average plus 95th percentile Text-based 20th, average, and 95th percentile Reliability Signposting Graphical “C” G. Graphical “D”
H.
Auditory “B”
A. B. C. D. E.
13
Demo of Experiments
Pre-experiment survey, commuting game, and mini post-experiment survey Often experiment completed within 60-70 minutes in Washington DC, while we ‘had’ participants for 90 minutes. Consequently, paper survey added in Chicago and Houston requests participants rank six of seven types of info (auditory was excluded)
Experimental Plan by Numbers
Reliability Data Content
X Two Distinct Experiments: (5 cities, 1 week each city) Experiment Version 1 2 3 4 Planned Participant Count 90 30 30 30 Completed Thus Far (DC & Houston) 70 18 13 17 A. Contol (no reliability info) P P P P TH B. Textual 95 P P TH C. Textual Avg + 95 P P P P TH TH D. Textual 20 + Avg + 95 P P P E. Visual and Textual signposting P F. Graphical presentation of "C" P P G. Graphical presentation of "D" P P H. Auditory presentation of "B" P P
Y (1 city, 4 weeks)
5 30 20 P
6 30 20
P
Experiment X: Pairwise Comparison Sample Size for Comparison of Different Types of Reliability Information (Experiment 1) A
B
C
D
E
F
G
A. Control B. 95th percentile text 83 C. Average and 95th percentile text 118 83 D. 20th, average, and 95th perctile text 105 70 105 E. average and signpost 69 70 70 70 F. Graphical C 31 13 31 18 18 G. Graphical D 35 0 35 35 18 18 H. Audio B 30 13 13 17 13 13 17 * each participant will make 5 trips using each type of reliability info
H
Preliminary Findings (83 observations)
Textual 95th v. Control
*90th percentile significance, others 95th percentile or higher significance EOW = End of week
Perceptions
Differences Not Stat. Significant Lower Post-trip and EOW willingness to pay Lower EOW understandability of info Higher EOW stress Higher EOW managing trip decisions
Outcomes
Statistically significant findings Lower week’s trip cost 50% lower late arrival frequency Higher pre-trip arrival confidence: 3.5 v. 3.3 Higher pre-trip but lower post-trip* usefulness Higher EOW usefulness: 3.4 v. 2.9
Text Avg+ 95th v. Control Statistically significant findings Lower week’s trip cost: $35 v $48 Lower late arrival frequency: 0.8 v 1.5 days/wk Higher pre-trip arrival confidence: 3.5 v 3.3 Higher pre-trip but lower post-trip* usefulness Higher EOW usefulness: 3.6 v 3.0 Higher post-trip and EOW* willingness to pay Higher EOW stress Higher EOW managing trip decisions Differences Not Stat. Significant Higher EOW understandability of info *95 percentile significance, others 99th percentile significance **End of week
Willingness to Pay for Info
Preliminary Findings (118 observations)
Preliminary Findings (105 observations)
Textual 20th+avg+95th v. Control Statistically significant findings Lower week’s trip cost $39 v $49 Lower late arrival frequency 1.1 v 1.5 Higher pre-trip arrival confidence Higher pre-trip & EOW, but lower post-trip usefulness Higher EOW managing trip decisions Higher EOW stress Higher post-trip and EOW* willingness to pay $0.32 and $0.18 more per trip, respectively
Differences Not Stat. Significant Higher EOW understandability of info
*90th percentile significance, others 95th percentile or higher significance EOW = End of week
Reliability Info Control Info
Preliminary Findings (70 observations)
Signpost v. Control Statistically significant findings Higher pre-trip and EOW* usefulness Higher post-trip stress 3.0 v 2.5 Higher EOW understandability 2.5 v 2.1 Differences Not Stat. Significant Lower week’s trip cost Lower post-trip and EOW willingness to pay for info Lower late arrival frequency Similar pre-trip on-time arrival confidence Lower Post-trip usefulness Lower EOW managing trip decisions Higher EOW stress *90th percentile significance, others 95th percentile or higher significance EOW = End of week 20
Preliminary Findings (31 observations)
Graphic Avg+95th v. Control Statistically significant findings Higher pre-trip and EOW* usefulness 3.6 v 2.9 and 4.1 v. 3.2, respectively Higher arrival confidence: 3.6 v 3.3 Higher post-trip willing to pay for info $0.48 more Higher post-trip and EOW stress 3.7 v 2.4 and 2.8 v 2.4 Higher EOW managing decisions 4.0 v 3.1 Differences Not Stat. Significant Lower week’s trip cost Lower late arrival frequency Lower Post-trip usefulness Higher EOW understandability Higher EOW willingness to pay for info *90th percentile significance, others 95th percentile or higher significance EOW = End of week 21
Preliminary Findings (31 observations)
Graphic 20th+Avg+95th v. Control Statistically significant findings Higher pre-trip usefulness Higher post-trip willingness to pay for info Higher EOW understandability Differences Not Stat. Significant Lower week’s trip cost Lower late arrival frequency Equal arrival confidence Higher Post-trip usefulness Higher EOW stress Higher EOW usefulness Higher EOW managing decisions Higher EOW willingness to pay for info
*90th percentile significance, others 95th percentile or higher significance EOW = End of week 22
Preliminary Findings (13 sample size)
Textual 95th v. Voice 95th Statistically significant findings
Voice has higher pre-trip usefulness:
4.0 v 3.1
Higher post-trip willingness to pay:
$3.9 v $3.1
Higher EOW usefulness:
4.7 v. 3.4
Higher EOW - better manage decisions: 4.7 v. 3.4
Lower EOW understandability:
1.7 v. 2.5
Higher EOW stress
4.0 v. 2.8
Differences Not Statistically Significant
Voice has higher late arrival freq.
Lower total cost
Higher on-time arrival confidence
Other preliminary comparisons…
Differences, based on a sample size of 31, for text versus graphic presentation of average and 95th percentile reliability information do not shows statistically significant differences in metrics at this preliminary juncture. Differences, based on a sample size of 35, for text versus graphic presentation of “good, typical, and bad” reliability information do not show statistically significant differences in metrics at this preliminary juncture.
24
Preliminary Implications on Hypotheses
Provision of reliability information improves on-time performance versus a control group that receives no reliability information Some
forms will be more effective than others Perceived benefit will trail actual benefit
Benefits of reliability information will decrease over time as participants build a larger experience base The provision of reliability signposts in real-time information will improve on-time performance
Next steps…
Merge in Chicago data and analyze results Analyze pre and post experiment surveys to determine
rankings among forms of presented information determine participant characteristics
Analyze Experiment “Y” to determine whether reliability benefits decrease over time
Remaining Project Timeline Beverly Kuhn, TTI
Next Steps
Preliminary findings for TETG
July 2012
Draft report to TETG, solicit TETG comments
August 2012
Final report with TETG comments incorporated
September 2012
Draft Guidebook updated to incorporate experiment findings, solicit TETG comments
September 2012
Final guidebook
December 2012
28
Questions Beverly Kuhn, TTI [email protected] 979-862-3558 Karl Wunderlich, Noblis [email protected] 202-488-5707 Vaishali Shah, Noblis [email protected] 202-488-5715 29
SHRP2 L14 TETG Briefing – July 18, 2012
Meeting Agenda
Welcome / Technology Check Introductions Meeting Objectives Project Objectives & Progress to Date Progress Report on Enhanced Experiment Remaining Project Timeline Final Remarks
2
Introductions
3
Meeting Objectives
INFORM TETG of the progress on the enhanced experiment. UPDATE the TETG on progress in the development of final deliverables. REVIEW the remaining project timeline.
4
Project Objectives & Progress to Date Beverly Kuhn, TTI
Phase 1: Foundational Research
SHRP2 L14: CONTRIBUTIONS OF EXPANDED EXPERIMENTATION
Innovation Scan, State of Practice Report, Expert Interviews
Definition of travel time reliability information Current forms and media for reliability information Focus Groups/ Surveys Preferred terms Alternative media
Precursor Experiment Evaluation methodology Serenity-sensitive utility function
Phase 2: Focused Experiments
Expanded Experimentation
8
Evaluated forms and terms Valuation of reliability info Guidebook • • • • •
What is reliability information? How does it differ from real-time traveler information? How can reliability information be conveyed? How can the impacts of reliability information be valued? What forms of reliability information are most effective?
Phase 1: Foundational Research
SHRP2 L14: RELIABILITY INFORMAITON GUIDEBOOK DEVELOPMENT
Innovation Scan, State of Practice Report, Expert Interviews
Definition of travel time reliability information Current forms and media for reliability information Focus Groups/ Surveys Preferred terms Alternative media
Precursor Experiment Evaluation methodology Serenity-sensitive utility function
Phase 2: Focused Experiments
Expanded Experimentation
Evaluated forms and terms Valuation of reliability info Guidebook • • • • •
What is reliability information? How does it differ from real-time traveler information? How can reliability information be conveyed? How can the impacts of reliability information be valued? What forms of reliability information are most effective?
9
Lexicon
8
Guidebook Outline
Introduction Key Concepts Travel
Time Reliability Reliability Information
Relevant Travel Time Reliability Terms Methods of Communication (Lexicon) Measuring Information Effectiveness Final Remarks References 9
Progress Report on Enhanced Experiment Karl Wunderlich, Noblis Vaishali Shah, Noblis
Experiment Hypotheses
Provision of reliability information improves on-time performance versus a control group that receives no reliability information Some
forms will be more effective than others Perceived benefit will trail actual benefit
Benefits of reliability information will decrease over time as participants build a larger experience base The provision of reliability signposts in real-time information will improve on-time performance
Two Distinct Experiments
Experiment X –Compare different forms and content of data to unfamiliar travelers
5 sets of reliability information tested by each participant through 5 weeks of simulated commuting, each week in a different city.
Experiment Y – Compare the learning curve for unfamiliar travelers with or without reliability information Half of participants receive simulated reliability information over a 4 week period in one city Other half do not receive any reliability information.
Experiment X: Reliability Terms to be Tested (based on terminology document)
F.
Baseline radio information presented textually Text-based 95th percentile Text-based average plus 95th percentile Text-based 20th, average, and 95th percentile Reliability Signposting Graphical “C” G. Graphical “D”
H.
Auditory “B”
A. B. C. D. E.
13
Demo of Experiments
Pre-experiment survey, commuting game, and mini post-experiment survey Often experiment completed within 60-70 minutes in Washington DC, while we ‘had’ participants for 90 minutes. Consequently, paper survey added in Chicago and Houston requests participants rank six of seven types of info (auditory was excluded)
Experimental Plan by Numbers
Reliability Data Content
X Two Distinct Experiments: (5 cities, 1 week each city) Experiment Version 1 2 3 4 Planned Participant Count 90 30 30 30 Completed Thus Far (DC & Houston) 70 18 13 17 A. Contol (no reliability info) P P P P TH B. Textual 95 P P TH C. Textual Avg + 95 P P P P TH TH D. Textual 20 + Avg + 95 P P P E. Visual and Textual signposting P F. Graphical presentation of "C" P P G. Graphical presentation of "D" P P H. Auditory presentation of "B" P P
Y (1 city, 4 weeks)
5 30 20 P
6 30 20
P
Experiment X: Pairwise Comparison Sample Size for Comparison of Different Types of Reliability Information (Experiment 1) A
B
C
D
E
F
G
A. Control B. 95th percentile text 83 C. Average and 95th percentile text 118 83 D. 20th, average, and 95th perctile text 105 70 105 E. average and signpost 69 70 70 70 F. Graphical C 31 13 31 18 18 G. Graphical D 35 0 35 35 18 18 H. Audio B 30 13 13 17 13 13 17 * each participant will make 5 trips using each type of reliability info
H
Preliminary Findings (83 observations)
Textual 95th v. Control
*90th percentile significance, others 95th percentile or higher significance EOW = End of week
Perceptions
Differences Not Stat. Significant Lower Post-trip and EOW willingness to pay Lower EOW understandability of info Higher EOW stress Higher EOW managing trip decisions
Outcomes
Statistically significant findings Lower week’s trip cost 50% lower late arrival frequency Higher pre-trip arrival confidence: 3.5 v. 3.3 Higher pre-trip but lower post-trip* usefulness Higher EOW usefulness: 3.4 v. 2.9
Text Avg+ 95th v. Control Statistically significant findings Lower week’s trip cost: $35 v $48 Lower late arrival frequency: 0.8 v 1.5 days/wk Higher pre-trip arrival confidence: 3.5 v 3.3 Higher pre-trip but lower post-trip* usefulness Higher EOW usefulness: 3.6 v 3.0 Higher post-trip and EOW* willingness to pay Higher EOW stress Higher EOW managing trip decisions Differences Not Stat. Significant Higher EOW understandability of info *95 percentile significance, others 99th percentile significance **End of week
Willingness to Pay for Info
Preliminary Findings (118 observations)
Preliminary Findings (105 observations)
Textual 20th+avg+95th v. Control Statistically significant findings Lower week’s trip cost $39 v $49 Lower late arrival frequency 1.1 v 1.5 Higher pre-trip arrival confidence Higher pre-trip & EOW, but lower post-trip usefulness Higher EOW managing trip decisions Higher EOW stress Higher post-trip and EOW* willingness to pay $0.32 and $0.18 more per trip, respectively
Differences Not Stat. Significant Higher EOW understandability of info
*90th percentile significance, others 95th percentile or higher significance EOW = End of week
Reliability Info Control Info
Preliminary Findings (70 observations)
Signpost v. Control Statistically significant findings Higher pre-trip and EOW* usefulness Higher post-trip stress 3.0 v 2.5 Higher EOW understandability 2.5 v 2.1 Differences Not Stat. Significant Lower week’s trip cost Lower post-trip and EOW willingness to pay for info Lower late arrival frequency Similar pre-trip on-time arrival confidence Lower Post-trip usefulness Lower EOW managing trip decisions Higher EOW stress *90th percentile significance, others 95th percentile or higher significance EOW = End of week 20
Preliminary Findings (31 observations)
Graphic Avg+95th v. Control Statistically significant findings Higher pre-trip and EOW* usefulness 3.6 v 2.9 and 4.1 v. 3.2, respectively Higher arrival confidence: 3.6 v 3.3 Higher post-trip willing to pay for info $0.48 more Higher post-trip and EOW stress 3.7 v 2.4 and 2.8 v 2.4 Higher EOW managing decisions 4.0 v 3.1 Differences Not Stat. Significant Lower week’s trip cost Lower late arrival frequency Lower Post-trip usefulness Higher EOW understandability Higher EOW willingness to pay for info *90th percentile significance, others 95th percentile or higher significance EOW = End of week 21
Preliminary Findings (31 observations)
Graphic 20th+Avg+95th v. Control Statistically significant findings Higher pre-trip usefulness Higher post-trip willingness to pay for info Higher EOW understandability Differences Not Stat. Significant Lower week’s trip cost Lower late arrival frequency Equal arrival confidence Higher Post-trip usefulness Higher EOW stress Higher EOW usefulness Higher EOW managing decisions Higher EOW willingness to pay for info
*90th percentile significance, others 95th percentile or higher significance EOW = End of week 22
Preliminary Findings (13 sample size)
Textual 95th v. Voice 95th Statistically significant findings
Voice has higher pre-trip usefulness:
4.0 v 3.1
Higher post-trip willingness to pay:
$3.9 v $3.1
Higher EOW usefulness:
4.7 v. 3.4
Higher EOW - better manage decisions: 4.7 v. 3.4
Lower EOW understandability:
1.7 v. 2.5
Higher EOW stress
4.0 v. 2.8
Differences Not Statistically Significant
Voice has higher late arrival freq.
Lower total cost
Higher on-time arrival confidence
Other preliminary comparisons…
Differences, based on a sample size of 31, for text versus graphic presentation of average and 95th percentile reliability information do not shows statistically significant differences in metrics at this preliminary juncture. Differences, based on a sample size of 35, for text versus graphic presentation of “good, typical, and bad” reliability information do not show statistically significant differences in metrics at this preliminary juncture.
24
Preliminary Implications on Hypotheses
Provision of reliability information improves on-time performance versus a control group that receives no reliability information Some
forms will be more effective than others Perceived benefit will trail actual benefit
Benefits of reliability information will decrease over time as participants build a larger experience base The provision of reliability signposts in real-time information will improve on-time performance
Next steps…
Merge in Chicago data and analyze results Analyze pre and post experiment surveys to determine
rankings among forms of presented information determine participant characteristics
Analyze Experiment “Y” to determine whether reliability benefits decrease over time
Remaining Project Timeline Beverly Kuhn, TTI
Next Steps
Preliminary findings for TETG
July 2012
Draft report to TETG, solicit TETG comments
August 2012
Final report with TETG comments incorporated
September 2012
Draft Guidebook updated to incorporate experiment findings, solicit TETG comments
September 2012
Final guidebook
December 2012
28
Questions Beverly Kuhn, TTI [email protected] 979-862-3558 Karl Wunderlich, Noblis [email protected] 202-488-5707 Vaishali Shah, Noblis [email protected] 202-488-5715 29
