Defining Space amazonaws com
Defining Space How do we combine what we see and hear?
The McGurk Effect: Integration of vision and hearing
Ventriloquism
Vision and Hearing
Line of sight
Visual signals are “eye-centered”
Auditory information about sound location • Interaural timing differences depend on sound location with respect to the head
Auditory information about sound location
• Interaural level differences depend on sound location with respect to the head
Auditory information about sound location • Spectral cues are “earcentered” • Which is the same as “headcentered” if your ears don’t move
Vision and Hearing
Line of sight
What this means
rustle
quack quack
quack
ripple 1. Potential correspondence between every point in visual space and every point in auditory space 2. True correspondence governed by eye position relative to the head
Communication between visual and auditory systems: • Requires information about eye position • Eye position modulates activity at multiple levels of the auditory pathway (all that have been tested) • Suggests widespread interactions between visual and auditory pathways
Auditory pathway
Medial Geniculate Nucleus of the Thalamus Inferior Colliculus Superior Olivary Complex Cochlear Nucleus (brain) Cochlea (ear)
Information flow è
Information flow è
Auditory Cortex
Medial Geniculate Nucleus of the Thalamus Inferior Colliculus Superior Olivary Complex Cochlear Nucleus (brain) Cochlea (ear)
Information flow è
Information flow è
Auditory Cortex
Schematicly…
Number of “spikes”
Response to a given sound
Eye position
Ventriloquism
Ventriloquism AfterEffect depends on eye position Kopco N, Lin IF, ShinnCunningham BG, Groh JM (2009) Reference frame of the ventriloquism aftereffect. J Neurosci 29:13809-13814
Shift is ventriloquism aftereffect +
Ventriloquism aftereffect biggest when sound location and eye position match the original exposure
+
Ventriloquism aftereffect biggest when sound location and eye position both match the original exposure
+
References (page 1) • Groh JM, Sparks DL (1992) Two models for transforming auditory signals from head-centered to eye- centered coordinates. Biological Cybernetics 67:291-302. • Groh JM, Trause AS, Underhill AM, Clark KR, Inati S (2001) Eye position influences auditory responses in primate inferior colliculus. Neuron 29:509-518. • Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, Groh JM (2003) Eye position affects activity in primary auditory cortex of primates. Current Biology 13:554-562. • Fu KM, Shah AS, O'Connell MN, McGinnis T, Eckholdt H, Lakatos P, Smiley J, Schroeder CE (2004) Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. J Neurophysiol 92:3522-3531. .
References (page 2) • Zwiers MP, Versnel H, Van Opstal AJ (2004) Involvement of monkey inferior colliculus in spatial hearing. J Neurosci 24:4145-4156. • Porter KK, Metzger RR, Groh JM (2006) Representation of eye position in primate inferior colliculus. J Neurophysiol 95:1826-1842. • Kopco N, Lin IF, Shinn-Cunningham BG, Groh JM (2009) Reference frame of the ventriloquism aftereffect. J Neurosci 29:13809-13814 • Bulkin DA, Groh JM (2012) Distribution of eye position information in the monkey inferior colliculus. J Neurophysiol 107:785-795.
Next up: Encoding sound location with respect to the eyes
The McGurk Effect: Integration of vision and hearing
Ventriloquism
Vision and Hearing
Line of sight
Visual signals are “eye-centered”
Auditory information about sound location • Interaural timing differences depend on sound location with respect to the head
Auditory information about sound location
• Interaural level differences depend on sound location with respect to the head
Auditory information about sound location • Spectral cues are “earcentered” • Which is the same as “headcentered” if your ears don’t move
Vision and Hearing
Line of sight
What this means
rustle
quack quack
quack
ripple 1. Potential correspondence between every point in visual space and every point in auditory space 2. True correspondence governed by eye position relative to the head
Communication between visual and auditory systems: • Requires information about eye position • Eye position modulates activity at multiple levels of the auditory pathway (all that have been tested) • Suggests widespread interactions between visual and auditory pathways
Auditory pathway
Medial Geniculate Nucleus of the Thalamus Inferior Colliculus Superior Olivary Complex Cochlear Nucleus (brain) Cochlea (ear)
Information flow è
Information flow è
Auditory Cortex
Medial Geniculate Nucleus of the Thalamus Inferior Colliculus Superior Olivary Complex Cochlear Nucleus (brain) Cochlea (ear)
Information flow è
Information flow è
Auditory Cortex
Schematicly…
Number of “spikes”
Response to a given sound
Eye position
Ventriloquism
Ventriloquism AfterEffect depends on eye position Kopco N, Lin IF, ShinnCunningham BG, Groh JM (2009) Reference frame of the ventriloquism aftereffect. J Neurosci 29:13809-13814
Shift is ventriloquism aftereffect +
Ventriloquism aftereffect biggest when sound location and eye position match the original exposure
+
Ventriloquism aftereffect biggest when sound location and eye position both match the original exposure
+
References (page 1) • Groh JM, Sparks DL (1992) Two models for transforming auditory signals from head-centered to eye- centered coordinates. Biological Cybernetics 67:291-302. • Groh JM, Trause AS, Underhill AM, Clark KR, Inati S (2001) Eye position influences auditory responses in primate inferior colliculus. Neuron 29:509-518. • Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, Groh JM (2003) Eye position affects activity in primary auditory cortex of primates. Current Biology 13:554-562. • Fu KM, Shah AS, O'Connell MN, McGinnis T, Eckholdt H, Lakatos P, Smiley J, Schroeder CE (2004) Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. J Neurophysiol 92:3522-3531. .
References (page 2) • Zwiers MP, Versnel H, Van Opstal AJ (2004) Involvement of monkey inferior colliculus in spatial hearing. J Neurosci 24:4145-4156. • Porter KK, Metzger RR, Groh JM (2006) Representation of eye position in primate inferior colliculus. J Neurophysiol 95:1826-1842. • Kopco N, Lin IF, Shinn-Cunningham BG, Groh JM (2009) Reference frame of the ventriloquism aftereffect. J Neurosci 29:13809-13814 • Bulkin DA, Groh JM (2012) Distribution of eye position information in the monkey inferior colliculus. J Neurophysiol 107:785-795.
Next up: Encoding sound location with respect to the eyes
