the relation between v1 neuronal responses and ... - Semantic Scholar
THE RELATION BETWEEN V1 NEURONAL RESPONSES AND EYE MOVEMENT-LIKE STIMULUS PRESENTATIONS. Barry J. Richmond Laboratory of Neuropsychology National Institute of Mental Health Bethesda MD [email protected]
John A. Hertz Nordita DK-2100 Copenhagen Denmark [email protected]
Timothy J. Gawne Department of Physiological Optics School of Optometry University of Alabama Birmingham, AL [email protected] October 15, 1998
Abstract Primates normally make 2-3 saccadic eye movements/sec to explore the environment. To investigate how these eye movements might influence visual responses, we compared the dynamics of stimuli arriving on V1 complex cell receptive fields by switching stimuli in sequence while a monkey fixated to the responses occurring when the stimulus appears due to saccadic eye movements. During the image sequences, information was greater when each image remained on the receptive fields longer, up to 200 ms; information was greatest when there was a gap of 50 ms between images. Responses were more variable when the image appeared due to a saccadic eye movement. The amount of stimulus-related information was lower in the early phase of the postsaccadic time, but increased during the post-saccadic fixation, so that after 400 ms there was almost as much stimulus-related information available as during the image switching. Eye position showed much larger variability after saccades, with the variability decreasing over 350-400 ms to reach the level seen during long fixations. The dynamics of information accumulation in V1 complex cells appear to be well matched to the manner in which the environment is normally viewed. Correspondence to: BJR, Bldg. 49, Room 1B80, Bethesda, MD 20892-4415. Phone: (301) 496-5625 Fax: (301) 402-0046
1
Richmond, Hertz and Gawne
2
INTRODUCTION Humans and primates normally fixate for short periods of time, and then make a saccadic eye movement to fixate some other location, either a different part of the same object or a different object. Thus, during normal viewing visual stimuli arrive suddenly on V1 receptive fields as a consequence of these saccadic eye movements. This happens 2-3 times per second, meaning that fixations last 200-400 ms, with each saccade taking another 20-40 ms. Despite these dynamics, recordings of single neurons in the early visual system are usually carried out while animals are either anesthetized or while they are fixating. It is easy to understand why this is done; the eyes are relatively stationary so there is good control over where images fall on the retinal receptors. However, it is possible that the responsivity of single neurons is different when animals are looking around, instead of fixating. Although the physiological correlates of the lack of vision during saccadic eye movements has been studied [Judge et al, 1980, Macknik and Livingstone, 1998], it is only recently there has been efforts to study visual responses during natural viewing conditions [Gallant et al, 1998]. There are two aspects to investigate. The first is the effect of the stimulus changing, isolated from the effect of eye movements. The second is the effect of the eye movements.
METHODS We recorded from 29 supragranular complex cells in monkeys. To study the effect of the stimulus changing without the influence of eye movements we constructed sequences with 16 successive images. Two sets of stimuli were used, oriented grating patches and Walsh patterns. Stimuli were positioned so that they covered the receptive field and part of the near surround. The images in the sequences were independent of each other; they were not explicitly related so as to elicit perceptions of motion such as is done for animation. Images from the two sets were not mixed in the same movie. To study the visual responses starting after saccades, the monkeys were required to make saccades to fixate a target jumping randomly among four locations. The same stimulus sets were used. Individual stimuli were positioned so that each saccadic eye movement brought a new stimulus onto the receptive field of the neuron. Finally to allow direct comparisons between the saccadic condition and the movie condition, a new stimulus appeared on the receptive field 400 ms after the monkey’s eyes arrived within 3 degrees of the fixation target, as in the longer image sequences. In addition to looking at rasters, we calculated the mutual or transmitted information between the stimuli and the response. Mutual or transmitted information is defined as I (S; R) = ∑ S
p(sjr) p(r) p(sjr)log2 p(s)
;
r
where I (S; R) is the information transmitted about the stimuli, S, given in the responses, R. s is the stimulus related to response r. p(s) is the a priori probability of the stimulus, which is known in the experiment. p(sjr) is the probability of s given r, i.e., the conditional probability of the condition being selected on the basis of the response. The
Richmond, Hertz and Gawne
3
maximum amount of information that could be transmitted regarding the condition is its entropy, H (S) = ; ∑ p(c)log2 p(s). A neural network was used to carry out a nominal regression of the stimulus on the neural response to estimate p(sjr) [Kjaer et al, 1994]. The spike counts were used as inputs to the neural network, and the stimuli were used as target outputs.
RESULTS The responses to patterns presented in a movie while the monkey fixates appear quite variable (Figure 1). To obtain data for analysis we randomly change the stimulus order
Figure 1: Example of data from a movie with the same image sequence presented repeatedly. Notice that the there are some big and some smaller responses. The apparent sequential decrease in response from the trigger line onward is related to the particular stimuli (note that the burst just before the trigger is also small).
-100
100
300
500
700
-100
100
300
500
700
in successive movies. When we measured the stimulus-related information available in the responses there were two effects. First, the information rose to higher levels when each image in the movie was presented for a longer time (Figure 2A). The was even more information when there was a short gap between the movie frames (on the order of 30-50 ms) (Figure 2B). Second, there was a another burst of information after the stimulus disappeared (Figure 2). This probably corresponds to the increased response variability seen in the random order movies. When we use the time that the eye enters a 1.5 degree region around the fixation target as the time of response onset (as if the stimulus came on then), the responses show some characteristics similar to those seen when stationary stimuli are flashed onto the receptive field (i.e., crisp onsets to optimal stimuli and tunings to orientation and spatial frequency). However, when the stimulus arrives on the receptive field due to a saccadic eye movement, responses appear less consistent than when they appear as the monkey is fixating (Figure 3A). The end-saccade triggered responses appear to differ from the responses that are elicited by the flashed stimulus in three ways. First, the latency appears more variable. Second, the large responses, which tend to show a large initial burst in the flashed condition, do not seem to have this burst when the neuron responds after the end of the saccade. Third, the strong responses tend to be more prolonged in time when the neuron starts firing after the saccade. The orientation tuning curves for the grating patches are similar for the whether the stimuli appear at the end of a saccade or appear by appearing while the monkey is fixating (Figure 3B).
Richmond, Hertz and Gawne
stimuli were brought into the receptive field of a neuron via a saccadic eye movement vs when the stimulus is flashed onto the receptive. The responses appear more variable in strength and time of onset (Saccades) than when the stimulus appears while the eye is stationary (Flashed). B. Tuning surfaces for orientation and spatial frequency. The tuning peak is at the same orientation. The tuning is less smoothly low pass in spatial frequency and more variable across spatial frequency when the stimulus is flashed onto the receptive rather than appearing because of a saccade.
0.7 CUMLATIVE INFORMATION 0.1 0.2 0.3 0.4 0.5 0.6
170 ms
136 ms
136 ms per pattern 170 ms per pattern
0
100 200 300 TIME FROM STIMULUS ONSET
400
CUMLATIVE INFORMATION 0.2 0.4 0.6 0.8
1.0
B
51 ms gap
136 ms
0.0
0 ms between patterns 51 ms between patterns
0
100 200 300 TIME FROM STIMULUS ONSET
400
A -100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
Saccades
-100
100
300
500
-100
100
Flashed
B 2
no. of spikes 0 5 10 15 20 2
Figure 3: A. Comparing responses when
A
0 5 10 15 20 25
Figure 2: A. Information in a growing window. Zero is the time of stimulus appearance. The rising lines show how the information grows starting at the time of stimulus appearance. The curves are for two different movie frame times, 136 and 170 ms. The information starts rising approximately 60 ms after the stimulus appears; it rises rapidly and then the rise slows until 60 ms after the stimulus disappears, at which time another short rise in information occurs. We speculate that this short rise after the stimulus disappears (and when the next stimulus appears) is related to the classic off-response. Information drops late when the response includes periods where the response is no longer related to the stimulus (adding noise by making the time window too wide). B. Information accumulation when there is a short gap (here 51 ms) between stimulus frames. Most of the information in the stimulus-elicited response of this neuron was present after the stimulus disappeared, but ended quickly when the next stimulus appeared, again suggesting that the off-response might carry stimulusrelated information.
4
Saccades
1.8
2
1.6
1.4
80
601
1.2 1
401 201
001
40
60
1 80
Flashed
1.8 Sp 1.6 at 1.4 ial fre 1.2 qu en 1 cy
80
1601 140 120 100 on 80 tati 60 Orien 40
Richmond, Hertz and Gawne
5
Comparing the stimulus-related information after the end of a saccade to the information when the stimulus changes while the animal fixates (flashed), the information in the flashed condition approaches an asymptotic value earlier (Figure 2). However, by the late part of the fixation the information is nearly equal in the two conditions.
vert position (deg) 0 5 10
100 ms ••••••••••• •••• •• ••••••• •••• ••••• ••• ••• ••••••••• •• ••••••• • ••••••• •••• •••• •••••• •• • ••••••••• •
••• ••• •••• •••• ••••••• •••• •• ••••• ••• ••• •• ••••• ••••• •••• •• ••• •• •••• •••••• •••• •••••• •••• ••• ••• ••••• ••••• ••••••••••••••• ••••••• • •
•••••••••• • •• • •••• ••••••• ••• •••• ••• ••••••••••••••••• •••••••• •••••••• •••••• •••••• ••• ••• ••••• •••••• •••• •••• ••• ••• ••• •••• •
• • •••••••• •••••••••••••••••••• •• •••••••••••••• ••••••• •••• ••••••••••••••••••••••••••••• ••••••• •• ••••••••• •••••• •• ••••• •
400 ms ••••••••• ••••• ••••• •• ••• •• •• •• ••••• ••• • ••••• •••••• •
••••• ••• •••• •• ••••• ••• •••••• ••• ••••••• ••• •••• •••••••• •••••• •••• ••• ••••••••••
•••• •• • ••••• •• ••• ••••••• ••••• •••• •• •• ••••• ••• • ••••••• •• •••••••
•• ••• ••••• •••••••• •• ••• • •• •••• ••••• •••••••••••• • •••••••• •• •
•
0 5 10 horiz position (deg)
0 5 10 horiz position (deg)
B
••• ••• ••• •• •••• ••••• •• ••• •••• ••••• •••••• •••••••••• ••••••••• •
• ••••••••• ••• ••••••• •• •••• •••••••• ••••••••• •••••••
0
5
10
400 ms • ••• •••••••••• • •••• •••••• •••• ••••• ••••
••• •• ••• •• ••• •••• ••••••••••• ••• ••• •••• ••••••• ••• •• ••• •••• ••••• ••• ••
•••• ••• ••••••• •••• ••• ••••• •••• • •••• •••• ••••••• •••••• •• ••• •••• •
•••• ••••••••••• ••••• ••••••• ••• ••••• ••• •• ••••••• •••• ••••• •• ••• •
5
10
••• ••• •• ••• ••• ••• ••••••••••••••••• •••• ••••• •••• ••• •••••• •• •••••• •••• ••••
5
• ••• ••• •••• •••••• •• ••• •• •• •••• ••• •••• •• •• ••••••• • ••• ••• ••• •••
0
vert position (deg)
sacmv006
100 ms
0
vert position (deg)
Figure 4: Scatter plots of eye position at different times (shown on each plot) (A) after the end of a saccade, or (B) after the stimulus changed as the monkey fixated. The scatter or variance in eye position is largest immediately after the saccade ends and becomes progressively smaller as the fixation period becomes longer, stabilizing about 350 or 400 ms after the saccade has ended. There is no discernible effect on fixation by changing the stimulus as the animal fixates.
vert position (deg) 0 5 10
A
•
10
0
horiz position (deg)
5
10
horiz position (deg)
•
After Flash Information (bits) 0.2 0.4 0.6
end of saccade and after switching the stimulus during fixation. The information rises more during the initial burst after switching. After approximately 400 ms it reaches nearly the same level after saccades. The increase seen in the movies after stimulus disappearance is seen in the curve labeled ’After Saccades’ at the time the stimulus switches while the monkey fixates.
•
•
•
•
•
•
• •
•
•
•
•
•
•
•
• •
•
•
After Saccades
•
•
• •
0.0
Figure 5: Information accumulation after
0.8
Accumulated information after trigger event
•
0
100
200 300 400 Time after trigger
500
600
The variability in the response dynamics when a saccade ends raises the question of what the monkeys eyes are doing at this time. Examination of the fixation shows that the eye positions are more scattered immediately after the saccades end (even though the eye is not moving much), and the scatter in eye position becomes much less as the fixation continues (Figure 4). However, it is not until 350 or 400 ms after the end of the saccade before the eye position scatter matches that seen during continual fixation.
DISCUSSION The responsivity of V1 supragranular complex cells appears to be well-adapted to carry information as we make saccades to explore the visual environment. In the experiments with the movies, the stimulus-related information grew as the period that each image remained on increased. There was a second burst of stimulus-related information after the stimulus disappeared. This burst of information was greatest if there was a gap lasting 50 ms between successive movie frames. We speculate that this burst of informa-
REFERENCES
6
tion is related to the off-response. This analysis suggests that the off-response, which might arise in the retinal receptors, has stimulus selectivity. The very small amount of stimulus-related information that is present when the stimuli are being changed at a rapid rate (faster than one stimulus/50-75ms) suggests that there is an effect a stimulus disappearing will affect the response of new stimulus. A great deal of this might arise from mixing effects of off-responses with the subsequent on-response. The sequence of timings that give the largest information is one that is consistent with the retinal events that occur during saccadic exploration of the environment, i.e. an image arrives on a receptive field, remains there for 200 or more milliseconds, and then a saccadic eye movement creates a smear (which might be like our grey background) lasting from 20-50 ms, followed by a new image. The greater variability that occurs in responses starting after a saccade ends vs after a flashed stimulus could be related to the greater variability of the stimulus position on the receptive field. However, it is also clear that the latency is more variable. We do not know what the relation between the end of the saccade and the onset of the neuronal response is. Nonetheless, the amount of information that is available by the end of a normal fixation period is similar to that available after the same amount of time when a stimulus appears on the retina during fixation. Thus, it appears that the time between normal eye movements allows information in V1 neuronal responses to accumulate to near its apparent asymptotic values.
References [Gallant et al, 1998]
Gallant, J.L., Connor, C.E. and Van Essen, D.C. (1998) Neural activity in V1, V2 and V4 during free viewing of natural scenes compared to controlled viewing. Neuroreport 9: 2153-2158.
[Judge et al, 1980]
Judge, S.J., Wurtz, R.H and Richmond, B.J. (1980) Vision during saccadic eye movements: I. Visual interactions in striate cortex. J. Neurophysiol. 43: 1133-1155.
[Kjaer et al, 1994]
Kjaer, T.W., Hertz, J.A. and Richmond, B.J. (1994) Decoding Cortical Neuronal Signals: Network Models, Information Estimation and Spatial Tuning. J Computational Neuroscience 1: 109-139.
[Macknik and Livingstone, 1998] Macknik, S.L. and Livingtone, M.S. (1998) Neuronal correlates of visibility and invisibility in the primate visual system. Nature Neurosci. 1: 144149.
Barry J. Richmond Barry Richmond received his M.D. in 1971 from Case-Western Reserve University, where he did a pediatric residency. After a three year neurology residency at the Harvard-Longwood Program, he went to the NIH to do a postdoc with Bob Wurtz. In 1980, he joined the Laboratory of Neuropsychology of NIMH, where he has been investigating the structure and content of neuronal codes in the visual system, and more recently, also the physiological basis of motivation and reward.
John A. Hertz John Hertz received his PhD from the University of Pennsylvania in 1970. He served on the faculty of the University of Chicago for seven years before moving to Nordita in Copenhagen in 1980. He has worked on problems in statistical physics, including critical phenomena and spin glasses, with the emphasis shifting in
recent years toward problems in biological systems, including neural networks. He collaborates extensively with colleagues in the Laboratory of Neuropsychology, NIMH.
Tim J. Gawne Tim Gawne received his Ph.D. in physiology from the Uniformed Services University of the Health Sciences in 1985 studying cardiovascular reflexes related to exercise and shock. He changed to neuroscience when he joined Barry Richmond at the NIH in 1987. Ever since he has been trying to understand how the coordinated activity of the many neurons in the visual system allows us to perceive and recognize objects. He has been in the Dept. of Physiological Optics at the University of Alabama, Birmingham since 1996.
7
John A. Hertz Nordita DK-2100 Copenhagen Denmark [email protected]
Timothy J. Gawne Department of Physiological Optics School of Optometry University of Alabama Birmingham, AL [email protected] October 15, 1998
Abstract Primates normally make 2-3 saccadic eye movements/sec to explore the environment. To investigate how these eye movements might influence visual responses, we compared the dynamics of stimuli arriving on V1 complex cell receptive fields by switching stimuli in sequence while a monkey fixated to the responses occurring when the stimulus appears due to saccadic eye movements. During the image sequences, information was greater when each image remained on the receptive fields longer, up to 200 ms; information was greatest when there was a gap of 50 ms between images. Responses were more variable when the image appeared due to a saccadic eye movement. The amount of stimulus-related information was lower in the early phase of the postsaccadic time, but increased during the post-saccadic fixation, so that after 400 ms there was almost as much stimulus-related information available as during the image switching. Eye position showed much larger variability after saccades, with the variability decreasing over 350-400 ms to reach the level seen during long fixations. The dynamics of information accumulation in V1 complex cells appear to be well matched to the manner in which the environment is normally viewed. Correspondence to: BJR, Bldg. 49, Room 1B80, Bethesda, MD 20892-4415. Phone: (301) 496-5625 Fax: (301) 402-0046
1
Richmond, Hertz and Gawne
2
INTRODUCTION Humans and primates normally fixate for short periods of time, and then make a saccadic eye movement to fixate some other location, either a different part of the same object or a different object. Thus, during normal viewing visual stimuli arrive suddenly on V1 receptive fields as a consequence of these saccadic eye movements. This happens 2-3 times per second, meaning that fixations last 200-400 ms, with each saccade taking another 20-40 ms. Despite these dynamics, recordings of single neurons in the early visual system are usually carried out while animals are either anesthetized or while they are fixating. It is easy to understand why this is done; the eyes are relatively stationary so there is good control over where images fall on the retinal receptors. However, it is possible that the responsivity of single neurons is different when animals are looking around, instead of fixating. Although the physiological correlates of the lack of vision during saccadic eye movements has been studied [Judge et al, 1980, Macknik and Livingstone, 1998], it is only recently there has been efforts to study visual responses during natural viewing conditions [Gallant et al, 1998]. There are two aspects to investigate. The first is the effect of the stimulus changing, isolated from the effect of eye movements. The second is the effect of the eye movements.
METHODS We recorded from 29 supragranular complex cells in monkeys. To study the effect of the stimulus changing without the influence of eye movements we constructed sequences with 16 successive images. Two sets of stimuli were used, oriented grating patches and Walsh patterns. Stimuli were positioned so that they covered the receptive field and part of the near surround. The images in the sequences were independent of each other; they were not explicitly related so as to elicit perceptions of motion such as is done for animation. Images from the two sets were not mixed in the same movie. To study the visual responses starting after saccades, the monkeys were required to make saccades to fixate a target jumping randomly among four locations. The same stimulus sets were used. Individual stimuli were positioned so that each saccadic eye movement brought a new stimulus onto the receptive field of the neuron. Finally to allow direct comparisons between the saccadic condition and the movie condition, a new stimulus appeared on the receptive field 400 ms after the monkey’s eyes arrived within 3 degrees of the fixation target, as in the longer image sequences. In addition to looking at rasters, we calculated the mutual or transmitted information between the stimuli and the response. Mutual or transmitted information is defined as I (S; R) = ∑ S
p(sjr) p(r) p(sjr)log2 p(s)
;
r
where I (S; R) is the information transmitted about the stimuli, S, given in the responses, R. s is the stimulus related to response r. p(s) is the a priori probability of the stimulus, which is known in the experiment. p(sjr) is the probability of s given r, i.e., the conditional probability of the condition being selected on the basis of the response. The
Richmond, Hertz and Gawne
3
maximum amount of information that could be transmitted regarding the condition is its entropy, H (S) = ; ∑ p(c)log2 p(s). A neural network was used to carry out a nominal regression of the stimulus on the neural response to estimate p(sjr) [Kjaer et al, 1994]. The spike counts were used as inputs to the neural network, and the stimuli were used as target outputs.
RESULTS The responses to patterns presented in a movie while the monkey fixates appear quite variable (Figure 1). To obtain data for analysis we randomly change the stimulus order
Figure 1: Example of data from a movie with the same image sequence presented repeatedly. Notice that the there are some big and some smaller responses. The apparent sequential decrease in response from the trigger line onward is related to the particular stimuli (note that the burst just before the trigger is also small).
-100
100
300
500
700
-100
100
300
500
700
in successive movies. When we measured the stimulus-related information available in the responses there were two effects. First, the information rose to higher levels when each image in the movie was presented for a longer time (Figure 2A). The was even more information when there was a short gap between the movie frames (on the order of 30-50 ms) (Figure 2B). Second, there was a another burst of information after the stimulus disappeared (Figure 2). This probably corresponds to the increased response variability seen in the random order movies. When we use the time that the eye enters a 1.5 degree region around the fixation target as the time of response onset (as if the stimulus came on then), the responses show some characteristics similar to those seen when stationary stimuli are flashed onto the receptive field (i.e., crisp onsets to optimal stimuli and tunings to orientation and spatial frequency). However, when the stimulus arrives on the receptive field due to a saccadic eye movement, responses appear less consistent than when they appear as the monkey is fixating (Figure 3A). The end-saccade triggered responses appear to differ from the responses that are elicited by the flashed stimulus in three ways. First, the latency appears more variable. Second, the large responses, which tend to show a large initial burst in the flashed condition, do not seem to have this burst when the neuron responds after the end of the saccade. Third, the strong responses tend to be more prolonged in time when the neuron starts firing after the saccade. The orientation tuning curves for the grating patches are similar for the whether the stimuli appear at the end of a saccade or appear by appearing while the monkey is fixating (Figure 3B).
Richmond, Hertz and Gawne
stimuli were brought into the receptive field of a neuron via a saccadic eye movement vs when the stimulus is flashed onto the receptive. The responses appear more variable in strength and time of onset (Saccades) than when the stimulus appears while the eye is stationary (Flashed). B. Tuning surfaces for orientation and spatial frequency. The tuning peak is at the same orientation. The tuning is less smoothly low pass in spatial frequency and more variable across spatial frequency when the stimulus is flashed onto the receptive rather than appearing because of a saccade.
0.7 CUMLATIVE INFORMATION 0.1 0.2 0.3 0.4 0.5 0.6
170 ms
136 ms
136 ms per pattern 170 ms per pattern
0
100 200 300 TIME FROM STIMULUS ONSET
400
CUMLATIVE INFORMATION 0.2 0.4 0.6 0.8
1.0
B
51 ms gap
136 ms
0.0
0 ms between patterns 51 ms between patterns
0
100 200 300 TIME FROM STIMULUS ONSET
400
A -100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
-100
100
300
500
Saccades
-100
100
300
500
-100
100
Flashed
B 2
no. of spikes 0 5 10 15 20 2
Figure 3: A. Comparing responses when
A
0 5 10 15 20 25
Figure 2: A. Information in a growing window. Zero is the time of stimulus appearance. The rising lines show how the information grows starting at the time of stimulus appearance. The curves are for two different movie frame times, 136 and 170 ms. The information starts rising approximately 60 ms after the stimulus appears; it rises rapidly and then the rise slows until 60 ms after the stimulus disappears, at which time another short rise in information occurs. We speculate that this short rise after the stimulus disappears (and when the next stimulus appears) is related to the classic off-response. Information drops late when the response includes periods where the response is no longer related to the stimulus (adding noise by making the time window too wide). B. Information accumulation when there is a short gap (here 51 ms) between stimulus frames. Most of the information in the stimulus-elicited response of this neuron was present after the stimulus disappeared, but ended quickly when the next stimulus appeared, again suggesting that the off-response might carry stimulusrelated information.
4
Saccades
1.8
2
1.6
1.4
80
601
1.2 1
401 201
001
40
60
1 80
Flashed
1.8 Sp 1.6 at 1.4 ial fre 1.2 qu en 1 cy
80
1601 140 120 100 on 80 tati 60 Orien 40
Richmond, Hertz and Gawne
5
Comparing the stimulus-related information after the end of a saccade to the information when the stimulus changes while the animal fixates (flashed), the information in the flashed condition approaches an asymptotic value earlier (Figure 2). However, by the late part of the fixation the information is nearly equal in the two conditions.
vert position (deg) 0 5 10
100 ms ••••••••••• •••• •• ••••••• •••• ••••• ••• ••• ••••••••• •• ••••••• • ••••••• •••• •••• •••••• •• • ••••••••• •
••• ••• •••• •••• ••••••• •••• •• ••••• ••• ••• •• ••••• ••••• •••• •• ••• •• •••• •••••• •••• •••••• •••• ••• ••• ••••• ••••• ••••••••••••••• ••••••• • •
•••••••••• • •• • •••• ••••••• ••• •••• ••• ••••••••••••••••• •••••••• •••••••• •••••• •••••• ••• ••• ••••• •••••• •••• •••• ••• ••• ••• •••• •
• • •••••••• •••••••••••••••••••• •• •••••••••••••• ••••••• •••• ••••••••••••••••••••••••••••• ••••••• •• ••••••••• •••••• •• ••••• •
400 ms ••••••••• ••••• ••••• •• ••• •• •• •• ••••• ••• • ••••• •••••• •
••••• ••• •••• •• ••••• ••• •••••• ••• ••••••• ••• •••• •••••••• •••••• •••• ••• ••••••••••
•••• •• • ••••• •• ••• ••••••• ••••• •••• •• •• ••••• ••• • ••••••• •• •••••••
•• ••• ••••• •••••••• •• ••• • •• •••• ••••• •••••••••••• • •••••••• •• •
•
0 5 10 horiz position (deg)
0 5 10 horiz position (deg)
B
••• ••• ••• •• •••• ••••• •• ••• •••• ••••• •••••• •••••••••• ••••••••• •
• ••••••••• ••• ••••••• •• •••• •••••••• ••••••••• •••••••
0
5
10
400 ms • ••• •••••••••• • •••• •••••• •••• ••••• ••••
••• •• ••• •• ••• •••• ••••••••••• ••• ••• •••• ••••••• ••• •• ••• •••• ••••• ••• ••
•••• ••• ••••••• •••• ••• ••••• •••• • •••• •••• ••••••• •••••• •• ••• •••• •
•••• ••••••••••• ••••• ••••••• ••• ••••• ••• •• ••••••• •••• ••••• •• ••• •
5
10
••• ••• •• ••• ••• ••• ••••••••••••••••• •••• ••••• •••• ••• •••••• •• •••••• •••• ••••
5
• ••• ••• •••• •••••• •• ••• •• •• •••• ••• •••• •• •• ••••••• • ••• ••• ••• •••
0
vert position (deg)
sacmv006
100 ms
0
vert position (deg)
Figure 4: Scatter plots of eye position at different times (shown on each plot) (A) after the end of a saccade, or (B) after the stimulus changed as the monkey fixated. The scatter or variance in eye position is largest immediately after the saccade ends and becomes progressively smaller as the fixation period becomes longer, stabilizing about 350 or 400 ms after the saccade has ended. There is no discernible effect on fixation by changing the stimulus as the animal fixates.
vert position (deg) 0 5 10
A
•
10
0
horiz position (deg)
5
10
horiz position (deg)
•
After Flash Information (bits) 0.2 0.4 0.6
end of saccade and after switching the stimulus during fixation. The information rises more during the initial burst after switching. After approximately 400 ms it reaches nearly the same level after saccades. The increase seen in the movies after stimulus disappearance is seen in the curve labeled ’After Saccades’ at the time the stimulus switches while the monkey fixates.
•
•
•
•
•
•
• •
•
•
•
•
•
•
•
• •
•
•
After Saccades
•
•
• •
0.0
Figure 5: Information accumulation after
0.8
Accumulated information after trigger event
•
0
100
200 300 400 Time after trigger
500
600
The variability in the response dynamics when a saccade ends raises the question of what the monkeys eyes are doing at this time. Examination of the fixation shows that the eye positions are more scattered immediately after the saccades end (even though the eye is not moving much), and the scatter in eye position becomes much less as the fixation continues (Figure 4). However, it is not until 350 or 400 ms after the end of the saccade before the eye position scatter matches that seen during continual fixation.
DISCUSSION The responsivity of V1 supragranular complex cells appears to be well-adapted to carry information as we make saccades to explore the visual environment. In the experiments with the movies, the stimulus-related information grew as the period that each image remained on increased. There was a second burst of stimulus-related information after the stimulus disappeared. This burst of information was greatest if there was a gap lasting 50 ms between successive movie frames. We speculate that this burst of informa-
REFERENCES
6
tion is related to the off-response. This analysis suggests that the off-response, which might arise in the retinal receptors, has stimulus selectivity. The very small amount of stimulus-related information that is present when the stimuli are being changed at a rapid rate (faster than one stimulus/50-75ms) suggests that there is an effect a stimulus disappearing will affect the response of new stimulus. A great deal of this might arise from mixing effects of off-responses with the subsequent on-response. The sequence of timings that give the largest information is one that is consistent with the retinal events that occur during saccadic exploration of the environment, i.e. an image arrives on a receptive field, remains there for 200 or more milliseconds, and then a saccadic eye movement creates a smear (which might be like our grey background) lasting from 20-50 ms, followed by a new image. The greater variability that occurs in responses starting after a saccade ends vs after a flashed stimulus could be related to the greater variability of the stimulus position on the receptive field. However, it is also clear that the latency is more variable. We do not know what the relation between the end of the saccade and the onset of the neuronal response is. Nonetheless, the amount of information that is available by the end of a normal fixation period is similar to that available after the same amount of time when a stimulus appears on the retina during fixation. Thus, it appears that the time between normal eye movements allows information in V1 neuronal responses to accumulate to near its apparent asymptotic values.
References [Gallant et al, 1998]
Gallant, J.L., Connor, C.E. and Van Essen, D.C. (1998) Neural activity in V1, V2 and V4 during free viewing of natural scenes compared to controlled viewing. Neuroreport 9: 2153-2158.
[Judge et al, 1980]
Judge, S.J., Wurtz, R.H and Richmond, B.J. (1980) Vision during saccadic eye movements: I. Visual interactions in striate cortex. J. Neurophysiol. 43: 1133-1155.
[Kjaer et al, 1994]
Kjaer, T.W., Hertz, J.A. and Richmond, B.J. (1994) Decoding Cortical Neuronal Signals: Network Models, Information Estimation and Spatial Tuning. J Computational Neuroscience 1: 109-139.
[Macknik and Livingstone, 1998] Macknik, S.L. and Livingtone, M.S. (1998) Neuronal correlates of visibility and invisibility in the primate visual system. Nature Neurosci. 1: 144149.
Barry J. Richmond Barry Richmond received his M.D. in 1971 from Case-Western Reserve University, where he did a pediatric residency. After a three year neurology residency at the Harvard-Longwood Program, he went to the NIH to do a postdoc with Bob Wurtz. In 1980, he joined the Laboratory of Neuropsychology of NIMH, where he has been investigating the structure and content of neuronal codes in the visual system, and more recently, also the physiological basis of motivation and reward.
John A. Hertz John Hertz received his PhD from the University of Pennsylvania in 1970. He served on the faculty of the University of Chicago for seven years before moving to Nordita in Copenhagen in 1980. He has worked on problems in statistical physics, including critical phenomena and spin glasses, with the emphasis shifting in
recent years toward problems in biological systems, including neural networks. He collaborates extensively with colleagues in the Laboratory of Neuropsychology, NIMH.
Tim J. Gawne Tim Gawne received his Ph.D. in physiology from the Uniformed Services University of the Health Sciences in 1985 studying cardiovascular reflexes related to exercise and shock. He changed to neuroscience when he joined Barry Richmond at the NIH in 1987. Ever since he has been trying to understand how the coordinated activity of the many neurons in the visual system allows us to perceive and recognize objects. He has been in the Dept. of Physiological Optics at the University of Alabama, Birmingham since 1996.
7
