A learning rule to model the development of orientation ... - geb.uma.es
A learning rule to model the development of orientation selectivity in visual cortex Jose M. Jerez1 , Miguel Atencia2 , Francisco J. Vico1 , and Enrique Dominguez1 1
Escuela T´ecnica Superior de Ingenier´ıa en Inform´ atica Departamento de Lenguajes y Ciencias de la Computaci´ on Universidad de M´ alaga [email protected] 2 Escuela T´ecnica Superior de Ingenier´ıa en Inform´ atica Departamento de Matem´ atica Aplicada Universidad de M´ alaga
Abstract. This paper presents a learning rule, CBA, to develop oriented receptive fields similar to those founded in cat striate cortex. The inherent complexity of the development of selectivity in visual cortex has led most authors to test their models by using a restricted input environment. Only recently, some learning rules (PCA and BCM rules) have been studied in a realistic visual environment. The CBA rule proposed in this work is tested in different input visual environments and the results are compared to those achieved by the BCM and PCA rules. The final results show that the CBA rule is appropriate for studying the biologically process of receptive field formation in visual cortex.
1
Introduction
Among the different approaches to imitate the perceptual capabilities of biological systems, neural-based models have been proposed in the last decades [1–5], and some have been tested in natural scenarios [6, 7]. Stimulating a single neuron model with natural images, PCA [8] and BCM [3] learning rules were shown to develop receptive fields (RFs) similar to those found in visual cortex in the early experiments of Hubel and Wiesel [9, 10]. Exposing the neuron to some stimulation trials transformed a random receptive field in one selective to orientation. Each trial included the presentation of a patch of size 13x13 pixels, obtained from a set of 24 grey-scale 256x256 pixels images, that has been processed with a DOG filter. Preferred orientations of the resulting RFs spread out widely, and concentrated slightly in the range from 80 to 120 since the images contained vegetal forms, that aligned more in vertical orientation. The resulting RFs contained excitatory and inhibitory regions arranged in a preferred orientation. The emergence of these regions have to do with the potentiating (LTP) and depressing (LTD) character of the learning rule. According to the Hebbian postulate, both rules include LTP terms, but differ in the way they implement LTD. While PCA incorporates heterosynaptic competition, BCM produces a similar effect through homosynaptic competition. These ´ J. Mira and J.R. Alvarez (Eds.): IWANN 2003, LNCS 2686, pp. 190-197, 2003. c Springer-Verlag Berlin Heidelberg 2003
A Learning Rule to Model the Development of Orientation Selectivity
191
two forms of LTD reinforce inhibition by means of spatial competition among the afferents of a neuron in the case of PCA, or temporal competition in the case of BCM. The fact that each of these learning rules rely on a single mechanism for LTD strongly influences the final shape of the RFs, and, consequently, the type of processing performed by the neuron. The RFs resulting of a PCA training are sensitive to low spatial frequencies (only two regions are differentiated), while those obtained with CBA show selectivity to high frequencies (three or more bands). Both, homosynaptic and heterosynaptic competition have been described in the nervous system [11–13], and its combined effect might yield the wide range of spatial frequencies that are captured by the RFs of the striate cortex cells [14]. Although, in principle, the BCM learning rule seems to be more suitable to achieve sensitivity to both low and high frequencies with a proper parameter set, the temporal competition that implements its LTD mechanism makes hard the fitting process. This problem arises when the BCM theory is tested using the images from a camera mounted on a freely moving robot [15]. Taking into account these functional limitations and biological constraints, we propose here a new learning rule that incorporates homosynaptic and heterosynaptic competition. This rule is derived from the one proposed in [16] for neural assemblies formation, with the only difference that incorporates a decay term. The rest of this paper is organized as follows. In Section 2 the model is presented. In Section 3, first, the rule is simulated within a restricted visual environment and then, realistic images are presented to the model. Both experiments suggest that receptive fields are formed, which mix properties of the BCM and the PCA rules. Finally, Section 4 summarizes the main conclusions, and some lines for future research are provided.
2
The model
The neuron single model consists of a vector x of inputs, representing an averaged presynaptic activity originated from another cell, a vector w of synaptic weights, and a scalar output y, given by y = w · x, that represents an averaged postsynaptic activity. The weight vector w can take negative values, since they can be considered as effective synapses, made up of multiple excitatory and inhibitory connections. Once the activation equation is defined, we face the problem of modelling the weight modification process that represents learning. In a previous work [16] we proposed a new correlational learning rule (BA, for bounded activity) that formed stable neural attractors in a recurrent network. The CBA learning rule is essentially a modification of the BA rule in which an extra term to implement the heterosynaptic LTD has been incorporated. Thus, the resulting synaptic modification equation for the CBA rule is a Hebbian-type learning rule with an specific form of stabilization, defined as d wi = α xi y (y − τ )(λ − y) − β y wi = f (w) dt
(1)
192
Jose M. Jerez et al.
where α is the learning rate. The rationale behind the introduction of the remaining parameters is now explained. The term λ avoids the unbounded weight growing in a plausible way, and can be interpreted as a neuronal parameter representing the maximum level of activity at which the neuron might work. The CBA modification equation also defines a threshold τ that determines whether depression or potentiation occurs when both the pre- and postsynaptic neurons fire. Finally, the parameter β controls the heterosynaptic competition effect, such that the strength of synapses can change even in the absence of presynaptic activity to those synapses. The β value should be lower than α to preserve the dominant character of LTP and homosynaptic LTD over heterosynaptic LTD. All these parameters adopt positive values. The effect of this adaptation mechanism in the receptive fields formation process is that the graded response elicited after stimulus presentation leads the neural activity either to high or resting levels.
3
Simulation results
Before doing simulations in a realistic visual environment, the simulations performed on a simpler input environment (one and two dimensions) will provide a qualitative insight on the system. In this context, dimension means number of afferent synaptic connections. 3.1
Low dimensional environment
In the one-dimensional case we have only one differential equation, where both the input x and the weight y are scalars. The fixed points would be the weight values w that satisfy the condition f (w) = 0: w0 = 0 ,
w1 =
γ+R , 2 x3 α
w2 =
γ−R 2 x3 α
where the parameters γ, ρ and R have been defined as √ γ = −β + x2 α(λ + τ ) , ρ = 2 x2 α λ τ , R = γ 2 − ρ2
(2)
(3)
It is instructive to observe the function f (w) in equation (1), which has been drawn in Figure 1. The geometrical intuition suggests that if f (a) > 0 (e.g. if a is largely negative) and the initial state of the system is w = a, w will increase. On the other hand, starting from w = a, w will decrease if f (a) < 0, e.g. if a is largely positive. The increasing or decreasing evolution will continue until a fixed point is reached, but the system ”corrects” itself so that its state does not blow up to ±∞. Although the one dimensional model gives us an idea about the system dynamics, one cannot obtain selectivity with this restricted environment. In this sense, we define a two-dimensional environment composed by two input patterns, x1 and x2 , presented to the neuron with equal probabilities. Figure 2 illustrates
A Learning Rule to Model the Development of Orientation Selectivity
193
f(w)
w1 − stable
w0 − stable
w w2 − unstable
Fig. 1. Nonlinear differential equation modelling the behaviour of the system.
the trajectories followed by different weight initializations (drawn as circles) in a states space, what provides an description about the weights dynamics in this restricted environment. In this figure we can observe one attractor fixed point, w1 , one saddle point, labelled as w2 , and a set of initial stabled points, w0 , located in a perpendicular plane to the attactor point. The figure shows an attractor basin towards a line of points crossing by zero, such a way that every weight initialization inside the region located between this line and the parallel one crossing by the unstable point, will yield the system not to develop selectivity. Blais [17] proposed the analysis of the output distribution of the neuron at the fixed points as a useful tool to compare the behavior of different learning rules. In Figure 3 the output distributions for PCA, BCM and CBA learning rules are compared for the two-input environment. The results show that the PCA rule is trying to have most of its responses strong, BCM rule tries to have a small subset of its responses strong and the others weak, and CBA gives the maximum response strong to an input pattern. These results might help us to predict the structures of the receptive fields achieved by these three learning rules trained in a more realistic visual environment. 3.2
Realistic visual environment
The visual environment used in this section is similar to that described in Law et al. [6], and it is composed by 24 natural images scanned into 256x256 pixel images, where man-made objects have been avoided, since they would make easier to achieve receptive fields, given their sharp edges and straight lines characteristics. The retina model is composed of square arrays of receptors which have antagonistic center-surround receptive fields that approximate a difference of
194
Jose M. Jerez et al. 1.5
w1 1
0.5
w2
0
w0 −0.5
−1
−1.5 −1.5
−1
−0.5
0
0.5
1
1.5
Fig. 2. States-space indicating the weights dynamics from different initial values (represented as circles). These results were achieved setting the learning constant, α, to the value 0.05, the maximum activity level, λ, was set to the value 1.0, the threshold, τ , was 0.25, and the heterosynaptic competition term, β, was set to 0.0025. BCM
Probability of response
0.8
CBA
PCA
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0
0
0.2
0.4
0.6
0.8
activity (response to input patterns)
1
0
0.1
0
0.2
0.4
0.6
0.8
activity (response to input patterns)
1
0
0
0.2
0.4
0.6
0.8
1
activity (response to input patterns)
Fig. 3. Output distributions for BCM, PCA and CBA learning rules. BCM seeks orthogonality to one of the input vectors, PCA tries to maximize responses to the set of input vectors, while CBA maximizes the projection to one input vector (giving the maximum response strong).
Gaussian (DOG) filter. The ratio of the surround to the center of the Gaussian distribution is approximately 3:1, which has been biologically observed in [18]. The model neuron was trained with 13x13 pixels patches randomly taken from the images. For every simulation step, the activity of the input cells in the retina is determined by randomly picking one of the 24 images and randomly shifting the receptive field mask. The activity of each input in the model is determined by the intensity of a pixel in the image. The exact time course of these simulations depend on the parameter chosen, so we have examined these over a large range. Table 1 shows the range of parameters used to obtained the results presented at Figure 4.
A Learning Rule to Model the Development of Orientation Selectivity
195
Table 1. Setting of learning rule parameters for simulations in a realistic visual environment. Learning constant, α 0.005 Maximum level of activity, λ 1.0 Threshold level, τ 0.15 Heterosynaptic competition term, β 0.00025 Input values range, x [−0.10, 0.10] [−0.15, 0.15] Weights initialization range, w0 Number of iterations 250000
Figure 4 shows the weights resulting from these simulations starting from different initial conditions. With this realistic input environment, the CBA neuron develops receptive fields with distinct excitatory and inhibitory regions. Notice, also, that the variety of oriented receptive fields structures obtained is significative enough.
Fig. 4. Different types of cortical receptive fields arising from the CBA learning rule. The individual plots show the weights vector for two-dimensional receptive field with white denoting positive values and black negative values (synaptic efficacies).
Figure 5 shows examples receptive for BCM and PCA rules trained in the same visual environment as CBA rule. These oriented receptive fields are similar to those experimentally observed by Hubel and Wiesel [9, 10]. However, BCM receptive fields are clearly selective to bars of lights at different orientations, whereas PCA develops receptive fields always divided into two antagonist regions, one of them with synaptic potentiation and the other one with synaptic depression. At this point, establishing a comparison to the receptive fields structures in Figure 4, we can assess that the CBA learning rule can develop receptive fields with properties similar to those achieved by both PCA and BCM rules. Effectively, Figure 4 shows examples receptive fields with the same structure as PCA receptive fields, and others becoming selective to bars of lights at different positions, but with an spatial frequency less than the receptive fields achieved by the BCM rule.
196
Jose M. Jerez et al.
Fig. 5. Examples receptive fields achieved by BCM (top) and PCA (bottom) trained in a realistic visual environment composed by natural images.
4
Conclusions and future work
This paper has shown that the CBA learning rule is approppiate to develop cells with oriented receptive fields in visual cortex. This learning rule contributes with a term for controlling the synaptic growing, such that any additional weight saturation and normalization constraint is avoided. Besides, the CBA rule integrates both heterosynaptic and homosynaptic methods through different parameters in the synaptic modification equation. The results have shown that, in a realistic visual environment, the CBA rule develops oriented receptive fields similar to those achieved by both BCM and PCA learning rules. In addition, the simulation results presented robustness and a high level of stability on a wide range of parameter values. Two immediate steps arise from this research as future works. On the one hand, it is preceptive to study the properties of CBA modification dynamics and the influence of the learning rule parameters in normal and deprived environments through experiments of visual deprivation. Also, the process of direction selective receptive fields formation in visual complex cells can be studied in terms of the CBA rule. On the other hand, the receptive fields achieved by this learning rule might be considered as filters susceptible of being applied as the first stage in the features extraction process carried out in image processing and artificial vision tasks. Finally, an exhaustive mathematical analysis of the CBA rule must be done in both one- and n-dimensional environment, determining the stability conditions as well as the basins of attraction for the system fixed points. This analysis will provide a better understanding of the CBA fundamental properties, and a mathematical relation among the parameters of the learning rule identifying a region where the system works properly.
References 1. Sejnowsky, T.: Storing covariance with nonlinearly interacting neurons. Journal of Math. Biology 4 (1977) 303–321 2. Von der Malsburg, C.: Self-organization of orientation sensivity cells in striate cortex. Kybernetik 14 (1973) 85–100
A Learning Rule to Model the Development of Orientation Selectivity
197
3. Bienenstock, E., Cooper, L., Munro, P.: Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience 2 (1982) 32–48 4. Linsker, R.: From basic network principles to neural architecture: Emergence of orientation-selective cells. Proceedings of the National Academy of Science 83 (1986) 8390–8394 5. Miller, K.: A model for the development of simple cell receptive fields and the ordered arrangement of orientation columns through activity-dependent competition between on-and off-center inputs. Journal of Neuroscience 14 (1994) 409–441 6. Law, C., Cooper, L.: Formation of receptive fields according to the bcm theory of synaptic plasiticity in realistic visual environment. Proceedings of the National Academy of Science 91 (1994) 7797–7801 7. Shouval, H., Liu, Y.: Principal component neurons in a realistic visual environment. Network 7 (1996) 501–515 8. Oja, E.: A simplified neuron model as a principal component analyzer. Mathematical Biology 15 (1982) 267–273 9. Hubbel, D., Wiesel, T.: Receptive fields, binocular interaction and functional architecture in the cats visual cortex. Journal of Physiology 160 (1962) 106–154 10. Hubbel, D., Wiesel, T.: Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology 195 (1968) 215–243 11. Artola, A., Brcher, S., Singer, W.: Different voltage-dependent threshold for inducing long-term depression and long-term potentiation in slices of rat visual cortex. Nature 347 (1990) 69–72 12. Abraham, W., Goddard, G.: Asymmetric relationships between homosynaptic long-term potentiation and heterosynaptic long-term depression. Nature 305 (1983) 717–719 13. Lynch, G., Dunwiddie, T., Gribkoff, V.: Heterosynaptic depression: A postsynaptic correlate of long-term potentiation. Nature 266 (1977) 737–739 14. Miller, K.: Synaptic economics: Competition and cooperation in correlation-based synaptic plasticity. Neuron 17 (1996) 371–374 15. Neskovic, P., Verschure, P.: Evaluation of a biologically realistic learning rule, bcm, in an active vision system. (1998) 16. Vico, F., Jerez, J.: Stable neural attractors formation: Learning rules and network dynamics. Neural Processing Letters to appear (2003) 17. Blais, B.: The role of the environment in synaptic plasticity: Towards an understanding of learning and memory. PhD thesis, Brown University (1998) 18. Linsenmeier, R., Frishman, L., Jakiela, H., Enroth-Cugell, C.: Receptive field properties of x and y cells in the cat retina derived from contrast sensitivity measurements. Vision Research 22 (1982) 11731183
Escuela T´ecnica Superior de Ingenier´ıa en Inform´ atica Departamento de Lenguajes y Ciencias de la Computaci´ on Universidad de M´ alaga [email protected] 2 Escuela T´ecnica Superior de Ingenier´ıa en Inform´ atica Departamento de Matem´ atica Aplicada Universidad de M´ alaga
Abstract. This paper presents a learning rule, CBA, to develop oriented receptive fields similar to those founded in cat striate cortex. The inherent complexity of the development of selectivity in visual cortex has led most authors to test their models by using a restricted input environment. Only recently, some learning rules (PCA and BCM rules) have been studied in a realistic visual environment. The CBA rule proposed in this work is tested in different input visual environments and the results are compared to those achieved by the BCM and PCA rules. The final results show that the CBA rule is appropriate for studying the biologically process of receptive field formation in visual cortex.
1
Introduction
Among the different approaches to imitate the perceptual capabilities of biological systems, neural-based models have been proposed in the last decades [1–5], and some have been tested in natural scenarios [6, 7]. Stimulating a single neuron model with natural images, PCA [8] and BCM [3] learning rules were shown to develop receptive fields (RFs) similar to those found in visual cortex in the early experiments of Hubel and Wiesel [9, 10]. Exposing the neuron to some stimulation trials transformed a random receptive field in one selective to orientation. Each trial included the presentation of a patch of size 13x13 pixels, obtained from a set of 24 grey-scale 256x256 pixels images, that has been processed with a DOG filter. Preferred orientations of the resulting RFs spread out widely, and concentrated slightly in the range from 80 to 120 since the images contained vegetal forms, that aligned more in vertical orientation. The resulting RFs contained excitatory and inhibitory regions arranged in a preferred orientation. The emergence of these regions have to do with the potentiating (LTP) and depressing (LTD) character of the learning rule. According to the Hebbian postulate, both rules include LTP terms, but differ in the way they implement LTD. While PCA incorporates heterosynaptic competition, BCM produces a similar effect through homosynaptic competition. These ´ J. Mira and J.R. Alvarez (Eds.): IWANN 2003, LNCS 2686, pp. 190-197, 2003. c Springer-Verlag Berlin Heidelberg 2003
A Learning Rule to Model the Development of Orientation Selectivity
191
two forms of LTD reinforce inhibition by means of spatial competition among the afferents of a neuron in the case of PCA, or temporal competition in the case of BCM. The fact that each of these learning rules rely on a single mechanism for LTD strongly influences the final shape of the RFs, and, consequently, the type of processing performed by the neuron. The RFs resulting of a PCA training are sensitive to low spatial frequencies (only two regions are differentiated), while those obtained with CBA show selectivity to high frequencies (three or more bands). Both, homosynaptic and heterosynaptic competition have been described in the nervous system [11–13], and its combined effect might yield the wide range of spatial frequencies that are captured by the RFs of the striate cortex cells [14]. Although, in principle, the BCM learning rule seems to be more suitable to achieve sensitivity to both low and high frequencies with a proper parameter set, the temporal competition that implements its LTD mechanism makes hard the fitting process. This problem arises when the BCM theory is tested using the images from a camera mounted on a freely moving robot [15]. Taking into account these functional limitations and biological constraints, we propose here a new learning rule that incorporates homosynaptic and heterosynaptic competition. This rule is derived from the one proposed in [16] for neural assemblies formation, with the only difference that incorporates a decay term. The rest of this paper is organized as follows. In Section 2 the model is presented. In Section 3, first, the rule is simulated within a restricted visual environment and then, realistic images are presented to the model. Both experiments suggest that receptive fields are formed, which mix properties of the BCM and the PCA rules. Finally, Section 4 summarizes the main conclusions, and some lines for future research are provided.
2
The model
The neuron single model consists of a vector x of inputs, representing an averaged presynaptic activity originated from another cell, a vector w of synaptic weights, and a scalar output y, given by y = w · x, that represents an averaged postsynaptic activity. The weight vector w can take negative values, since they can be considered as effective synapses, made up of multiple excitatory and inhibitory connections. Once the activation equation is defined, we face the problem of modelling the weight modification process that represents learning. In a previous work [16] we proposed a new correlational learning rule (BA, for bounded activity) that formed stable neural attractors in a recurrent network. The CBA learning rule is essentially a modification of the BA rule in which an extra term to implement the heterosynaptic LTD has been incorporated. Thus, the resulting synaptic modification equation for the CBA rule is a Hebbian-type learning rule with an specific form of stabilization, defined as d wi = α xi y (y − τ )(λ − y) − β y wi = f (w) dt
(1)
192
Jose M. Jerez et al.
where α is the learning rate. The rationale behind the introduction of the remaining parameters is now explained. The term λ avoids the unbounded weight growing in a plausible way, and can be interpreted as a neuronal parameter representing the maximum level of activity at which the neuron might work. The CBA modification equation also defines a threshold τ that determines whether depression or potentiation occurs when both the pre- and postsynaptic neurons fire. Finally, the parameter β controls the heterosynaptic competition effect, such that the strength of synapses can change even in the absence of presynaptic activity to those synapses. The β value should be lower than α to preserve the dominant character of LTP and homosynaptic LTD over heterosynaptic LTD. All these parameters adopt positive values. The effect of this adaptation mechanism in the receptive fields formation process is that the graded response elicited after stimulus presentation leads the neural activity either to high or resting levels.
3
Simulation results
Before doing simulations in a realistic visual environment, the simulations performed on a simpler input environment (one and two dimensions) will provide a qualitative insight on the system. In this context, dimension means number of afferent synaptic connections. 3.1
Low dimensional environment
In the one-dimensional case we have only one differential equation, where both the input x and the weight y are scalars. The fixed points would be the weight values w that satisfy the condition f (w) = 0: w0 = 0 ,
w1 =
γ+R , 2 x3 α
w2 =
γ−R 2 x3 α
where the parameters γ, ρ and R have been defined as √ γ = −β + x2 α(λ + τ ) , ρ = 2 x2 α λ τ , R = γ 2 − ρ2
(2)
(3)
It is instructive to observe the function f (w) in equation (1), which has been drawn in Figure 1. The geometrical intuition suggests that if f (a) > 0 (e.g. if a is largely negative) and the initial state of the system is w = a, w will increase. On the other hand, starting from w = a, w will decrease if f (a) < 0, e.g. if a is largely positive. The increasing or decreasing evolution will continue until a fixed point is reached, but the system ”corrects” itself so that its state does not blow up to ±∞. Although the one dimensional model gives us an idea about the system dynamics, one cannot obtain selectivity with this restricted environment. In this sense, we define a two-dimensional environment composed by two input patterns, x1 and x2 , presented to the neuron with equal probabilities. Figure 2 illustrates
A Learning Rule to Model the Development of Orientation Selectivity
193
f(w)
w1 − stable
w0 − stable
w w2 − unstable
Fig. 1. Nonlinear differential equation modelling the behaviour of the system.
the trajectories followed by different weight initializations (drawn as circles) in a states space, what provides an description about the weights dynamics in this restricted environment. In this figure we can observe one attractor fixed point, w1 , one saddle point, labelled as w2 , and a set of initial stabled points, w0 , located in a perpendicular plane to the attactor point. The figure shows an attractor basin towards a line of points crossing by zero, such a way that every weight initialization inside the region located between this line and the parallel one crossing by the unstable point, will yield the system not to develop selectivity. Blais [17] proposed the analysis of the output distribution of the neuron at the fixed points as a useful tool to compare the behavior of different learning rules. In Figure 3 the output distributions for PCA, BCM and CBA learning rules are compared for the two-input environment. The results show that the PCA rule is trying to have most of its responses strong, BCM rule tries to have a small subset of its responses strong and the others weak, and CBA gives the maximum response strong to an input pattern. These results might help us to predict the structures of the receptive fields achieved by these three learning rules trained in a more realistic visual environment. 3.2
Realistic visual environment
The visual environment used in this section is similar to that described in Law et al. [6], and it is composed by 24 natural images scanned into 256x256 pixel images, where man-made objects have been avoided, since they would make easier to achieve receptive fields, given their sharp edges and straight lines characteristics. The retina model is composed of square arrays of receptors which have antagonistic center-surround receptive fields that approximate a difference of
194
Jose M. Jerez et al. 1.5
w1 1
0.5
w2
0
w0 −0.5
−1
−1.5 −1.5
−1
−0.5
0
0.5
1
1.5
Fig. 2. States-space indicating the weights dynamics from different initial values (represented as circles). These results were achieved setting the learning constant, α, to the value 0.05, the maximum activity level, λ, was set to the value 1.0, the threshold, τ , was 0.25, and the heterosynaptic competition term, β, was set to 0.0025. BCM
Probability of response
0.8
CBA
PCA
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0
0
0.2
0.4
0.6
0.8
activity (response to input patterns)
1
0
0.1
0
0.2
0.4
0.6
0.8
activity (response to input patterns)
1
0
0
0.2
0.4
0.6
0.8
1
activity (response to input patterns)
Fig. 3. Output distributions for BCM, PCA and CBA learning rules. BCM seeks orthogonality to one of the input vectors, PCA tries to maximize responses to the set of input vectors, while CBA maximizes the projection to one input vector (giving the maximum response strong).
Gaussian (DOG) filter. The ratio of the surround to the center of the Gaussian distribution is approximately 3:1, which has been biologically observed in [18]. The model neuron was trained with 13x13 pixels patches randomly taken from the images. For every simulation step, the activity of the input cells in the retina is determined by randomly picking one of the 24 images and randomly shifting the receptive field mask. The activity of each input in the model is determined by the intensity of a pixel in the image. The exact time course of these simulations depend on the parameter chosen, so we have examined these over a large range. Table 1 shows the range of parameters used to obtained the results presented at Figure 4.
A Learning Rule to Model the Development of Orientation Selectivity
195
Table 1. Setting of learning rule parameters for simulations in a realistic visual environment. Learning constant, α 0.005 Maximum level of activity, λ 1.0 Threshold level, τ 0.15 Heterosynaptic competition term, β 0.00025 Input values range, x [−0.10, 0.10] [−0.15, 0.15] Weights initialization range, w0 Number of iterations 250000
Figure 4 shows the weights resulting from these simulations starting from different initial conditions. With this realistic input environment, the CBA neuron develops receptive fields with distinct excitatory and inhibitory regions. Notice, also, that the variety of oriented receptive fields structures obtained is significative enough.
Fig. 4. Different types of cortical receptive fields arising from the CBA learning rule. The individual plots show the weights vector for two-dimensional receptive field with white denoting positive values and black negative values (synaptic efficacies).
Figure 5 shows examples receptive for BCM and PCA rules trained in the same visual environment as CBA rule. These oriented receptive fields are similar to those experimentally observed by Hubel and Wiesel [9, 10]. However, BCM receptive fields are clearly selective to bars of lights at different orientations, whereas PCA develops receptive fields always divided into two antagonist regions, one of them with synaptic potentiation and the other one with synaptic depression. At this point, establishing a comparison to the receptive fields structures in Figure 4, we can assess that the CBA learning rule can develop receptive fields with properties similar to those achieved by both PCA and BCM rules. Effectively, Figure 4 shows examples receptive fields with the same structure as PCA receptive fields, and others becoming selective to bars of lights at different positions, but with an spatial frequency less than the receptive fields achieved by the BCM rule.
196
Jose M. Jerez et al.
Fig. 5. Examples receptive fields achieved by BCM (top) and PCA (bottom) trained in a realistic visual environment composed by natural images.
4
Conclusions and future work
This paper has shown that the CBA learning rule is approppiate to develop cells with oriented receptive fields in visual cortex. This learning rule contributes with a term for controlling the synaptic growing, such that any additional weight saturation and normalization constraint is avoided. Besides, the CBA rule integrates both heterosynaptic and homosynaptic methods through different parameters in the synaptic modification equation. The results have shown that, in a realistic visual environment, the CBA rule develops oriented receptive fields similar to those achieved by both BCM and PCA learning rules. In addition, the simulation results presented robustness and a high level of stability on a wide range of parameter values. Two immediate steps arise from this research as future works. On the one hand, it is preceptive to study the properties of CBA modification dynamics and the influence of the learning rule parameters in normal and deprived environments through experiments of visual deprivation. Also, the process of direction selective receptive fields formation in visual complex cells can be studied in terms of the CBA rule. On the other hand, the receptive fields achieved by this learning rule might be considered as filters susceptible of being applied as the first stage in the features extraction process carried out in image processing and artificial vision tasks. Finally, an exhaustive mathematical analysis of the CBA rule must be done in both one- and n-dimensional environment, determining the stability conditions as well as the basins of attraction for the system fixed points. This analysis will provide a better understanding of the CBA fundamental properties, and a mathematical relation among the parameters of the learning rule identifying a region where the system works properly.
References 1. Sejnowsky, T.: Storing covariance with nonlinearly interacting neurons. Journal of Math. Biology 4 (1977) 303–321 2. Von der Malsburg, C.: Self-organization of orientation sensivity cells in striate cortex. Kybernetik 14 (1973) 85–100
A Learning Rule to Model the Development of Orientation Selectivity
197
3. Bienenstock, E., Cooper, L., Munro, P.: Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience 2 (1982) 32–48 4. Linsker, R.: From basic network principles to neural architecture: Emergence of orientation-selective cells. Proceedings of the National Academy of Science 83 (1986) 8390–8394 5. Miller, K.: A model for the development of simple cell receptive fields and the ordered arrangement of orientation columns through activity-dependent competition between on-and off-center inputs. Journal of Neuroscience 14 (1994) 409–441 6. Law, C., Cooper, L.: Formation of receptive fields according to the bcm theory of synaptic plasiticity in realistic visual environment. Proceedings of the National Academy of Science 91 (1994) 7797–7801 7. Shouval, H., Liu, Y.: Principal component neurons in a realistic visual environment. Network 7 (1996) 501–515 8. Oja, E.: A simplified neuron model as a principal component analyzer. Mathematical Biology 15 (1982) 267–273 9. Hubbel, D., Wiesel, T.: Receptive fields, binocular interaction and functional architecture in the cats visual cortex. Journal of Physiology 160 (1962) 106–154 10. Hubbel, D., Wiesel, T.: Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology 195 (1968) 215–243 11. Artola, A., Brcher, S., Singer, W.: Different voltage-dependent threshold for inducing long-term depression and long-term potentiation in slices of rat visual cortex. Nature 347 (1990) 69–72 12. Abraham, W., Goddard, G.: Asymmetric relationships between homosynaptic long-term potentiation and heterosynaptic long-term depression. Nature 305 (1983) 717–719 13. Lynch, G., Dunwiddie, T., Gribkoff, V.: Heterosynaptic depression: A postsynaptic correlate of long-term potentiation. Nature 266 (1977) 737–739 14. Miller, K.: Synaptic economics: Competition and cooperation in correlation-based synaptic plasticity. Neuron 17 (1996) 371–374 15. Neskovic, P., Verschure, P.: Evaluation of a biologically realistic learning rule, bcm, in an active vision system. (1998) 16. Vico, F., Jerez, J.: Stable neural attractors formation: Learning rules and network dynamics. Neural Processing Letters to appear (2003) 17. Blais, B.: The role of the environment in synaptic plasticity: Towards an understanding of learning and memory. PhD thesis, Brown University (1998) 18. Linsenmeier, R., Frishman, L., Jakiela, H., Enroth-Cugell, C.: Receptive field properties of x and y cells in the cat retina derived from contrast sensitivity measurements. Vision Research 22 (1982) 11731183
