i = 1;...; n (1) x we wij; j 2 P+ 0; j 2 P0. - Semantic Scholar
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
175
Analysis of Continuous Attractors for 2-D Linear Threshold Neural Networks
or its equivalent form
Lan Zou, Huajin Tang, Kay Chen Tan, and Weinian Zhang
where each xi denotes the activity of neuron i, x = (x1 ; . . . ; xn )T 2 n , h 2 n denotes external inputs, W = (wij )n2n is a constant real matrix, each entry of which denotes the synaptic weight representing the strength of the synaptic connection from neuron j to neuron i, and (x) is the LT transfer function defined by (x) = maxf0; xg. The global attractivity and multistability were examined in the theoretical studies [6], [11], and more complicated dynamics, such as limit cycle was investigated with analytical results in [13]. The continuous attractors as limit cycle were investigated in our previous work [13], and the dynamic properties and geometrical structures of nondegenerate equilibria were studied in [11]. As an extension of the previous works, this work is focused on the degenerate equilibria of the LT network. The relationships between features of equilibria and network parameters (synaptic weights and external inputs) are revealed so that sufficient and necessary conditions for coexistence of both nondegenerate equilibria and degenerate ones are drawn.
x_ (t) = 0x(t) + W (x(t)) + h
R
Abstract—This brief investigates continuous attractors of the well-developed model in visual cortex, i.e., the linear threshold (LT) neural networks, based on a parameterized 2-D model. On the basis of existing results on nondegenerate equilibria in mathematics, we further discuss degenerate equilibria for such networks and present properties and distributions of the equilibria, which enables us to draw the coexistence conditions of nondegenerate and degenerate equilibria (e.g., singular lines). Our theoretical results provide a useful framework for precise tuning on the network parameters, e.g., the feedbacks and visual inputs. Simulations are also presented to illustrate the theoretical findings. Index Terms—Continuous attractor, degenerate equilibria, linear threshold (LT) network, multistable neural network, singular line.
I. INTRODUCTION The recurrent neural networks with linear threshold (LT) transfer functions have received extensive research interest in recent years from both aspects of mathematics and neuroscience [1]–[7]. Some features underling the attractor recurrent neural networks, such as stability, attractivity, and winner-takes-all (WTA) behavior have also been studied recently [8]–[10]. The LT networks exhibit an important dynamic property, namely, multistability, which underlies the mathematical principles in visual cortex and eye memory networks. In our previous work [11], the features of nondegenerate equilibria and global attractivity have been discussed, and theoretical proofs for the coexistence of multiple nondegenerate equilibria were also given. However, there is a lack of theoretical studies on the degenerate equilibria of the LT networks, which would form a continuous attractor (e.g., singular lines and limit cycles). As reported in the neural model [5], [12], continuous attractor is an important dynamical property in the memory networks, because the oculomotor control in the brain requires storing continuous eye positions. The relationship between the network parameters and such degenerate equilibria remains unknown. Thus, it is necessary to explore the dynamical properties of the degenerate equilibria and to reveal the parameter conditions. We consider the recurrent neural networks with LT transfer functions described by n
x_ i (t) = 0xi (t) + j
=1
wij (xj (t)) + hi ;
i = 1; . . . ; n
(1)
Manuscript received April 09, 2008; accepted October 31, 2008. First published December 09, 2008; current version published January 05, 2009. This work was supported in part by the National Science Foundation of China under Grants 10825104 and 10571127. L. Zou and W. Zhang are with the Yangtze Center of Mathematics and the Department of Mathematics, Sichuan University, Chengdu 610064, China (e-mail: [email protected]; [email protected]). H. Tang is with the Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore 138632, Singapore (e-mail: [email protected]). K. C. Tan is with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore 119260, Singapore (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TNN.2008.2009535
(2)
R
II. PRELIMINARIES As stated in [11], an equilibrium of network (2) is a vector x3 that satisfies
F (x3 ) := 0x3 + W (x3 ) + h 0:
(3)
The equilibrium x3 is said to be nondegenerate if the Jacobian matrix DF (x3 ) := (dF=dx)jx=x is invertible; otherwise, x3 is said to be degenerate. System (2) is nonlinear but, other than Jacobian matrix, the convenient way on its linearization is to consider the so-called effective recurrence matrix W e (see [1]), defined by
W e x(t) = W (x(t)) :
(4)
The matrix W e describes the connectivity of the operating network, where only active neurons are dynamically relevant. More intuitively, at any time t, each LT neuron is either firing (active) or silent (inactive). So the whole ensemble of the LT neurons can be divided into a partition P + (t) of neurons with positive states xi (t) 0 for i 2 P + (t) and a partition P 0 (t) of neurons with negative states xi (t) < 0 for i 2 P 0 (t). Clearly, P + (t) [ P 0 (t) = f1; . . . ; ng. There are totally n e 2 possible partitions for x. The matrix W e = (wij ) is equivalently defined by e wij
=
wij ; j 2 P + 0; j 2 P 0.
Consequently, the nonlinear dynamics of the LT neural network is simplified as
x_ (t) = (W e 0 I )x(t) + h
(5)
where I is the identity matrix. As indicated in [11], for any fixed partition, the LT network is described as a linear system, but nonlinearities arise when a partition is switching to another. For 2-D system (1) or (2), i.e., n = 2, let J denote the Jacobian matrix of the LT network, i.e.,J = W e 0 I , and let J1 ; . . . ; J4 denote
1045-9227/$25.00 © 2008 IEEE
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
176
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
the restrictions of J to the four quadrants D1 ; . . . ; D4 , respectively, where
D1 = D2 = D3 = D4 = Clearly,
R2
=
J1 = J3 =
D1
x1 ; x2 ) 2 R2 : x1 2 (x1 ; x2 ) 2 R : x1 2 (x1 ; x2 ) 2 R : x1 2 (x1 ; x2 ) 2 R : x1 (
D2
D3
01 0
0
01
< 0; x2 < 0 0; x2 < 0 :
01
J2 =
w12 w22 0 1
0
w11 0 1 0 w21 01 :
J4 =
Let := (w11 ; w12 ; w21 ; w22 ; h1 ; h2 ). The equilibria properties and distributions for nondegenerate dynamical systems have been given in [11]. Unlike the nondegenerate case in [11], a degenerate system of differential equations may have infinitely many equilibria. Consider the linear system
dx dt :=
Ax
= (
degenerate linear systems [14]. Lemma 1: Suppose that = 0 in system (6). 1) If A = 0, each point in 2 is an equilibrium of system (6). Moreover, each of them is stable and neutral, i.e., no orbits tend to the equilibrium as t ! 61. 2) If = 0 and A 6= 0, system (6) has a unique singular line ` : a11 x1 + a12 x2 = 0 (particularly, `1 : x2 = 0 and `2 : x1 = 0), on which every point is an equilibrium, if A is of the form
R
a11 a12 a21 a22
particularly
0
a12
0
0
and
0
0
a21
0
particularly
a11 a12 0
0
R
J
;
0
0
a21 a22 0 0
;
a12 a ; 11 a22 a21
0 0
=
J1 =
w11 0 1 w12 w21 w22 0 1
and equilibrium x3 is computed by (3), i.e.,
w11 0 1)x13 + w12 x23 + h1 = 0 w21 x13 + (w22 0 1)x23 + h2 = 0: (
Because J1 is degenerate, the parameters wij (i; j one of the following conditions:
w11 = 1; w12 = 0 w21 = 0; w22 = 1 w11 = 1; w12 = 0 (1c) 2 2 + (w22 0 1) 6= 0 w21 (1a)
(1e)
where a11 a22 = a12 a21 and a11 + a22 = 0. Moreover, off the singular line, the system has no other equilibria and all orbits are parallel to the line, but on the line, each point is an unstable equilibrium. 3) If < 0, system (6) has a unique singular line ` : a11 x1 + a12 x2 = 0 or `0 : a21 x1 + a22 x2 = 0 (particularly, `1 : x2 = 0 and `2 : x1 = 0) if A is of the form
a11 a12 a21 a22
Theorem 1: Suppose the matrix Ji is degenerate, i.e., det Ji = 0, where i = 1; . . . ; 4. Then, the qualitative properties and distribution of the 2-D LT network equilibrium x3 and the corresponding parameter conditions are described in Table I. Proof: Let the neuron states x 2 2 be divided into the four quadrants D1 , D2 , D3 , and D4 . We generally consider an equilibrium x3 of the system and discuss the following four cases. Case 1: x3 = (x13 ; x32 ) 2 D1 : In this case, the effective recurrence matrix W e is equal to W . So J = W 0 I , which implies that
(6)
x1 ; x2 )T 2 R2 and A = (aij ) is a 2 2 2 matrix. Let det A and = tr A. The following results are elementary for
where x
=
To discuss the degenerate equilibria of system (2), let
L11 : (w11 0 1)x1 + w12 x2 + h1 = 0; for (x1 ; x2 ) 2 D1 L12 : w21 x1 + (w22 0 1)x2 + h2 = 0; for (x1 ; x2 ) 2 D1 L13 : (w11 0 1)x1 + w12 x2 = 0; for(x1 ; x2 ) 2 D1 L14 : w21 x1 + (w22 0 1)x2 = 0; for (x1 ; x2 ) 2 D1 for (x1 ; x2 ) 2 D1 L15 : w12 x2 + h1 = 0; for (x1 ; x2 ) 2 D1 L16 : w21 x1 + h2 = 0; L2 : 0x1 + w12 x2 + h1 = 0; for (x1 ; x2 ) 2 D2 L4 : w21 x1 0 x2 + h2 = 0; for (x1 ; x2 ) 2 D4 :
0; x2 0 < 0; x2 0
D4 and
w11 0 1 w12 w21 w22 0 1
III. DEGENERATE EQUILIBRIA
w11 = 1; w12 6= 0 w21 = 0; w22 6= 1
(7)
;
= 1 2)
in J1 satisfy
2 w11 0 1)2 + w12 6= 0 w21 = 0; w22 = 1 w11 6= 1; w12 = 0 (1d) w21 6= 0; w22 = 1 w11 6= 1; w12 6= 0 w 21 6= 0; w22 6= 1 (1f) w12 w11 0 1 = : w21 w22 0 1
(1b)
(
Under condition (1a), the necessary condition for existence of equilibria in D1 is that h1 0 and h2 0. Thus, we obtain the conditions
w11 = w22 = 1 w12 = w21 = 0 h1 = h2 = 0 as shown in class I 0 (1) in Table I, under which all points in D1 are equilibria. Furthermore, those parameters in this case satisfy conditions in Lemma 1i), so each point in D1 is stable and neutral. Under condition (1b), the necessary condition for existence of equilibria of the system is h2 0. Therefore, x3 := (x13 ; x32 ) is determined by (7) as follows. When h1 = 0, x3 = (0; 0) is the unique equilibrium if
w11 0 1)w12 > 0
(8)
(
where a11 a22 = a12 a21 and a11 + a22 < 0. Moreover, off the singular line, the system has no other equilibria and all orbits straightly go and approach the line as t ! +1 in a parallel manner, but on the line, each point is a stable equilibrium. 4) If > 0, the system has the same singular line as in 3) but the asymptotic behavior is reverse.
and, therefore, the conditions are summarized as
w22 = 1 w21 = 0 h1 = h2 = 0
w11 0 1)w12 > 0:
(
This equilibrium is either stable if w11 < 1 or unstable if w11 > 1, by 3) and 4) of Lemma 1. If (8) does not hold, then the system has
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
177
TABLE I PROPERTIES AND DISTRIBUTIONS OF DEGENERATE EQUILIBRIA
a singular line L13 , which is either stable if w11 < 1 or unstable if w11 1, by 2), 3), and 4) of Lemma 1. The conditions in this two situations are summarized, respectively, as
w22 = 1; w21 = 0 w22 = 1; w21 = 0 h1 = h2 = 0 h1 = h2 = 0 and w11 1; w12 < 0 w11 < 1; w12 0: When h1 = 6 0, the existence of equilibria implies minf(w11 0 1)=h1 ; w12 =h1 g < 0. Together with (1b), this inequality gives the con-
dition of equilibria
w22 = 1 w21 = h2 = 0
min
w11 0 1 ; w12 < 0 h1 h1
as indicated in classes I 0 (7)–I 0 (9) in Table I. In such a situation, those equilibria form either the singular line L11 when w11 6= 1 or the singular line L15 when w11 = 1. By Lemma 1, L15 is unstable and L11 is either stable when w11 < 1 or unstable when w11 > 1. The case under condition (1c) can be discussed similarly to the case of (1b). Under condition (1d), w11 6= 1. The necessary condition for existence of equilibria is h1 =(w11 0 1) = h2 =w21 . In this situation, all equilibria of the system form a singular line L16 , which is either stable when w11 < 1 or unstable when w11 > 1, by 3) and 4) of Lemma 1. The case of (1e) can be discussed similarly to the case of (1d). Under condition (1f), the necessary condition for equilibria is that
(w11 0 1)
w21
=
w12
(w22 0 1)
h1 w12
=
h2
(w22 0 1)
:
This condition combined with the inequality minfw12 =(w11 0 1); h1 =(w11 0 1)g < 0 gives existence of equilibria in D1 , as shown in classes I 0 (22)–I 0 (24) in Table I. By Lemma 1, those equilibria actually form a singular line L11 , which is either stable when w11 + w22 < 2 or unstable when w11 + w22 2.
Case 2: x3 = (x13 ; x32 )
2 D2 : In this case J = J2 = 001 w w120 1 : 22
Then, det J2 = 0 if and only if w22 = 1. Thus, from (7), a degenerate equilibrium exists if and only if h2 0 and 0x1 + w12 x2 + h1 = 0. Clearly, those degenerate equilibria lie on the singular line L2 , which intersects the region D2 if and only if minfh1 ; w12 g < 0. By iii) of Lemma 1, L2 is stable. Case 3: x3 = (x13 ; x32 ) 2 D3 : In this case, the system has no degenerate equilibria since J = J3 = diag(01; 01). Case 4: x3 = (x13 ; x32 ) 2 D4 : This case can be discussed similarly as in Case 2 and a stable singular line L4 is obtained under the condition
w11 0 1 = h1 = 0
minfh2 ; w21 g < 0:
The proof is completed. IV. COEXISTENCE
OF
DEGENERATE EQUILIBRIA
AND
NONDEGENERATE
Multistability depends on coexistence of equilibria. Table I only gives the relations between the parameters (synaptic weights and external inputs) and the features of equilibria for the network. The results of Table I allow us to derive the sufficient and necessary conditions for the coexistence of more than one equilibria by solving the intersections of the parameter sets defined by the corresponding inequalities. Let Si denote the parameter set that corresponds to the class of equilibria indexed by i = I (1); . . . ; I (6); II (1); II (2); III; IV (1); IV (2) (the nondegenerate ones in [11, Table I]) and I 0 (1); I 0 (2); . . . ; I 0 (24);II 0 ; III 0 ; IV 0 (Table I). Theorem 2: The 2-D network (5) has at least two degenerate equilibria if and only if the parameter lies in one of the sets: SII \ SIV ,
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
178
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
SI (2) [ ([9i=5 SI (i) ) \ SII
, and SI (10) [ ([i17 =13 SI (i) ) \ SIV (formulations can be found in the Appendix). Proof: The condition of coexistence is given by intersection of those conditions of existence for each equilibrium (or singular line). For the system to have at least two singular lines, it suffices to consider the intersections of all pairs of sets: SI (1) ; . . . ; SI (24) , SII , SIII . and SIV , peeling off the empty ones and simplifying those nonempty intersections. One can check that the intersection of each pair of sets SI (1) ; . . . ; SI (24) is empty. In fact, the condition w11 = 1 in SI (1) contradicts the condition w11 < 1 in SI (2) , so SI (1) \ SI (2) = ;. It is similar for others. Thus, we only need to discuss the three cases: C1) ([i24 =1 SI (i) ) \ SII ; C2) ([i24 =1 SI (i) ) \ SIV ; C3) SII \ SIV . In case C1), the set SII is described by Fig. 1. Degenerate equilibrium and a singular line coexist.
w22 = 1; h2 = 0 minfh1 ; w12 g < 0:
(9)
Obviously, the sets SI (20) ; . . . ; SI (24) all contradict one of the equalities in the first row of (9), i.e., w22 = 1, h2 = 0. Moreover, the sets SI (1) ; SI (10) ; . . . ; SI (17) all contradict the condition in the second row of (9). Besides, both SI (18) \ SII and SI (19) \ SII are empty. In fact, h2 = 0 and h1 =(w11 0 1) = h2 =w21 implies h1 = 0, which together with w12 = 0 contradict minfh1 ; w12 g < 0. SI (i) SII , i = 3; 4, are also empty because the conditions h1 = 0 and w12 > 0 (resp., w12 0) contradict the condition minfh1 ; w12 g < 0. It implies that the intersection in case C1) is equal to (SI (2) [ ([i9=5 SI (i) )) SII . Computing the intersection SI (2) SII yields
w11 < 1; w12 < 0; w21 = 0; w22 = 1 h1 = h2 = 0 minfh1 ; w12 g < 0: It is equivalent to
Proof: We consider the intersections of sets in [11, Th. 4] and sets
SI (1) ; . . . ; SI (24) ; SII ; SIV . SI (j ) in Table I and SI (i) in [11, Table 1] imply that SI (i) \ SI (j ) = ; for all i = 1; . . . ; 6, j = 1; . . . ; 24, and Si(j) \Si = ; for all i = II , IV , j = 1; 2. Furthermore, Theorem 3 implies that SIII \ SII = ;, SIII \ SIV = ;, and SII (i) \ SI (j ) = ; for all i = 1; 2, j = 1; . . . ; 24. It follows that
SII (1) [ SII (2)
Thus, we only need to discuss the remaining intersections: SI (1) \ SII (1) \ SIV , SI (1) \ SIV (1) \ SII , (SI (3) [ SI (5) ) \ SII (2) \ SIV , and (SI (3) [ SI (5) ) \ SIV (2) \ SII . It is easy to derive the intersection SI (3) \ SIV (2) \ SII as
w11 < 1; w12 < 0; w21 = 0; w22 = 1 h1 = h2 = 0: Similarly, the intersections of other pairs SI (i) SII , i = 5; . . . ; 9 can be computed. The cases C2) and C3) can be analyzed in the same procedure as in case C1). The expressions of intersections are also given in the Appendix. Using the same idea, we can obtain all conditions under which at least one nondegenerate equilibrium and either one singular line or two singular lines coexist. We only state the results and omit the similar analysis. Theorem 3: The 2-D network (5) has one nondegenerate equilibrium and one singular line if lies in one of the sets: (([6i=1 SI (i) ) [ 2 6 2 ([i=1 SIV (i) )) \ SII , (([i=1 SI (i) ) [ ([i=1 SII (i) )) \ SIV , and 24 SIII \ ([i=18 SI (i) ) (formulations can be found in the Appendix). Theorem 4: The 2-D network (5) has one nondegenerate equilibrium and two singular lines if lies in the set SI (1) \ SII \ SIV , which is formulated by f 2 6 jw11 = w22 = 1; h1 = h2 = 0; w12 < 0;w21 < 0g. More complicated equalities and inequalities are involved in finding conditions of coexistence of two nondegenerate equilibria and a singular line, although the same idea is used. Theorem 5: The 2-D network (5) has two nondegenerate equilibria and one singular line if lies in one of the sets: SI (1) \ SII (1) \ SIV , SI (1) \ SIV (1) \ SII , (SI (3) [ SI (5) ) \ SII (2) \ SIV , and (SI (3) [ SI (5) ) \ SIV (2) \ SII (formulations can be found in the Appendix).
R
[i6=2 SI (i) \ SIII \ (SII [ SIV ) = ; SII (2) \ SIII \ (SII [ SIV ) = ; \ SIV (1) [ SIV (2) \ [j24=1 SI (j) = ; SIII \ SIV (2) \ [i24=1 SI (i) [ SII = ;:
0 14 (w11 0 w22 )2 < w12 w21 < (w11 0 1)(w22 0 1) (w22 0 1)h1 w12 h2 h1 0; h2 < w21 h1 =(w11 0 1)
(10)
w11 > 1; w11 + w22 > 2 w22 = 1; h2 = 0; minfh1 ; w21 g < 0 where the inequalities w11 > 1 and w22 = 1 imply that w11 + w22 > 2.
Moreover
01=4(w11 0 1)2 < w12 w21 < 0:
(11)
Thus, (10) can be simplified further. Note that h1 6= 0; otherwise, h2 < by the second inequality on the third line of (10), which contradicts h2 = 0. Since it has been proved that h1 < 0, we naturally have minfh1 ; w21 g < 0. Furthermore, since 0 = h2 < w21 h1 =(w11 0 1), w11 > 1, and h1 < 0, we assure that w21 < 0. It follows from (11) that w12 > 0. It is also clear that the inequality on the second line of (10) is a vain restriction because (w22 0 1)h1 = 0 and w12 h2 = 0. Therefore, (10) is finally simplified as 0
w11 > 1; w12 > 0; w21 < 0; w22 = 1 h1 0; h2 = 0 0 14 (w11 0 1)2 < w12 w21 as given for SI (3) \ SIV (2)
\ SII
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
in the Appendix.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
Computation of the intersections SI (5) \ SIV (2) \ SII and those intersections SI (1) \ SII (1) \ SIV , SI (1) \ SIV (1) \ SII , and (SI (3) [ SI (5) ) \ SII (2) \ SIV can be proceeded similarly (simplified expressions are shown in the Appendix). Corollary 1: None of the following equilibria coexist: 1) three singular lines; 2) a degenerate equilibria and two singular lines; 3) two equilibria and two singular lines; and 4) three equilibria and a singular line. Proof: By Theorem 5, ([9i=1 SI (i) ) [ ([i12 =11 SI (i) ) [ 24 4 24 ([i=18 SI (i) ) \ SIV = ; and SI 1 [ ([i=3 SI (i) ) [ ([i=10 SI (i) ) \ 24 SII = ;. It follows that ([i=1 SI (i) ) \ SII \ SIV = ;, which means that either any three singular lines or a degenerate equilibria and two singular lines cannot coexist. By Theorem 5, SI (1) \ SII (1) \ ([i24 =1 SI (i) [ SII ) = ;, SI (1) \ SIV (1) \ ([i24=1 SI (i) [ SIV ) = ;, (SI (3) [ SI (5) ) \ SII (2) \ 24 24 ([i=1 SI (i) [ SII ) = ;, and (SI (3) [ SI (5) ) \ SIV (2) \ ([i=1 SI (i) [ SIV ) = ;, implying that neither any two equilibria and any two singular lines nor three equilibria and a singular line can coexist. V. SIMULATIONS AND DISCUSSIONS In this section, simulations are presented to illustrate the theoretical results. Example 1: Consider a 2-D neural network (2) with
W
=
1
0
01 01
h=
0 0
:
One can check that the parameters lie in the intersection SI (10) SIV . By Theorem 2, the network has a stable degenerate equilibrium (0; 0) in D1 and a stable singular line L4 : 0x1 0 x2 = 0; (x1 ; x2 ) 2 D4 . Its phase portrait is illustrated in Fig. 1, where some orbits tend to the origin and others are attracted by the line L4 . The attractor composed by the origin and the line L4 is not a compact set. Example 2: Consider a 2-D neural network (2) with
W
=
04
3
0
1
h=
01 : 0
It is verified that the parameters belong to the intersection SI (7) SII . By Theorem 2, the network has two degenerate equilibria: two stable lines L11 : 05x1 + 3x2 0 1 = 0; (x1 ; x2 ) 2 D1 and L2 : 0x1 + 3x2 0 1 = 0; (x1 ; x2 ) 2 D2 (Fig. 2). It is seen that some orbits tend to the line L11 and others are attracted by the line L2 . It is noted that the attractor composed by the line L11 and the line L2 is not a compact set. As shown in [1], in the cases of degenerate equilibria, some neurons can get latched at a nonzero activity for zero input h = 0 and there are an infinite number of equilibria. An example of stable singular line (i.e., an infinite number of stable equilibria) was given in [10]. In visual perception, visual images are stored as a number of continuous fixed points (i.e., a continuous attractor) and the brain is able to hold the eyes still because its memory network stores the eye position as a line attractor [5], [12]. On the other hand, as noted in [15], more than 25 years of experimental and theoretical work indicate that the onset of oscillations in neurons and in neuron populations is characterized by multistability. VI. CONCLUSION In this paper, degenerate equilibria and their dynamic properties have been discussed for the recurrent neural networks with LT transfer func-
179
Fig. 2. Two stable singular lines coexist.
tions. In order to reveal the geometrical structures of the attractors, the dynamical behaviors and the conditions of coexistence of more than one (degenerate and nondegenerate) equilibria have been examined. Based on those results, all conditions of existence of continuous attractors that contain more than one equilibria have been obtained. Those conditions enrich and enhance the analytical results for the attractors’ geometrical structures in previous work [11]. They provide an insightful understanding about the continuous attractors in the system that composes of the recently explored memory network model and give a useful basis for precise tuning of the parameters. APPENDIX A. Sets of Parameters in Theorem 2 S
S
S
S
S
S
S
\S
:
w22 = 1 w21 = h1 = h2 = 0 w11 < 1; w12 < 0 \S
:
w11 = 1; w22 = 1 w12 = h1 = 0 w21 =h2 < 0
S
S
\S
:
S
\S
:
S
w22 = 1; w11 < 1 w21 = h2 = 0 h1 w12 < 0 w22 = 1; w11 = 1 w21 = h2 = 0 w12 =h1 < 0 \S
:
w12 < 0; w21 < 0 w11 = w22 = 1 h1 = h2 = 0 \S
:
\S
:
w11 = 1 w12 = h1 = h2 = 0 w22 = 1; w21 < 0 w11 = 1 w12 = h1 = h2 = 0 w22 > 1; w21 < 0:
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
S
S
\S
:
\S
:
w22 = 1 w21 = h1 = h2 = 0 w11 = 1; w12 < 0 w22 = 1 w21 = h1 = h2 = 0 w11 > 1; w12 < 0 \S
:
w11 = 1; w22 > 1 w12 = h1 = 0; h2 6= 0 minfh2 ; w21 g < 0 \S
:
w22 = 1; w11 > 1 w21 = h2 = 0; h1 6= 0 minfh1 ; w12 g < 0 \S
:
\S
:
w11 = 1 w12 = h1 = h2 = 0 w22 < 1; w21 < 0 w11 = 1; w22 < 1 w12 = h1 = 0 h2 w21 < 0
180
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
B. Sets of Parameters in Theorem 3
C. Sets of Parameters in Theorem 5 \S : S S \S
S
\S
:
w = 1; h = 0 w < 0(> 0); w < 0(> 0) h 0(< 0) \S 0 (w 0 1) w w w = 1; w < 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S 0 (w 0 1) w w w = 1; w > 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S w w < 0 (w 0 1) w < 1; w = 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S w w < 0 (w 0 1) w > 1; w = 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S w =w =1 h 0(< 0); h = 0 w < 0(> 0); w > 0(< 0) \S w = 1; h = 0 w < 1; h > 0 w < 0; w < 0 22 12
S
1
:
2
11
4
22
12
11
12
21
1
2
:
1
2
11
4
22
12
21
1
2
:
12
1
21
22
12
21
1
2
:
12
1
21
22
12
21
1
2
:
22
11
1
S
S
2
12
21
:
22
2
11
1
21
\S
12
:
w = 1; h = 0 w < 1; h > 0 w < 0; w < 0 11
S
2
=
1
4
\S w 0
1
:
= ww 0 = hh w +w >2 h < 0; h < 0 min ww 0 ; w h 0 < 0 1
w
11
1
22
1
2
\S
1
12
21
22
12
21
1
2
:
1
2
11
4
11
12
21
22
12
21
1
2
:
12
1
21
11
4
11
2
22
12
21
1
2
:
12
1
21
11
4
11
2
22
12
21
1
2
:
22
11
12
21
1
2
:
22
2
11
1
21
:
S
12
2
21
1
2
\S w 01 w 1
S
1
:
= ww 0 = hh w +w =2 h < 0; h < 0 min ww 0 ; w h 0 < 0 11
2
1
2
22
11
22
1
S
=
h h
w +w 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S 0 (w 0 1) w w w = 1; w > 1 w > 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S w w < 0 (w 0 1) w = 1; w < 1 w > 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S w w < 0 (w 0 1) w = 1; w > 1 w > 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S w =w =1 w < 0(> 0); w > 0(< 0) h = 0; h 0(< 0) S \S w = 1; h = 0 w > 1; h < 0 0)
11
2
21
:
22
1
22
S
2
11
4
11
S
2
11
4
11
S
21
11
12
S
21
\S
11
21
1
S
S
2
1
22 2
\S
1
:
w = 1; w = 0 w >1 h < 0; h < 0 11
21
22
1
h h
2
= ww 0
1
w = 1; h = 0 w < 1; h > 0 w < 0; w < 0 \S S \S w w < 0 (w 0 1) h < 0; w < 0 w > 1; w > 0 w = 1; h = 0 \S S \S w w < 0 (w 0 1) w = 1; h = 0 w > 1; h < 0 w > 0; w < 0: 11
1
22
2
21
12
:
12
1
21
1
4
2
11
21
11
12
22
2
:
12
1
21
4
11
1
22
2
21
11
1 2
:
21
1
11
22
2
:
1 4
11
2
21
2
22
12
21
1
22
12
12
ACKNOWLEDGMENT The authors would like to thank the anonymous reviewers for their valuable comments and suggestions that helped to improve this brief significantly.
REFERENCES [1] R. L. T. Hahnloser, “On the piecewise analysis of linear threshold neural networks,” Neural Netw., vol. 11, pp. 691–697, 1998. [2] R. Douglas, C. Koch, M. Mahowald, and H. Suarez, “Recurrent excitation in neocortical circuits,” Science, vol. 269, pp. 981–985, 1995. [3] H. Wersing, W.-J. Beyn, and H. Ritter, “Dynamical stability conditions for recurrent neural networks with unsaturating piecewise linear transfer functions,” Neural Comput., vol. 13, pp. 1811–1825, 2001. [4] R. Hahnloser, R. Sarpeshkar, M. A. Mahowald, R. J. Douglas, and H. S. Seung, “Digital selection and analogue amplification coexist in a cortex-inspired silicon circuits,” Nature, vol. 405, pp. 947–951, 2000. [5] H. S. Seung, “Continuous attractors and oculomotor control,” Science, vol. 290, pp. 2268–2269, 2000. [6] Z. Yi, K. K. Tan, and T. H. Lee, “Multistability analysis for recurrent neural networks with unsaturating piecewise linear transfer functions,” Neural Comput., vol. 15, no. 3, pp. 639–662, 2003. [7] L. Zhang, Z. Yi, and J. Yu, “Multiperiodicity and attractivity of delayed recurrent neural networks with unsaturating piecewise linear transfer functions,” IEEE Trans. Neural Netw., vol. 19, no. 1, pp. 158–167, Jan. 2008. [8] A. Meyer-Base and V. Thummler, “Local and global stability analysis of an unsupervised competitive neural network,” IEEE Trans. Neural Netw., vol. 19, no. 2, pp. 346–351, Feb. 2008. [9] A. A. Frolov, D. Husek, I. P. Muraviev, and P. Y. Polyakov, “Boolean factor analysis by attractor neural network,” IEEE Trans. Neural Netw., vol. 18, no. 3, pp. 698–707, May 2007. [10] Z.-H. Mao and S. G. Massaquoi, “Dynamics of winner-take-all competition in recurrent neural networks with lateral inhibition,” IEEE Trans. Neural Netw., vol. 18, no. 1, pp. 55–69, Jan. 2007. [11] K. C. Tan, H. Tang, and W. Zhang, “Qualitative analysis for recurrent neural networks with linear threshold transfer functions,” IEEE Trans. Circuit Syst. I, Reg. Papers, vol. 52, no. 5, pp. 1003–1012, May 2005. [12] H. S. Seung, “How the brain keeps the eyes still,” Proc. Nat. Acad. Sci. USA, vol. 93, pp. 13339–13344, 1996. [13] H. Tang, K. C. Tan, and W. Zhang, “Cyclic dynamics analysis for networks of linear threshold neurons,” Neural Comput., vol. 17, no. 1, pp. 97–114, 2005. [14] Z.-F. Zhang, T.-R. Ding, W.-Z. Huang, and Z.-X. Dong, Qualitative Theory of Differential Equations. Providence, RI: Transl. Math. Monographs, Amer. Math. Soc., 1992. [15] J. Milton, “EPILESPY: Multistability in a dynamic disease,” in SelfOrganized Biological Dynamics Nonlinear Control: Toward Understanding Complexity, Chaos, and Emergent Function in Living Systems, J. Walleczek, Ed. Cambridge, U.K.: Cambridge Univ. Press, 2000, pp. 374–386.
1
w = 1; h = 0 w > 1; h < 0 w 0; w < 1 w = 1; h = 0 \S S \S 0 (w 0 1) w w w = 1; h = 0 h < 0; w > 1 w > 0; w < 0
12
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
175
Analysis of Continuous Attractors for 2-D Linear Threshold Neural Networks
or its equivalent form
Lan Zou, Huajin Tang, Kay Chen Tan, and Weinian Zhang
where each xi denotes the activity of neuron i, x = (x1 ; . . . ; xn )T 2 n , h 2 n denotes external inputs, W = (wij )n2n is a constant real matrix, each entry of which denotes the synaptic weight representing the strength of the synaptic connection from neuron j to neuron i, and (x) is the LT transfer function defined by (x) = maxf0; xg. The global attractivity and multistability were examined in the theoretical studies [6], [11], and more complicated dynamics, such as limit cycle was investigated with analytical results in [13]. The continuous attractors as limit cycle were investigated in our previous work [13], and the dynamic properties and geometrical structures of nondegenerate equilibria were studied in [11]. As an extension of the previous works, this work is focused on the degenerate equilibria of the LT network. The relationships between features of equilibria and network parameters (synaptic weights and external inputs) are revealed so that sufficient and necessary conditions for coexistence of both nondegenerate equilibria and degenerate ones are drawn.
x_ (t) = 0x(t) + W (x(t)) + h
R
Abstract—This brief investigates continuous attractors of the well-developed model in visual cortex, i.e., the linear threshold (LT) neural networks, based on a parameterized 2-D model. On the basis of existing results on nondegenerate equilibria in mathematics, we further discuss degenerate equilibria for such networks and present properties and distributions of the equilibria, which enables us to draw the coexistence conditions of nondegenerate and degenerate equilibria (e.g., singular lines). Our theoretical results provide a useful framework for precise tuning on the network parameters, e.g., the feedbacks and visual inputs. Simulations are also presented to illustrate the theoretical findings. Index Terms—Continuous attractor, degenerate equilibria, linear threshold (LT) network, multistable neural network, singular line.
I. INTRODUCTION The recurrent neural networks with linear threshold (LT) transfer functions have received extensive research interest in recent years from both aspects of mathematics and neuroscience [1]–[7]. Some features underling the attractor recurrent neural networks, such as stability, attractivity, and winner-takes-all (WTA) behavior have also been studied recently [8]–[10]. The LT networks exhibit an important dynamic property, namely, multistability, which underlies the mathematical principles in visual cortex and eye memory networks. In our previous work [11], the features of nondegenerate equilibria and global attractivity have been discussed, and theoretical proofs for the coexistence of multiple nondegenerate equilibria were also given. However, there is a lack of theoretical studies on the degenerate equilibria of the LT networks, which would form a continuous attractor (e.g., singular lines and limit cycles). As reported in the neural model [5], [12], continuous attractor is an important dynamical property in the memory networks, because the oculomotor control in the brain requires storing continuous eye positions. The relationship between the network parameters and such degenerate equilibria remains unknown. Thus, it is necessary to explore the dynamical properties of the degenerate equilibria and to reveal the parameter conditions. We consider the recurrent neural networks with LT transfer functions described by n
x_ i (t) = 0xi (t) + j
=1
wij (xj (t)) + hi ;
i = 1; . . . ; n
(1)
Manuscript received April 09, 2008; accepted October 31, 2008. First published December 09, 2008; current version published January 05, 2009. This work was supported in part by the National Science Foundation of China under Grants 10825104 and 10571127. L. Zou and W. Zhang are with the Yangtze Center of Mathematics and the Department of Mathematics, Sichuan University, Chengdu 610064, China (e-mail: [email protected]; [email protected]). H. Tang is with the Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore 138632, Singapore (e-mail: [email protected]). K. C. Tan is with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore 119260, Singapore (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TNN.2008.2009535
(2)
R
II. PRELIMINARIES As stated in [11], an equilibrium of network (2) is a vector x3 that satisfies
F (x3 ) := 0x3 + W (x3 ) + h 0:
(3)
The equilibrium x3 is said to be nondegenerate if the Jacobian matrix DF (x3 ) := (dF=dx)jx=x is invertible; otherwise, x3 is said to be degenerate. System (2) is nonlinear but, other than Jacobian matrix, the convenient way on its linearization is to consider the so-called effective recurrence matrix W e (see [1]), defined by
W e x(t) = W (x(t)) :
(4)
The matrix W e describes the connectivity of the operating network, where only active neurons are dynamically relevant. More intuitively, at any time t, each LT neuron is either firing (active) or silent (inactive). So the whole ensemble of the LT neurons can be divided into a partition P + (t) of neurons with positive states xi (t) 0 for i 2 P + (t) and a partition P 0 (t) of neurons with negative states xi (t) < 0 for i 2 P 0 (t). Clearly, P + (t) [ P 0 (t) = f1; . . . ; ng. There are totally n e 2 possible partitions for x. The matrix W e = (wij ) is equivalently defined by e wij
=
wij ; j 2 P + 0; j 2 P 0.
Consequently, the nonlinear dynamics of the LT neural network is simplified as
x_ (t) = (W e 0 I )x(t) + h
(5)
where I is the identity matrix. As indicated in [11], for any fixed partition, the LT network is described as a linear system, but nonlinearities arise when a partition is switching to another. For 2-D system (1) or (2), i.e., n = 2, let J denote the Jacobian matrix of the LT network, i.e.,J = W e 0 I , and let J1 ; . . . ; J4 denote
1045-9227/$25.00 © 2008 IEEE
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
176
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
the restrictions of J to the four quadrants D1 ; . . . ; D4 , respectively, where
D1 = D2 = D3 = D4 = Clearly,
R2
=
J1 = J3 =
D1
x1 ; x2 ) 2 R2 : x1 2 (x1 ; x2 ) 2 R : x1 2 (x1 ; x2 ) 2 R : x1 2 (x1 ; x2 ) 2 R : x1 (
D2
D3
01 0
0
01
< 0; x2 < 0 0; x2 < 0 :
01
J2 =
w12 w22 0 1
0
w11 0 1 0 w21 01 :
J4 =
Let := (w11 ; w12 ; w21 ; w22 ; h1 ; h2 ). The equilibria properties and distributions for nondegenerate dynamical systems have been given in [11]. Unlike the nondegenerate case in [11], a degenerate system of differential equations may have infinitely many equilibria. Consider the linear system
dx dt :=
Ax
= (
degenerate linear systems [14]. Lemma 1: Suppose that = 0 in system (6). 1) If A = 0, each point in 2 is an equilibrium of system (6). Moreover, each of them is stable and neutral, i.e., no orbits tend to the equilibrium as t ! 61. 2) If = 0 and A 6= 0, system (6) has a unique singular line ` : a11 x1 + a12 x2 = 0 (particularly, `1 : x2 = 0 and `2 : x1 = 0), on which every point is an equilibrium, if A is of the form
R
a11 a12 a21 a22
particularly
0
a12
0
0
and
0
0
a21
0
particularly
a11 a12 0
0
R
J
;
0
0
a21 a22 0 0
;
a12 a ; 11 a22 a21
0 0
=
J1 =
w11 0 1 w12 w21 w22 0 1
and equilibrium x3 is computed by (3), i.e.,
w11 0 1)x13 + w12 x23 + h1 = 0 w21 x13 + (w22 0 1)x23 + h2 = 0: (
Because J1 is degenerate, the parameters wij (i; j one of the following conditions:
w11 = 1; w12 = 0 w21 = 0; w22 = 1 w11 = 1; w12 = 0 (1c) 2 2 + (w22 0 1) 6= 0 w21 (1a)
(1e)
where a11 a22 = a12 a21 and a11 + a22 = 0. Moreover, off the singular line, the system has no other equilibria and all orbits are parallel to the line, but on the line, each point is an unstable equilibrium. 3) If < 0, system (6) has a unique singular line ` : a11 x1 + a12 x2 = 0 or `0 : a21 x1 + a22 x2 = 0 (particularly, `1 : x2 = 0 and `2 : x1 = 0) if A is of the form
a11 a12 a21 a22
Theorem 1: Suppose the matrix Ji is degenerate, i.e., det Ji = 0, where i = 1; . . . ; 4. Then, the qualitative properties and distribution of the 2-D LT network equilibrium x3 and the corresponding parameter conditions are described in Table I. Proof: Let the neuron states x 2 2 be divided into the four quadrants D1 , D2 , D3 , and D4 . We generally consider an equilibrium x3 of the system and discuss the following four cases. Case 1: x3 = (x13 ; x32 ) 2 D1 : In this case, the effective recurrence matrix W e is equal to W . So J = W 0 I , which implies that
(6)
x1 ; x2 )T 2 R2 and A = (aij ) is a 2 2 2 matrix. Let det A and = tr A. The following results are elementary for
where x
=
To discuss the degenerate equilibria of system (2), let
L11 : (w11 0 1)x1 + w12 x2 + h1 = 0; for (x1 ; x2 ) 2 D1 L12 : w21 x1 + (w22 0 1)x2 + h2 = 0; for (x1 ; x2 ) 2 D1 L13 : (w11 0 1)x1 + w12 x2 = 0; for(x1 ; x2 ) 2 D1 L14 : w21 x1 + (w22 0 1)x2 = 0; for (x1 ; x2 ) 2 D1 for (x1 ; x2 ) 2 D1 L15 : w12 x2 + h1 = 0; for (x1 ; x2 ) 2 D1 L16 : w21 x1 + h2 = 0; L2 : 0x1 + w12 x2 + h1 = 0; for (x1 ; x2 ) 2 D2 L4 : w21 x1 0 x2 + h2 = 0; for (x1 ; x2 ) 2 D4 :
0; x2 0 < 0; x2 0
D4 and
w11 0 1 w12 w21 w22 0 1
III. DEGENERATE EQUILIBRIA
w11 = 1; w12 6= 0 w21 = 0; w22 6= 1
(7)
;
= 1 2)
in J1 satisfy
2 w11 0 1)2 + w12 6= 0 w21 = 0; w22 = 1 w11 6= 1; w12 = 0 (1d) w21 6= 0; w22 = 1 w11 6= 1; w12 6= 0 w 21 6= 0; w22 6= 1 (1f) w12 w11 0 1 = : w21 w22 0 1
(1b)
(
Under condition (1a), the necessary condition for existence of equilibria in D1 is that h1 0 and h2 0. Thus, we obtain the conditions
w11 = w22 = 1 w12 = w21 = 0 h1 = h2 = 0 as shown in class I 0 (1) in Table I, under which all points in D1 are equilibria. Furthermore, those parameters in this case satisfy conditions in Lemma 1i), so each point in D1 is stable and neutral. Under condition (1b), the necessary condition for existence of equilibria of the system is h2 0. Therefore, x3 := (x13 ; x32 ) is determined by (7) as follows. When h1 = 0, x3 = (0; 0) is the unique equilibrium if
w11 0 1)w12 > 0
(8)
(
where a11 a22 = a12 a21 and a11 + a22 < 0. Moreover, off the singular line, the system has no other equilibria and all orbits straightly go and approach the line as t ! +1 in a parallel manner, but on the line, each point is a stable equilibrium. 4) If > 0, the system has the same singular line as in 3) but the asymptotic behavior is reverse.
and, therefore, the conditions are summarized as
w22 = 1 w21 = 0 h1 = h2 = 0
w11 0 1)w12 > 0:
(
This equilibrium is either stable if w11 < 1 or unstable if w11 > 1, by 3) and 4) of Lemma 1. If (8) does not hold, then the system has
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
177
TABLE I PROPERTIES AND DISTRIBUTIONS OF DEGENERATE EQUILIBRIA
a singular line L13 , which is either stable if w11 < 1 or unstable if w11 1, by 2), 3), and 4) of Lemma 1. The conditions in this two situations are summarized, respectively, as
w22 = 1; w21 = 0 w22 = 1; w21 = 0 h1 = h2 = 0 h1 = h2 = 0 and w11 1; w12 < 0 w11 < 1; w12 0: When h1 = 6 0, the existence of equilibria implies minf(w11 0 1)=h1 ; w12 =h1 g < 0. Together with (1b), this inequality gives the con-
dition of equilibria
w22 = 1 w21 = h2 = 0
min
w11 0 1 ; w12 < 0 h1 h1
as indicated in classes I 0 (7)–I 0 (9) in Table I. In such a situation, those equilibria form either the singular line L11 when w11 6= 1 or the singular line L15 when w11 = 1. By Lemma 1, L15 is unstable and L11 is either stable when w11 < 1 or unstable when w11 > 1. The case under condition (1c) can be discussed similarly to the case of (1b). Under condition (1d), w11 6= 1. The necessary condition for existence of equilibria is h1 =(w11 0 1) = h2 =w21 . In this situation, all equilibria of the system form a singular line L16 , which is either stable when w11 < 1 or unstable when w11 > 1, by 3) and 4) of Lemma 1. The case of (1e) can be discussed similarly to the case of (1d). Under condition (1f), the necessary condition for equilibria is that
(w11 0 1)
w21
=
w12
(w22 0 1)
h1 w12
=
h2
(w22 0 1)
:
This condition combined with the inequality minfw12 =(w11 0 1); h1 =(w11 0 1)g < 0 gives existence of equilibria in D1 , as shown in classes I 0 (22)–I 0 (24) in Table I. By Lemma 1, those equilibria actually form a singular line L11 , which is either stable when w11 + w22 < 2 or unstable when w11 + w22 2.
Case 2: x3 = (x13 ; x32 )
2 D2 : In this case J = J2 = 001 w w120 1 : 22
Then, det J2 = 0 if and only if w22 = 1. Thus, from (7), a degenerate equilibrium exists if and only if h2 0 and 0x1 + w12 x2 + h1 = 0. Clearly, those degenerate equilibria lie on the singular line L2 , which intersects the region D2 if and only if minfh1 ; w12 g < 0. By iii) of Lemma 1, L2 is stable. Case 3: x3 = (x13 ; x32 ) 2 D3 : In this case, the system has no degenerate equilibria since J = J3 = diag(01; 01). Case 4: x3 = (x13 ; x32 ) 2 D4 : This case can be discussed similarly as in Case 2 and a stable singular line L4 is obtained under the condition
w11 0 1 = h1 = 0
minfh2 ; w21 g < 0:
The proof is completed. IV. COEXISTENCE
OF
DEGENERATE EQUILIBRIA
AND
NONDEGENERATE
Multistability depends on coexistence of equilibria. Table I only gives the relations between the parameters (synaptic weights and external inputs) and the features of equilibria for the network. The results of Table I allow us to derive the sufficient and necessary conditions for the coexistence of more than one equilibria by solving the intersections of the parameter sets defined by the corresponding inequalities. Let Si denote the parameter set that corresponds to the class of equilibria indexed by i = I (1); . . . ; I (6); II (1); II (2); III; IV (1); IV (2) (the nondegenerate ones in [11, Table I]) and I 0 (1); I 0 (2); . . . ; I 0 (24);II 0 ; III 0 ; IV 0 (Table I). Theorem 2: The 2-D network (5) has at least two degenerate equilibria if and only if the parameter lies in one of the sets: SII \ SIV ,
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
178
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
SI (2) [ ([9i=5 SI (i) ) \ SII
, and SI (10) [ ([i17 =13 SI (i) ) \ SIV (formulations can be found in the Appendix). Proof: The condition of coexistence is given by intersection of those conditions of existence for each equilibrium (or singular line). For the system to have at least two singular lines, it suffices to consider the intersections of all pairs of sets: SI (1) ; . . . ; SI (24) , SII , SIII . and SIV , peeling off the empty ones and simplifying those nonempty intersections. One can check that the intersection of each pair of sets SI (1) ; . . . ; SI (24) is empty. In fact, the condition w11 = 1 in SI (1) contradicts the condition w11 < 1 in SI (2) , so SI (1) \ SI (2) = ;. It is similar for others. Thus, we only need to discuss the three cases: C1) ([i24 =1 SI (i) ) \ SII ; C2) ([i24 =1 SI (i) ) \ SIV ; C3) SII \ SIV . In case C1), the set SII is described by Fig. 1. Degenerate equilibrium and a singular line coexist.
w22 = 1; h2 = 0 minfh1 ; w12 g < 0:
(9)
Obviously, the sets SI (20) ; . . . ; SI (24) all contradict one of the equalities in the first row of (9), i.e., w22 = 1, h2 = 0. Moreover, the sets SI (1) ; SI (10) ; . . . ; SI (17) all contradict the condition in the second row of (9). Besides, both SI (18) \ SII and SI (19) \ SII are empty. In fact, h2 = 0 and h1 =(w11 0 1) = h2 =w21 implies h1 = 0, which together with w12 = 0 contradict minfh1 ; w12 g < 0. SI (i) SII , i = 3; 4, are also empty because the conditions h1 = 0 and w12 > 0 (resp., w12 0) contradict the condition minfh1 ; w12 g < 0. It implies that the intersection in case C1) is equal to (SI (2) [ ([i9=5 SI (i) )) SII . Computing the intersection SI (2) SII yields
w11 < 1; w12 < 0; w21 = 0; w22 = 1 h1 = h2 = 0 minfh1 ; w12 g < 0: It is equivalent to
Proof: We consider the intersections of sets in [11, Th. 4] and sets
SI (1) ; . . . ; SI (24) ; SII ; SIV . SI (j ) in Table I and SI (i) in [11, Table 1] imply that SI (i) \ SI (j ) = ; for all i = 1; . . . ; 6, j = 1; . . . ; 24, and Si(j) \Si = ; for all i = II , IV , j = 1; 2. Furthermore, Theorem 3 implies that SIII \ SII = ;, SIII \ SIV = ;, and SII (i) \ SI (j ) = ; for all i = 1; 2, j = 1; . . . ; 24. It follows that
SII (1) [ SII (2)
Thus, we only need to discuss the remaining intersections: SI (1) \ SII (1) \ SIV , SI (1) \ SIV (1) \ SII , (SI (3) [ SI (5) ) \ SII (2) \ SIV , and (SI (3) [ SI (5) ) \ SIV (2) \ SII . It is easy to derive the intersection SI (3) \ SIV (2) \ SII as
w11 < 1; w12 < 0; w21 = 0; w22 = 1 h1 = h2 = 0: Similarly, the intersections of other pairs SI (i) SII , i = 5; . . . ; 9 can be computed. The cases C2) and C3) can be analyzed in the same procedure as in case C1). The expressions of intersections are also given in the Appendix. Using the same idea, we can obtain all conditions under which at least one nondegenerate equilibrium and either one singular line or two singular lines coexist. We only state the results and omit the similar analysis. Theorem 3: The 2-D network (5) has one nondegenerate equilibrium and one singular line if lies in one of the sets: (([6i=1 SI (i) ) [ 2 6 2 ([i=1 SIV (i) )) \ SII , (([i=1 SI (i) ) [ ([i=1 SII (i) )) \ SIV , and 24 SIII \ ([i=18 SI (i) ) (formulations can be found in the Appendix). Theorem 4: The 2-D network (5) has one nondegenerate equilibrium and two singular lines if lies in the set SI (1) \ SII \ SIV , which is formulated by f 2 6 jw11 = w22 = 1; h1 = h2 = 0; w12 < 0;w21 < 0g. More complicated equalities and inequalities are involved in finding conditions of coexistence of two nondegenerate equilibria and a singular line, although the same idea is used. Theorem 5: The 2-D network (5) has two nondegenerate equilibria and one singular line if lies in one of the sets: SI (1) \ SII (1) \ SIV , SI (1) \ SIV (1) \ SII , (SI (3) [ SI (5) ) \ SII (2) \ SIV , and (SI (3) [ SI (5) ) \ SIV (2) \ SII (formulations can be found in the Appendix).
R
[i6=2 SI (i) \ SIII \ (SII [ SIV ) = ; SII (2) \ SIII \ (SII [ SIV ) = ; \ SIV (1) [ SIV (2) \ [j24=1 SI (j) = ; SIII \ SIV (2) \ [i24=1 SI (i) [ SII = ;:
0 14 (w11 0 w22 )2 < w12 w21 < (w11 0 1)(w22 0 1) (w22 0 1)h1 w12 h2 h1 0; h2 < w21 h1 =(w11 0 1)
(10)
w11 > 1; w11 + w22 > 2 w22 = 1; h2 = 0; minfh1 ; w21 g < 0 where the inequalities w11 > 1 and w22 = 1 imply that w11 + w22 > 2.
Moreover
01=4(w11 0 1)2 < w12 w21 < 0:
(11)
Thus, (10) can be simplified further. Note that h1 6= 0; otherwise, h2 < by the second inequality on the third line of (10), which contradicts h2 = 0. Since it has been proved that h1 < 0, we naturally have minfh1 ; w21 g < 0. Furthermore, since 0 = h2 < w21 h1 =(w11 0 1), w11 > 1, and h1 < 0, we assure that w21 < 0. It follows from (11) that w12 > 0. It is also clear that the inequality on the second line of (10) is a vain restriction because (w22 0 1)h1 = 0 and w12 h2 = 0. Therefore, (10) is finally simplified as 0
w11 > 1; w12 > 0; w21 < 0; w22 = 1 h1 0; h2 = 0 0 14 (w11 0 1)2 < w12 w21 as given for SI (3) \ SIV (2)
\ SII
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
in the Appendix.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
Computation of the intersections SI (5) \ SIV (2) \ SII and those intersections SI (1) \ SII (1) \ SIV , SI (1) \ SIV (1) \ SII , and (SI (3) [ SI (5) ) \ SII (2) \ SIV can be proceeded similarly (simplified expressions are shown in the Appendix). Corollary 1: None of the following equilibria coexist: 1) three singular lines; 2) a degenerate equilibria and two singular lines; 3) two equilibria and two singular lines; and 4) three equilibria and a singular line. Proof: By Theorem 5, ([9i=1 SI (i) ) [ ([i12 =11 SI (i) ) [ 24 4 24 ([i=18 SI (i) ) \ SIV = ; and SI 1 [ ([i=3 SI (i) ) [ ([i=10 SI (i) ) \ 24 SII = ;. It follows that ([i=1 SI (i) ) \ SII \ SIV = ;, which means that either any three singular lines or a degenerate equilibria and two singular lines cannot coexist. By Theorem 5, SI (1) \ SII (1) \ ([i24 =1 SI (i) [ SII ) = ;, SI (1) \ SIV (1) \ ([i24=1 SI (i) [ SIV ) = ;, (SI (3) [ SI (5) ) \ SII (2) \ 24 24 ([i=1 SI (i) [ SII ) = ;, and (SI (3) [ SI (5) ) \ SIV (2) \ ([i=1 SI (i) [ SIV ) = ;, implying that neither any two equilibria and any two singular lines nor three equilibria and a singular line can coexist. V. SIMULATIONS AND DISCUSSIONS In this section, simulations are presented to illustrate the theoretical results. Example 1: Consider a 2-D neural network (2) with
W
=
1
0
01 01
h=
0 0
:
One can check that the parameters lie in the intersection SI (10) SIV . By Theorem 2, the network has a stable degenerate equilibrium (0; 0) in D1 and a stable singular line L4 : 0x1 0 x2 = 0; (x1 ; x2 ) 2 D4 . Its phase portrait is illustrated in Fig. 1, where some orbits tend to the origin and others are attracted by the line L4 . The attractor composed by the origin and the line L4 is not a compact set. Example 2: Consider a 2-D neural network (2) with
W
=
04
3
0
1
h=
01 : 0
It is verified that the parameters belong to the intersection SI (7) SII . By Theorem 2, the network has two degenerate equilibria: two stable lines L11 : 05x1 + 3x2 0 1 = 0; (x1 ; x2 ) 2 D1 and L2 : 0x1 + 3x2 0 1 = 0; (x1 ; x2 ) 2 D2 (Fig. 2). It is seen that some orbits tend to the line L11 and others are attracted by the line L2 . It is noted that the attractor composed by the line L11 and the line L2 is not a compact set. As shown in [1], in the cases of degenerate equilibria, some neurons can get latched at a nonzero activity for zero input h = 0 and there are an infinite number of equilibria. An example of stable singular line (i.e., an infinite number of stable equilibria) was given in [10]. In visual perception, visual images are stored as a number of continuous fixed points (i.e., a continuous attractor) and the brain is able to hold the eyes still because its memory network stores the eye position as a line attractor [5], [12]. On the other hand, as noted in [15], more than 25 years of experimental and theoretical work indicate that the onset of oscillations in neurons and in neuron populations is characterized by multistability. VI. CONCLUSION In this paper, degenerate equilibria and their dynamic properties have been discussed for the recurrent neural networks with LT transfer func-
179
Fig. 2. Two stable singular lines coexist.
tions. In order to reveal the geometrical structures of the attractors, the dynamical behaviors and the conditions of coexistence of more than one (degenerate and nondegenerate) equilibria have been examined. Based on those results, all conditions of existence of continuous attractors that contain more than one equilibria have been obtained. Those conditions enrich and enhance the analytical results for the attractors’ geometrical structures in previous work [11]. They provide an insightful understanding about the continuous attractors in the system that composes of the recently explored memory network model and give a useful basis for precise tuning of the parameters. APPENDIX A. Sets of Parameters in Theorem 2 S
S
S
S
S
S
S
\S
:
w22 = 1 w21 = h1 = h2 = 0 w11 < 1; w12 < 0 \S
:
w11 = 1; w22 = 1 w12 = h1 = 0 w21 =h2 < 0
S
S
\S
:
S
\S
:
S
w22 = 1; w11 < 1 w21 = h2 = 0 h1 w12 < 0 w22 = 1; w11 = 1 w21 = h2 = 0 w12 =h1 < 0 \S
:
w12 < 0; w21 < 0 w11 = w22 = 1 h1 = h2 = 0 \S
:
\S
:
w11 = 1 w12 = h1 = h2 = 0 w22 = 1; w21 < 0 w11 = 1 w12 = h1 = h2 = 0 w22 > 1; w21 < 0:
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
S
S
\S
:
\S
:
w22 = 1 w21 = h1 = h2 = 0 w11 = 1; w12 < 0 w22 = 1 w21 = h1 = h2 = 0 w11 > 1; w12 < 0 \S
:
w11 = 1; w22 > 1 w12 = h1 = 0; h2 6= 0 minfh2 ; w21 g < 0 \S
:
w22 = 1; w11 > 1 w21 = h2 = 0; h1 6= 0 minfh1 ; w12 g < 0 \S
:
\S
:
w11 = 1 w12 = h1 = h2 = 0 w22 < 1; w21 < 0 w11 = 1; w22 < 1 w12 = h1 = 0 h2 w21 < 0
180
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 1, JANUARY 2009
B. Sets of Parameters in Theorem 3
C. Sets of Parameters in Theorem 5 \S : S S \S
S
\S
:
w = 1; h = 0 w < 0(> 0); w < 0(> 0) h 0(< 0) \S 0 (w 0 1) w w w = 1; w < 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S 0 (w 0 1) w w w = 1; w > 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S w w < 0 (w 0 1) w < 1; w = 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S w w < 0 (w 0 1) w > 1; w = 1 w < 0(> 0); w > 0(< 0) h 0(< 0); h = 0 \S w =w =1 h 0(< 0); h = 0 w < 0(> 0); w > 0(< 0) \S w = 1; h = 0 w < 1; h > 0 w < 0; w < 0 22 12
S
1
:
2
11
4
22
12
11
12
21
1
2
:
1
2
11
4
22
12
21
1
2
:
12
1
21
22
12
21
1
2
:
12
1
21
22
12
21
1
2
:
22
11
1
S
S
2
12
21
:
22
2
11
1
21
\S
12
:
w = 1; h = 0 w < 1; h > 0 w < 0; w < 0 11
S
2
=
1
4
\S w 0
1
:
= ww 0 = hh w +w >2 h < 0; h < 0 min ww 0 ; w h 0 < 0 1
w
11
1
22
1
2
\S
1
12
21
22
12
21
1
2
:
1
2
11
4
11
12
21
22
12
21
1
2
:
12
1
21
11
4
11
2
22
12
21
1
2
:
12
1
21
11
4
11
2
22
12
21
1
2
:
22
11
12
21
1
2
:
22
2
11
1
21
:
S
12
2
21
1
2
\S w 01 w 1
S
1
:
= ww 0 = hh w +w =2 h < 0; h < 0 min ww 0 ; w h 0 < 0 11
2
1
2
22
11
22
1
S
=
h h
w +w 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S 0 (w 0 1) w w w = 1; w > 1 w > 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S w w < 0 (w 0 1) w = 1; w < 1 w > 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S w w < 0 (w 0 1) w = 1; w > 1 w > 0(< 0); w < 0(> 0) h = 0; h 0(< 0) S \S w =w =1 w < 0(> 0); w > 0(< 0) h = 0; h 0(< 0) S \S w = 1; h = 0 w > 1; h < 0 0)
11
2
21
:
22
1
22
S
2
11
4
11
S
2
11
4
11
S
21
11
12
S
21
\S
11
21
1
S
S
2
1
22 2
\S
1
:
w = 1; w = 0 w >1 h < 0; h < 0 11
21
22
1
h h
2
= ww 0
1
w = 1; h = 0 w < 1; h > 0 w < 0; w < 0 \S S \S w w < 0 (w 0 1) h < 0; w < 0 w > 1; w > 0 w = 1; h = 0 \S S \S w w < 0 (w 0 1) w = 1; h = 0 w > 1; h < 0 w > 0; w < 0: 11
1
22
2
21
12
:
12
1
21
1
4
2
11
21
11
12
22
2
:
12
1
21
4
11
1
22
2
21
11
1 2
:
21
1
11
22
2
:
1 4
11
2
21
2
22
12
21
1
22
12
12
ACKNOWLEDGMENT The authors would like to thank the anonymous reviewers for their valuable comments and suggestions that helped to improve this brief significantly.
REFERENCES [1] R. L. T. Hahnloser, “On the piecewise analysis of linear threshold neural networks,” Neural Netw., vol. 11, pp. 691–697, 1998. [2] R. Douglas, C. Koch, M. Mahowald, and H. Suarez, “Recurrent excitation in neocortical circuits,” Science, vol. 269, pp. 981–985, 1995. [3] H. Wersing, W.-J. Beyn, and H. Ritter, “Dynamical stability conditions for recurrent neural networks with unsaturating piecewise linear transfer functions,” Neural Comput., vol. 13, pp. 1811–1825, 2001. [4] R. Hahnloser, R. Sarpeshkar, M. A. Mahowald, R. J. Douglas, and H. S. Seung, “Digital selection and analogue amplification coexist in a cortex-inspired silicon circuits,” Nature, vol. 405, pp. 947–951, 2000. [5] H. S. Seung, “Continuous attractors and oculomotor control,” Science, vol. 290, pp. 2268–2269, 2000. [6] Z. Yi, K. K. Tan, and T. H. Lee, “Multistability analysis for recurrent neural networks with unsaturating piecewise linear transfer functions,” Neural Comput., vol. 15, no. 3, pp. 639–662, 2003. [7] L. Zhang, Z. Yi, and J. Yu, “Multiperiodicity and attractivity of delayed recurrent neural networks with unsaturating piecewise linear transfer functions,” IEEE Trans. Neural Netw., vol. 19, no. 1, pp. 158–167, Jan. 2008. [8] A. Meyer-Base and V. Thummler, “Local and global stability analysis of an unsupervised competitive neural network,” IEEE Trans. Neural Netw., vol. 19, no. 2, pp. 346–351, Feb. 2008. [9] A. A. Frolov, D. Husek, I. P. Muraviev, and P. Y. Polyakov, “Boolean factor analysis by attractor neural network,” IEEE Trans. Neural Netw., vol. 18, no. 3, pp. 698–707, May 2007. [10] Z.-H. Mao and S. G. Massaquoi, “Dynamics of winner-take-all competition in recurrent neural networks with lateral inhibition,” IEEE Trans. Neural Netw., vol. 18, no. 1, pp. 55–69, Jan. 2007. [11] K. C. Tan, H. Tang, and W. Zhang, “Qualitative analysis for recurrent neural networks with linear threshold transfer functions,” IEEE Trans. Circuit Syst. I, Reg. Papers, vol. 52, no. 5, pp. 1003–1012, May 2005. [12] H. S. Seung, “How the brain keeps the eyes still,” Proc. Nat. Acad. Sci. USA, vol. 93, pp. 13339–13344, 1996. [13] H. Tang, K. C. Tan, and W. Zhang, “Cyclic dynamics analysis for networks of linear threshold neurons,” Neural Comput., vol. 17, no. 1, pp. 97–114, 2005. [14] Z.-F. Zhang, T.-R. Ding, W.-Z. Huang, and Z.-X. Dong, Qualitative Theory of Differential Equations. Providence, RI: Transl. Math. Monographs, Amer. Math. Soc., 1992. [15] J. Milton, “EPILESPY: Multistability in a dynamic disease,” in SelfOrganized Biological Dynamics Nonlinear Control: Toward Understanding Complexity, Chaos, and Emergent Function in Living Systems, J. Walleczek, Ed. Cambridge, U.K.: Cambridge Univ. Press, 2000, pp. 374–386.
1
w = 1; h = 0 w > 1; h < 0 w 0; w < 1 w = 1; h = 0 \S S \S 0 (w 0 1) w w w = 1; h = 0 h < 0; w > 1 w > 0; w < 0
12
Authorized licensed use limited to: ASTAR. Downloaded on January 21, 2009 at 02:19 from IEEE Xplore. Restrictions apply.
