Reproducing Chaos by Variable Structure ... - Semantic Scholar
1450
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
Reproducing Chaos by Variable Structure Recurrent Neural Networks Ramon A. Felix, Edgar N. Sanchez, Senior Member, IEEE, and Guanrong Chen, Fellow, IEEE
Abstract—In this paper, we present a new approach for chaos reproduction using variable structure recurrent neural networks (VSRNN). A neural network identifier is designed, with a variable structure that will change according to its output performance as compared to the given orbits of an unknown chaotic systems. A tradeoff between identification errors and computational complexity is discussed. Index Terms—Chaos generation, identification, recurrent neural networks, variable structure system.
(VSRNN) [11] is proposed, for general nonlinear systems identification. In this approach, an initial configuration for the neural identifier is first assumed. If a prespecified error bound is not reached, more synaptic connections will be added, and another cycle of simulation will begin, until the output performance of the network satisfies the predesired criterion. Complex chaotic systems are used to illustrate the capability and applicability of the proposed identification scheme [12]. II. RECURRENT HIGH-ORDER NEURAL NETWORKS
I. INTRODUCTION
R
ECENTLY, controlling and ordering chaos has received increasing attention from various scientific and engineering communities [3]. To control a chaotic system, similarly to controlling a general nonlinear system, an important requirement usually is to have a good model of the underlying system. For many nonlinear systems, particularly chaotic systems in applications, it is often difficult to obtain accurate and faithful mathematical models, regarding their physically complex structures and hidden parameters as discussed in [3]. Therefore, system identification becomes important and even necessary before system control can be considered. A challenger problem for nonlinear systems identification is to select a suitable structure for the identifier, capable of approximating the unknown nonlinear dynamics. In this consideration, it is notable that recurrent neural networks offer the advantage of well approximating a nonlinear system to an arbitrarily accurate level [2], provided that the neural identifier has a sufficiently large number of synaptic connections [7]. However, it is generally quite difficult to determine the number of sufficient synaptic connections to approximate a given dynamical system. If the neural identifier does not have enough synaptic connections, it is impossible to assure that the parameters converge to their optimal values, even using persistently excited inputs, and in many cases the identification error does not converge to zero. On the other hand, if there are too many synaptic connections, computational burden will be huge and the suggested solution becomes impractical. In this paper, to alleviate the aforementioned troublesome situation, a specially designed variable structure neural network Manuscript received February 7, 2002; revised September 28, 2003. This work was supported by the CONACYT, Mexico, under Grants 32059A and 39866Y. R. A. Felix and E. N. Sanchez are with the CINVESTAV, Unidad Guadalajara, Mexico. G. Chen is with the Department of Electronic Engineering, City University of Hong Kong. Digital Object Identifier 10.1109/TNN.2004.836236
The recurrent neural network structure, used to define the VSRNN, is presented in [7]. For completeness, a brief review of this type of neural networks is first given. The recurrent highorder recurrent neural networks (RHONN) are defined as (1) where is the th neuron state, is the number of high-order connections, is a collection of nonordered sub, , are adjustable weights sets of are nonnegative integers, and is a vector of the network, defined by
with and
.. .
.. .
.. .
.. .
being the input to the networks, a smooth sigmoid function of the form
where is a positive constant. Based on experimental eviis selected as follows; first, define dences, the parameter as the maximum value attained by the respective system state . to be identified, then select As can be seen, (1) allows the inclusion of high-order terms. Define
1045-9227/04$20.00 © 2004 IEEE
.. .
.. .
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
Then, (1) can be rewritten as
or (2) where is the vector of adaptive weights. It is clear that the synaptic weights depend on time. Each is called a high-order connection, and each , a high-order term.
1451
It is noted that for the VSRNN, the time function , defined for the th neuron, is increasing, with . The function is determined by an external agent, called supervisor. This agent evaluates the VSRNN performance and, depending on this evaluation, it calculates the values of online. According to the definitions for the family of functions , and for the switching function , the indexes sequence , is defined offline. Hence, only the switching time sequence is determined by the supervisor online. As stated above, the weights are time functions, hence, every , for the initial structure, is a solution of the differential equation (5)
III. VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
, the weights are given by
For
A. Switched Systems Let the family of controlled vector fields be , where is an index set, , for all , and for each , is locally Lispchitz [8]. Given the family , consider a switched system [1], [13]
(6) is a bounded time function defined in Section IV-A, where and is the unit step function. The above equations imply that all the weights which are not included are zeros, until the respective high-order connections are added.
(3) , ,and is the switching signal, where defined as a continuous function ; associated with the signal , there is a sequence of real numbers , called switching time sequence, and a sequence of indexes, , such that for all .
IV. NONLINEAR SYSTEMS IDENTIFICATION Consider the problem of identifying a nonlinear system of the form (7) , , is a smooth vector field. In order to where identify system (7), the VSRNN described in Section III will be used. As discussed in [7], it is assumed that system (7) is fully represented by an RHONN, with each neuron state given by
B. Structure of the VSRNNs Define the dynamics of each neuron of a VSRNN by [11] (4) For the th neuron, consider a family of functions: , where , is the switching signal, and is the maximal number, which is finite, for the high-order connections of the th neuron. Here, it is suggested to select such that
(8) where is denoted as
. The optimal (unknown) parameters vector
Assumption 1: The optimal weight vector lowing expression: where , and is the number of initial highorder connections. Hence, the maximal number of high-order connections is . For simplicity, is rewritten as
where
,
,
and , with . Let and be the initial or minimal structure and the maximal structure, respectively, for the th neuron state. Let the high-order connections, which have not been connected at , be denoted by .
has the fol-
where the entries of can be of any finite value, and the ones of are zeros. This assumption implies that there could exist a neural structure, simpler than the maximal one, which can arbitrarily well approximate system (7). It should be noted that the dimensions of and here are unknown. In order to identify system (7), it is assumed that it can be represented by the proposed RHONN, hence, two possible models can be built: • Parallel model • Cascade-Parallel model where is the th component of the RHONN, and is the state of system (7).
1452
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
for all and for all , where is a positive semidefinite function. Then, all trajectories of (10) are bounded for and , where is a positive and bounded constant. Moreover
Using Lemma 2, one can propose a positive definite function , bounded from below, and a parameter adaptive law, such that exists, and is uniformly continuous in , with . Consider the Lyapunov function candidate (11) where is the learning rate. Differentiating (11) along the trajectories of (9) gives
Fig. 1. Block diagram of the proposed scheme.
To develop an online weights update law, the Cascade-Parallel model is used. The idea is to propose an initial structure for the RHONN and then adapt the neural parameters in such a way that if an error criterion is not satisfied, then an extra high-order connection will be added. It continues to add different high-order connections in this way, until the error criterion is satisfied or the maximal neural structure is reached. Fig. 1 shows a block diagram of this scheme, where the supervisor evaluates the VSRNN’s performance: if the error criterion is not satisfied, a high-order connections will be added. For simplicity of notation, the subindex in the switching function will be dropped below. As in the definition of a VSRNN, one can define , where are the connections being added to the VSRNN, while are the connections which are not added for all . Define the th identification error as
and the th parameter error at
as
Then, from (4) and (8), one can obtain the error equation as follows: (9)
A. Online Identification Lemma 2: [6] Consider the system
(12) If one defines the weight adaptive law by [9]
(13) then (5) and (6) are satisfied, and (12) becomes
Considering Assumption 1, it is possible to find some intermediate structure, simpler than the maximal structure, such that . Hence, there exists some finite time, , such that
for all . This means that, after the –th connection has been added to the VSRNN, one will not consider to add more high-order terms to the network. Using Lemma 2 and Assumption 1, it is easy to see that if one further reduces the modeling error term by adding sufficient connections at a finite time , then with the adaptive law (13) one can guarantee that the weights are bounded and the identification error converge to zero after , where , , and . B. Robust Online Identification When the final parameterization has not been reached, the adaptive law (13) does not guarantee either the boundedness of the weights or the convergence of the identification error to zero. Therefore, the learning law (13) has to be modified in order to avoid the parameters drift problem. For this, the well-known -modification scheme [5] is applied to (13): (14)
(10) where and is locally Lipschitz in and uniformly continuous in . Suppose that there exists a function , radially unbounded and continuously differentiable, such that
where
is given as if if if
with integer
, and
and
are positive constants.
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
1453
Lemma 3: Consider system (8) and VSRNN (4), whose pa. rameters are adapted using law (14). Then, Proof: The differential of along the trajectories of (9) and (14) is given by
Applying the inequality
and defining
, one obtains
Since
, one has Fig. 2.
Therefore
Define
. Then
Substituting
from (11) into the above inequality yields
Considering the worst case when lect , so that
, one can se-
and x trajectories of Chen’s system.
, then the supervisor adds the next high-order connection; otherwise, the neuron keeps its current structure, with being a prespecified positive bound for . Remark 2: The supervisor parameters are selected depending on the accuracy expected from the neural identifier; a smaller means a better accuracy but a more complex neural structure. The evaluation period can be selected arbitrarily. If is too short, the connections may be added too fast and a more complex neural structure will be created; on the other hand, if is too long, the selection of an adequate neural structure may take a longer time. About the selection of , we propose to check the nominal parameters of the real system and select to be close to these values. Nevertheless, when no model is available for the system, this checking cannot be done; for these cases, the selection of the parameter is still an open problem. V. CHAOS REPRODUCTION A. Chaotic Chen’s System
Therefore, , and the proof is completed. Remark 1: Notice that the update law (13) has a similar structure as the backpropagation learning law for a single layer perceptron [4]. If we consider as an updating rate, then the righthand term of (13) corresponds exactly to the backpropagation scheme. On the other hand, (14) can be viewed as a backpropagation learning law with a conditioned varying momentum term. C. VSRNN Supervisor The aforementioned supervisor criterion for evaluating the VSRNN performance is described as follows. The time is divided over the evaluation intervals, each one with a length , called the evaluation period. Define the evaluation function as (15) Obviously,
, and it is monotonically increasing for . If, at the end of each evaluation interval,
In order to test the capability and applicability of the proposed scheme, the complex chaotic Chen’s system is selected as an example. This chaotic system is described by the following differential equations [10]:
which is chaotic when , , . The RHONN fixed parameters used in the simulation are , , ,
1454
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
Fig. 3. Identification error e
= 0x
of Chen’s system. Fig. 5.
Fig. 4.
Phase portrait of the chaotic Chen’s system.
Phase portrait of the VSRNN for the chaotic Chen’s system.
Fig. 6. Supervisor evaluation functions E , E , and E for Chen’s system.
The supervisor parameters used are: , , , , and the index sequences . are As can be seen from Figs. 2–5, the proposed scheme is able to reproduce the Chen’s chaotic attractor. Fig. 6 shows the evaluation functions , , and . The initial and added weights are plotted in Fig. 7. Finally, in Fig. 8, the switching signals , and are shown. B. Chaotic Chua’s Circuit The complex chaotic Chua’s circuit is simulated here. This chaotic system is topologically not equivalent to Chen’s system, and is given by the following differential equations [10]:
Fig. 7.
Weights of x for Chen’s system.
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
Fig. 8. Switching signals s , s , and s for Chen’s system.
Fig. 9.
1455
= 0x
Fig. 10.
Identification error e
Fig. 11.
Phase portrait of the chaotic Chua’s system.
Fig. 12.
Phase portrait of the VSRNN for the chaotic Chua’s system.
of Chua’s system.
and x trajectories of Chua’s system.
which is chaotic when
,
and , with
and . The RHONN fixed parameters used in the simulation are: , , ,
For simplicity, the initial structure is chosen as a first-order one. It should be mentioned that the selection of was made heuristically for this system as well as for the Chen’s system. , The supervisor parameters used are: , , , and the index sequences . are As well as for the first simulation reported above, Figs. 9–12 show that the proposed scheme is able to reproduce the chaotic
1456
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
Chua’s attractor. The evaluation functions , and are weights and switching signals are shown in Fig. 13, and the plotted in Figs. 14 and 15, respectively. The computations in our examples include at most eleven additions and thirty multiplications. VI. CONCLUSION
Fig. 13.
Supervisor evaluation functions E , E , and E for Chua’s system.
A VSRNN has been proposed for nonlinear system identification. The results are very encouraging, particularly when the scheme is applied to the relatively difficult task of chaos reproduction. This scheme trades off between the identifier performance and the computational complexity. As mentioned in Section III-B, an indexes sequence, , is defined before the identification process starts. In order to further improve the VSRNN identification performance, the sequence could be given by the supervisor. In doing so, a difficulty is to design an appropriate criterion for the addition of high-order connections to the network. This is left as a research topic for future study. The VSRNN identification scheme can be considered as a first attempt for structural neural identification of nonlinear dynamical systems. This scheme imposes several constrains: the structures are ordered according to their increasing complexity, the index sequence is fixed offline, the error criterion is based on the identification error, and high-order connections can only be added but not be removed. Hence, in future research, we propose to relax such constrains, allowing more flexibility such as giving index sequence online, not using ordered structures, letting the error criterion be based on modeling errors, and checking for redundant high-order connections in order to disconnect them. ACKNOWLEDGMENT The authors thank the anonymous reviewers for their helpful comments. REFERENCES
Fig. 14.
Weights of x for Chua’s system.
Fig. 15. Switching signals s , s , and s for Chua’s system.
[1] M. S. Branicky, “Multiple Lyapunov functions and other analysis tool for switched and hybrid systems,” IEEE Trans. Automat. Contr., vol. 43, pp. 475–481, Apr. 1998. [2] N. E. Cotter, “The Stone–Weiertrass theorem and its application to neural networks,” IEEE Trans. Neural Networks, vol. 1, pp. 290–295, July 1990. [3] G. Chen and X. Dong, From Chaos to Order: Methodologies, Perspectives, and Applications, Singapore: World Scientific, 1998. [4] K. Hunt, G. Irwin, and K. Warwick, Eds., Neural Networks Engineering in Dynamic Control Systems. New York: Springer-Verlag, 1995. [5] P. A. Ioannou and J. Sun, Robust Adaptive Control. Englewood Cliffs, NJ: Prentice-Hall, 1996. [6] H. Khalil, Nonlinear System. Englewood Cliffs, NJ: Prentice-Hall, 1996. [7] E. B. Kosmatopoulos et al., “Dynamical neural networks that ensure exponential identification error convergence,” Neural Netw., vol. 10, no. 2, pp. 299–314, 1997. [8] J. L. Mancilla and R. A. Garcia, “On the existence of common Lyapunov triples for ISS and iISS switched systems,” in Proc. IEEE CDC 2000, Sydney, Australia, Dec. 2000. [9] G. A. Rovithakis and M. A. Christodolou, “Adaptive control of unknown plants using dynamical neural networks,” IEEE Trans. Syst., Man, Cybern., vol. 24, pp. 400–412, Mar. 1994. [10] E. N. Sanchez, J. P. Perez, and G. Chen, “Using dynamical networks to generate chaos: An inverse optimal control approach,” Int. J. Bifurcation Chaos, vol. 11, no. 3, pp. 857–863, 2001. [11] E. N. Sanchez and R. A. Felix, “Nonlinear identification via variable structure recurrent neural networks,” in IFAC World Congr., Barcelona, Spain, July 2002.
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
[12]
, “Chaos identification using variable structure recurrent neural networks,” in 9th Int. Conf. Neural Information Processing (ICONIP’02), Singapore, Nov. 2002. [13] H. Ye, A. N. Michel, and L. Hou, “Stability theory for hybrid dynamical systems,” IEEE Trans. Automat. Contr., vol. 43, pp. 461–474, Apr. 1998.
Ramon A. Felix was born in Guamuchil, Sinaloa, Mexico, in 1976. He received the B.Sc. degree from Instituto Tecnológico del Mar (ITMAR), Mazatlan Campus, Sinaloa, Mexico in 1998, and the M.Sc. and Ph.D. degree in electrical engineering from the Centro de Investigación y Estudios Avanzados del Instituto Politécnico Nacional (CINVESTAV), Guadalajara Campus, Jalisco, Mexico. In 2002, he visited the Laboratoire des signaux et systèmes (LSS), Ecole Supérieur d’Electricité (SUPELEC), Paris, France, by invitation of Romeo Ortega. He has published more than ten technical papers in international conferences and journals. He is currently associated as a Postdoctoral Scientist at CINVESTAV Guadalajara Campus. His reseach interest focuses on neural control and variable structure control and theirs real-time applications to electrical machines and power electronic systems.
Edgar N. Sanchez (M’85–SM’95) was born in Sardinata, Colombia, South America, in 1949. He received the B.S.E.E. degree in power systems from Universidad Industrial de Santander (UIS), Bucaramanga, Colombia, in 1971, the M.S.E.E. degree in automatic control from Advanced Studies and Research Center of the National Polytechnic Institute (CINVESTAV-IPN), Mexico City, Mexico, in 1974, and the Dr. Ing. degree in automatic control from Institut Nationale Polytechnique de Grenoble, Grenoble, France, in 1980. In 1971, 1972, 1975, and 1976, he worked for different electrical engineering consulting companies in Bogota, Colombia. In 1974, he was a Professor in the Electrical Engineering Department, UIS, Colombia. From 1981 to 1990, he worked as a Researcher at the Electrical Research Institute, Cuernavaca, Mexico. He was a Professor of the graduate program in Electrical Engineering of the Universidad Autonoma de Nuevo Leon (UANL), Monterrey, Mexico, from 1990 to 1996. Since 1997, he has been with CINVESTAV-IPN, Guadalajara Campus, Mexico, where he is a Professor of Electrical Engineering graduate programs. His research interest center in neural networks and fuzzy logic as applied to automatic control systems. He has published more than 100 technical papers in international journal and conferences, and has served as reviewer for different international journals and conferences. Dr. Sanchez was granted an USA National Research Council Award as a Research Associate at NASA Langley Research Center, Hampton, Virginia. He is also member of the Mexican National Research System, the Mexican Academy of Science, and the Mexican Academy of Engineering.
1457
Guanrong Chen (M’87–SM’92–F’96) received the M.Sc. degree in computer science from the Sun Yat-sen (Zhongshan) University, China and the Ph.D. degree in applied mathematics from Texas A&M University, College Station. Currently, he is a Chair Professor and the Founding Director of the Center for Chaos Control and Synchronization at the City University of Hong Kong. He is Honorary Professor of the Central Queensland University, Australia, as well as Honorary Guest-Chair Professor of more than ten Universities in China. He has authored or coauthored 15 research monographs and advanced textbooks, more than 300 SCI journal papers, and about 180 refereed conference papers, published since 1981 in the fields of nonlinear system dynamics and controls. Among his publications are the research monographs and edited books entitled Hopf Bifurcation Analysis: A Frequency Domain Approach (Singapore: World Scientific, 1996), From Chaos to Order: Methodologies, Perspectives and Applications (Singapore: World Scientific, 1998), Controlling Chaos and Bifurcations in Engineering Systems (Boca Raton, FL: CRC Press, 1999), Chaos in Circuits and Systems (Singapore: World Scientific, 2002), and Chaos Control and Bifurcation Control (New York: Springer-Verlag, 2003). Prof. Chen has served as Chief Editor, Deputy Chief Editor, Advisory Editor, Features Editor, and Associate Editors for 8 international journals including the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS, IEEE TRANSACTIONS ON AUTOMATIC CONTROL, and the International Journal of Bifurcation and Chaos. He received the 1998 Harden-Simons Prize for the Outstanding Journal Paper Award from the American Society of Engineering Education and the 2001 M. Barry Carlton Best Annual Transactions Paper Award from the IEEE Aerospace and Electronic Systems Society.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
Reproducing Chaos by Variable Structure Recurrent Neural Networks Ramon A. Felix, Edgar N. Sanchez, Senior Member, IEEE, and Guanrong Chen, Fellow, IEEE
Abstract—In this paper, we present a new approach for chaos reproduction using variable structure recurrent neural networks (VSRNN). A neural network identifier is designed, with a variable structure that will change according to its output performance as compared to the given orbits of an unknown chaotic systems. A tradeoff between identification errors and computational complexity is discussed. Index Terms—Chaos generation, identification, recurrent neural networks, variable structure system.
(VSRNN) [11] is proposed, for general nonlinear systems identification. In this approach, an initial configuration for the neural identifier is first assumed. If a prespecified error bound is not reached, more synaptic connections will be added, and another cycle of simulation will begin, until the output performance of the network satisfies the predesired criterion. Complex chaotic systems are used to illustrate the capability and applicability of the proposed identification scheme [12]. II. RECURRENT HIGH-ORDER NEURAL NETWORKS
I. INTRODUCTION
R
ECENTLY, controlling and ordering chaos has received increasing attention from various scientific and engineering communities [3]. To control a chaotic system, similarly to controlling a general nonlinear system, an important requirement usually is to have a good model of the underlying system. For many nonlinear systems, particularly chaotic systems in applications, it is often difficult to obtain accurate and faithful mathematical models, regarding their physically complex structures and hidden parameters as discussed in [3]. Therefore, system identification becomes important and even necessary before system control can be considered. A challenger problem for nonlinear systems identification is to select a suitable structure for the identifier, capable of approximating the unknown nonlinear dynamics. In this consideration, it is notable that recurrent neural networks offer the advantage of well approximating a nonlinear system to an arbitrarily accurate level [2], provided that the neural identifier has a sufficiently large number of synaptic connections [7]. However, it is generally quite difficult to determine the number of sufficient synaptic connections to approximate a given dynamical system. If the neural identifier does not have enough synaptic connections, it is impossible to assure that the parameters converge to their optimal values, even using persistently excited inputs, and in many cases the identification error does not converge to zero. On the other hand, if there are too many synaptic connections, computational burden will be huge and the suggested solution becomes impractical. In this paper, to alleviate the aforementioned troublesome situation, a specially designed variable structure neural network Manuscript received February 7, 2002; revised September 28, 2003. This work was supported by the CONACYT, Mexico, under Grants 32059A and 39866Y. R. A. Felix and E. N. Sanchez are with the CINVESTAV, Unidad Guadalajara, Mexico. G. Chen is with the Department of Electronic Engineering, City University of Hong Kong. Digital Object Identifier 10.1109/TNN.2004.836236
The recurrent neural network structure, used to define the VSRNN, is presented in [7]. For completeness, a brief review of this type of neural networks is first given. The recurrent highorder recurrent neural networks (RHONN) are defined as (1) where is the th neuron state, is the number of high-order connections, is a collection of nonordered sub, , are adjustable weights sets of are nonnegative integers, and is a vector of the network, defined by
with and
.. .
.. .
.. .
.. .
being the input to the networks, a smooth sigmoid function of the form
where is a positive constant. Based on experimental eviis selected as follows; first, define dences, the parameter as the maximum value attained by the respective system state . to be identified, then select As can be seen, (1) allows the inclusion of high-order terms. Define
1045-9227/04$20.00 © 2004 IEEE
.. .
.. .
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
Then, (1) can be rewritten as
or (2) where is the vector of adaptive weights. It is clear that the synaptic weights depend on time. Each is called a high-order connection, and each , a high-order term.
1451
It is noted that for the VSRNN, the time function , defined for the th neuron, is increasing, with . The function is determined by an external agent, called supervisor. This agent evaluates the VSRNN performance and, depending on this evaluation, it calculates the values of online. According to the definitions for the family of functions , and for the switching function , the indexes sequence , is defined offline. Hence, only the switching time sequence is determined by the supervisor online. As stated above, the weights are time functions, hence, every , for the initial structure, is a solution of the differential equation (5)
III. VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
, the weights are given by
For
A. Switched Systems Let the family of controlled vector fields be , where is an index set, , for all , and for each , is locally Lispchitz [8]. Given the family , consider a switched system [1], [13]
(6) is a bounded time function defined in Section IV-A, where and is the unit step function. The above equations imply that all the weights which are not included are zeros, until the respective high-order connections are added.
(3) , ,and is the switching signal, where defined as a continuous function ; associated with the signal , there is a sequence of real numbers , called switching time sequence, and a sequence of indexes, , such that for all .
IV. NONLINEAR SYSTEMS IDENTIFICATION Consider the problem of identifying a nonlinear system of the form (7) , , is a smooth vector field. In order to where identify system (7), the VSRNN described in Section III will be used. As discussed in [7], it is assumed that system (7) is fully represented by an RHONN, with each neuron state given by
B. Structure of the VSRNNs Define the dynamics of each neuron of a VSRNN by [11] (4) For the th neuron, consider a family of functions: , where , is the switching signal, and is the maximal number, which is finite, for the high-order connections of the th neuron. Here, it is suggested to select such that
(8) where is denoted as
. The optimal (unknown) parameters vector
Assumption 1: The optimal weight vector lowing expression: where , and is the number of initial highorder connections. Hence, the maximal number of high-order connections is . For simplicity, is rewritten as
where
,
,
and , with . Let and be the initial or minimal structure and the maximal structure, respectively, for the th neuron state. Let the high-order connections, which have not been connected at , be denoted by .
has the fol-
where the entries of can be of any finite value, and the ones of are zeros. This assumption implies that there could exist a neural structure, simpler than the maximal one, which can arbitrarily well approximate system (7). It should be noted that the dimensions of and here are unknown. In order to identify system (7), it is assumed that it can be represented by the proposed RHONN, hence, two possible models can be built: • Parallel model • Cascade-Parallel model where is the th component of the RHONN, and is the state of system (7).
1452
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
for all and for all , where is a positive semidefinite function. Then, all trajectories of (10) are bounded for and , where is a positive and bounded constant. Moreover
Using Lemma 2, one can propose a positive definite function , bounded from below, and a parameter adaptive law, such that exists, and is uniformly continuous in , with . Consider the Lyapunov function candidate (11) where is the learning rate. Differentiating (11) along the trajectories of (9) gives
Fig. 1. Block diagram of the proposed scheme.
To develop an online weights update law, the Cascade-Parallel model is used. The idea is to propose an initial structure for the RHONN and then adapt the neural parameters in such a way that if an error criterion is not satisfied, then an extra high-order connection will be added. It continues to add different high-order connections in this way, until the error criterion is satisfied or the maximal neural structure is reached. Fig. 1 shows a block diagram of this scheme, where the supervisor evaluates the VSRNN’s performance: if the error criterion is not satisfied, a high-order connections will be added. For simplicity of notation, the subindex in the switching function will be dropped below. As in the definition of a VSRNN, one can define , where are the connections being added to the VSRNN, while are the connections which are not added for all . Define the th identification error as
and the th parameter error at
as
Then, from (4) and (8), one can obtain the error equation as follows: (9)
A. Online Identification Lemma 2: [6] Consider the system
(12) If one defines the weight adaptive law by [9]
(13) then (5) and (6) are satisfied, and (12) becomes
Considering Assumption 1, it is possible to find some intermediate structure, simpler than the maximal structure, such that . Hence, there exists some finite time, , such that
for all . This means that, after the –th connection has been added to the VSRNN, one will not consider to add more high-order terms to the network. Using Lemma 2 and Assumption 1, it is easy to see that if one further reduces the modeling error term by adding sufficient connections at a finite time , then with the adaptive law (13) one can guarantee that the weights are bounded and the identification error converge to zero after , where , , and . B. Robust Online Identification When the final parameterization has not been reached, the adaptive law (13) does not guarantee either the boundedness of the weights or the convergence of the identification error to zero. Therefore, the learning law (13) has to be modified in order to avoid the parameters drift problem. For this, the well-known -modification scheme [5] is applied to (13): (14)
(10) where and is locally Lipschitz in and uniformly continuous in . Suppose that there exists a function , radially unbounded and continuously differentiable, such that
where
is given as if if if
with integer
, and
and
are positive constants.
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
1453
Lemma 3: Consider system (8) and VSRNN (4), whose pa. rameters are adapted using law (14). Then, Proof: The differential of along the trajectories of (9) and (14) is given by
Applying the inequality
and defining
, one obtains
Since
, one has Fig. 2.
Therefore
Define
. Then
Substituting
from (11) into the above inequality yields
Considering the worst case when lect , so that
, one can se-
and x trajectories of Chen’s system.
, then the supervisor adds the next high-order connection; otherwise, the neuron keeps its current structure, with being a prespecified positive bound for . Remark 2: The supervisor parameters are selected depending on the accuracy expected from the neural identifier; a smaller means a better accuracy but a more complex neural structure. The evaluation period can be selected arbitrarily. If is too short, the connections may be added too fast and a more complex neural structure will be created; on the other hand, if is too long, the selection of an adequate neural structure may take a longer time. About the selection of , we propose to check the nominal parameters of the real system and select to be close to these values. Nevertheless, when no model is available for the system, this checking cannot be done; for these cases, the selection of the parameter is still an open problem. V. CHAOS REPRODUCTION A. Chaotic Chen’s System
Therefore, , and the proof is completed. Remark 1: Notice that the update law (13) has a similar structure as the backpropagation learning law for a single layer perceptron [4]. If we consider as an updating rate, then the righthand term of (13) corresponds exactly to the backpropagation scheme. On the other hand, (14) can be viewed as a backpropagation learning law with a conditioned varying momentum term. C. VSRNN Supervisor The aforementioned supervisor criterion for evaluating the VSRNN performance is described as follows. The time is divided over the evaluation intervals, each one with a length , called the evaluation period. Define the evaluation function as (15) Obviously,
, and it is monotonically increasing for . If, at the end of each evaluation interval,
In order to test the capability and applicability of the proposed scheme, the complex chaotic Chen’s system is selected as an example. This chaotic system is described by the following differential equations [10]:
which is chaotic when , , . The RHONN fixed parameters used in the simulation are , , ,
1454
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
Fig. 3. Identification error e
= 0x
of Chen’s system. Fig. 5.
Fig. 4.
Phase portrait of the chaotic Chen’s system.
Phase portrait of the VSRNN for the chaotic Chen’s system.
Fig. 6. Supervisor evaluation functions E , E , and E for Chen’s system.
The supervisor parameters used are: , , , , and the index sequences . are As can be seen from Figs. 2–5, the proposed scheme is able to reproduce the Chen’s chaotic attractor. Fig. 6 shows the evaluation functions , , and . The initial and added weights are plotted in Fig. 7. Finally, in Fig. 8, the switching signals , and are shown. B. Chaotic Chua’s Circuit The complex chaotic Chua’s circuit is simulated here. This chaotic system is topologically not equivalent to Chen’s system, and is given by the following differential equations [10]:
Fig. 7.
Weights of x for Chen’s system.
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
Fig. 8. Switching signals s , s , and s for Chen’s system.
Fig. 9.
1455
= 0x
Fig. 10.
Identification error e
Fig. 11.
Phase portrait of the chaotic Chua’s system.
Fig. 12.
Phase portrait of the VSRNN for the chaotic Chua’s system.
of Chua’s system.
and x trajectories of Chua’s system.
which is chaotic when
,
and , with
and . The RHONN fixed parameters used in the simulation are: , , ,
For simplicity, the initial structure is chosen as a first-order one. It should be mentioned that the selection of was made heuristically for this system as well as for the Chen’s system. , The supervisor parameters used are: , , , and the index sequences . are As well as for the first simulation reported above, Figs. 9–12 show that the proposed scheme is able to reproduce the chaotic
1456
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 6, NOVEMBER 2004
Chua’s attractor. The evaluation functions , and are weights and switching signals are shown in Fig. 13, and the plotted in Figs. 14 and 15, respectively. The computations in our examples include at most eleven additions and thirty multiplications. VI. CONCLUSION
Fig. 13.
Supervisor evaluation functions E , E , and E for Chua’s system.
A VSRNN has been proposed for nonlinear system identification. The results are very encouraging, particularly when the scheme is applied to the relatively difficult task of chaos reproduction. This scheme trades off between the identifier performance and the computational complexity. As mentioned in Section III-B, an indexes sequence, , is defined before the identification process starts. In order to further improve the VSRNN identification performance, the sequence could be given by the supervisor. In doing so, a difficulty is to design an appropriate criterion for the addition of high-order connections to the network. This is left as a research topic for future study. The VSRNN identification scheme can be considered as a first attempt for structural neural identification of nonlinear dynamical systems. This scheme imposes several constrains: the structures are ordered according to their increasing complexity, the index sequence is fixed offline, the error criterion is based on the identification error, and high-order connections can only be added but not be removed. Hence, in future research, we propose to relax such constrains, allowing more flexibility such as giving index sequence online, not using ordered structures, letting the error criterion be based on modeling errors, and checking for redundant high-order connections in order to disconnect them. ACKNOWLEDGMENT The authors thank the anonymous reviewers for their helpful comments. REFERENCES
Fig. 14.
Weights of x for Chua’s system.
Fig. 15. Switching signals s , s , and s for Chua’s system.
[1] M. S. Branicky, “Multiple Lyapunov functions and other analysis tool for switched and hybrid systems,” IEEE Trans. Automat. Contr., vol. 43, pp. 475–481, Apr. 1998. [2] N. E. Cotter, “The Stone–Weiertrass theorem and its application to neural networks,” IEEE Trans. Neural Networks, vol. 1, pp. 290–295, July 1990. [3] G. Chen and X. Dong, From Chaos to Order: Methodologies, Perspectives, and Applications, Singapore: World Scientific, 1998. [4] K. Hunt, G. Irwin, and K. Warwick, Eds., Neural Networks Engineering in Dynamic Control Systems. New York: Springer-Verlag, 1995. [5] P. A. Ioannou and J. Sun, Robust Adaptive Control. Englewood Cliffs, NJ: Prentice-Hall, 1996. [6] H. Khalil, Nonlinear System. Englewood Cliffs, NJ: Prentice-Hall, 1996. [7] E. B. Kosmatopoulos et al., “Dynamical neural networks that ensure exponential identification error convergence,” Neural Netw., vol. 10, no. 2, pp. 299–314, 1997. [8] J. L. Mancilla and R. A. Garcia, “On the existence of common Lyapunov triples for ISS and iISS switched systems,” in Proc. IEEE CDC 2000, Sydney, Australia, Dec. 2000. [9] G. A. Rovithakis and M. A. Christodolou, “Adaptive control of unknown plants using dynamical neural networks,” IEEE Trans. Syst., Man, Cybern., vol. 24, pp. 400–412, Mar. 1994. [10] E. N. Sanchez, J. P. Perez, and G. Chen, “Using dynamical networks to generate chaos: An inverse optimal control approach,” Int. J. Bifurcation Chaos, vol. 11, no. 3, pp. 857–863, 2001. [11] E. N. Sanchez and R. A. Felix, “Nonlinear identification via variable structure recurrent neural networks,” in IFAC World Congr., Barcelona, Spain, July 2002.
FELIX et al.: REPRODUCING CHAOS BY VARIABLE STRUCTURE RECURRENT NEURAL NETWORKS
[12]
, “Chaos identification using variable structure recurrent neural networks,” in 9th Int. Conf. Neural Information Processing (ICONIP’02), Singapore, Nov. 2002. [13] H. Ye, A. N. Michel, and L. Hou, “Stability theory for hybrid dynamical systems,” IEEE Trans. Automat. Contr., vol. 43, pp. 461–474, Apr. 1998.
Ramon A. Felix was born in Guamuchil, Sinaloa, Mexico, in 1976. He received the B.Sc. degree from Instituto Tecnológico del Mar (ITMAR), Mazatlan Campus, Sinaloa, Mexico in 1998, and the M.Sc. and Ph.D. degree in electrical engineering from the Centro de Investigación y Estudios Avanzados del Instituto Politécnico Nacional (CINVESTAV), Guadalajara Campus, Jalisco, Mexico. In 2002, he visited the Laboratoire des signaux et systèmes (LSS), Ecole Supérieur d’Electricité (SUPELEC), Paris, France, by invitation of Romeo Ortega. He has published more than ten technical papers in international conferences and journals. He is currently associated as a Postdoctoral Scientist at CINVESTAV Guadalajara Campus. His reseach interest focuses on neural control and variable structure control and theirs real-time applications to electrical machines and power electronic systems.
Edgar N. Sanchez (M’85–SM’95) was born in Sardinata, Colombia, South America, in 1949. He received the B.S.E.E. degree in power systems from Universidad Industrial de Santander (UIS), Bucaramanga, Colombia, in 1971, the M.S.E.E. degree in automatic control from Advanced Studies and Research Center of the National Polytechnic Institute (CINVESTAV-IPN), Mexico City, Mexico, in 1974, and the Dr. Ing. degree in automatic control from Institut Nationale Polytechnique de Grenoble, Grenoble, France, in 1980. In 1971, 1972, 1975, and 1976, he worked for different electrical engineering consulting companies in Bogota, Colombia. In 1974, he was a Professor in the Electrical Engineering Department, UIS, Colombia. From 1981 to 1990, he worked as a Researcher at the Electrical Research Institute, Cuernavaca, Mexico. He was a Professor of the graduate program in Electrical Engineering of the Universidad Autonoma de Nuevo Leon (UANL), Monterrey, Mexico, from 1990 to 1996. Since 1997, he has been with CINVESTAV-IPN, Guadalajara Campus, Mexico, where he is a Professor of Electrical Engineering graduate programs. His research interest center in neural networks and fuzzy logic as applied to automatic control systems. He has published more than 100 technical papers in international journal and conferences, and has served as reviewer for different international journals and conferences. Dr. Sanchez was granted an USA National Research Council Award as a Research Associate at NASA Langley Research Center, Hampton, Virginia. He is also member of the Mexican National Research System, the Mexican Academy of Science, and the Mexican Academy of Engineering.
1457
Guanrong Chen (M’87–SM’92–F’96) received the M.Sc. degree in computer science from the Sun Yat-sen (Zhongshan) University, China and the Ph.D. degree in applied mathematics from Texas A&M University, College Station. Currently, he is a Chair Professor and the Founding Director of the Center for Chaos Control and Synchronization at the City University of Hong Kong. He is Honorary Professor of the Central Queensland University, Australia, as well as Honorary Guest-Chair Professor of more than ten Universities in China. He has authored or coauthored 15 research monographs and advanced textbooks, more than 300 SCI journal papers, and about 180 refereed conference papers, published since 1981 in the fields of nonlinear system dynamics and controls. Among his publications are the research monographs and edited books entitled Hopf Bifurcation Analysis: A Frequency Domain Approach (Singapore: World Scientific, 1996), From Chaos to Order: Methodologies, Perspectives and Applications (Singapore: World Scientific, 1998), Controlling Chaos and Bifurcations in Engineering Systems (Boca Raton, FL: CRC Press, 1999), Chaos in Circuits and Systems (Singapore: World Scientific, 2002), and Chaos Control and Bifurcation Control (New York: Springer-Verlag, 2003). Prof. Chen has served as Chief Editor, Deputy Chief Editor, Advisory Editor, Features Editor, and Associate Editors for 8 international journals including the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS, IEEE TRANSACTIONS ON AUTOMATIC CONTROL, and the International Journal of Bifurcation and Chaos. He received the 1998 Harden-Simons Prize for the Outstanding Journal Paper Award from the American Society of Engineering Education and the 2001 M. Barry Carlton Best Annual Transactions Paper Award from the IEEE Aerospace and Electronic Systems Society.
