A Recurrent Neural Network for Nonlinear Optimization with a ...
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
1251
A Recurrent Neural Network for Nonlinear Optimization with a Continuously Differentiable Objective Function and Bound Constraints Xue-Bin Liang, Student Member, IEEE, and Jun Wang, Senior Member, IEEE
Abstract—This paper presents a continuous-time recurrent neural-network model for nonlinear optimization with any continuously differentiable objective function and bound constraints. Quadratic optimization with bound constraints is a special problem which can be solved by the recurrent neural network. The proposed recurrent neural network has the following characteristics. 1) It is regular in the sense that any optimum of the objective function with bound constraints is also an equilibrium point of the neural network. If the objective function to be minimized is convex, then the recurrent neural network is complete in the sense that the set of optima of the function with bound constraints coincides with the set of equilibria of the neural network. 2) The recurrent neural network is primal and quasiconvergent in the sense that its trajectory cannot escape from the feasible region and will converge to the set of equilibria of the neural network for any initial point in the feasible bound region. 3) The recurrent neural network has an attractivity property in the sense that its trajectory will eventually converge to the feasible region for any initial states even at outside of the bounded feasible region. 4) For minimizing any strictly convex quadratic objective function subject to bound constraints, the recurrent neural network is globally exponentially stable for almost any positive network parameters. Simulation results are given to demonstrate the convergence and performance of the proposed recurrent neural network for nonlinear optimization with bound constraints. Index Terms—Bound constraints, continuously differentiable objective functions, continuous-time recurrent neural networks, global exponential stability, optimization method, quasi-convergence.
I. INTRODUCTION
B
OUND-CONSTRAINED quadratic programming problems have numerous applications in many fields of science and engineering and attracted much attention [1]. It is known that the conventional algorithms are time-consuming for solving large-scale quadratic optimization problems with bound constraints. Since the seminal work of Hopfield and Tank [2], [3], the neural-network approach to optimization has been investigated extensively; e.g., [2]–[9]. See [10] and [11] for overviews and paradigms of recurrent neural networks (RNNs) Manuscript received February 3, 1999; revised August 30, 1999 and January 6, 2000. This work was supported by the Hong Kong Research Grants Council under Grant CUHK381/96E and the National Natural Science Foundation of China under Grant 69702001. X.-B. Liang is with the Department of Electrical and Computer Engineering, University of Delaware, Newark, DE 19716 USA. J. Wang is with the Department of Automation and Computer-Aided Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong. Publisher Item Identifier S 1045-9227(00)03007-1.
for optimization. Unlike most numerical optimization methods, recurrent neural networks for solving optimization problems are readily hardware-implementable. Thus, neural networks are a top-choice of real-time solvers for bound-constrained optimization problems. In the neural-network literature, there exist a few RNN models for solving quadratic optimization problems with bound constraints; e.g., [12]–[14]). An RNN model is proposed in [12] for solving unconstrained and strictly convex quadratic programming problems, which is shown to be globally exponentially stable (GES) and can find the unique optimal solution. In [13], a simple RNN model is presented for solving bound-constrained quadratic programs, which is GES for strictly convex quadratic programs and quasiconvergent for convex quadratic programs under given conditions. In a recent paper [14], a discrete-time RNN model is proposed to solve strictly convex quadratic programs with bound constraints. Sufficient conditions for the global exponential convergence of the RNN model and several corresponding neuron updating rules are presented [14]. The discrete-time RNN model has some advantages over the continuous-time counterpart in numerical simulation and digital implementation. In this paper, we propose a continuous-time RNN model for bound-constrained nonlinear optimization with any continuously differentiable objective function which is not necessarily quadratic or convex. Quadratic optimization with bound constraints is then a special problem which can also be solved by using the proposed RNN model. The proposed RNN model in the present paper has the following features. 1) The RNN model is regular in the sense that any optimum of the objective function subject to bound constraints is also an equilibrium point of the RNN. If the minimized function is convex, then the RNN model is complete in the sense that the set of optima of the objective function with bound constraints is equal to the set of equilibria of the RNN. 2) The RNN model is a primal method in the sense that for any initial point in the feasible bound region, its trajectory can never escape from the feasible region. It has the quasiconvergence property that all trajectories starting from the feasible bound region will converge to the set of equilibria of the RNN. 3) The RNN model has an attractivity property in the sense that its trajectory will eventually converge to the feasible region for any initial point even at outside of the feasible bound region. 4) For strictly convex quadratic optimization problems with bound constraints, the RNN model is GES with almost any network parameters. These desirable
1045–9227/00$10.00 © 2000 IEEE
1252
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
properties demonstrate that the RNN model is suitable for solving bound-constrained optimization problems. The organization of this paper is as follows. In Section II, we formulate the bound-constrained optimization problem and present the continuous-time RNN model. The regular and complete properties of the RNN model are proved in Section III. In Section IV, the primal and quasiconvergence properties of the RNN model are discussed. In Section V, we prove that the RNN model is actually GES for solving strictly convex quadratic programs with bound constraints. Illustrative examples are presented in Section VI. In Section VII, we summarize the main results of the paper and make a concluding remark. II. PROBLEM FORMULATION AND THE NEURAL-NETWORK MODEL Consider the optimization problem of minimizing any conin tinuously differentiable objective function subject to bound constraints (1) is a decision vector; and are, respectively, given constant lower and upper bound with ( ), and the superscript denotes the transpose operator. is An important special case of the objective function the quadratic function; i.e.,
Fig. 1.
Functional block diagram of the RNN model (N).
paper is to construct an RNN model which can find the solutions in . The proposed RNN model is as follows:
where
(2)
(N) and where activation function
are any positive constants, and the is defined by if if if
is a symmetric matrix, and is a constant vector. We do not assume that is positive definite or semidefinite unless we point out this property explicitly somewhere. As demonstrated in [13], in the more general case
where
where
is nonsingular, by a variable transformation we can obtain the form of (1) where the objective of function to be minimized is with the bound constraints . Let the nonempty and compact (bounded and closed) set in be denoted as
then problem (1) is to minimize subject to . A point is called a feasible solution to the problem and the set is called the feasible region of the problem. If and for any , then is called an optimal solution, a global optimal solution, or simply a solution to the problem (1). ; i.e., Let the set of solutions of the problem be , . The primary goal of the
for and . The activation function can be regarded as the projection of on the closed in. We define the vector-valued function terval as , which is the proonto . jection operator of When the RNN model (N) is used to solve the optimization problem (2), the initial state point is required to be mapped into the feasible region . That is, for any , the initial point of the is setrajectory , or equivalently for lected as . The functional block diagram of the RNN model (N) is depicted in Fig. 1. When the RNN model (N) is used to perform quadratic optimization with bound constraints (1), the RNN model (N) can be rewritten in the explicit form
( ) for which the functional block diagram is shown in Fig. 2. We can define the following two positive diagonal matrices diag and diag . Then, the RNN model (N) for solving the optimization problem
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1253
III. REGULAR AND COMPLETE PROPERTIES
Fig. 2. Functional block diagram of the RNN model (Ñ).
(1) and the RNN model (Ñ) for solving the quadratic optimization problem (2) can be represented, respectively, as the following compact matrix forms: (3) and (4) in the RNN model (N) The positive constants and or (Ñ) are, respectively, time constants and scaling parameters, which can be used to control the convergence rate of the trajectories to the set of equilibria of the RNN model. Moreover, in the RNN model (Ñ), these constants can also be utilized as the preconditioners of the matrix to reduce the condition number of and hence improve the numerical stability of simulations , of the RNN model (Ñ). To see this, let a new variable then from (4), we can obtain a differential equation in terms of
(5) can be used where the two positive diagonal matrices and as the left and right diagonal preconditioners of , respectively. A widely used choice of the left and right diagonal preconditioners of is [13], [15] and [16] diag where is a positive constant. Let the solution found by the preconditioned RNN model (5) be , then the solution of . the above bound constrained optimization problem is defined Let the equilibrium set of the RNN model (N) be by
In the next section, we will explore the relationship between of problem (1) and the equilibrium set the minimizer set of the RNN model (N).
To characterize the properties of an energy function for binary-valued neural networks, three useful concepts are proposed in [17]. An energy function or a neural network is called to be regular or normal if the set of minimizers of the energy function is a subset or superset of the set of stable states of the network, respectively. If the above two sets of minimizers and stable states are the same, then the energy function or the neural network is said to be complete. The regular and normal properties of an energy function or a neural network imply the reliability and effectiveness of the neural network for optimization, respectively. Since a complete neural network is both regular and normal, it is both reliable and effective. We can generalize these concepts for binary neural networks to the continuous-time RNN model (N) for solving optimization problem , and , we (1). In the three cases of say that the RNN model (N) for optimization is regular, normal, and complete, respectively. In the regular case, the RNN for optimization is reliable but not guarantee to be effective, while the RNN for optimization is effective but not guarantee to be reliable in the normal case. In the complete case, the RNN for optimization is both reliable and effective. and are both nonempty. We now show that the two sets Since is compact, the continuously differentiable objective has at least one minimizer on . Thus, problem function . (1) has at least one solution, i.e., that is a projection operator of Noting that the function onto , it is obvious that the function is a continuous mapping from to . By the Brouwer fixed point theorem (e.g., [20, p. 10]), the above continuous mapping has at least one fixed point on . By (3), the fixed point of the above continuous mapping on is also an equilibrium point of the . RNN model (N). Thus, Theorem 1: The RNN model (N) is regular; i.e., that . In order to give the proof of Theorem 1, we need the following lemma. ( ) be a continuously differLemma 1: Let with , if entiable function. For any given is a minimizer of on , then satisfies (6) , then and hence Proof: If satisfies (6). If , then for . So, , where denotes the right derivative of at . Thus, satisfies (6). , by virtue of Similarly, if where denotes the left derivative of at , (6) . still holds for Proof of Theorem 1: Let , , for any . In particular, for then
1254
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
where is the th component variable and the symbols of “ ” for . represent the omitted component variables , for , then Let is continuously differentiable in and satisfies . is a minimizer of on . By Lemma 1, Thus, we have
Noting that , we obtain
and
This inequality implies the following nonlinear variational inequality:
By Peano’s local existence theorem for solutions of ordinary differential equations (ODEs) (see [20, p. 14]), for any , there exists a positive number such that for the autonomous system (N) has a solution satisfying , where or such that is the maximal right exis. tence interval of the solution of RNN model (N) Proposition 1: The solution is bounded for with the initial condition . Proof: The proof method is similar to one used in [13, p. 303]. for ) of The solution RNN model (N) or (3) with initial condition is given by
which is equivalent to (see [18])
Thus, . If the objective function to be minimized is convex on , it is shown below that the RNN model (N) is complete; i.e., . that Lemma 2 [19, Corollary 2, p. 103]: Let be a differentiable convex function on , and any given . Consider the problem of mininonempty and convex set in subject to . Then, is an optimal solution to mizing for all . the problem if and only if is convex on , Theorem 2: If the objective function . then the RNN model (N) is complete, i.e., that . Proof: By Theorem 1, it suffices to prove that , . This means Suppose that
which is equivalent to (see [18])
Because of
, we have
Thus, we get
Since the objective function 2, we obtain
Thus,
is convex on
, by Lemma
.
It is obvious that the mapping ( ) is for some positive number bounded; i.e., that , where is the Euclidean norm defined by . Let and , from the above expression of the solution , we have
Thus, the solution of the RNN model (N) is bounded . for By the continuation theorem for the solutions of ODE (see and [20, pp. 17–18]), the local existence of the solution its boundness property shown in Proposition 1 actually imply the global existence of the solution. Thus, we can conclude for any . In the following, let that , for be a solution of the RNN model (N) with ini. tial condition Now, we show that is a positive invariant and attractive set of the RNN model (N). That is, for any solution trajectory of the RNN model (N) starting from , the trajectory can not escape from . Moreover, for any solution trajectory of the RNN model (N) starting from the outside of , it will converge to . is a positive invariant and attractive set of the Theorem 3: RNN model (N). be any given positive number. For Proof: Let , define the set as
IV. PRIMALITY AND QUASICONVERGENCE PROPERTIES We first show that the RNN model (N) has a solution trajectory for any initial point in .
and
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1255
We first prove that for , if the initial point satisfies , then the th component has the property that of the solution for . In this sense, is said to be positive invariant. Let
By the theorem of differential inequalities (see, e.g., [20, Th. 6.1, p. 31]), we get
We can show that by a contradiction method. If oth. Then, for erwise, we have and there exists a positive number such that
and . for Integrating the above two proved properties of , we know that for any given positive number , is positive invariant and that for any solution the set with the initial point , it will enter into in a finite time after which will stay in . If taking , we can conclude that is a positive invariant the limit and attractive set of the RNN model (N). Let be any point in , if the solution of the , by Theorem 3, RNN model (N) satisfies for . Next, we will demonstrate that then will converge to the equilibrium set of the RNN model (N). For the purpose, we need two lemmas described below. The first is the well-known Barbalat lemma. Lemma 4 [21, p. 123]: If the differentiable function ( ) has a finite limit as , and if the derivative is uniformly continuous, then as function . , we have Lemma 5: For
Without loss of generality, we can assume that (7) Consider the th variable of the RNN model (N) (8) ( ) satisfies. which By the definition of activation function (7), we get
Since
and the assumption
Taking into account the above definition of
, we can obtain
, the above inequality yields (10)
So, that
is strictly increasing for
and hence
where the equality holds if and only if , Proof: Since tone increasing property of and have
. . Noting the mono, we
(9) Noting that assumption (7), we know that obtain
for
and the above . By (9), we
This is in contradiction with (7). Therefore, . This is positive invariant. means that , if the initial point Second, we show that for satisfies , then the trajectory will in some finite time. Because of the proved positive enter into will stay in invariant property of , we know that after a finite period of time. . We can assume that Suppose that because the proof for the same conclusion in the case of is similar. Let
Considering (8) with
(
If , then the equality in (10) holds. Con, then versely, suppose that the equality in (10) holds. If . If , then , i.e., that . Theorem 4: For any , the solution ( ) of the RNN model (N) with initial condition is convergent to in the sense that as Proof: By Theorem 3, if , then for . Let for . along the solution of the RNN model Differentiating (N), by Lemma 5, we have
), we have Thus,
is monotone decreasing.
1256
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
Because is continuous on , it can attain its minimal has a lower bound for . value on . This implies that Combining this property with the monotone decreasing one of , it can be concluded that has a finite limit as . for be defined by Let
then is continuous in . Since is nonempty and comis uniformly continuous and bounded on . pact, is uniformly continuous for . Now, we prove that , as a composite function of two uniThen, and , is also uniformly formly continuous functions of . continuous for The right-hand side of the RNN model (N), as a continuous , function of on , is bounded. Thus, for is bounded for , which implies that is . Thus, is uniformly continuniformly continuous for . uous for ( ) satisfies the From the above results, the function two conditions in Lemma 4. Therefore as
(11)
In the following, we show that Theorem 4 holds by using a contradiction method. If otherwise, there exists a positive constant such that for any positive number there satisfying exists a corresponding positive number
From this and the boundness property of ( exists a convergent subsequence such that function
for
, there ) of the
(12) ,
where
as such
, and there exists that
Because of the compactness of . have is continuous for Since
Then, by the definition of
which means that
and
for
, we
, from (11), we get
and Lemma 5, we can obtain
. From this, it follows that
Taking the limit , we get , which contradicts (12). Therefore, the proof of Theorem 4 is complete. From Theorems 3 and 4 and their proofs, it is known that the RNN model (N) solves bound-constrained optimization problems through its improved solution trajectory starting from the feasible region, and that the solution trajectory does not escape from the feasible region. That is, if the initial point is feasible, then the objective value along the corresponding solution trajectory decreases as the time elapses, while the solution trajectory remains in the feasible region. Thus, any solution trajectory of the RNN model (N) starting from the feasible region is a continuous-time optimization process. Since the primal feasibility is maintained during the continuous-time optimization process, the RNN model (N) can be referred to as the continuous-time primal approach (see [19, p. 408]), or simply to be primal. According to Theorem 4, all the initial points in the feasible of the RNN model region converge to the equilibrium set of the RNN (N). In the sense, we say that the equilibrium set model (N), or simply the RNN model (N) itself, is quasiconvergent in the feasible region. From the above obtained results, we can draw the following conclusions. For the minimization of any continuously differentiable objective function with bound constraints, we can use the RNN model (N) to find the optimal solutions of problem (1). If the objective function is convex, then any trajectory of the RNN model (N) starting from the feasible region is convergent to the set of minimizers. If the objective function is strictly convex, then any trajectory of the RNN model (N) starting from the feasible region is convergent to the unique minimizer. However, it is worthy pointing out that the RNN model (N) for the minimization of a nonconvex objective function can only guarantee that each trajectory starting from the feasible region converges to an equilibrium of the network, which may or may not be the minimum of the objective function. For quadratic optimization with bound constraints (2), the RNN model (Ñ) can be employed to find the optimal soluis tions. If the quadratic objective function is convex (i.e., positive semidefinite), then any trajectory of the RNN model (Ñ) starting from the feasible region is convergent to the set of minimizers of the bound-constrained quadratic optimization. A continuous-time RNN model with simple network structure and quasiconvergence property is proposed in [13] for convex quadratic optimization with bound constraints. While the trajec, tory of the neural network can start from the whole space the convergence of the trajectory to the set of minimizers of the problem (2) is guaranteed under the condition that all the diagonal elements of are positive [13, Theorem 3] which is not required for the quasiconvergence in the feasible region of the RNN model (Ñ) in the present paper. If the quadratic objective function is strictly convex (i.e., that is positive definite), then any trajectory of the RNN model (Ñ) starting from the feasible region is convergent to the unique minimizer of the problem. In Section V, we will show that, in the strictly convex case, the RNN model (Ñ) with equal time constants is, in fact, GES for for . almost any positive scaling parameters Contrary to this, the GES property of the neural network in [13] is under a condition which does not hold almost anywhere [13, Theorem 1, Lemma 1, and Corollary 1].
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1257
V. GES PROPERTY FOR STRICTLY CONVEX QUADRATIC OPTIMIZATION WITH BOUND CONSTRAINTS In the section, we consider an important case: strictly convex quadratic optimization with the bound constraints where is positive definite. This . problem has a unique minimizer in and ( ) in the RNN Let the time constants model (Ñ) be equal to , then the RNN model (Ñ) for solving the strictly convex quadratic optimization with bound constraints can be described by
(Q) which can also be written as the following compact matrix form:
with
as its unique equilibrium point, where for . Let the vector-valued function , . Then, for ( ) is monotone increasing and satisfies
For the RNN model (Q) and its equivalent system (13), the for is uniquely determined by its initial solution because the right-hand side of condition (13) is obviously a globally Lipschitz continuous mapping from to . Let be the unique solution of (15) with the initial condition . Construct the following Lyapunov function of the generalized Lur’e-Postnikov type [8]
(13) When the RNN model (Q) is used to find the unique minof the strictly convex quadratic optimization with imizer bound constraints, the initial point for the solution trajectory of , whereas the initial point the RNN (Q) can be any one in is required to be located in the feasible region in the RNN model (Ñ) for general quadratic optimization with bound constraints. The functional block diagram of the RNN model (Q) for solving strictly convex quadratic optimization problems with bound constraints is identical to the one shown in Fig. 2 expect that the former does not appear the saturation functions applied and that the time constants for to the initial point are all equal to . In the following, we will show that the RNN model (Q) is for actually GES for almost any scaling parameters . is nonsingular, where Theorem 5: If the matrix is the identity matrix, then the RNN model (Q) is GES. and such That is, there exist two positive constants and that for any
with
where and denote the maximal and minimal eigenvalues of the symmetric matrix arguments, respectively. It , since is symmetric and positive is noted that definite. Thus, can be well defined as above. , and Let
Then, we have
Computing the derivative of system (15), we have
along the solution
where is the solution of the RNN model (Q) with initial condition . , and the Proof: Let the nonsingular matrix . vector , the RNN Making the variable transformation model (Q) or (13) can be rewritten as the following system of : ODE in terms of (14) as its unique equiwith librium point. , then Let a new variable (14) can be reduced to a system of ODE in terms of of the form (15)
where the fact that is used in the above first inequality. It follows from the above result that
for
of
1258
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
00 2 0 9)
Fig. 3. Convergence to a minimizer of the initial point ( Example 1.
: ;
:
in in
Fig. 4. Convergence to a minimizer of the initial point (
in Example 1.
01 5 02 5) : ;
:
not in
By the theorem of differential inequalities, we obtain
and hence
Noting that and it follows from the above inequality that for
,
cond (16) is where cond is not singular, the condition number of the matrix . Since , which guarantees the well-definiteness of . Let cond and , cond then Theorem 5 is proved. Since , as a function of , is continuously differentiable, the set of its zero points has , where the set zero measure in . Thus, for almost any scaling parameters for , the condition in Theorem 5 is satisfied and hence the conclusion of GES property of the RNN model (Q) is true. From inequality (16), we know that the global exponential convergence rate of the RNN model (Q) for strictly convex quadratic optimization with bound constraints can be made . arbitrarily large by decreasing the time constant However, this may induce the RNN model (Q) more sensitive to input noise and round-off errors in analog implementation and numerical simulation.
Fig. 5. Convergence to the minimizers of 30 random initial points in Example 1.
in
VI. SIMULATION RESULTS In this section, several numerical simulation examples are presented to demonstrate the characteristics and performance of
Fig. 6.
Spatial representation of the trajectories starting from in Example 1.
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1259
the proposed RNN model (N), (Ñ), or (Q) for solving optimization problems with bound constraints. Example 1: Consider a convex quadratic optimization problem with bound constraints:
It is obvious that is a convex quadratic function in with the positive semidefinite matrix
According to Theorems 2 and 4, the RNN model (Ñ) is complete and quasiconvergent. That is, all the solution trajectories of system (Ñ) starting from the feasible region are convergent to the set of minimizers of the bound-constrained optimization problem. Moreover, all the minimizers of this bounded optimization problem can be reached via the solution trajectories of the RNN model (Ñ) starting from . and scaling In the simulation, time constants were used. As shown in Figs. 3 parameters and 4, two different initial points can converge to different minimizers of the bounded quadratic program. Note that the two initial points selected in Figs. 3 and 4 are inside and outside of , respectively. generated based on uniform Next, 30 random points in distribution are used as initial states of the solution trajectories. It can be seen from Fig. 5 that all the trajectories corresponding to the above 30 random initial points are convergent to the minimizers of the bound-constrained quadratic program, since all the objective values converge to the minimal value as time approaches infinity. To observe the global convergent behavior of the RNN model (Ñ) in , 300 random points generated uniformly in are used as the initial states of the RNN model (Ñ). The spatial representation of these trajectories is described in Fig. 6. It is evident that the set of minimizers of the optimization problem, i.e., , is an attractive set of the RNN model (Ñ). , Next, 300 random points from a superset of , are used as initial states, and find that all the corresponding trajectories starting from the inside and outside of converge to the attractive set of the RNN model (Ñ), as shown in Fig. 7. Example 2: Consider a strictly convex quadratic optimization problem with bound constraints as follows:
Fig. 7. Spatial representation of the trajectories starting from the inside and outside of in Example 1.
Fig. 8. Spatial representation of solution trajectories in the plane (x Example 2.
; x
) in
Fig. 9. Spatial representation of solution trajectories in the plane (x Example 2.
; x
) in
with the positive definite matrix
and
This example is the same as the first example in [14] where a discrete-time RNN model is employed to
1260
Fig. 10. Spatial representation of solution trajectories in the plane (x Example 2.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
; x
) in
Fig. 11. Convergence to the unique minimizer of 30 random initial points in Example 2.
solve the optimization problem. In the unconstrained case, the optimization problem has a unique minimizer , outside of and the . minimal value of The RNN model (Q) is used to find the unique minimizer of the above optimization problem with the time constant and the scaling constants . According to Theorem 5, the RNN model (Q) is GES. We select 100 random points in the feasible region as the initial states of the RNN (Q). The spatial representations of these trajectories in the , and are shown coordinate planes in Figs. 8–10, respectively. It is found that the unique minimizer of the above bound-constrained optimization problem is . Next, 30 random initial points taken uniformly in the superset as the initial states of the RNN model (Q). As illustrated in Fig. 11, all the solution trajec-
Fig. 12.
Convergence to the unique minimizer of the initial point in in Example 2.
(15 10 010) ;
;
Fig. 13. Convergence to the unique constrained minimizer of the unconstrained minimizer as the initial point not in in Example 2.
tories from these initial points approach the unique minimizer with the value of . Since of some initial points are by far larger than the values of those of other initial ones during the initial period, to illustrate clearly the long-term convergence behaviors of all trajectories instead of for the vertical in the same graph, we use axis. In Figs. 12 and 13, the trajectories of two particular initial points are shown to be both convergent to the unique minimizer . In Fig. 13, the initial point is the unconstrained minimizer . The objective value increases at first for a short period of time and then keeps decreasing and eventually converges to the unique constrained minimizer . Example 3: In this example, the optimization problem is to minimize a nonconvex objective function
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1261
00 5 00 8)
Fig. 14. Convergence to a minimizer of the initial point ( in Example 3.
Fig. 15. Convergence to a minimizer of the initial point ( Example 3.
:
in
00 8 0 9)
in in
: ;
: ;
:
subject to . The minimal value of the above optimization problem is obviously . The RNN model (N) can be used to solve such an optimization problem. According to Theorems 3 and 4, the feasible region is an attractive set of the RNN model and all the trajectories starting from will converge to the set of equilibria of the RNN model (N). Through a simple computation, the second partial derivative at is . of is not convex in . So, the Thus, the objective function RNN model (N) is regular but not complete. That is, a trajectory of the RNN system (N) starting from the feasible region may converge to an equilibrium point which is not a minimizer of the above optimization problem. As shown in Figs. 14 and 15, two different initial states in converge to two different minimizers of the above optimization problem. From 20 initial random initial states taking from
Fig. 16. Convergence to the minimizers of 20 random initial points in Example 3.
Fig. 17. 3.
in
Spatial representation of the trajectories starting from in Example
uniformly the feasible region , Fig. 16 shows that they all converge to the minimizers of the above optimization problem. To observe the global dynamical behavior of the RNN model in , 1000 random initial states generated uniformly in . As illustrated in a Y-shape curve appears and is a part of the attractive equilibrium set of the RNN model (N). In the region , the objective value is larger than and hence that there is no any minimizer of the above optimization problem in . It can be seen from Fig. 17 that all the initial points in converge to the Y-shape curve. We select 10 000 random initial states based on uniform distribution in the feasible region and find that 65 initial points fail to converge to the minimizers of the above optimization being more than 10 problem with the objective value when the simulation algorithm terminates. Thus, the success ratio or the effectiveness of the RNN model (N) in finding the minimizers of the above optimization problem is 99.35% statis-
1262
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
tically. It is worth pointing out here that the above high success ratio may be the consequence of the particular characteristics of the nonconvex objective function. In other words, the success ratio may become more or less for other chosen nonconvex objective functions. VII. CONCLUSION We have proposed a continuous-time RNN for optimizing any continuously differentiable objective function subject to bound constraints. The proposed RNN has the desirable regularity, primality, quasiconvergence, and attractivity properties. In solving general convex programs with bound constraints, all the solution trajectories of the RNN starting from the feasible region are convergent to set of the minimizers of the optimization problem. In solving strictly convex quadratic programs with bound constraints, the RNN is globally exponentially stable and can find the unique minimizer of the optimization problems. The simulation results have demonstrated the convergence behaviors and characteristics of the RNN for solving bound-constrained optimization problems. REFERENCES [1] J. J. Mor´e and G. Toraldo, “On the solution of large quadratic programming problems with bound constraints,” SIAM J. Optim., vol. 1, pp. 93–113, 1991. [2] J. J. Hopfield and D. W. Tank, “Neural computation of decision in optimization problem,” Biol. Cybern., vol. 52, pp. 141–152, 1985. [3] D. W. Tank and J. J. Hopfield, “Simple ‘neural’ optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit,” IEEE Trans. Circuits Syst., vol. CS-33, pp. 533–541, 1986. [4] M. P. Kennedy and L. O. Chua, “Neural networks for nonlinear programming,” IEEE Trans. Circuits Syst., vol. 35, pp. 554–562, 1988. [5] A. Rodríguez-Vázquez, R. Domínguer, A. Ruda, J. L. Huertas, and E. Sánchez-Sinencio, “Nonlinear switched-capacitor ‘neural’ networks for optimization problems,” IEEE Trans. Circuits Syst., vol. 37, pp. 384–397, 1990. [6] C. Y. Maa and M. Shanblatt, “Linear and quadratic programming neural network analysis,” IEEE Trans. Neural Networks, vol. 3, pp. 580–594, 1992. [7] J. Wang, “A deterministic annealing neural network for convex programming,” Neural Networks, vol. 7, pp. 629–641, 1994. [8] M. Forti and A. Tesi, “New conditions for global stability of neural networks with applications to linear and quadratic programming problems,” IEEE Trans. Circuits Syst. I, vol. 42, pp. 354–366, 1995. [9] Y. Xia and J. Wang, “A general methodology for designing globally convergent optimization neural networks,” IEEE Trans. Neural Networks, vol. 9, pp. 1331–1343, 1998. [10] A. Cichocki and R. Unbehauen, Neural Networks for Optimization and Signal Processing, U.K.: Wiley, 1993. [11] J. Wang, “Recurrent neural networks for optimization,” in Fuzzy Logic and Neural Network Handbook, C. H. Chen, Ed. New York: McGrawHill, 1996, pp. 4.1–4.35. [12] S. I. Sudharsanan and M. K. Sundareshan, “Exponential stability and a systematic synthesis of a neural network for quadratic minimization,” Neural Networks, vol. 4, pp. 599–613, 1991.
[13] A. Bouzerdoum and T. R. Pattison, “Neural network for quadratic optimization with bound constraints,” IEEE Trans. Neural Networks, vol. 4, pp. 293–303, 1993. [14] M. J. Pérez-Ilzarbe, “Convergence analysis of a discrete-time recurrent neural network to perform quadratic real optimization with bound constraints,” IEEE Trans. Neural Networks, vol. 9, pp. 1344–1351, 1998. [15] A. Greenbaum and G. H. Rodrigue, “Optimal preconditioners of a given sparsity pattern,” Comput. Sci. Numer. Math., vol. 29, pp. 610–634, 1989. [16] G. H. Golub and C. F. Van Loan, Matrix Computation, 2nd ed. Baltimore, MD: Johns Hopkins Univ. Press, 1989. [17] Z. B. Xu, G. Q. Hu, and C. P. Kwong, “Asymmetric Hopfield-type networks: Theory and applications,” Neural Networks, vol. 9, pp. 483–501, 1996. [18] B. C. Eaves, “On the basic theorem of complementarity,” Math. Programming, vol. 1, pp. 68–75, 1971. [19] M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear Programming: Theory and Algorithms, 2nd ed. New York: Wiley, 1993. [20] J. K. Hale, Ordinary Differential Equations. New York: Wiley, 1969. [21] J.-J. E. Slotine and W. Li, Applied Nonlinear Control. Englewood Cliffs, NJ: Prentice-Hall, 1991.
Xue-Bin Liang (S’99) received the B.S. and M.S. degrees in mathematics from Nankai University, Tianjin, and Fudan University, Shanghai, China, in 1988 and 1991, respectively. From 1991 to 1999, he held research and/or teaching positions in the Department of Mathematics at Anhui University, Hefei, China, the Department of Computer Science at Fudan University, Shanghai, China; the Department of Information Science at Utsunomiya University, Utsunomiya, Japan; the Department of Mechanical and Automation Engineering at Chinese University of Hong Kong; the Department of Manufacturing Engineering and Engineering Management at City University of Hong Kong; and the Department of Electrical Engineering at Arizona State University, Tempe, AZ. He is currently with the Department of Electrical and Computer Engineering, University of Delaware, Newark. His current research interests include digital and wireless communications, data communication and networking, multimedia processing and transportation, digital and statistical signal processing, and computational intelligence.
Jun Wang (S’89–M’90–SM’93) received the B.S. degree in electrical engineering and the M.S. degree in systems engineering from Dalian Institute of Technology (now renamed as Dalian University of Technology), Dalian, China. He received the Ph.D. degree in systems engineering from Case Western Reserve University, Cleveland, OH. He was an Associate Professor at the University of North Dakota, Grand Forks. Presently, he is an Associate Professor in the Department of Automation and Computer-Aided Engineering, the Chinese University of Hong Kong. He is the author or coauthor of more than 50 journal papers, nine book chapters, two edited books, and numerous conference papers. His current research interests include theory and methodology of neural networks and their engineering applications.
1251
A Recurrent Neural Network for Nonlinear Optimization with a Continuously Differentiable Objective Function and Bound Constraints Xue-Bin Liang, Student Member, IEEE, and Jun Wang, Senior Member, IEEE
Abstract—This paper presents a continuous-time recurrent neural-network model for nonlinear optimization with any continuously differentiable objective function and bound constraints. Quadratic optimization with bound constraints is a special problem which can be solved by the recurrent neural network. The proposed recurrent neural network has the following characteristics. 1) It is regular in the sense that any optimum of the objective function with bound constraints is also an equilibrium point of the neural network. If the objective function to be minimized is convex, then the recurrent neural network is complete in the sense that the set of optima of the function with bound constraints coincides with the set of equilibria of the neural network. 2) The recurrent neural network is primal and quasiconvergent in the sense that its trajectory cannot escape from the feasible region and will converge to the set of equilibria of the neural network for any initial point in the feasible bound region. 3) The recurrent neural network has an attractivity property in the sense that its trajectory will eventually converge to the feasible region for any initial states even at outside of the bounded feasible region. 4) For minimizing any strictly convex quadratic objective function subject to bound constraints, the recurrent neural network is globally exponentially stable for almost any positive network parameters. Simulation results are given to demonstrate the convergence and performance of the proposed recurrent neural network for nonlinear optimization with bound constraints. Index Terms—Bound constraints, continuously differentiable objective functions, continuous-time recurrent neural networks, global exponential stability, optimization method, quasi-convergence.
I. INTRODUCTION
B
OUND-CONSTRAINED quadratic programming problems have numerous applications in many fields of science and engineering and attracted much attention [1]. It is known that the conventional algorithms are time-consuming for solving large-scale quadratic optimization problems with bound constraints. Since the seminal work of Hopfield and Tank [2], [3], the neural-network approach to optimization has been investigated extensively; e.g., [2]–[9]. See [10] and [11] for overviews and paradigms of recurrent neural networks (RNNs) Manuscript received February 3, 1999; revised August 30, 1999 and January 6, 2000. This work was supported by the Hong Kong Research Grants Council under Grant CUHK381/96E and the National Natural Science Foundation of China under Grant 69702001. X.-B. Liang is with the Department of Electrical and Computer Engineering, University of Delaware, Newark, DE 19716 USA. J. Wang is with the Department of Automation and Computer-Aided Engineering, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong. Publisher Item Identifier S 1045-9227(00)03007-1.
for optimization. Unlike most numerical optimization methods, recurrent neural networks for solving optimization problems are readily hardware-implementable. Thus, neural networks are a top-choice of real-time solvers for bound-constrained optimization problems. In the neural-network literature, there exist a few RNN models for solving quadratic optimization problems with bound constraints; e.g., [12]–[14]). An RNN model is proposed in [12] for solving unconstrained and strictly convex quadratic programming problems, which is shown to be globally exponentially stable (GES) and can find the unique optimal solution. In [13], a simple RNN model is presented for solving bound-constrained quadratic programs, which is GES for strictly convex quadratic programs and quasiconvergent for convex quadratic programs under given conditions. In a recent paper [14], a discrete-time RNN model is proposed to solve strictly convex quadratic programs with bound constraints. Sufficient conditions for the global exponential convergence of the RNN model and several corresponding neuron updating rules are presented [14]. The discrete-time RNN model has some advantages over the continuous-time counterpart in numerical simulation and digital implementation. In this paper, we propose a continuous-time RNN model for bound-constrained nonlinear optimization with any continuously differentiable objective function which is not necessarily quadratic or convex. Quadratic optimization with bound constraints is then a special problem which can also be solved by using the proposed RNN model. The proposed RNN model in the present paper has the following features. 1) The RNN model is regular in the sense that any optimum of the objective function subject to bound constraints is also an equilibrium point of the RNN. If the minimized function is convex, then the RNN model is complete in the sense that the set of optima of the objective function with bound constraints is equal to the set of equilibria of the RNN. 2) The RNN model is a primal method in the sense that for any initial point in the feasible bound region, its trajectory can never escape from the feasible region. It has the quasiconvergence property that all trajectories starting from the feasible bound region will converge to the set of equilibria of the RNN. 3) The RNN model has an attractivity property in the sense that its trajectory will eventually converge to the feasible region for any initial point even at outside of the feasible bound region. 4) For strictly convex quadratic optimization problems with bound constraints, the RNN model is GES with almost any network parameters. These desirable
1045–9227/00$10.00 © 2000 IEEE
1252
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
properties demonstrate that the RNN model is suitable for solving bound-constrained optimization problems. The organization of this paper is as follows. In Section II, we formulate the bound-constrained optimization problem and present the continuous-time RNN model. The regular and complete properties of the RNN model are proved in Section III. In Section IV, the primal and quasiconvergence properties of the RNN model are discussed. In Section V, we prove that the RNN model is actually GES for solving strictly convex quadratic programs with bound constraints. Illustrative examples are presented in Section VI. In Section VII, we summarize the main results of the paper and make a concluding remark. II. PROBLEM FORMULATION AND THE NEURAL-NETWORK MODEL Consider the optimization problem of minimizing any conin tinuously differentiable objective function subject to bound constraints (1) is a decision vector; and are, respectively, given constant lower and upper bound with ( ), and the superscript denotes the transpose operator. is An important special case of the objective function the quadratic function; i.e.,
Fig. 1.
Functional block diagram of the RNN model (N).
paper is to construct an RNN model which can find the solutions in . The proposed RNN model is as follows:
where
(2)
(N) and where activation function
are any positive constants, and the is defined by if if if
is a symmetric matrix, and is a constant vector. We do not assume that is positive definite or semidefinite unless we point out this property explicitly somewhere. As demonstrated in [13], in the more general case
where
where
is nonsingular, by a variable transformation we can obtain the form of (1) where the objective of function to be minimized is with the bound constraints . Let the nonempty and compact (bounded and closed) set in be denoted as
then problem (1) is to minimize subject to . A point is called a feasible solution to the problem and the set is called the feasible region of the problem. If and for any , then is called an optimal solution, a global optimal solution, or simply a solution to the problem (1). ; i.e., Let the set of solutions of the problem be , . The primary goal of the
for and . The activation function can be regarded as the projection of on the closed in. We define the vector-valued function terval as , which is the proonto . jection operator of When the RNN model (N) is used to solve the optimization problem (2), the initial state point is required to be mapped into the feasible region . That is, for any , the initial point of the is setrajectory , or equivalently for lected as . The functional block diagram of the RNN model (N) is depicted in Fig. 1. When the RNN model (N) is used to perform quadratic optimization with bound constraints (1), the RNN model (N) can be rewritten in the explicit form
( ) for which the functional block diagram is shown in Fig. 2. We can define the following two positive diagonal matrices diag and diag . Then, the RNN model (N) for solving the optimization problem
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1253
III. REGULAR AND COMPLETE PROPERTIES
Fig. 2. Functional block diagram of the RNN model (Ñ).
(1) and the RNN model (Ñ) for solving the quadratic optimization problem (2) can be represented, respectively, as the following compact matrix forms: (3) and (4) in the RNN model (N) The positive constants and or (Ñ) are, respectively, time constants and scaling parameters, which can be used to control the convergence rate of the trajectories to the set of equilibria of the RNN model. Moreover, in the RNN model (Ñ), these constants can also be utilized as the preconditioners of the matrix to reduce the condition number of and hence improve the numerical stability of simulations , of the RNN model (Ñ). To see this, let a new variable then from (4), we can obtain a differential equation in terms of
(5) can be used where the two positive diagonal matrices and as the left and right diagonal preconditioners of , respectively. A widely used choice of the left and right diagonal preconditioners of is [13], [15] and [16] diag where is a positive constant. Let the solution found by the preconditioned RNN model (5) be , then the solution of . the above bound constrained optimization problem is defined Let the equilibrium set of the RNN model (N) be by
In the next section, we will explore the relationship between of problem (1) and the equilibrium set the minimizer set of the RNN model (N).
To characterize the properties of an energy function for binary-valued neural networks, three useful concepts are proposed in [17]. An energy function or a neural network is called to be regular or normal if the set of minimizers of the energy function is a subset or superset of the set of stable states of the network, respectively. If the above two sets of minimizers and stable states are the same, then the energy function or the neural network is said to be complete. The regular and normal properties of an energy function or a neural network imply the reliability and effectiveness of the neural network for optimization, respectively. Since a complete neural network is both regular and normal, it is both reliable and effective. We can generalize these concepts for binary neural networks to the continuous-time RNN model (N) for solving optimization problem , and , we (1). In the three cases of say that the RNN model (N) for optimization is regular, normal, and complete, respectively. In the regular case, the RNN for optimization is reliable but not guarantee to be effective, while the RNN for optimization is effective but not guarantee to be reliable in the normal case. In the complete case, the RNN for optimization is both reliable and effective. and are both nonempty. We now show that the two sets Since is compact, the continuously differentiable objective has at least one minimizer on . Thus, problem function . (1) has at least one solution, i.e., that is a projection operator of Noting that the function onto , it is obvious that the function is a continuous mapping from to . By the Brouwer fixed point theorem (e.g., [20, p. 10]), the above continuous mapping has at least one fixed point on . By (3), the fixed point of the above continuous mapping on is also an equilibrium point of the . RNN model (N). Thus, Theorem 1: The RNN model (N) is regular; i.e., that . In order to give the proof of Theorem 1, we need the following lemma. ( ) be a continuously differLemma 1: Let with , if entiable function. For any given is a minimizer of on , then satisfies (6) , then and hence Proof: If satisfies (6). If , then for . So, , where denotes the right derivative of at . Thus, satisfies (6). , by virtue of Similarly, if where denotes the left derivative of at , (6) . still holds for Proof of Theorem 1: Let , , for any . In particular, for then
1254
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
where is the th component variable and the symbols of “ ” for . represent the omitted component variables , for , then Let is continuously differentiable in and satisfies . is a minimizer of on . By Lemma 1, Thus, we have
Noting that , we obtain
and
This inequality implies the following nonlinear variational inequality:
By Peano’s local existence theorem for solutions of ordinary differential equations (ODEs) (see [20, p. 14]), for any , there exists a positive number such that for the autonomous system (N) has a solution satisfying , where or such that is the maximal right exis. tence interval of the solution of RNN model (N) Proposition 1: The solution is bounded for with the initial condition . Proof: The proof method is similar to one used in [13, p. 303]. for ) of The solution RNN model (N) or (3) with initial condition is given by
which is equivalent to (see [18])
Thus, . If the objective function to be minimized is convex on , it is shown below that the RNN model (N) is complete; i.e., . that Lemma 2 [19, Corollary 2, p. 103]: Let be a differentiable convex function on , and any given . Consider the problem of mininonempty and convex set in subject to . Then, is an optimal solution to mizing for all . the problem if and only if is convex on , Theorem 2: If the objective function . then the RNN model (N) is complete, i.e., that . Proof: By Theorem 1, it suffices to prove that , . This means Suppose that
which is equivalent to (see [18])
Because of
, we have
Thus, we get
Since the objective function 2, we obtain
Thus,
is convex on
, by Lemma
.
It is obvious that the mapping ( ) is for some positive number bounded; i.e., that , where is the Euclidean norm defined by . Let and , from the above expression of the solution , we have
Thus, the solution of the RNN model (N) is bounded . for By the continuation theorem for the solutions of ODE (see and [20, pp. 17–18]), the local existence of the solution its boundness property shown in Proposition 1 actually imply the global existence of the solution. Thus, we can conclude for any . In the following, let that , for be a solution of the RNN model (N) with ini. tial condition Now, we show that is a positive invariant and attractive set of the RNN model (N). That is, for any solution trajectory of the RNN model (N) starting from , the trajectory can not escape from . Moreover, for any solution trajectory of the RNN model (N) starting from the outside of , it will converge to . is a positive invariant and attractive set of the Theorem 3: RNN model (N). be any given positive number. For Proof: Let , define the set as
IV. PRIMALITY AND QUASICONVERGENCE PROPERTIES We first show that the RNN model (N) has a solution trajectory for any initial point in .
and
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1255
We first prove that for , if the initial point satisfies , then the th component has the property that of the solution for . In this sense, is said to be positive invariant. Let
By the theorem of differential inequalities (see, e.g., [20, Th. 6.1, p. 31]), we get
We can show that by a contradiction method. If oth. Then, for erwise, we have and there exists a positive number such that
and . for Integrating the above two proved properties of , we know that for any given positive number , is positive invariant and that for any solution the set with the initial point , it will enter into in a finite time after which will stay in . If taking , we can conclude that is a positive invariant the limit and attractive set of the RNN model (N). Let be any point in , if the solution of the , by Theorem 3, RNN model (N) satisfies for . Next, we will demonstrate that then will converge to the equilibrium set of the RNN model (N). For the purpose, we need two lemmas described below. The first is the well-known Barbalat lemma. Lemma 4 [21, p. 123]: If the differentiable function ( ) has a finite limit as , and if the derivative is uniformly continuous, then as function . , we have Lemma 5: For
Without loss of generality, we can assume that (7) Consider the th variable of the RNN model (N) (8) ( ) satisfies. which By the definition of activation function (7), we get
Since
and the assumption
Taking into account the above definition of
, we can obtain
, the above inequality yields (10)
So, that
is strictly increasing for
and hence
where the equality holds if and only if , Proof: Since tone increasing property of and have
. . Noting the mono, we
(9) Noting that assumption (7), we know that obtain
for
and the above . By (9), we
This is in contradiction with (7). Therefore, . This is positive invariant. means that , if the initial point Second, we show that for satisfies , then the trajectory will in some finite time. Because of the proved positive enter into will stay in invariant property of , we know that after a finite period of time. . We can assume that Suppose that because the proof for the same conclusion in the case of is similar. Let
Considering (8) with
(
If , then the equality in (10) holds. Con, then versely, suppose that the equality in (10) holds. If . If , then , i.e., that . Theorem 4: For any , the solution ( ) of the RNN model (N) with initial condition is convergent to in the sense that as Proof: By Theorem 3, if , then for . Let for . along the solution of the RNN model Differentiating (N), by Lemma 5, we have
), we have Thus,
is monotone decreasing.
1256
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
Because is continuous on , it can attain its minimal has a lower bound for . value on . This implies that Combining this property with the monotone decreasing one of , it can be concluded that has a finite limit as . for be defined by Let
then is continuous in . Since is nonempty and comis uniformly continuous and bounded on . pact, is uniformly continuous for . Now, we prove that , as a composite function of two uniThen, and , is also uniformly formly continuous functions of . continuous for The right-hand side of the RNN model (N), as a continuous , function of on , is bounded. Thus, for is bounded for , which implies that is . Thus, is uniformly continuniformly continuous for . uous for ( ) satisfies the From the above results, the function two conditions in Lemma 4. Therefore as
(11)
In the following, we show that Theorem 4 holds by using a contradiction method. If otherwise, there exists a positive constant such that for any positive number there satisfying exists a corresponding positive number
From this and the boundness property of ( exists a convergent subsequence such that function
for
, there ) of the
(12) ,
where
as such
, and there exists that
Because of the compactness of . have is continuous for Since
Then, by the definition of
which means that
and
for
, we
, from (11), we get
and Lemma 5, we can obtain
. From this, it follows that
Taking the limit , we get , which contradicts (12). Therefore, the proof of Theorem 4 is complete. From Theorems 3 and 4 and their proofs, it is known that the RNN model (N) solves bound-constrained optimization problems through its improved solution trajectory starting from the feasible region, and that the solution trajectory does not escape from the feasible region. That is, if the initial point is feasible, then the objective value along the corresponding solution trajectory decreases as the time elapses, while the solution trajectory remains in the feasible region. Thus, any solution trajectory of the RNN model (N) starting from the feasible region is a continuous-time optimization process. Since the primal feasibility is maintained during the continuous-time optimization process, the RNN model (N) can be referred to as the continuous-time primal approach (see [19, p. 408]), or simply to be primal. According to Theorem 4, all the initial points in the feasible of the RNN model region converge to the equilibrium set of the RNN (N). In the sense, we say that the equilibrium set model (N), or simply the RNN model (N) itself, is quasiconvergent in the feasible region. From the above obtained results, we can draw the following conclusions. For the minimization of any continuously differentiable objective function with bound constraints, we can use the RNN model (N) to find the optimal solutions of problem (1). If the objective function is convex, then any trajectory of the RNN model (N) starting from the feasible region is convergent to the set of minimizers. If the objective function is strictly convex, then any trajectory of the RNN model (N) starting from the feasible region is convergent to the unique minimizer. However, it is worthy pointing out that the RNN model (N) for the minimization of a nonconvex objective function can only guarantee that each trajectory starting from the feasible region converges to an equilibrium of the network, which may or may not be the minimum of the objective function. For quadratic optimization with bound constraints (2), the RNN model (Ñ) can be employed to find the optimal soluis tions. If the quadratic objective function is convex (i.e., positive semidefinite), then any trajectory of the RNN model (Ñ) starting from the feasible region is convergent to the set of minimizers of the bound-constrained quadratic optimization. A continuous-time RNN model with simple network structure and quasiconvergence property is proposed in [13] for convex quadratic optimization with bound constraints. While the trajec, tory of the neural network can start from the whole space the convergence of the trajectory to the set of minimizers of the problem (2) is guaranteed under the condition that all the diagonal elements of are positive [13, Theorem 3] which is not required for the quasiconvergence in the feasible region of the RNN model (Ñ) in the present paper. If the quadratic objective function is strictly convex (i.e., that is positive definite), then any trajectory of the RNN model (Ñ) starting from the feasible region is convergent to the unique minimizer of the problem. In Section V, we will show that, in the strictly convex case, the RNN model (Ñ) with equal time constants is, in fact, GES for for . almost any positive scaling parameters Contrary to this, the GES property of the neural network in [13] is under a condition which does not hold almost anywhere [13, Theorem 1, Lemma 1, and Corollary 1].
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1257
V. GES PROPERTY FOR STRICTLY CONVEX QUADRATIC OPTIMIZATION WITH BOUND CONSTRAINTS In the section, we consider an important case: strictly convex quadratic optimization with the bound constraints where is positive definite. This . problem has a unique minimizer in and ( ) in the RNN Let the time constants model (Ñ) be equal to , then the RNN model (Ñ) for solving the strictly convex quadratic optimization with bound constraints can be described by
(Q) which can also be written as the following compact matrix form:
with
as its unique equilibrium point, where for . Let the vector-valued function , . Then, for ( ) is monotone increasing and satisfies
For the RNN model (Q) and its equivalent system (13), the for is uniquely determined by its initial solution because the right-hand side of condition (13) is obviously a globally Lipschitz continuous mapping from to . Let be the unique solution of (15) with the initial condition . Construct the following Lyapunov function of the generalized Lur’e-Postnikov type [8]
(13) When the RNN model (Q) is used to find the unique minof the strictly convex quadratic optimization with imizer bound constraints, the initial point for the solution trajectory of , whereas the initial point the RNN (Q) can be any one in is required to be located in the feasible region in the RNN model (Ñ) for general quadratic optimization with bound constraints. The functional block diagram of the RNN model (Q) for solving strictly convex quadratic optimization problems with bound constraints is identical to the one shown in Fig. 2 expect that the former does not appear the saturation functions applied and that the time constants for to the initial point are all equal to . In the following, we will show that the RNN model (Q) is for actually GES for almost any scaling parameters . is nonsingular, where Theorem 5: If the matrix is the identity matrix, then the RNN model (Q) is GES. and such That is, there exist two positive constants and that for any
with
where and denote the maximal and minimal eigenvalues of the symmetric matrix arguments, respectively. It , since is symmetric and positive is noted that definite. Thus, can be well defined as above. , and Let
Then, we have
Computing the derivative of system (15), we have
along the solution
where is the solution of the RNN model (Q) with initial condition . , and the Proof: Let the nonsingular matrix . vector , the RNN Making the variable transformation model (Q) or (13) can be rewritten as the following system of : ODE in terms of (14) as its unique equiwith librium point. , then Let a new variable (14) can be reduced to a system of ODE in terms of of the form (15)
where the fact that is used in the above first inequality. It follows from the above result that
for
of
1258
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
00 2 0 9)
Fig. 3. Convergence to a minimizer of the initial point ( Example 1.
: ;
:
in in
Fig. 4. Convergence to a minimizer of the initial point (
in Example 1.
01 5 02 5) : ;
:
not in
By the theorem of differential inequalities, we obtain
and hence
Noting that and it follows from the above inequality that for
,
cond (16) is where cond is not singular, the condition number of the matrix . Since , which guarantees the well-definiteness of . Let cond and , cond then Theorem 5 is proved. Since , as a function of , is continuously differentiable, the set of its zero points has , where the set zero measure in . Thus, for almost any scaling parameters for , the condition in Theorem 5 is satisfied and hence the conclusion of GES property of the RNN model (Q) is true. From inequality (16), we know that the global exponential convergence rate of the RNN model (Q) for strictly convex quadratic optimization with bound constraints can be made . arbitrarily large by decreasing the time constant However, this may induce the RNN model (Q) more sensitive to input noise and round-off errors in analog implementation and numerical simulation.
Fig. 5. Convergence to the minimizers of 30 random initial points in Example 1.
in
VI. SIMULATION RESULTS In this section, several numerical simulation examples are presented to demonstrate the characteristics and performance of
Fig. 6.
Spatial representation of the trajectories starting from in Example 1.
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1259
the proposed RNN model (N), (Ñ), or (Q) for solving optimization problems with bound constraints. Example 1: Consider a convex quadratic optimization problem with bound constraints:
It is obvious that is a convex quadratic function in with the positive semidefinite matrix
According to Theorems 2 and 4, the RNN model (Ñ) is complete and quasiconvergent. That is, all the solution trajectories of system (Ñ) starting from the feasible region are convergent to the set of minimizers of the bound-constrained optimization problem. Moreover, all the minimizers of this bounded optimization problem can be reached via the solution trajectories of the RNN model (Ñ) starting from . and scaling In the simulation, time constants were used. As shown in Figs. 3 parameters and 4, two different initial points can converge to different minimizers of the bounded quadratic program. Note that the two initial points selected in Figs. 3 and 4 are inside and outside of , respectively. generated based on uniform Next, 30 random points in distribution are used as initial states of the solution trajectories. It can be seen from Fig. 5 that all the trajectories corresponding to the above 30 random initial points are convergent to the minimizers of the bound-constrained quadratic program, since all the objective values converge to the minimal value as time approaches infinity. To observe the global convergent behavior of the RNN model (Ñ) in , 300 random points generated uniformly in are used as the initial states of the RNN model (Ñ). The spatial representation of these trajectories is described in Fig. 6. It is evident that the set of minimizers of the optimization problem, i.e., , is an attractive set of the RNN model (Ñ). , Next, 300 random points from a superset of , are used as initial states, and find that all the corresponding trajectories starting from the inside and outside of converge to the attractive set of the RNN model (Ñ), as shown in Fig. 7. Example 2: Consider a strictly convex quadratic optimization problem with bound constraints as follows:
Fig. 7. Spatial representation of the trajectories starting from the inside and outside of in Example 1.
Fig. 8. Spatial representation of solution trajectories in the plane (x Example 2.
; x
) in
Fig. 9. Spatial representation of solution trajectories in the plane (x Example 2.
; x
) in
with the positive definite matrix
and
This example is the same as the first example in [14] where a discrete-time RNN model is employed to
1260
Fig. 10. Spatial representation of solution trajectories in the plane (x Example 2.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
; x
) in
Fig. 11. Convergence to the unique minimizer of 30 random initial points in Example 2.
solve the optimization problem. In the unconstrained case, the optimization problem has a unique minimizer , outside of and the . minimal value of The RNN model (Q) is used to find the unique minimizer of the above optimization problem with the time constant and the scaling constants . According to Theorem 5, the RNN model (Q) is GES. We select 100 random points in the feasible region as the initial states of the RNN (Q). The spatial representations of these trajectories in the , and are shown coordinate planes in Figs. 8–10, respectively. It is found that the unique minimizer of the above bound-constrained optimization problem is . Next, 30 random initial points taken uniformly in the superset as the initial states of the RNN model (Q). As illustrated in Fig. 11, all the solution trajec-
Fig. 12.
Convergence to the unique minimizer of the initial point in in Example 2.
(15 10 010) ;
;
Fig. 13. Convergence to the unique constrained minimizer of the unconstrained minimizer as the initial point not in in Example 2.
tories from these initial points approach the unique minimizer with the value of . Since of some initial points are by far larger than the values of those of other initial ones during the initial period, to illustrate clearly the long-term convergence behaviors of all trajectories instead of for the vertical in the same graph, we use axis. In Figs. 12 and 13, the trajectories of two particular initial points are shown to be both convergent to the unique minimizer . In Fig. 13, the initial point is the unconstrained minimizer . The objective value increases at first for a short period of time and then keeps decreasing and eventually converges to the unique constrained minimizer . Example 3: In this example, the optimization problem is to minimize a nonconvex objective function
LIANG AND WANG: A RECURRENT NEURAL NETWORK
1261
00 5 00 8)
Fig. 14. Convergence to a minimizer of the initial point ( in Example 3.
Fig. 15. Convergence to a minimizer of the initial point ( Example 3.
:
in
00 8 0 9)
in in
: ;
: ;
:
subject to . The minimal value of the above optimization problem is obviously . The RNN model (N) can be used to solve such an optimization problem. According to Theorems 3 and 4, the feasible region is an attractive set of the RNN model and all the trajectories starting from will converge to the set of equilibria of the RNN model (N). Through a simple computation, the second partial derivative at is . of is not convex in . So, the Thus, the objective function RNN model (N) is regular but not complete. That is, a trajectory of the RNN system (N) starting from the feasible region may converge to an equilibrium point which is not a minimizer of the above optimization problem. As shown in Figs. 14 and 15, two different initial states in converge to two different minimizers of the above optimization problem. From 20 initial random initial states taking from
Fig. 16. Convergence to the minimizers of 20 random initial points in Example 3.
Fig. 17. 3.
in
Spatial representation of the trajectories starting from in Example
uniformly the feasible region , Fig. 16 shows that they all converge to the minimizers of the above optimization problem. To observe the global dynamical behavior of the RNN model in , 1000 random initial states generated uniformly in . As illustrated in a Y-shape curve appears and is a part of the attractive equilibrium set of the RNN model (N). In the region , the objective value is larger than and hence that there is no any minimizer of the above optimization problem in . It can be seen from Fig. 17 that all the initial points in converge to the Y-shape curve. We select 10 000 random initial states based on uniform distribution in the feasible region and find that 65 initial points fail to converge to the minimizers of the above optimization being more than 10 problem with the objective value when the simulation algorithm terminates. Thus, the success ratio or the effectiveness of the RNN model (N) in finding the minimizers of the above optimization problem is 99.35% statis-
1262
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 6, NOVEMBER 2000
tically. It is worth pointing out here that the above high success ratio may be the consequence of the particular characteristics of the nonconvex objective function. In other words, the success ratio may become more or less for other chosen nonconvex objective functions. VII. CONCLUSION We have proposed a continuous-time RNN for optimizing any continuously differentiable objective function subject to bound constraints. The proposed RNN has the desirable regularity, primality, quasiconvergence, and attractivity properties. In solving general convex programs with bound constraints, all the solution trajectories of the RNN starting from the feasible region are convergent to set of the minimizers of the optimization problem. In solving strictly convex quadratic programs with bound constraints, the RNN is globally exponentially stable and can find the unique minimizer of the optimization problems. The simulation results have demonstrated the convergence behaviors and characteristics of the RNN for solving bound-constrained optimization problems. REFERENCES [1] J. J. Mor´e and G. Toraldo, “On the solution of large quadratic programming problems with bound constraints,” SIAM J. Optim., vol. 1, pp. 93–113, 1991. [2] J. J. Hopfield and D. W. Tank, “Neural computation of decision in optimization problem,” Biol. Cybern., vol. 52, pp. 141–152, 1985. [3] D. W. Tank and J. J. Hopfield, “Simple ‘neural’ optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit,” IEEE Trans. Circuits Syst., vol. CS-33, pp. 533–541, 1986. [4] M. P. Kennedy and L. O. Chua, “Neural networks for nonlinear programming,” IEEE Trans. Circuits Syst., vol. 35, pp. 554–562, 1988. [5] A. Rodríguez-Vázquez, R. Domínguer, A. Ruda, J. L. Huertas, and E. Sánchez-Sinencio, “Nonlinear switched-capacitor ‘neural’ networks for optimization problems,” IEEE Trans. Circuits Syst., vol. 37, pp. 384–397, 1990. [6] C. Y. Maa and M. Shanblatt, “Linear and quadratic programming neural network analysis,” IEEE Trans. Neural Networks, vol. 3, pp. 580–594, 1992. [7] J. Wang, “A deterministic annealing neural network for convex programming,” Neural Networks, vol. 7, pp. 629–641, 1994. [8] M. Forti and A. Tesi, “New conditions for global stability of neural networks with applications to linear and quadratic programming problems,” IEEE Trans. Circuits Syst. I, vol. 42, pp. 354–366, 1995. [9] Y. Xia and J. Wang, “A general methodology for designing globally convergent optimization neural networks,” IEEE Trans. Neural Networks, vol. 9, pp. 1331–1343, 1998. [10] A. Cichocki and R. Unbehauen, Neural Networks for Optimization and Signal Processing, U.K.: Wiley, 1993. [11] J. Wang, “Recurrent neural networks for optimization,” in Fuzzy Logic and Neural Network Handbook, C. H. Chen, Ed. New York: McGrawHill, 1996, pp. 4.1–4.35. [12] S. I. Sudharsanan and M. K. Sundareshan, “Exponential stability and a systematic synthesis of a neural network for quadratic minimization,” Neural Networks, vol. 4, pp. 599–613, 1991.
[13] A. Bouzerdoum and T. R. Pattison, “Neural network for quadratic optimization with bound constraints,” IEEE Trans. Neural Networks, vol. 4, pp. 293–303, 1993. [14] M. J. Pérez-Ilzarbe, “Convergence analysis of a discrete-time recurrent neural network to perform quadratic real optimization with bound constraints,” IEEE Trans. Neural Networks, vol. 9, pp. 1344–1351, 1998. [15] A. Greenbaum and G. H. Rodrigue, “Optimal preconditioners of a given sparsity pattern,” Comput. Sci. Numer. Math., vol. 29, pp. 610–634, 1989. [16] G. H. Golub and C. F. Van Loan, Matrix Computation, 2nd ed. Baltimore, MD: Johns Hopkins Univ. Press, 1989. [17] Z. B. Xu, G. Q. Hu, and C. P. Kwong, “Asymmetric Hopfield-type networks: Theory and applications,” Neural Networks, vol. 9, pp. 483–501, 1996. [18] B. C. Eaves, “On the basic theorem of complementarity,” Math. Programming, vol. 1, pp. 68–75, 1971. [19] M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear Programming: Theory and Algorithms, 2nd ed. New York: Wiley, 1993. [20] J. K. Hale, Ordinary Differential Equations. New York: Wiley, 1969. [21] J.-J. E. Slotine and W. Li, Applied Nonlinear Control. Englewood Cliffs, NJ: Prentice-Hall, 1991.
Xue-Bin Liang (S’99) received the B.S. and M.S. degrees in mathematics from Nankai University, Tianjin, and Fudan University, Shanghai, China, in 1988 and 1991, respectively. From 1991 to 1999, he held research and/or teaching positions in the Department of Mathematics at Anhui University, Hefei, China, the Department of Computer Science at Fudan University, Shanghai, China; the Department of Information Science at Utsunomiya University, Utsunomiya, Japan; the Department of Mechanical and Automation Engineering at Chinese University of Hong Kong; the Department of Manufacturing Engineering and Engineering Management at City University of Hong Kong; and the Department of Electrical Engineering at Arizona State University, Tempe, AZ. He is currently with the Department of Electrical and Computer Engineering, University of Delaware, Newark. His current research interests include digital and wireless communications, data communication and networking, multimedia processing and transportation, digital and statistical signal processing, and computational intelligence.
Jun Wang (S’89–M’90–SM’93) received the B.S. degree in electrical engineering and the M.S. degree in systems engineering from Dalian Institute of Technology (now renamed as Dalian University of Technology), Dalian, China. He received the Ph.D. degree in systems engineering from Case Western Reserve University, Cleveland, OH. He was an Associate Professor at the University of North Dakota, Grand Forks. Presently, he is an Associate Professor in the Department of Automation and Computer-Aided Engineering, the Chinese University of Hong Kong. He is the author or coauthor of more than 50 journal papers, nine book chapters, two edited books, and numerous conference papers. His current research interests include theory and methodology of neural networks and their engineering applications.
