Markov Chains, Entropy, and Fundamental ... - Semantic Scholar
784
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
Markov Chains, Entropy, and Fundamental Limitations in Nonlinear Stabilization Prashant G. Mehta, Umesh Vaidya, and Andrzej Banaszuk Abstract—In this paper, we propose a novel methodology for establishing fundamental limitations in nonlinear stabilization. To aid the analysis, we express the stabilization problem as control of Markov chains. Using Markov chains, we derive the limitations as certain maximum probability bounds or as positive conditional entropy of the certain signals in the feedback loop. The former is related to the infeasibility of the asymptotic stabilization in the presence of quantization and the latter to the Bode integral formula. In either cases, it is shown that uncertainty—associated here with the unstable eigenvalues of the linearization—leads to fundamental limitations. Index Terms—Ergodic theory, fundamental limitations, Markov chains, nonlinear systems, stabilization.
I. INTRODUCTION Recently, there have been several important works relating fundamental limitations in control (Bode formula) to entropy rates Hc (x) − Hc (d) =
1 2π
π
log |S(ei ω )|dω = log (a)
(1)
−π
where S(ei ω ) is the sensitivity transfer function of the feedback loop from the disturbance d to the output x, Hc (·) denotes the conditional entropy [1], [2] of random process, and a is the expansion rate with . log(a) = k log(|pk |), pk being the unstable poles of the open-loop plant (|pk | > 1). Martins and Dahleh employ methods of information theory in [3], to derive the entropy bound for the linear disturbance rejection problem. Zang and Iglesias obtain entropy estimates for the disturbance rejection problem in [4], where the open-loop plant is globally asymptotically stable and furthermore has a finite or fading memory property. It is shown in [5] that the rate of instability of the open-loop plant must be compensated by the information transmission rate over the communication channel in any stabilizing feedback. Entropy is also relevant to the study of deterministic and stochastic nonlinear dynamical systems via methods of Ergodic theory; cf. [6]. One important notion is the topological entropy that is used to estimate the growth rate of the number of distinguishable orbits on taking finer and finer partitions of the phase space. Nair et al. extend this notion to feedback systems in [7], by defining topological feedback entropy (TFE). The authors express fundamental limitation results in nonlinear stabilization as bounds on TFE. These results, taken together with results of [3]–[5], have laid a foundation for expressing limitations in control of nonlinear plants with a well-defined linearization. In this paper, we revisit the study of fundamental limitations by developing an explicit probabilistic framework for the study of deterministic control problems. The proposed methodological framework Manuscript received October 2, 2006; revised June 22, 2007. Recommended by Associate Editor A. Astolfi. The work of Prashant G. Mehta was supported by the National Science Fund (NSF) under Grant CMS 05-56352. The work of Andrzej Banaszuk was supported by the Air Force Office of Scientific Research (AFOSR) under Grant FA9550-07-C-0045. P. G. Mehta is with the Department of Mechanical Science and Engineering and the Coordinated Science Laboratory, University of Illinois, Urbana, IL 61801 USA (e-mail: [email protected]). U. Vaidya is with the Department of Electrical and Computer Engineering, Iowa State University, Ames, IA 50014 USA (e-mail: [email protected]). A. Banaszuk is with the Systems Department, United Technologies Research Center, East Hartford, CT 06108 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TAC.2008.917640
is based upon methods of ergodic theory whereby the deterministic nonlinear dynamical system is replaced by its stochastic counterpart, the so-called Perron–Frobenius (P–F) operator [8]. While the dynamical model propagates the initial condition, the P–F operator propagates uncertainty in initial condition. There are three advantages in doing so. One, it is now easier to compute stochastic quantities such as entropy relevant to the study of fundamental limitations. Two, the P–F operator is linear enabling analysis in very general settings. Finally, the stabilization problem with finite partition, or quantization, is easily considered using the discretization of the P–F operator. The novel PF-based framework represents the main contribution of our research. This paper is the first to consider measure-theoretic notions of entropy and bridging the statistical entropy notion of [3]–[5] with the dynamical systems notion of the entropy [7]. The measure theoretic notions appear to be particularly well suited for uncertainty characterization with disturbance whereby the nonlinear stabilization is interpreted as a suitable limit. Our research also establishes rigorous connections to Markov chains, thereby creating an opportunity to bridge performance limitations in control with performance limitations in other literature, such as queueing networks, where Markovian models are the norm. Discretization of the P–F operator also allows one to establish connections to the literature on control with quanitization. To do so, we introduce the probabilistic notion of q-stability. Although, it is well known that asymptotic stabilization of a linearly unstable equilibrium is infeasible in the presence of discrete quantized feedback [9], qstability provides for a more refined characterization of this result. Our main result, expressed in Theorem 6, shows that the quantized interval can at best be made stable with probability 1/a where a > 1 is the expansion rate. This bound is also shown to be control-independent and thus fundamental. Finally, we apply the proposed framework to rederive Bode type entropy estimates for the problem of nonlinear stabilization. Our results, expressed in Theorems 8 and 11, are the same as (and motivated by) results of Nair et al. [7] (although the authors there used topological notions of entropy) and the results pertaining to stabilization of Martins and Dahleh [3] for the linear case. Also, as in both these researches, the estimates are control-independent and, in fact, quite general with respect to the nature of the stabilizing control used (linear, nonlinear, quantized or even certain probabilistic controls). The outline of this paper is as follows. In Section II, we begin by describing the P–F formalism for a nonlinear dynamical system. In Section III, we formulate the nonlinear stabilization problem and in Section IV, we present the probability and entropy bounds for this problem. II. PRELIMINARIES In this paper, discrete mappings of the form xn + 1 = T (xn )
(2)
are considered. T : X → Y ⊂ Rm is assumed to be continuous and X, Y ⊂ Rm are compact sets; often X = Y . B(X) denotes the Borel σ-algebra on X and M(X) the vector space of bounded real valued measures on B(X). The P–F operator for (2) is given by
δT x (A)dµ(x) = µ(T −1 (A))
P[µ](A) =
(3)
X
where µ ∈ M(X) and A ∈ B(X) and δT x (A) is the stochastic transition function for the deterministic map (2); cf. [8]. By taking finite partitions of the compact sets X and Y , one can approximate the
0018-9286/$25.00 © 2008 IEEE Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
Example 1: Consider the closed-loop system with linear equations
infinite-dimensional P–F operator by a finite-dimensional matrix Pi j =
m(T −1 (Ej ) ∩ Di ) m(Di )
(4)
where ∪i Di = X, ∪j Ej = Y , and m is the Lebesgue measure; cf. [10]. This matrix is Markov (row stochastic) for stochastic P and subMarkov (with row sum less than equal to 1) for substochastic P. By defining an appropriate stochastic transition function [integrand in (3)], these considerations also extend to a class of randomly perturbed dynamical systems (5) xn + 1 = T (xn , dn ) where dn ∈ D models a disturbance assumed in this paper to be independent identically distributed (i.i.d.) with probability measure Ω(D); cf. [11]. Notationally, P is used to represent a finite-dimensional Markov matrix and P a continuous P–F operator. We will use the underline notation to distinguish a finite-dimensional measure µ ∈ RL (a vector) from an infinite dimensional µ ∈ M(X); the ith element of the finitedimensional vector µ is denoted as µi . We refer the reader to our paper [12] (and references therein) for additional details on the P–F formalism. The stabilization problem is considered for the closed-loop equation .
xn + 1 = T ◦ (1 + K)(xn ) = TK (xn )
(6)
where T is the plant and 1 + K is the control; 1 denotes the identity . mapping and (1 + K)(xn ) = xn + K(xn ). It is assumed that T (0) = 0 and that (1 + K)(0) = 0. The control is said to be inactive if K = 0. The control K is said to be stabilizing if the linearization of TK taken at the equilibrium 0 has eigenvalues strictly inside the unit circle. For such a case, we note that the following holds for sufficiently small ε: there is an ε0 > 0 such that, for any ε ∈ [0, ε0 ], there exists a compact set X ε with 0 ∈ X ε and volume m(X ε ) = ε (or O(ε)) such that TK : X ε → X ε . This may be proved by using a local Lyapunov function in conjunction with the Grobman–Hartman theorem [6]. III. CONTROL PROBLEM FORMULATION For the sake of exposition in this section, we consider the closedloop equation (6) on some compact set X ⊂ Rn where 0 ∈ X, where T ◦ (1 + K) : X → X, the mapping T is assumed to be well defined for all x ∈ X, and furthermore, it is assumed that (1 + K) : X → X. The assumption (1 + K) : X → X is stronger than what is needed. However, the exposition is simpler with only a single compact set X, and the scalar stabilization problem and certain multistate problems can be handled with this formulation. In order to prove the multistate stabilization result, we will extend it to handle that particular case. In addition to (6), a randomized or perturbed version of the closedloop equation (7) xn + 1 = Td ◦ (1 + Kd )(xn ) is also considered, where Td and 1 + Kd are randomly perturbed maps. Often . (8) Td (u) = T (u) + d where d is a random variable taking values from a given distribution. In fact, (7) then corresponds to a closed loop with disturbance xn + 1 = T ◦
1 + K)(xn ) + dK n
+ dTn + 1
785
(9)
state :
zn + 1 = azn + bun
output :
xn = zn + dn
control :
˜ n ) + r˜n . un = k(x
(10)
The closed-loop equation for the output is given by xn + 1 = a((1 + k)xn + rn − dn ) + dn + 1
(11)
rn . These considerations also exwhere k = (b/a)k˜ and rn = (b/a)˜ tend to the multistate case. The perturbed closed loop in (9) is thus the nonlinear generalization of the linear disturbance rejection problem with full-state feedback. With no disturbance, one recovers the nonlinear stabilization problem in (6) as the special case of (9). Denote PT and PK to be the infinite-dimensional P–F operators corresponding to mappings T and 1 + K, respectively. The propagation of measures [in M(X)] for the composition in (6) is easily verified to be given by (12) µ n + 1 = µ n PK PT where the ordering reflects the fact that controller (PK ) acts before the plant (PT ). This formalism is useful because: 1) PF operators are linear and 2) nonlinear composition of two mappings lead to linear multiplication of corresponding operators. If control is inactive (K = 0), then PK = 1, the identity operator. A similar stochastic description also exists for the composition in (7), where the P–F operators now correspond to the random mappings. Closely related to (12) is its finitedimensional approximation that is discussed next. A. Conditional Dither In order to pose a stabilization problem with respect to a finite partition, we propose the use of conditional dither that is defined with respect to the partition. Given a partition with a quantization X ={D1 , . . . , DL }
(13)
Q ={θ(1), . . . , θ(L)}
(14)
θ(i) ∈ Di being the quantized value in cell Di , conditional dither is defined using a random vector D = {∂(1), . . . , ∂(L)}
(15)
where ∂(i) represents a uniformly distributed random perturbation with support on cell Di . In particular, given z ∈ X, define a random variable y(z) = θ(|z|) + ∂(|z|)
(16)
where |z| = i, the subscript of the cell Di where z lies. Intuitively, ∂(|z|) randomizes the state z with in the cell. The variables y and z are both located in the same cell Di and contain the same information modulo the partition. We will refer to (13)–(15) as a finite partition with conditional dither. With a finite partition, the stabilization problem is posed with respect to the perturbed closed-loop equation xn + 1 = Td ◦ (1 + Kd )(xn )
(17)
where the subscript d corresponds to the conditional dither. In particular, we use D to define .
T where {dK n } and {dn } are disturbance processes, each modeled here as an i.i.d random process.
Td (u) = θ(|T (u)|) + ∂(|T (u)|) .
(1 + Kd )(u) = θ(|(1 + K)(u)|) + ∂(|(1 + K)(u)|).
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
(18)
786
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
Formally, Td and Kd approximates T and K in the limit of taking finer partitions. More precisely, Td corresponds to the stochastic transition function dm(y) (19) χD |T ( x ) | (y) pd (x, A) = m(D |T (x )| ) A where m is the Lebesgue measure representing the fact that disturbance is taken from a uniform distribution. The term m(D|T (x )| ) is used for normalization, so pd (x, ·) is a probability measure for each x. The stochastic transition function corresponds to a small random perturbation of the deterministic dynamical system T , i.e., pd (x, ·) → δT (x ) in a weak-* sense in the limit as m(Dj ) → 0 for all cells. For Td , the Markov operator is given by
Pd [µ](A) =
pd (x, A)dµ(x)
(20)
X
whose invariant measure converges to the physical measure of T as d → 0; cf. [11], [13]. The following lemma clarifies the relationship between stochastic aspects of Td and the Markov chain PT . Lemma 2: Consider a dynamical system T : X → X with a finite partition XL and an associated Markov chain PT . Using conditional dither D, consider next a randomly perturbed dynamical system Td : X → X and the associated Markov operator Pd . 1) Suppose µ ∈ M(X) and ν = Pd [µ].Then, there exists νi such that L χD i (x) dm(x). (21) νi dν(x) = m(Di ) 2) If νj =
i= 1
µ P , then ν = Pd [µ], where i i ij
χD (x) dµ i (x) = , µi dm m(Di ) L
i= 1
χD (x) dν i (x) = . νi dm m(Di ) L
i= 1
(22) 3) Suppose PT admits a unique invariant measure µ = [µ1 , . . . , µL ]. Then starting with Lebesgue a.e. set of initial conditions x0 ∈ X, the trajectory {xn } of the map Td has a stationary distribution given by Prob(xn ∈ Di ) = µi
(23)
and it is uniformly distributed in Di . Proof: We note that Td : X → X as T : Di → X and the conditional dither perturbs only within the cells of the partition. For µ ∈ M(X),
ν(A) = Pd [µ](A) =
χD |T ( x ) | (y) X
A
dm(y) dµ(x). m(D|T (x )| )
(24)
L
Now, (1/m(D|T (x )| ))χD |T ( x ) | (y) = χ (y) · χD j (T x)(1/ j=1 Dj m(Dj )) and substituting this in (24), we have ν(A) = .
L
A j=1
χD j (T x)dµ(x)
X
χD j (y) dm(y). m(Dj )
(25)
Denoting, νj = X χD j (T x)dµ(x), (21) and part 1) follows. In order to show part 2), use (22) to evaluate these coefficients explicitly
νj =
χD j (T x) X
i= 1
L
=
i= 1
L
µi
1 m(Di )
µi χD i (x)
dm(x) m(Di )
χD j (T x)χD i (x)dm(x) = X
L
µi Pi j
i= 1
(26)
as desired. From part 2), if µPT = µ, then Pd [µ] = µ, where
L
dµ/dm(x) = i = 1 µi (χD i /(m(Di )))(x). Using part 1), if Pd [µ] = µ, then µ too has this form, and thus, µPT = µ. As a result, invariant measures of PT are in one–one correspondence with invariant measures of Pd . Finally, we note that the stationary density of the random sequence {xn } is given by f (x) =
µi , m(Di )
for x ∈ Di .
(27)
The following theorem, which follows from the Lemma 2, clarifies the relationship between the asymptotic behavior of the perturbed closed loop (17) and the invariant measure of the discrete formulation PK PT . Theorem 3: Consider a partition XL with disturbance D. Let PT and PK be the sub-Markov (or Markov) chain for T and (1 + K), respectively. Suppose PK PT is a Markov chain (with row sum as unity), then: 1) the perturbed closed-loop map Td ◦ (1 + Kd ) : X → X; 2) if PK PT has a unique invariant measure µ = [µ1 , . . . , µL ], then starting with (Lebesgue) a.e. initial conditions x0 ∈ X, the trajectory {xn } for the closed loop (17) has a stationary distribution given by (28) Prob(xn ∈ Di ) = µi and it is uniform in Di . Proof: We prove the first part by contradiction. Suppose there is a set B ⊂ X of positive Lebesgue measure such that T ◦ (1 + Kd )(B) ⊂ X for some value of disturbance d. Now, because there are only finitely many cells {Di }, there must be at least one cell Dl such that m(B ∩ Dl ) > 0. So, without loss of generality, we assume B ⊂ Dl such that m(B) > 0 and T ◦ (1 + Kd )(B) ⊂ X. Next, because (1 + K) : X → X, let {Dj r }R r = 1 be the R cells such that (1 + K)(Dm ) ∩ Dj r = {φ} and m((1 + K)(Dm ) ∩ Dj r ) > 0. We . denote Dq = Dj 1 and note that the last statement implies that [PK ]l q > 0.
(29)
[PT ]q = [p1 , . . . , pL ]
(30)
We use the notation
to denote the qth row and note that because T ◦ (1 + Kd )(B) ⊂ X, we L have i = 1 pi = βq < 1. Equations (29)–(30) will be used to derive a contradiction by showing that the lth row sum of PK PT is strictly less than 1. Indeed, on multiplying PT on the right by column vector 1, one obtains (31) PT 1 = [β1 , . . . , βq , . . . , βL ] .
where βi ≤ 1 and βq < 1. Next denote the lth row [PK ]l = [α1 , . . . , αq , . . . , αL ] and αq > 0 by (29). The lth row sum is now easily seen to be L i= 1
αi βi ≤ max(βi )
α i + βq α q ≤ 1
i = q
αi + βq αq < 1.
i = q
(32) As a result, PK PT is not a Markov chain providing the contradiction we seek. This proves part 1). The part 2) of the theorem follows from the final part of Lemma 2. Assume an invariant measure µPK PT = µ and denote µPK = ν, so νPT = µ. Using Lemma 2, µPK = ν and νPT = µ so µPK PT =
L
µ where (dµ/dm)(x) = i = 1 µi χD i /(m(Di )(x)). The result then follows by using (assuming) a suitable Ergodic hypothesis for invariant measures.
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
The significance of the aforementioned theorem is just as PT was the stochastic counterpart of the randomly perturbed map Td , PK PT is the stochastic counterpart of closed loop Td ◦ (1 + Kd ) with conditional dither D. The analysis, including derivation of probability and entropy estimates, for finite Markov chains is particularly straightforward. In the limit of taking finer partitions (or letting d → 0), one recovers the stabilization problem T ◦ (1 + K). We close this section with an example. Example 4: Consider a linear expanding map zn + 1 = a(zn + un )
where ad and (1 + kd ) are perturbed versions of a and possibly nonlinear control mapping k. We define this perturbed problem wrt a partition X = {D1 , D2 , D3 } with three cells D1 = [−ε/2, ε/2], D2 = [−3ε/2, −ε/2], and D3 = [ε/2, 3ε2]. On X , the 3 × 3 subMarkov matrix is 1 1 1 2 4 4 (35) PT = 0 0 0 . 0
0
With respect to the partition X , the control objective is to “stabilize” the cell D1 containing 0. On X , denote by PK the Markov chain corresponding to control (1 + k). The Markov chain for the closedloop equation (34) is formally written as µn + 1 = µn PK PT .
(36)
Note that (36) provides a linear description of (34) irrespective of whether 1 + k is linear, nonlinear, or even stochastic map. As an example, for
PK =
1 1 1
1 2 1 PK PT = 2 1 2
0 0 0 1 4 1 4 1 4
A. Probability Bounds In order to obtain probability bounds, it is useful to extend the notion of stability in terms of invariant measures of PK PT as follows. Definition 5: Consider a closed-loop system Td ◦ (1 + Kd ) : X → X together with its unique invariant measure µ with support A ⊂ X. A subset S ⊆ A is q-stable if µ(S) = q.
(39)
A set S is stable if it is q-stable with q = 1. Intuitively, q-stability provides for a weaker notion of stabilty whereby any “typical” trajectory of the dynamical system lies in set S only a fraction (equal to q) of the time N −1 1 χS (xn ) = q. Prob(xn ∈ S) = lim N →∞ N .
(40)
k=0
[PT ]1 = [p1 , p2 , . . . , pL ]
.
is assumed to be given. By virtueof the fact that PT is a Markov or a sub-Markov matrix, pi ≥ 0 and i pi ≤ 1. Additionally, assume that the first column denoted as
(37)
[PT ]1 = [p1 , 0, . . .]
For the choice of PK , the control may be a linear gain k = −0.5, a random gain where k is a random variable chosen from set [−0.5, −1) or a nonlinear-quantization-based feedback controller. Note that ad (1 + kd ) : X → X [in (34)] for all these cases. The only invariant measure for PK PT in this example is given by [1/2, 1/4, 1/4]. This implies that asymptotically a typical trajectory {xn } lies in the cell D1 with probability 1/2, i.e., .
dynamical system where the 0 equilibrium is unstable, and furthermore, all of the eigenvalues of its linearization are outside the unit circle. For such a problem, control-independent probability and conditional entropy estimates are obtained in the following two sections, respectively.
For the stabilization problem with respect to a finite partition, suppose XL = {D1 , D2 , . . . , DL } denotes a finite partition with a matrix PT whose first row denoted as
0 0 , the closed-loop P–F is given by 0 1 4 1 4 1 4
Fig. 1 (a)Markov chain where the first row and column are given by (41) and (42), respectively. (b) Finite partition with cell D 1 containing the origin 0.
(33)
where a = 2 is the expansion rate. The perturbed closed loop is given by (34) xn + 1 = ad (1 + kd )(xn )
0
787
N −1 1 1 χD 1 (xn ) = N →∞ N 2
Prob(xn ∈ D1 ) = lim
(38)
k=0
and the cell D1 is not stable in the conventional sense.
(41)
(42)
has atmost one nonzero entry p1 . The resulting Markov chain is drawn in Fig. 1. It is a model of a dynamical system with an unstable equilibrium in cell D1 . For instance, the Markov chain for the example in Section III where T (x) = 2x has as its first row [1/2, 1/4, 1/4] and first column [1/2, 0, 0] . The control objective is to design PK to q-stabilize D1 with, say, maximal possible value of q. For this, we have a following fundamental—control-independent—result on the qstabilization: Theorem 6: Consider PT defined on a partition XL with the structure in Fig. 1 and the first row and column given in (41) and (42), respectively. Let PK denote a control Markov matrix on XL . If µ = [µ1 , . . .] is an invariant measure for the closed-loop Markov matrix PK PT , then
IV. FUNDAMENTAL LIMITATIONS
µ1 ≤ p1
In this paper, fundamental limitations are expressed within the stochastic framework, i.e., by replacing Td and a stabilizing Kd by PT and PK , respectively. It is assumed that the T is an expanding
i.e., the maximum possible value for q-stabilization of D1 is q = p1 . Suppose 1 is made p1 -stable. Then: D L p = 1; 1) i= 1 i
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
(43)
788
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
2) for every i with pi = 0, the ith row of the control [PK ]i = [1, 0, . . .]; 3) µ with µj = pj gives an invariant probability measure of the controlled markov chain PK PT . Proof: Suppose µ = µPK PT . Due to the assumption on the first column of PT (42) [PK ]1 > p1
µ1 =< µ,
D = {∂(1), . . . , ∂(L)} the conditional dither. Associated with the discrete partition, the first row and column of the sub-Markov chain PT (for the state evolution T z = b(z)) are m(DL ) 1 m(D2 ) , ,..., a aε aε
[PT ]1 =
"1
[PT ]1 =
(44)
a
!
# , 0, . . . , 0
(53)
1
where [PK ] denotes the first column of the Markov chain PK and denotes the standard inner product. If µ is a probability measure, then i µi = 1 and µ1 ≤
µi max [PK ]i 1 p1 ≤ p1 . i
xn + 1 = bd ◦ (1 + kd )xn (45)
i
This gives the desired inequality (43). Now, suppose PK is such that D1 is made p1 -stable. Using (44) together with the fact that i µi = 1, we have whenever µi = 0.
µ1 = p1 implies [PK ]i 1 = 1,
(46)
In particular, because µ1 = p1 > 0, [PK ]1 1 = 1 and because PK is a Markov matrix, its first row [PK ]1 = [1, 0, . . .]. This shows part 2) for the particular case of i = 1. By matrix multiplication, [PK PT ]1 = [p1 , p2 , . . . , pL ]. As PK PT is a Markov matrix, necessarily 1). Since µ is an invariant measure
p1 [µ1 , µ2 , . . . , µL ] × .. .
p2 × .. .
...
i
(47)
pL × = [µ1 , µ2 , . . . , µL ] .. .
(48)
µi = µ1 pi + · · · ≥ p1 pi .
As a result, µi > 0 for all i such that pi > 0. Using the condition in (46) whenever pi > 0. (50) [PK ]i = [1, 0, . . .],
whenever pi > 0.
(51)
Finally, because of part 1) and (51), it is easy to verify that µj = pj is an invariant probability measure for the Markov chain PK PT . The theorem gives limitations on: 1) maximum achievable value of q for q-stability of D1 and 2) the resulting invariant measure. Either of these are a function of only the properties of the open-loop Markov chain PT . The key assumption needed for the conclusions is the structure of PT with respect to D1 —as expressed by (42). B. Entropy Bounds First consider a scalar linear dynamical system zn + 1 = b(zn + un ) .
|a(1 + k)| < 1
(56)
which necessarily implies that (1 + k) : a(Di ) → D1 for i = 1, . . . , L. The Markov chain for the control and the closed loop are then given by 1 . PK = .. 1
0 .. . 0
1
··· 0 .. .. . . , ··· 0
a
PK PT = .. .
1 a
m (D 2 ) aε
.. . m (D 2 ) aε
··· .. . ···
m (D L ) aε
.. .
m (D L ) aε
(57) respectively, and its closed-loop invariant measure is
This shows part 2). Once again, by matrix multiplication
state :
where PK is the Markov chain constructed from the control k. With discrete Markov chains, the stabilization problem was posed as the qstabilization of cell D1 . The following lemma provides the relationship between this and the original problem. Lemma 7: Suppose un = k(xn ) be any linear stabilizing control of (52) and XL be a finite partition with conditional dither D. Then, for the closed loop (54), the cell D1 is q-stable with maximal value of q = 1/a. Proof: The condition for closed loop stability is
(49)
(54)
where: 1) k(·) is any stabilizing control and 2) the disturbance {dn } arises due to conditional dither D. The stochastic analogue of (54) is the now familiar (55) µn + 1 = µn PK PT
pi = 1. This shows part
that implies
[PK PT ]i = [p1 , p2 , . . . , pL ],
respectively. With respect to the partition, we consider the perturbed closed-loop equation
(52)
with the expansion rate a = |b| > 1. We will derive the limitations for a finite partition with cell D1 = [−ε/2, ε/2] denoting the εneighborhood of 0. For reasons that will become clear in the fol. lowing, we take X = a(D1 ) = [−a(ε/2), a(ε/2)] to be the compact set. As before, XL = {D1 , D2 , . . . , DL } denotes a finite partition and
µ=
1 m(D2 ) m(DL ) , ,..., a aε aε
! (58)
i.e., the cell D1 is 1/a-stable. Using Theorem 6, this is also the maximum value possible. Using Theorem 5, the random sequence {xn } for the perturbed closed loop (54) lies in the cell D1 with probability 1/a. The inability to q-stabilize the cell D1 for arbitrary value of q (an in particular, for q = 1) constitutes a fundamental limitation in stabilization. This depends only upon the open-loop dynamics, expansion rate a here, and is independent of the choice of the feedback control gain k. Conversely, as the lemma shows, any stabilizing control achieves the upper bound 1/a. Next, the larger the value of a, the larger the uncertainty in the state {xn } of the closed-loop system—it could be anywhere in cell a(D1 ) of length aε. This uncertainty is best expressed in terms of the entropy metric and relates to the Bode formula. In order to make this correspondence precise, we need to define entropy rate for the disturbance. One possibility is to take it to be the entropy rate for the conditional dither acting on the control map (1 + kd ) .
H(d) = H(∂(1)) = ln(ε).
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
(59)
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
This choice is dictated by our choice of cell D1 (with volume ε) and the fact that a partition of a(D1 ) \ D1 is taken to be arbitrary. In effect, the conditional dither in cell D1 determines the uncertainty of the output x and one obtains a version of Bode formula. Theorem 8: Consider the perturbed closed loop (54) with the expansion rate a > 1. For any stabilizing control gain k, the random output sequence {xn } has entropy given by Hc (xn ) = ln(a) + H(d).
(60)
Proof: Using Theorem 6 and the expression for the invariant measure µ in (58) Prob(xn ∈ D1 ) =
1 a
Prob(xn ∈ Di ) =
m(Di ) , aε
789
invariant measure of P . Then, the sequence {xn } for the closed loop (54) has entropy Hc (xn ) = −
L
µi Pi j ln(Pi j ) +
i,j = 1
L
µi Pi j ln(m(Dj ))
where m(Di ) is the volume of cell Di . If P comprises of repeated rows [P ]j = [p1 , . . . , pL ] for j = 1, . . . , L, then Hc (xn ) = −
L
pj ln
j=1
pj . m(Dj )
f (x1 ) =
µi , m(Di )
for x1 ∈ Di
where xn is uniformly distributed within each cell and it follows that the stationary probability density function (pdf) of xn is
f (x2 | x1 ) =
Pi j , m(Dj )
for x1 ∈ Di ,
=
1 , aε
for i = 2, . . . , L
for x ∈ D1
1 m(Di ) = , aεm(Di ) aε
for x ∈ X − D1
(62)
i.e., f (x) = 1/aε is the uniform distribution for x ∈ a(D1 ). Next, using (57) and a similar calculation as earlier, the conditional pdf for the stationary case is f (x2 | x1 ) = =
1 , aε
for x2 ∈ D1
m(Dj ) 1 = , aεm(Dj ) aε
for x2 ∈ X − D1
(63)
i.e., f (x2 | x1 ) = 1/aε for x2 , x1 ∈ X. Now, applying the formula for relative entropy, we have
H(x2 | x1 ) =
f (x2 | x1 ) ln(f (x2 | x1 ))dx2 dx1
f (x1 ) X
(67)
Proof: Equation (66) follows from using the formula for the relative entropy and the relationships (61)
f (x) =
(66)
i,j = 1
X
= ln(a) + ln(ε)
H(xn | xn −1 , . . . , xn −m ) = H(xn | xn −1 ) = H(x2 | x1 )
H(x2 | s1 ) = −
νi Pi j ln
i,j
Pi j m(Dj )
.
(69)
This formula will be useful later. We note that the proof of Theorem 8 did not use linearity of maps b and (1 + k). In fact, we did not even use these mappings, rather only the Markov chains PK and PT . We next use this framework for the general multistate expanding nonlinear dynamical system state: zn + 1 = T (zn + un )
(70)
where the Jacobian DT (0) has only unstable eigenvalues |λi | > 1 . for i = 1, . . . , m, and expansion rate a = |DT (0)| > 1. In order to apply the framework, let K be a stabilizing control and X be O(ε)neighborhood of 0 with the property that T ◦ (1 + K) : X → X.
(71)
Now, for the multistate case, it is not necessary that a stabilizing control (1 + K) : X → X, a condition that has been assumed so far. Denote . Y = (1 + K)(X), so (1 + K) : X → Y.
(65)
where the first equality is due to the Markov assumption and the second equality is due to the stationarity [2]. Remark 9: We assumed a finite partition for X = a(D1 ) because it represents the support of the invariant measure with conditional dither in cell D1 and a stabilizing controller. Additional cells taken from a partition over any larger [than a(D1 )] set will not change the results as the relative entropy is calculated with respect to the invariant measure. Next, the estimate holds for an arbitrary partition of a(D1 ) \ D1 . Thus, one can express the result with respect to a partition {D1 , D2 , . . .} where {D2 , . . .} is only assumed to be sufficiently fine. With conditional dither, it is more convenient to express a formula for the relative entropy directly in terms of entries of the Markov matrix as summarized in the following lemma. Lemma 10: Consider a closed loop (54) together with its associated . Markov chain P = PK PT on XL = {D1 , . . . , DL }. Let {µi } be the
(68)
Equation (67) then follows as a special case. . Denoting s1 = (1 + kd )x1 , ν to be its invariant measure, P = PT , and assuming stationarity, we also note that
(64)
where ln(ε) = H(d) is the entropy of the conditional dither in cell D1 . The proof is completed by noting that xn depends only upon xn −1 and not on its entire history, i.e., {xn } is a Markov process. For a stationary Markov process, the conditional entropy
x2 ∈ D j .
(72)
Note that 0 ∈ Y and m(Y ) = O(ε). Equations (71)–(72) imply that T : Y → X. Consider next a finite partition of Y and X denoted as YM = {E1 , . . . , EM } and XL = {D1 , . . . , DL } to write the perturbed equations as (1+Kd ) : X → Y,
Td : Y → X,
xn+1 = Td ◦ (1 + Kd )(xn ). (73) We assume D1 to be the cell containing 0. Next, express xn + 1 = Td (sn ),
where
sn = (1 + Kd )xn
(74)
represents a stabilizing control input. Since x1 → s1 → x2 form a Markov chain (75) H(x2 | x1 ) ≥ H(x2 | s1 ) because of the data processing inequality; see [2, Sec. 2.8]. The righthand side of (75) is estimated using the Markov chain formalism. In
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
790
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
particular, let µ be the invariant measure of {xn } and ν be the invariant measure of control input {sn }. In the following, P indicates the Markov chain PT for the dynamical system T . By Lemma 2 and (74), these measures are related as µj =
M
νi Pi j ,
for j = 1, . . . , L.
(76)
i= 1
L
L
for (76) P = 1, for i = By taking a summation j=1 j = 1 ij 1, . . . , M . With this notation, (69) gives the formula for H(x2 | s1 ) = −
L M
νi Pi j ln
i= 1 j = 1
Pi j m(Dj )
lim Pi j
ε →0
= lim ε →0
=
1 m(Ei )
,
χT x (Dj )dm(x)
lim q ε =
ε →0
(78)
(79)
Next, assume the control K to be strongly stabilizing the cell D1 , i.e., (1 + K) : T (D1 ) → D1 . Then, as ε → 0 Hc (xn ) = ln(a) + H(d).
(80)
Proof: Using the estimate in (78) m(T (Ei ) ∩ Dj ) Pi j 1 1 = = Qi j m(Dj ) am(Ei ) m(Dj ) am(Ei )
(81)
where Qi j ≤ 1. Substituting this in (77)
Hc (xn ) = −q ln(q ) − (1 − q ) ln
T (E i )
Hc (xn ) ≥ ln(a) + H(d).
≥ ln(a) +
(85)
ε
χy (Dj ) | DT −1 (y) | dm(y)
νi Pi j ln(Qi j ) +
!
where q ε = [PT ]1 1 ; the superscript makes the dependence on ε explicit. Using the formula (67), one obtains an entropy estimate as
i= 1 j = 1
1 − qε 1 − qε
ε
ε
1 − qε m(D2ε )
.
(86)
To prove the result, we show that
Ei
because limε →0 |DT −1 (·)| = 1/a. Since, entropy rates are computed with respect to a given partition of X, we take M = 1 and Y1 = {Y } to be a singleton and rescale volumes, so m(Y ) = ε; an estimate with a general partition Y is also provided. The entropy of the disturbance is taken to be for the dither defined wrt Y; for Y1 , H(d) = ln(ε). Theorem 11: Consider an expanding dynamical system given by (70) with a = |DT (0)|. Let K be a any stabilizing control such that DT (0)(1 + DK)(0) has eigenvalues inside the unit circle. Then, in the limit of vanishing conditional dither (ε → 0), the output sequence {xn } of the closed-loop dynamical system Td ◦ (1 + Kd ) has entropy
H(x2 | s1 ) = −
qε qε
(77)
M L
(84)
With strong stabilization, Y = D1 , X = T (D1 ), and PK is given by (57). We derive the entropy estimate with the simplest two cell partition X2 = {D1 , D2 }, where D2 = T (D1 ) \ D1 . For such a partition, the closed-loop Markov matrix is given as PK PT =
m(T (Ei ) ∩ Dj ) 1 lim a ε →0 m(Ei )
M L
H(x2 | s1 ) ≥ ln(a) + ln(ε) = ln(a) + H(d).
In the limit that ε → 0, the individual entries Pi j are related to the local expansion rate as 1 = lim ε →0 m(Ei )
The term ν ln(m(Ei )) corresponds to the entropy of the coni i ditional dither defined with respect to the partition {E1 , . . . , EM }. Specifically with M = 1
νi Pi j ln(am(Ei ))
i= 1 j = 1
νi Pi j ln(m(Ei )).
(82)
1 , a
lim
ε →0
m(D2ε ) = (a − 1). ε
(87)
Indeed, the latter equation is clear by construction, and in the ε = 0 limit, q 0 = lim ε →0
m(T −1 D1 ∩ D1 ) m(T −1 D1 ) 1 = lim = |DT −1 (0)| = ε →0 m(D1 ) m(D1 ) a
where a = |DT (0)| is the expansion rate. Since the argument relies on a type of linearization, the estimate is insensitive to the choice of partition in the limiting case. In particular, a finer partition of T (D1 ) \ D1 will also yield the same conclusion (see also Theorem 8 and Remark 9). As a result of this theorem, the stabilization in the general multistate case only leads to an inequality for the entropy estimate. With strong stabilization, one obtains an equality. Note that strong stabilization is equivalent to stabilization for the scalar case. The general multistate result was first shown with the aid of TFE in Nair et al. [7] who also showed the bound to be tight with strong stabilization. Our result is motivated by [7] and is analogous with one important difference: using TFE, the uncertainty arises due to the initial condition while using conditional dither, the uncertainty arises due to disturbance. After taking appropriate limits in each case (finer partitions or vanishing conditional dither), one obtains a bound in terms of expansion rate a due to unstable dynamics. We further remark that the notion of stabilization needed to obtain these bounds is weaker than Lyapunov stability. The control K needs to be stabilizing only in the sense that (71)–(72) are valid, i.e., the closed-loop map is positively invariant for a certain small neighborhood of 0. Finally, we stress the important role of disturbance. At each point, one is indeed solving a perturbed problem where ln(ε) denotes the entropy rate due to conditional dither. In order to interpret the ε → 0 limit, we note that the output xn is a continuous random variable. Thus, it is meaningful to talk only about change in uncertainty, and this change is reflected here as the difference between uncertainty rates of xn and dn given by the Bode formula (79) for the general case and (80) for the scalar or the strong stabilization case.
i,j
Finally, using the fact that
j
ACKNOWLEDGMENT
Pi j = 1 and
M
H(x2 | s1 ) ≥ ln(a) +
i= 1
νi ln(m(Ei )).
(83)
The authors thank G. Froyland, M. Hessel, and C. Jacobson for useful discussions. They are grateful to anonymous reviewers whose many comments including suggestion of the term “conditional dither” helped improved this paper.
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
REFERENCES [1] A. Papoulis, Probability, Random Variables, and Stochastic Processes. New York: McGraw-Hill, 1984. [2] T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley Series in Telecommunications Series). New York: Wiley, 1991. [3] N. C. Martins and M. A. Dahleh, “Fundamental limitations of performance in the presence of finite capacity feedback,” in Proc. Amer. Control Conf., 2005, vol. 1, pp. 79–86. [4] G. Zang and P. A. Iglesias, “Nonlinear extension of Bode’s integral based on an information theoretic interpretation,” Syst. Control Lett., vol. 50, pp. 11–29, 2003. [5] N. Elia, “When Bode meets Shannon: Control-oriented feedback communication schemes,” IEEE Trans. Autom. Control, vol. 49, no. 9, pp. 1477– 1488, Sep. 2004. [6] A. Katok and B. Hasselblatt, Introduction to the Modern Theory of Dynamical Systems. Cambridge, U.K.: Cambridge Univ. Press, 1995. [7] G. N. Nair, R. J. Evans, and I. M. Y. Mareels, “Topological feedback entropy and nonlinear stabilization,” IEEE Trans. Autom. Control, vol. 49, no. 9, pp. 1585–1597, Sep. 2004. [8] A. Lasota and M. C. Mackey, Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics. New York: Springer-Verlag, 1994. [9] F. Fagnani and S. Zampieri, “Stability analysis and synthesis for scalar linear systems with quantized feedback,” IEEE Trans. Autom. Control, vol. 48, no. 9, pp. 1569–1584, Sep. 2003. [10] G. Froyland, “Extracting dynamical behaviour via Markov models,” in in Nonlinear Dynamics and Statistics: Proceedings, Newton Institute, Cambridge, 1998. Boston, MA: Birkhauser, 2001, pp. 283–324. [11] Y. Kifer, Ergodic Theory of Random Transformations (Progress of Probability and Statistics Series 10). Boston, MA: Birkhauser, 1986. [12] U. Vaidya and P. G. Mehta, “Lyapunov measure for almost everywhere stability,” IEEE Trans. Autom. Control, to be published. [13] M. Dellnitz and O. Junge, “On the approximation of complicated dynamical behavior,” SIAM J. Numer. Anal., vol. 36, pp. 491–515, 1999.
791
A Necessary and Sufficient Condition for Consensus Over Random Networks Alireza Tahbaz-Salehi, Student Member, IEEE, and Ali Jadbabaie, Senior Member, IEEE Abstract—We consider the consensus problem for stochastic discretetime linear dynamical systems. The underlying graph of such systems at a given time instance is derived from a random graph process, independent of other time instances. For such a framework, we present a necessary and sufficient condition for almost sure asymptotic consensus using simple ergodicity and probabilistic arguments. This easily verifiable condition uses the spectrum of the average weight matrix. Finally, we investigate a special case for which the linear dynamical system converges to a fixed vector with probability 1. Index Terms—Consensus problem, random graphs, tail events, weak ergodicity.
I. INTRODUCTION Decentralized iterative schemes such as agreement and consensus problems have an old history [1]–[4]. Over the past few years, they have attracted a significant amount of attention in various contexts such as motion coordination of autonomous agents [5], [6], distributed computation of averages and least squares among sensors [7]–[9], and rendezvous problems [10]. In all these cases, the dynamical system under study is deterministic. More recently, there has been some interest in the stochastic variants of the problem. In [11], the authors study the linear dynamical system x(k) = Wk x(k − 1), where the weight matrices Wk are independent, identically distributed (i.i.d.) stochastic matrices. It is shown that all the entries of x(k) converge to a common value almost surely (with probability 1), if each edge of G(Wk ), the graph corresponding to matrix Wk , is chosen independently with the same probability (Erd¨os–R´enyi random graph model). A more general model is studied in [12], where the edges of G(Wk ) are directed and not necessarily independent. However, the author proves only convergence to a consensus in probability, rather than the more general notion of almost sure convergence. Moreover, the assumption in [12] is the occurrence of scrambling matrices with positive probability, which can be weakened. The purpose of this note is to provide a necessary and sufficient condition for an almost sure consensus in the linear dynamical system x(k) = Wk x(k − 1), when the weight matrices are general i.i.d. stochastic matrices. Our results contain the results of [11] and [12] as special cases. This necessary and sufficient condition is easily verifiable and only depends on the spectrum of the average weight matrix EWk . Finally, for a special case, we state a variant of a theorem in [2] that provides the asymptotic consensus value. Even though it is possible to derive our main theorem by combining results from ergodic theory of Markov chains in random environments [13]–[15], our proofs are self-contained and are only based on simple linear algebra machinery and the concept of coefficients of ergodicity, as introduced by Dobrushin [16]. Manuscript received September 8, 2006; revised January 30, 2007, June 28, 2007, and August 31, 2007. Recommended by Associate Editor F. Bullo. This work was supported in part by the Department of Defense Multidiciplinary University Research Initiative (ARO/MURI) under Grant W911NF-051-0381, in part by the Office of Naval Research Young Investigator Program (ONR/YIP) under Grant N00014-04-1-0467, and in part by the National Science Foundation under Grant ECS-0347285. The authors are with the General Robotics, Automation, Sensing and Perception (GRASP) Laboratory, Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA 19104-6228 USA (e-mail: [email protected]; [email protected]). Digital Object Identifier 10.1109/TAC.2008.917743
0018-9286/$25.00 © 2008 IEEE Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
Markov Chains, Entropy, and Fundamental Limitations in Nonlinear Stabilization Prashant G. Mehta, Umesh Vaidya, and Andrzej Banaszuk Abstract—In this paper, we propose a novel methodology for establishing fundamental limitations in nonlinear stabilization. To aid the analysis, we express the stabilization problem as control of Markov chains. Using Markov chains, we derive the limitations as certain maximum probability bounds or as positive conditional entropy of the certain signals in the feedback loop. The former is related to the infeasibility of the asymptotic stabilization in the presence of quantization and the latter to the Bode integral formula. In either cases, it is shown that uncertainty—associated here with the unstable eigenvalues of the linearization—leads to fundamental limitations. Index Terms—Ergodic theory, fundamental limitations, Markov chains, nonlinear systems, stabilization.
I. INTRODUCTION Recently, there have been several important works relating fundamental limitations in control (Bode formula) to entropy rates Hc (x) − Hc (d) =
1 2π
π
log |S(ei ω )|dω = log (a)
(1)
−π
where S(ei ω ) is the sensitivity transfer function of the feedback loop from the disturbance d to the output x, Hc (·) denotes the conditional entropy [1], [2] of random process, and a is the expansion rate with . log(a) = k log(|pk |), pk being the unstable poles of the open-loop plant (|pk | > 1). Martins and Dahleh employ methods of information theory in [3], to derive the entropy bound for the linear disturbance rejection problem. Zang and Iglesias obtain entropy estimates for the disturbance rejection problem in [4], where the open-loop plant is globally asymptotically stable and furthermore has a finite or fading memory property. It is shown in [5] that the rate of instability of the open-loop plant must be compensated by the information transmission rate over the communication channel in any stabilizing feedback. Entropy is also relevant to the study of deterministic and stochastic nonlinear dynamical systems via methods of Ergodic theory; cf. [6]. One important notion is the topological entropy that is used to estimate the growth rate of the number of distinguishable orbits on taking finer and finer partitions of the phase space. Nair et al. extend this notion to feedback systems in [7], by defining topological feedback entropy (TFE). The authors express fundamental limitation results in nonlinear stabilization as bounds on TFE. These results, taken together with results of [3]–[5], have laid a foundation for expressing limitations in control of nonlinear plants with a well-defined linearization. In this paper, we revisit the study of fundamental limitations by developing an explicit probabilistic framework for the study of deterministic control problems. The proposed methodological framework Manuscript received October 2, 2006; revised June 22, 2007. Recommended by Associate Editor A. Astolfi. The work of Prashant G. Mehta was supported by the National Science Fund (NSF) under Grant CMS 05-56352. The work of Andrzej Banaszuk was supported by the Air Force Office of Scientific Research (AFOSR) under Grant FA9550-07-C-0045. P. G. Mehta is with the Department of Mechanical Science and Engineering and the Coordinated Science Laboratory, University of Illinois, Urbana, IL 61801 USA (e-mail: [email protected]). U. Vaidya is with the Department of Electrical and Computer Engineering, Iowa State University, Ames, IA 50014 USA (e-mail: [email protected]). A. Banaszuk is with the Systems Department, United Technologies Research Center, East Hartford, CT 06108 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TAC.2008.917640
is based upon methods of ergodic theory whereby the deterministic nonlinear dynamical system is replaced by its stochastic counterpart, the so-called Perron–Frobenius (P–F) operator [8]. While the dynamical model propagates the initial condition, the P–F operator propagates uncertainty in initial condition. There are three advantages in doing so. One, it is now easier to compute stochastic quantities such as entropy relevant to the study of fundamental limitations. Two, the P–F operator is linear enabling analysis in very general settings. Finally, the stabilization problem with finite partition, or quantization, is easily considered using the discretization of the P–F operator. The novel PF-based framework represents the main contribution of our research. This paper is the first to consider measure-theoretic notions of entropy and bridging the statistical entropy notion of [3]–[5] with the dynamical systems notion of the entropy [7]. The measure theoretic notions appear to be particularly well suited for uncertainty characterization with disturbance whereby the nonlinear stabilization is interpreted as a suitable limit. Our research also establishes rigorous connections to Markov chains, thereby creating an opportunity to bridge performance limitations in control with performance limitations in other literature, such as queueing networks, where Markovian models are the norm. Discretization of the P–F operator also allows one to establish connections to the literature on control with quanitization. To do so, we introduce the probabilistic notion of q-stability. Although, it is well known that asymptotic stabilization of a linearly unstable equilibrium is infeasible in the presence of discrete quantized feedback [9], qstability provides for a more refined characterization of this result. Our main result, expressed in Theorem 6, shows that the quantized interval can at best be made stable with probability 1/a where a > 1 is the expansion rate. This bound is also shown to be control-independent and thus fundamental. Finally, we apply the proposed framework to rederive Bode type entropy estimates for the problem of nonlinear stabilization. Our results, expressed in Theorems 8 and 11, are the same as (and motivated by) results of Nair et al. [7] (although the authors there used topological notions of entropy) and the results pertaining to stabilization of Martins and Dahleh [3] for the linear case. Also, as in both these researches, the estimates are control-independent and, in fact, quite general with respect to the nature of the stabilizing control used (linear, nonlinear, quantized or even certain probabilistic controls). The outline of this paper is as follows. In Section II, we begin by describing the P–F formalism for a nonlinear dynamical system. In Section III, we formulate the nonlinear stabilization problem and in Section IV, we present the probability and entropy bounds for this problem. II. PRELIMINARIES In this paper, discrete mappings of the form xn + 1 = T (xn )
(2)
are considered. T : X → Y ⊂ Rm is assumed to be continuous and X, Y ⊂ Rm are compact sets; often X = Y . B(X) denotes the Borel σ-algebra on X and M(X) the vector space of bounded real valued measures on B(X). The P–F operator for (2) is given by
δT x (A)dµ(x) = µ(T −1 (A))
P[µ](A) =
(3)
X
where µ ∈ M(X) and A ∈ B(X) and δT x (A) is the stochastic transition function for the deterministic map (2); cf. [8]. By taking finite partitions of the compact sets X and Y , one can approximate the
0018-9286/$25.00 © 2008 IEEE Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
Example 1: Consider the closed-loop system with linear equations
infinite-dimensional P–F operator by a finite-dimensional matrix Pi j =
m(T −1 (Ej ) ∩ Di ) m(Di )
(4)
where ∪i Di = X, ∪j Ej = Y , and m is the Lebesgue measure; cf. [10]. This matrix is Markov (row stochastic) for stochastic P and subMarkov (with row sum less than equal to 1) for substochastic P. By defining an appropriate stochastic transition function [integrand in (3)], these considerations also extend to a class of randomly perturbed dynamical systems (5) xn + 1 = T (xn , dn ) where dn ∈ D models a disturbance assumed in this paper to be independent identically distributed (i.i.d.) with probability measure Ω(D); cf. [11]. Notationally, P is used to represent a finite-dimensional Markov matrix and P a continuous P–F operator. We will use the underline notation to distinguish a finite-dimensional measure µ ∈ RL (a vector) from an infinite dimensional µ ∈ M(X); the ith element of the finitedimensional vector µ is denoted as µi . We refer the reader to our paper [12] (and references therein) for additional details on the P–F formalism. The stabilization problem is considered for the closed-loop equation .
xn + 1 = T ◦ (1 + K)(xn ) = TK (xn )
(6)
where T is the plant and 1 + K is the control; 1 denotes the identity . mapping and (1 + K)(xn ) = xn + K(xn ). It is assumed that T (0) = 0 and that (1 + K)(0) = 0. The control is said to be inactive if K = 0. The control K is said to be stabilizing if the linearization of TK taken at the equilibrium 0 has eigenvalues strictly inside the unit circle. For such a case, we note that the following holds for sufficiently small ε: there is an ε0 > 0 such that, for any ε ∈ [0, ε0 ], there exists a compact set X ε with 0 ∈ X ε and volume m(X ε ) = ε (or O(ε)) such that TK : X ε → X ε . This may be proved by using a local Lyapunov function in conjunction with the Grobman–Hartman theorem [6]. III. CONTROL PROBLEM FORMULATION For the sake of exposition in this section, we consider the closedloop equation (6) on some compact set X ⊂ Rn where 0 ∈ X, where T ◦ (1 + K) : X → X, the mapping T is assumed to be well defined for all x ∈ X, and furthermore, it is assumed that (1 + K) : X → X. The assumption (1 + K) : X → X is stronger than what is needed. However, the exposition is simpler with only a single compact set X, and the scalar stabilization problem and certain multistate problems can be handled with this formulation. In order to prove the multistate stabilization result, we will extend it to handle that particular case. In addition to (6), a randomized or perturbed version of the closedloop equation (7) xn + 1 = Td ◦ (1 + Kd )(xn ) is also considered, where Td and 1 + Kd are randomly perturbed maps. Often . (8) Td (u) = T (u) + d where d is a random variable taking values from a given distribution. In fact, (7) then corresponds to a closed loop with disturbance xn + 1 = T ◦
1 + K)(xn ) + dK n
+ dTn + 1
785
(9)
state :
zn + 1 = azn + bun
output :
xn = zn + dn
control :
˜ n ) + r˜n . un = k(x
(10)
The closed-loop equation for the output is given by xn + 1 = a((1 + k)xn + rn − dn ) + dn + 1
(11)
rn . These considerations also exwhere k = (b/a)k˜ and rn = (b/a)˜ tend to the multistate case. The perturbed closed loop in (9) is thus the nonlinear generalization of the linear disturbance rejection problem with full-state feedback. With no disturbance, one recovers the nonlinear stabilization problem in (6) as the special case of (9). Denote PT and PK to be the infinite-dimensional P–F operators corresponding to mappings T and 1 + K, respectively. The propagation of measures [in M(X)] for the composition in (6) is easily verified to be given by (12) µ n + 1 = µ n PK PT where the ordering reflects the fact that controller (PK ) acts before the plant (PT ). This formalism is useful because: 1) PF operators are linear and 2) nonlinear composition of two mappings lead to linear multiplication of corresponding operators. If control is inactive (K = 0), then PK = 1, the identity operator. A similar stochastic description also exists for the composition in (7), where the P–F operators now correspond to the random mappings. Closely related to (12) is its finitedimensional approximation that is discussed next. A. Conditional Dither In order to pose a stabilization problem with respect to a finite partition, we propose the use of conditional dither that is defined with respect to the partition. Given a partition with a quantization X ={D1 , . . . , DL }
(13)
Q ={θ(1), . . . , θ(L)}
(14)
θ(i) ∈ Di being the quantized value in cell Di , conditional dither is defined using a random vector D = {∂(1), . . . , ∂(L)}
(15)
where ∂(i) represents a uniformly distributed random perturbation with support on cell Di . In particular, given z ∈ X, define a random variable y(z) = θ(|z|) + ∂(|z|)
(16)
where |z| = i, the subscript of the cell Di where z lies. Intuitively, ∂(|z|) randomizes the state z with in the cell. The variables y and z are both located in the same cell Di and contain the same information modulo the partition. We will refer to (13)–(15) as a finite partition with conditional dither. With a finite partition, the stabilization problem is posed with respect to the perturbed closed-loop equation xn + 1 = Td ◦ (1 + Kd )(xn )
(17)
where the subscript d corresponds to the conditional dither. In particular, we use D to define .
T where {dK n } and {dn } are disturbance processes, each modeled here as an i.i.d random process.
Td (u) = θ(|T (u)|) + ∂(|T (u)|) .
(1 + Kd )(u) = θ(|(1 + K)(u)|) + ∂(|(1 + K)(u)|).
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
(18)
786
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
Formally, Td and Kd approximates T and K in the limit of taking finer partitions. More precisely, Td corresponds to the stochastic transition function dm(y) (19) χD |T ( x ) | (y) pd (x, A) = m(D |T (x )| ) A where m is the Lebesgue measure representing the fact that disturbance is taken from a uniform distribution. The term m(D|T (x )| ) is used for normalization, so pd (x, ·) is a probability measure for each x. The stochastic transition function corresponds to a small random perturbation of the deterministic dynamical system T , i.e., pd (x, ·) → δT (x ) in a weak-* sense in the limit as m(Dj ) → 0 for all cells. For Td , the Markov operator is given by
Pd [µ](A) =
pd (x, A)dµ(x)
(20)
X
whose invariant measure converges to the physical measure of T as d → 0; cf. [11], [13]. The following lemma clarifies the relationship between stochastic aspects of Td and the Markov chain PT . Lemma 2: Consider a dynamical system T : X → X with a finite partition XL and an associated Markov chain PT . Using conditional dither D, consider next a randomly perturbed dynamical system Td : X → X and the associated Markov operator Pd . 1) Suppose µ ∈ M(X) and ν = Pd [µ].Then, there exists νi such that L χD i (x) dm(x). (21) νi dν(x) = m(Di ) 2) If νj =
i= 1
µ P , then ν = Pd [µ], where i i ij
χD (x) dµ i (x) = , µi dm m(Di ) L
i= 1
χD (x) dν i (x) = . νi dm m(Di ) L
i= 1
(22) 3) Suppose PT admits a unique invariant measure µ = [µ1 , . . . , µL ]. Then starting with Lebesgue a.e. set of initial conditions x0 ∈ X, the trajectory {xn } of the map Td has a stationary distribution given by Prob(xn ∈ Di ) = µi
(23)
and it is uniformly distributed in Di . Proof: We note that Td : X → X as T : Di → X and the conditional dither perturbs only within the cells of the partition. For µ ∈ M(X),
ν(A) = Pd [µ](A) =
χD |T ( x ) | (y) X
A
dm(y) dµ(x). m(D|T (x )| )
(24)
L
Now, (1/m(D|T (x )| ))χD |T ( x ) | (y) = χ (y) · χD j (T x)(1/ j=1 Dj m(Dj )) and substituting this in (24), we have ν(A) = .
L
A j=1
χD j (T x)dµ(x)
X
χD j (y) dm(y). m(Dj )
(25)
Denoting, νj = X χD j (T x)dµ(x), (21) and part 1) follows. In order to show part 2), use (22) to evaluate these coefficients explicitly
νj =
χD j (T x) X
i= 1
L
=
i= 1
L
µi
1 m(Di )
µi χD i (x)
dm(x) m(Di )
χD j (T x)χD i (x)dm(x) = X
L
µi Pi j
i= 1
(26)
as desired. From part 2), if µPT = µ, then Pd [µ] = µ, where
L
dµ/dm(x) = i = 1 µi (χD i /(m(Di )))(x). Using part 1), if Pd [µ] = µ, then µ too has this form, and thus, µPT = µ. As a result, invariant measures of PT are in one–one correspondence with invariant measures of Pd . Finally, we note that the stationary density of the random sequence {xn } is given by f (x) =
µi , m(Di )
for x ∈ Di .
(27)
The following theorem, which follows from the Lemma 2, clarifies the relationship between the asymptotic behavior of the perturbed closed loop (17) and the invariant measure of the discrete formulation PK PT . Theorem 3: Consider a partition XL with disturbance D. Let PT and PK be the sub-Markov (or Markov) chain for T and (1 + K), respectively. Suppose PK PT is a Markov chain (with row sum as unity), then: 1) the perturbed closed-loop map Td ◦ (1 + Kd ) : X → X; 2) if PK PT has a unique invariant measure µ = [µ1 , . . . , µL ], then starting with (Lebesgue) a.e. initial conditions x0 ∈ X, the trajectory {xn } for the closed loop (17) has a stationary distribution given by (28) Prob(xn ∈ Di ) = µi and it is uniform in Di . Proof: We prove the first part by contradiction. Suppose there is a set B ⊂ X of positive Lebesgue measure such that T ◦ (1 + Kd )(B) ⊂ X for some value of disturbance d. Now, because there are only finitely many cells {Di }, there must be at least one cell Dl such that m(B ∩ Dl ) > 0. So, without loss of generality, we assume B ⊂ Dl such that m(B) > 0 and T ◦ (1 + Kd )(B) ⊂ X. Next, because (1 + K) : X → X, let {Dj r }R r = 1 be the R cells such that (1 + K)(Dm ) ∩ Dj r = {φ} and m((1 + K)(Dm ) ∩ Dj r ) > 0. We . denote Dq = Dj 1 and note that the last statement implies that [PK ]l q > 0.
(29)
[PT ]q = [p1 , . . . , pL ]
(30)
We use the notation
to denote the qth row and note that because T ◦ (1 + Kd )(B) ⊂ X, we L have i = 1 pi = βq < 1. Equations (29)–(30) will be used to derive a contradiction by showing that the lth row sum of PK PT is strictly less than 1. Indeed, on multiplying PT on the right by column vector 1, one obtains (31) PT 1 = [β1 , . . . , βq , . . . , βL ] .
where βi ≤ 1 and βq < 1. Next denote the lth row [PK ]l = [α1 , . . . , αq , . . . , αL ] and αq > 0 by (29). The lth row sum is now easily seen to be L i= 1
αi βi ≤ max(βi )
α i + βq α q ≤ 1
i = q
αi + βq αq < 1.
i = q
(32) As a result, PK PT is not a Markov chain providing the contradiction we seek. This proves part 1). The part 2) of the theorem follows from the final part of Lemma 2. Assume an invariant measure µPK PT = µ and denote µPK = ν, so νPT = µ. Using Lemma 2, µPK = ν and νPT = µ so µPK PT =
L
µ where (dµ/dm)(x) = i = 1 µi χD i /(m(Di )(x)). The result then follows by using (assuming) a suitable Ergodic hypothesis for invariant measures.
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
The significance of the aforementioned theorem is just as PT was the stochastic counterpart of the randomly perturbed map Td , PK PT is the stochastic counterpart of closed loop Td ◦ (1 + Kd ) with conditional dither D. The analysis, including derivation of probability and entropy estimates, for finite Markov chains is particularly straightforward. In the limit of taking finer partitions (or letting d → 0), one recovers the stabilization problem T ◦ (1 + K). We close this section with an example. Example 4: Consider a linear expanding map zn + 1 = a(zn + un )
where ad and (1 + kd ) are perturbed versions of a and possibly nonlinear control mapping k. We define this perturbed problem wrt a partition X = {D1 , D2 , D3 } with three cells D1 = [−ε/2, ε/2], D2 = [−3ε/2, −ε/2], and D3 = [ε/2, 3ε2]. On X , the 3 × 3 subMarkov matrix is 1 1 1 2 4 4 (35) PT = 0 0 0 . 0
0
With respect to the partition X , the control objective is to “stabilize” the cell D1 containing 0. On X , denote by PK the Markov chain corresponding to control (1 + k). The Markov chain for the closedloop equation (34) is formally written as µn + 1 = µn PK PT .
(36)
Note that (36) provides a linear description of (34) irrespective of whether 1 + k is linear, nonlinear, or even stochastic map. As an example, for
PK =
1 1 1
1 2 1 PK PT = 2 1 2
0 0 0 1 4 1 4 1 4
A. Probability Bounds In order to obtain probability bounds, it is useful to extend the notion of stability in terms of invariant measures of PK PT as follows. Definition 5: Consider a closed-loop system Td ◦ (1 + Kd ) : X → X together with its unique invariant measure µ with support A ⊂ X. A subset S ⊆ A is q-stable if µ(S) = q.
(39)
A set S is stable if it is q-stable with q = 1. Intuitively, q-stability provides for a weaker notion of stabilty whereby any “typical” trajectory of the dynamical system lies in set S only a fraction (equal to q) of the time N −1 1 χS (xn ) = q. Prob(xn ∈ S) = lim N →∞ N .
(40)
k=0
[PT ]1 = [p1 , p2 , . . . , pL ]
.
is assumed to be given. By virtueof the fact that PT is a Markov or a sub-Markov matrix, pi ≥ 0 and i pi ≤ 1. Additionally, assume that the first column denoted as
(37)
[PT ]1 = [p1 , 0, . . .]
For the choice of PK , the control may be a linear gain k = −0.5, a random gain where k is a random variable chosen from set [−0.5, −1) or a nonlinear-quantization-based feedback controller. Note that ad (1 + kd ) : X → X [in (34)] for all these cases. The only invariant measure for PK PT in this example is given by [1/2, 1/4, 1/4]. This implies that asymptotically a typical trajectory {xn } lies in the cell D1 with probability 1/2, i.e., .
dynamical system where the 0 equilibrium is unstable, and furthermore, all of the eigenvalues of its linearization are outside the unit circle. For such a problem, control-independent probability and conditional entropy estimates are obtained in the following two sections, respectively.
For the stabilization problem with respect to a finite partition, suppose XL = {D1 , D2 , . . . , DL } denotes a finite partition with a matrix PT whose first row denoted as
0 0 , the closed-loop P–F is given by 0 1 4 1 4 1 4
Fig. 1 (a)Markov chain where the first row and column are given by (41) and (42), respectively. (b) Finite partition with cell D 1 containing the origin 0.
(33)
where a = 2 is the expansion rate. The perturbed closed loop is given by (34) xn + 1 = ad (1 + kd )(xn )
0
787
N −1 1 1 χD 1 (xn ) = N →∞ N 2
Prob(xn ∈ D1 ) = lim
(38)
k=0
and the cell D1 is not stable in the conventional sense.
(41)
(42)
has atmost one nonzero entry p1 . The resulting Markov chain is drawn in Fig. 1. It is a model of a dynamical system with an unstable equilibrium in cell D1 . For instance, the Markov chain for the example in Section III where T (x) = 2x has as its first row [1/2, 1/4, 1/4] and first column [1/2, 0, 0] . The control objective is to design PK to q-stabilize D1 with, say, maximal possible value of q. For this, we have a following fundamental—control-independent—result on the qstabilization: Theorem 6: Consider PT defined on a partition XL with the structure in Fig. 1 and the first row and column given in (41) and (42), respectively. Let PK denote a control Markov matrix on XL . If µ = [µ1 , . . .] is an invariant measure for the closed-loop Markov matrix PK PT , then
IV. FUNDAMENTAL LIMITATIONS
µ1 ≤ p1
In this paper, fundamental limitations are expressed within the stochastic framework, i.e., by replacing Td and a stabilizing Kd by PT and PK , respectively. It is assumed that the T is an expanding
i.e., the maximum possible value for q-stabilization of D1 is q = p1 . Suppose 1 is made p1 -stable. Then: D L p = 1; 1) i= 1 i
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
(43)
788
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
2) for every i with pi = 0, the ith row of the control [PK ]i = [1, 0, . . .]; 3) µ with µj = pj gives an invariant probability measure of the controlled markov chain PK PT . Proof: Suppose µ = µPK PT . Due to the assumption on the first column of PT (42) [PK ]1 > p1
µ1 =< µ,
D = {∂(1), . . . , ∂(L)} the conditional dither. Associated with the discrete partition, the first row and column of the sub-Markov chain PT (for the state evolution T z = b(z)) are m(DL ) 1 m(D2 ) , ,..., a aε aε
[PT ]1 =
"1
[PT ]1 =
(44)
a
!
# , 0, . . . , 0
(53)
1
where [PK ] denotes the first column of the Markov chain PK and denotes the standard inner product. If µ is a probability measure, then i µi = 1 and µ1 ≤
µi max [PK ]i 1 p1 ≤ p1 . i
xn + 1 = bd ◦ (1 + kd )xn (45)
i
This gives the desired inequality (43). Now, suppose PK is such that D1 is made p1 -stable. Using (44) together with the fact that i µi = 1, we have whenever µi = 0.
µ1 = p1 implies [PK ]i 1 = 1,
(46)
In particular, because µ1 = p1 > 0, [PK ]1 1 = 1 and because PK is a Markov matrix, its first row [PK ]1 = [1, 0, . . .]. This shows part 2) for the particular case of i = 1. By matrix multiplication, [PK PT ]1 = [p1 , p2 , . . . , pL ]. As PK PT is a Markov matrix, necessarily 1). Since µ is an invariant measure
p1 [µ1 , µ2 , . . . , µL ] × .. .
p2 × .. .
...
i
(47)
pL × = [µ1 , µ2 , . . . , µL ] .. .
(48)
µi = µ1 pi + · · · ≥ p1 pi .
As a result, µi > 0 for all i such that pi > 0. Using the condition in (46) whenever pi > 0. (50) [PK ]i = [1, 0, . . .],
whenever pi > 0.
(51)
Finally, because of part 1) and (51), it is easy to verify that µj = pj is an invariant probability measure for the Markov chain PK PT . The theorem gives limitations on: 1) maximum achievable value of q for q-stability of D1 and 2) the resulting invariant measure. Either of these are a function of only the properties of the open-loop Markov chain PT . The key assumption needed for the conclusions is the structure of PT with respect to D1 —as expressed by (42). B. Entropy Bounds First consider a scalar linear dynamical system zn + 1 = b(zn + un ) .
|a(1 + k)| < 1
(56)
which necessarily implies that (1 + k) : a(Di ) → D1 for i = 1, . . . , L. The Markov chain for the control and the closed loop are then given by 1 . PK = .. 1
0 .. . 0
1
··· 0 .. .. . . , ··· 0
a
PK PT = .. .
1 a
m (D 2 ) aε
.. . m (D 2 ) aε
··· .. . ···
m (D L ) aε
.. .
m (D L ) aε
(57) respectively, and its closed-loop invariant measure is
This shows part 2). Once again, by matrix multiplication
state :
where PK is the Markov chain constructed from the control k. With discrete Markov chains, the stabilization problem was posed as the qstabilization of cell D1 . The following lemma provides the relationship between this and the original problem. Lemma 7: Suppose un = k(xn ) be any linear stabilizing control of (52) and XL be a finite partition with conditional dither D. Then, for the closed loop (54), the cell D1 is q-stable with maximal value of q = 1/a. Proof: The condition for closed loop stability is
(49)
(54)
where: 1) k(·) is any stabilizing control and 2) the disturbance {dn } arises due to conditional dither D. The stochastic analogue of (54) is the now familiar (55) µn + 1 = µn PK PT
pi = 1. This shows part
that implies
[PK PT ]i = [p1 , p2 , . . . , pL ],
respectively. With respect to the partition, we consider the perturbed closed-loop equation
(52)
with the expansion rate a = |b| > 1. We will derive the limitations for a finite partition with cell D1 = [−ε/2, ε/2] denoting the εneighborhood of 0. For reasons that will become clear in the fol. lowing, we take X = a(D1 ) = [−a(ε/2), a(ε/2)] to be the compact set. As before, XL = {D1 , D2 , . . . , DL } denotes a finite partition and
µ=
1 m(D2 ) m(DL ) , ,..., a aε aε
! (58)
i.e., the cell D1 is 1/a-stable. Using Theorem 6, this is also the maximum value possible. Using Theorem 5, the random sequence {xn } for the perturbed closed loop (54) lies in the cell D1 with probability 1/a. The inability to q-stabilize the cell D1 for arbitrary value of q (an in particular, for q = 1) constitutes a fundamental limitation in stabilization. This depends only upon the open-loop dynamics, expansion rate a here, and is independent of the choice of the feedback control gain k. Conversely, as the lemma shows, any stabilizing control achieves the upper bound 1/a. Next, the larger the value of a, the larger the uncertainty in the state {xn } of the closed-loop system—it could be anywhere in cell a(D1 ) of length aε. This uncertainty is best expressed in terms of the entropy metric and relates to the Bode formula. In order to make this correspondence precise, we need to define entropy rate for the disturbance. One possibility is to take it to be the entropy rate for the conditional dither acting on the control map (1 + kd ) .
H(d) = H(∂(1)) = ln(ε).
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
(59)
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
This choice is dictated by our choice of cell D1 (with volume ε) and the fact that a partition of a(D1 ) \ D1 is taken to be arbitrary. In effect, the conditional dither in cell D1 determines the uncertainty of the output x and one obtains a version of Bode formula. Theorem 8: Consider the perturbed closed loop (54) with the expansion rate a > 1. For any stabilizing control gain k, the random output sequence {xn } has entropy given by Hc (xn ) = ln(a) + H(d).
(60)
Proof: Using Theorem 6 and the expression for the invariant measure µ in (58) Prob(xn ∈ D1 ) =
1 a
Prob(xn ∈ Di ) =
m(Di ) , aε
789
invariant measure of P . Then, the sequence {xn } for the closed loop (54) has entropy Hc (xn ) = −
L
µi Pi j ln(Pi j ) +
i,j = 1
L
µi Pi j ln(m(Dj ))
where m(Di ) is the volume of cell Di . If P comprises of repeated rows [P ]j = [p1 , . . . , pL ] for j = 1, . . . , L, then Hc (xn ) = −
L
pj ln
j=1
pj . m(Dj )
f (x1 ) =
µi , m(Di )
for x1 ∈ Di
where xn is uniformly distributed within each cell and it follows that the stationary probability density function (pdf) of xn is
f (x2 | x1 ) =
Pi j , m(Dj )
for x1 ∈ Di ,
=
1 , aε
for i = 2, . . . , L
for x ∈ D1
1 m(Di ) = , aεm(Di ) aε
for x ∈ X − D1
(62)
i.e., f (x) = 1/aε is the uniform distribution for x ∈ a(D1 ). Next, using (57) and a similar calculation as earlier, the conditional pdf for the stationary case is f (x2 | x1 ) = =
1 , aε
for x2 ∈ D1
m(Dj ) 1 = , aεm(Dj ) aε
for x2 ∈ X − D1
(63)
i.e., f (x2 | x1 ) = 1/aε for x2 , x1 ∈ X. Now, applying the formula for relative entropy, we have
H(x2 | x1 ) =
f (x2 | x1 ) ln(f (x2 | x1 ))dx2 dx1
f (x1 ) X
(67)
Proof: Equation (66) follows from using the formula for the relative entropy and the relationships (61)
f (x) =
(66)
i,j = 1
X
= ln(a) + ln(ε)
H(xn | xn −1 , . . . , xn −m ) = H(xn | xn −1 ) = H(x2 | x1 )
H(x2 | s1 ) = −
νi Pi j ln
i,j
Pi j m(Dj )
.
(69)
This formula will be useful later. We note that the proof of Theorem 8 did not use linearity of maps b and (1 + k). In fact, we did not even use these mappings, rather only the Markov chains PK and PT . We next use this framework for the general multistate expanding nonlinear dynamical system state: zn + 1 = T (zn + un )
(70)
where the Jacobian DT (0) has only unstable eigenvalues |λi | > 1 . for i = 1, . . . , m, and expansion rate a = |DT (0)| > 1. In order to apply the framework, let K be a stabilizing control and X be O(ε)neighborhood of 0 with the property that T ◦ (1 + K) : X → X.
(71)
Now, for the multistate case, it is not necessary that a stabilizing control (1 + K) : X → X, a condition that has been assumed so far. Denote . Y = (1 + K)(X), so (1 + K) : X → Y.
(65)
where the first equality is due to the Markov assumption and the second equality is due to the stationarity [2]. Remark 9: We assumed a finite partition for X = a(D1 ) because it represents the support of the invariant measure with conditional dither in cell D1 and a stabilizing controller. Additional cells taken from a partition over any larger [than a(D1 )] set will not change the results as the relative entropy is calculated with respect to the invariant measure. Next, the estimate holds for an arbitrary partition of a(D1 ) \ D1 . Thus, one can express the result with respect to a partition {D1 , D2 , . . .} where {D2 , . . .} is only assumed to be sufficiently fine. With conditional dither, it is more convenient to express a formula for the relative entropy directly in terms of entries of the Markov matrix as summarized in the following lemma. Lemma 10: Consider a closed loop (54) together with its associated . Markov chain P = PK PT on XL = {D1 , . . . , DL }. Let {µi } be the
(68)
Equation (67) then follows as a special case. . Denoting s1 = (1 + kd )x1 , ν to be its invariant measure, P = PT , and assuming stationarity, we also note that
(64)
where ln(ε) = H(d) is the entropy of the conditional dither in cell D1 . The proof is completed by noting that xn depends only upon xn −1 and not on its entire history, i.e., {xn } is a Markov process. For a stationary Markov process, the conditional entropy
x2 ∈ D j .
(72)
Note that 0 ∈ Y and m(Y ) = O(ε). Equations (71)–(72) imply that T : Y → X. Consider next a finite partition of Y and X denoted as YM = {E1 , . . . , EM } and XL = {D1 , . . . , DL } to write the perturbed equations as (1+Kd ) : X → Y,
Td : Y → X,
xn+1 = Td ◦ (1 + Kd )(xn ). (73) We assume D1 to be the cell containing 0. Next, express xn + 1 = Td (sn ),
where
sn = (1 + Kd )xn
(74)
represents a stabilizing control input. Since x1 → s1 → x2 form a Markov chain (75) H(x2 | x1 ) ≥ H(x2 | s1 ) because of the data processing inequality; see [2, Sec. 2.8]. The righthand side of (75) is estimated using the Markov chain formalism. In
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
790
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
particular, let µ be the invariant measure of {xn } and ν be the invariant measure of control input {sn }. In the following, P indicates the Markov chain PT for the dynamical system T . By Lemma 2 and (74), these measures are related as µj =
M
νi Pi j ,
for j = 1, . . . , L.
(76)
i= 1
L
L
for (76) P = 1, for i = By taking a summation j=1 j = 1 ij 1, . . . , M . With this notation, (69) gives the formula for H(x2 | s1 ) = −
L M
νi Pi j ln
i= 1 j = 1
Pi j m(Dj )
lim Pi j
ε →0
= lim ε →0
=
1 m(Ei )
,
χT x (Dj )dm(x)
lim q ε =
ε →0
(78)
(79)
Next, assume the control K to be strongly stabilizing the cell D1 , i.e., (1 + K) : T (D1 ) → D1 . Then, as ε → 0 Hc (xn ) = ln(a) + H(d).
(80)
Proof: Using the estimate in (78) m(T (Ei ) ∩ Dj ) Pi j 1 1 = = Qi j m(Dj ) am(Ei ) m(Dj ) am(Ei )
(81)
where Qi j ≤ 1. Substituting this in (77)
Hc (xn ) = −q ln(q ) − (1 − q ) ln
T (E i )
Hc (xn ) ≥ ln(a) + H(d).
≥ ln(a) +
(85)
ε
χy (Dj ) | DT −1 (y) | dm(y)
νi Pi j ln(Qi j ) +
!
where q ε = [PT ]1 1 ; the superscript makes the dependence on ε explicit. Using the formula (67), one obtains an entropy estimate as
i= 1 j = 1
1 − qε 1 − qε
ε
ε
1 − qε m(D2ε )
.
(86)
To prove the result, we show that
Ei
because limε →0 |DT −1 (·)| = 1/a. Since, entropy rates are computed with respect to a given partition of X, we take M = 1 and Y1 = {Y } to be a singleton and rescale volumes, so m(Y ) = ε; an estimate with a general partition Y is also provided. The entropy of the disturbance is taken to be for the dither defined wrt Y; for Y1 , H(d) = ln(ε). Theorem 11: Consider an expanding dynamical system given by (70) with a = |DT (0)|. Let K be a any stabilizing control such that DT (0)(1 + DK)(0) has eigenvalues inside the unit circle. Then, in the limit of vanishing conditional dither (ε → 0), the output sequence {xn } of the closed-loop dynamical system Td ◦ (1 + Kd ) has entropy
H(x2 | s1 ) = −
qε qε
(77)
M L
(84)
With strong stabilization, Y = D1 , X = T (D1 ), and PK is given by (57). We derive the entropy estimate with the simplest two cell partition X2 = {D1 , D2 }, where D2 = T (D1 ) \ D1 . For such a partition, the closed-loop Markov matrix is given as PK PT =
m(T (Ei ) ∩ Dj ) 1 lim a ε →0 m(Ei )
M L
H(x2 | s1 ) ≥ ln(a) + ln(ε) = ln(a) + H(d).
In the limit that ε → 0, the individual entries Pi j are related to the local expansion rate as 1 = lim ε →0 m(Ei )
The term ν ln(m(Ei )) corresponds to the entropy of the coni i ditional dither defined with respect to the partition {E1 , . . . , EM }. Specifically with M = 1
νi Pi j ln(am(Ei ))
i= 1 j = 1
νi Pi j ln(m(Ei )).
(82)
1 , a
lim
ε →0
m(D2ε ) = (a − 1). ε
(87)
Indeed, the latter equation is clear by construction, and in the ε = 0 limit, q 0 = lim ε →0
m(T −1 D1 ∩ D1 ) m(T −1 D1 ) 1 = lim = |DT −1 (0)| = ε →0 m(D1 ) m(D1 ) a
where a = |DT (0)| is the expansion rate. Since the argument relies on a type of linearization, the estimate is insensitive to the choice of partition in the limiting case. In particular, a finer partition of T (D1 ) \ D1 will also yield the same conclusion (see also Theorem 8 and Remark 9). As a result of this theorem, the stabilization in the general multistate case only leads to an inequality for the entropy estimate. With strong stabilization, one obtains an equality. Note that strong stabilization is equivalent to stabilization for the scalar case. The general multistate result was first shown with the aid of TFE in Nair et al. [7] who also showed the bound to be tight with strong stabilization. Our result is motivated by [7] and is analogous with one important difference: using TFE, the uncertainty arises due to the initial condition while using conditional dither, the uncertainty arises due to disturbance. After taking appropriate limits in each case (finer partitions or vanishing conditional dither), one obtains a bound in terms of expansion rate a due to unstable dynamics. We further remark that the notion of stabilization needed to obtain these bounds is weaker than Lyapunov stability. The control K needs to be stabilizing only in the sense that (71)–(72) are valid, i.e., the closed-loop map is positively invariant for a certain small neighborhood of 0. Finally, we stress the important role of disturbance. At each point, one is indeed solving a perturbed problem where ln(ε) denotes the entropy rate due to conditional dither. In order to interpret the ε → 0 limit, we note that the output xn is a continuous random variable. Thus, it is meaningful to talk only about change in uncertainty, and this change is reflected here as the difference between uncertainty rates of xn and dn given by the Bode formula (79) for the general case and (80) for the scalar or the strong stabilization case.
i,j
Finally, using the fact that
j
ACKNOWLEDGMENT
Pi j = 1 and
M
H(x2 | s1 ) ≥ ln(a) +
i= 1
νi ln(m(Ei )).
(83)
The authors thank G. Froyland, M. Hessel, and C. Jacobson for useful discussions. They are grateful to anonymous reviewers whose many comments including suggestion of the term “conditional dither” helped improved this paper.
Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 3, APRIL 2008
REFERENCES [1] A. Papoulis, Probability, Random Variables, and Stochastic Processes. New York: McGraw-Hill, 1984. [2] T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley Series in Telecommunications Series). New York: Wiley, 1991. [3] N. C. Martins and M. A. Dahleh, “Fundamental limitations of performance in the presence of finite capacity feedback,” in Proc. Amer. Control Conf., 2005, vol. 1, pp. 79–86. [4] G. Zang and P. A. Iglesias, “Nonlinear extension of Bode’s integral based on an information theoretic interpretation,” Syst. Control Lett., vol. 50, pp. 11–29, 2003. [5] N. Elia, “When Bode meets Shannon: Control-oriented feedback communication schemes,” IEEE Trans. Autom. Control, vol. 49, no. 9, pp. 1477– 1488, Sep. 2004. [6] A. Katok and B. Hasselblatt, Introduction to the Modern Theory of Dynamical Systems. Cambridge, U.K.: Cambridge Univ. Press, 1995. [7] G. N. Nair, R. J. Evans, and I. M. Y. Mareels, “Topological feedback entropy and nonlinear stabilization,” IEEE Trans. Autom. Control, vol. 49, no. 9, pp. 1585–1597, Sep. 2004. [8] A. Lasota and M. C. Mackey, Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics. New York: Springer-Verlag, 1994. [9] F. Fagnani and S. Zampieri, “Stability analysis and synthesis for scalar linear systems with quantized feedback,” IEEE Trans. Autom. Control, vol. 48, no. 9, pp. 1569–1584, Sep. 2003. [10] G. Froyland, “Extracting dynamical behaviour via Markov models,” in in Nonlinear Dynamics and Statistics: Proceedings, Newton Institute, Cambridge, 1998. Boston, MA: Birkhauser, 2001, pp. 283–324. [11] Y. Kifer, Ergodic Theory of Random Transformations (Progress of Probability and Statistics Series 10). Boston, MA: Birkhauser, 1986. [12] U. Vaidya and P. G. Mehta, “Lyapunov measure for almost everywhere stability,” IEEE Trans. Autom. Control, to be published. [13] M. Dellnitz and O. Junge, “On the approximation of complicated dynamical behavior,” SIAM J. Numer. Anal., vol. 36, pp. 491–515, 1999.
791
A Necessary and Sufficient Condition for Consensus Over Random Networks Alireza Tahbaz-Salehi, Student Member, IEEE, and Ali Jadbabaie, Senior Member, IEEE Abstract—We consider the consensus problem for stochastic discretetime linear dynamical systems. The underlying graph of such systems at a given time instance is derived from a random graph process, independent of other time instances. For such a framework, we present a necessary and sufficient condition for almost sure asymptotic consensus using simple ergodicity and probabilistic arguments. This easily verifiable condition uses the spectrum of the average weight matrix. Finally, we investigate a special case for which the linear dynamical system converges to a fixed vector with probability 1. Index Terms—Consensus problem, random graphs, tail events, weak ergodicity.
I. INTRODUCTION Decentralized iterative schemes such as agreement and consensus problems have an old history [1]–[4]. Over the past few years, they have attracted a significant amount of attention in various contexts such as motion coordination of autonomous agents [5], [6], distributed computation of averages and least squares among sensors [7]–[9], and rendezvous problems [10]. In all these cases, the dynamical system under study is deterministic. More recently, there has been some interest in the stochastic variants of the problem. In [11], the authors study the linear dynamical system x(k) = Wk x(k − 1), where the weight matrices Wk are independent, identically distributed (i.i.d.) stochastic matrices. It is shown that all the entries of x(k) converge to a common value almost surely (with probability 1), if each edge of G(Wk ), the graph corresponding to matrix Wk , is chosen independently with the same probability (Erd¨os–R´enyi random graph model). A more general model is studied in [12], where the edges of G(Wk ) are directed and not necessarily independent. However, the author proves only convergence to a consensus in probability, rather than the more general notion of almost sure convergence. Moreover, the assumption in [12] is the occurrence of scrambling matrices with positive probability, which can be weakened. The purpose of this note is to provide a necessary and sufficient condition for an almost sure consensus in the linear dynamical system x(k) = Wk x(k − 1), when the weight matrices are general i.i.d. stochastic matrices. Our results contain the results of [11] and [12] as special cases. This necessary and sufficient condition is easily verifiable and only depends on the spectrum of the average weight matrix EWk . Finally, for a special case, we state a variant of a theorem in [2] that provides the asymptotic consensus value. Even though it is possible to derive our main theorem by combining results from ergodic theory of Markov chains in random environments [13]–[15], our proofs are self-contained and are only based on simple linear algebra machinery and the concept of coefficients of ergodicity, as introduced by Dobrushin [16]. Manuscript received September 8, 2006; revised January 30, 2007, June 28, 2007, and August 31, 2007. Recommended by Associate Editor F. Bullo. This work was supported in part by the Department of Defense Multidiciplinary University Research Initiative (ARO/MURI) under Grant W911NF-051-0381, in part by the Office of Naval Research Young Investigator Program (ONR/YIP) under Grant N00014-04-1-0467, and in part by the National Science Foundation under Grant ECS-0347285. The authors are with the General Robotics, Automation, Sensing and Perception (GRASP) Laboratory, Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA 19104-6228 USA (e-mail: [email protected]; [email protected]). Digital Object Identifier 10.1109/TAC.2008.917743
0018-9286/$25.00 © 2008 IEEE Authorized licensed use limited to: IEEE Xplore. Downloaded on November 20, 2008 at 18:31 from IEEE Xplore. Restrictions apply.
