A Generalization of the Borkar-Meyn Theorem for Stochastic ...
A Generalization of the Borkar-Meyn Theorem for Stochastic Recursive Inclusions
arXiv:1502.01953v2 [cs.SY] 2 Mar 2015
Arunselvan Ramaswamy
1
and Shalabh Bhatnagar
2
1
[email protected] 2 [email protected] 1,2 Department of Computer Science and Automation, Indian Institute of Science, Bangalore - 560012, India. March 3, 2015 Abstract In this paper the stability theorem of Borkar and Meyn is extended to include the case when the mean field is a set-valued map. Two different sets of sufficient conditions are presented that guarantee the stability and convergence of stochastic recursive inclusions. Our work builds on the works of Bena¨ım, Hofbauer and Sorin as well as Borkar and Meyn. As a corollary to one of the main theorems, a natural generalization of the Borkar and Meyn Theorem follows. In addition, the original theorem of Borkar and Meyn is shown to hold under slightly relaxed assumptions. Finally, as an application to one of the main theorems we discuss a solution to the ‘approximate drift problem’.
1
Introduction
Consider the following recursion in Rd (d ≥ 1): xn+1 = xn + a(n) [h(xn ) + Mn+1 ] , f or n ≥ 0, where
(i) h : Rd → Rd is a Lipschitz continuous function.
(ii) a(n) P > 0, for all n, is the step-size sequence satisfying ∞ and n=0 a(n)2 < ∞.
P∞
n=0
(1)
a(n) = ∞
(iii) Mn , n ≥ 0, is a sequence of martingale difference terms. The stochastic recursion given by (1) is often referred to as a stochastic recursive equation (SRE). A powerful method to analyze the limiting behavior of (1) is the ODE (Ordinary Differential Equation) method. Here the limiting behavior of the algorithm is described in terms of the asymptotics of the solution to the ODE x(t) ˙ = h(x(t)). This method was introduced by Ljung [10] in 1977. For a detailed exposition on the subject and a survey of results, the reader is referred to Kushner and Yin [9] as well as Borkar [7]. 1
In 1996, Benaim [4] showed that the asymptotic behavior of a stochastic recursive equation can be studied by analyzing the asymptotic behavior of the associated o.d.e.. However no assumptions were made on the dynamics of the o.d.e. Specifically, he developed sufficient conditions which guarantee that limit sets of the continuously interpolated stochastic iterates are compact, connected, internally chain transitive and invariant sets of the associated o.d.e. The results found in [4] are generalized in [3]. The assumptions made in [4] are sometimes referred to as the ‘classical assumptions’. One of the key assumptions used by Bena¨ım to prove this convergence theorem is the almost sure boundedness of the iterates. In other words the iterates are stable. In 1999, Borkar and Meyn [8] developed sufficient conditions which guarantee both the stability and convergence of stochastic recursive equations. These assumptions were consistent with those developed in [4]. In this paper we refer to the main result of Borkar and Meyn colloquially as the Borkar-Meyn Theorem. In the same paper [8], several applications to problems from reinforcement learning have also been discussed. An extention to the Borkar-Meyn result for asynchronous stochastic iterates was given by Bhatnagar [6]. In 2005, Bena¨ım, Hofbauer and Sorin [5] showed that the dynamical systems approach can be extended to the situation where the mean field is a set-valued map. In other words, they considered algorithms with iterates of the form: xn+1 = xn + a(n) [yn + Mn+1 ] , f or n ≥ 0, where (2) (i) yn ∈ h(xn ) and h : Rd → {subsets of Rd } is a Marchaud map. For a definition of Marchaud maps the reader is referred to section 2.1. P∞ (ii) a(n) P > 0, for all n, is the step-size sequence satisfying n=0 a(n) = ∞ ∞ and n=0 a(n)2 < ∞.
(iii) Mn , n ≥ 0, is a sequence of martingale difference terms.
Such iterates are referred to as stochastic recursive inclusions (SRI). An SRE given by (1) can be seen as a special case of an SRI given by (2). Also, a differential equation can be seen as a special case of a differential inclusion where the set {h(x)} has cardinality one for all x ∈ Rd . The main aim of this paper is to extend the original Borkar-Meyn theorem to the case of stochastic recursive inclusions. We present two overlapping yet different sets of assumptions, in Sections 2.2 and 5.1 respectively, that guarantee the stability and convergence of a SRI given by (2). We present several interesting consequences to the main results (Theorems 2 and 4). Specifically, in Section 5.2 we show that the Borkar-Meyn theorem can be naturally extended to include the case where the infinity system is more generally allowed to be the set of accumulation points. We present a unified set of assumptions that takes care of both the original theorem and the aforementioned extension. While doing so we show how the assumptions of the Borkar-Meyn theorem could be relaxed.
2
The organization of this paper is as follows: In section 2.1, we discuss the definitions and notations used in this paper. In section 2.2, we discuss the assumptions under which the iterates given by (2) are stable and converge to a closed, connected, internally chain transitive and invariant set. A preliminary result is also presented in this section. In section 3.1, we present auxiliary results needed to prove the main result. In section 3.2, a stability theorem (Theorem 2) for stochastic recursive inclusions is proved under the assumptions outlined in section 2.2. Theorem 2 is one of two stability theorems presented in this paper. In section 4 we discuss applications of Theorem 2. Specifically, in section 4.1 a solution to the problem of ‘approximate drift’ is discussed. For more details on the ‘approximate drift’ problem the reader is referred to Borkar [7]. In section 4.2, the Borkar-Meyn Theorem is proved under a relaxed set of assumptions. In section 5.1, another set of assumptions that guarantee the stability and convergence to a closed, connected, internally chain transitive and invariant set of a stochastic recursive inclusion is discussed. In the same section we provide a brief outline of the proof of stability under these set of assumptions. This yields our second stability result for SRI : Theorem 4. In section 5.2, as an application of Theorem 4, the Borkar-Meyn Theorem [8] is generalized to when one the key requirements does not hold. This result is summarized in Corollary 2. Finally, in section 5.3 we discuss how the assumptions described in sections 2.2, 5.1 could be relaxed.
2
Preliminaries and Assumptions
2.1
Definitions and Notations
The definitions and notations used in this paper are similar to those in Bena¨ım et. al. [5], Aubin et. al. [1], [2] and Borkar [7]. In this section, we present a few for easy reference. A set-valued map h : Rn → {subsets of Rm } is called a Marchaud map if it satisfies the following properties: (i) For each x ∈ Rn , h(x) is convex and compact. (ii) (point-wise boundedness) For each x ∈ Rn , sup kwk < K (1 + kxk) for w∈h(x)
some K > 0. (iii) h is an upper-semicontinuous map. We say that h is upper-semicontinuous, if given sequences {xn }n≥1 (in Rn ) and {yn }n≥1 (in Rm ) with xn → x, yn → y and yn ∈ h(xn ), n ≥ 1, implies that y ∈ h(x). In other words the graph of h, {(x, y) : y ∈ h(x), x ∈ Rn }, is closed in Rn × Rm . Let H be a Marchaud map on Rd . The differential inclusion (DI) given by x˙ ∈ H(x)
(3)
is guaranteed to have at least one solution that is absolutely P continuous. The reader is referred to [1] for more details. We say that x ∈ if x is an absolutely continuous map that satisfies (3). The set-valued semiflow Φ associated with 3
(3) is defined on [0, +∞) × Rd as: Φt (x) = {x(t) | x ∈ B × M ⊂ [0, +∞) × Rk and define [ ΦB (M ) = Φt (x).
P
, x(0) = x}. Let
t∈B, x∈M
T Let M ⊆ Rd , the ω − limit set be defined by ωΦ (M ) = t≥0 Φ[t,+∞) (M ). T Similarly the limit set of a solution x is given by L(x) = t≥0 x([t, +∞)). M ⊆ Rd is invariant if for every x ∈ MPthere exists a trajectory, x, entirely in M with x(0) = x. In other words, x ∈ with x(t) ∈ M , for all t ≥ 0. Internally Chain Transitive Set : M ⊂ Rd is said to be internally chain transitive if M is compact and for every x, y ∈ M , ǫ > 0 and T > 0 we have the following: There exist Φ1 , . . . , Φn that are n solutions to the differential inclusion x(t) ˙ ∈ h(x(t)), a sequence x1 (= x), . . . , xn+1 (= y) ⊂ M and n real numbers t1 , t2 , . . . , tn greater than T such that: Φiti (xi ) ∈ N ǫ (xi+1 ) and Φi[0,ti ] (xi ) ⊂ M for 1 ≤ i ≤ n. The sequence (x1 (= x), . . . , xn+1 (= y)) is called an (ǫ, T ) chain in M from x to y. Let x ∈ Rd and A ⊆ Rd , then d(x, A) := inf{ka − yk | y ∈ A}. We define the δ-open neighborhood of A by N δ (A) := {x | d(x, A) < δ}. The δ-closed neighborhood of A is defined by N δ (A) := {x | d(x, A) ≤ δ}. A ⊆ Rd is an attracting set if it is compact and there exists a neighborhood U such that for any ǫ > 0, ∃ T (ǫ) ≥ 0 such that Φ[T (ǫ),+∞) (U ) ⊂ N ǫ (A). Such a U is called the fundamental neighborhood of A. In addition to being compact if the attracting set is also invariant then it is called an attractor. The basin of attraction of A is given by B(A) = {x | ωΦ (x) ⊂ A}. It is called Lyapunov stable if for all δ > 0, ∃ ǫ > 0 such that Φ[0,+∞) (N ǫ (A)) ⊆ N δ (A). We use T (ǫ) and Tǫ interchangeably to denote the dependence of T on ǫ. The open ball of radius r around 0 is represented by Br (0), while the closed ball is represented by B r (0). We define the lower and upper limits of sequences of sets. Let {Kn }n≥1 be a sequence of sets in Rd . 1. The lower limit of {Kn }n≥1 is given by, Liminfn→∞Kn := {x | lim d(x, Kn ) = 0}. n→∞
2. The upper-limit of {Kn }n≥1 given by, Limsupn→∞Kn := {y | lim d(y, Kn ) = 0}. We may interpret that the n→∞
upper-limit collects the accumulation points of {Kn }n≥1 .
2.2
The assumptions
Recall that we have the following recursion in Rd : xn+1 = xn + a(n) [yn + Mn+1 ], where yn ∈ h(xn ). We state our assumptions below:
(A1) h : Rd → {subsets of Rd } is a Marchaud map. P a(n) = ∞ and (A2) {a(n)}n≥0 is a scalar sequence such that: a(n) ≥ 0 ∀n, n≥0 P a(n)2 < ∞. Without loss of generality we let sup a(n) ≤ 1. n
n≥0
4
(A3)
{Mn }n≥1 is a martingale difference sequence with respect to the filtration Fn := σ (x0 , M1 , . . . , Mn ), n ≥ 0. (i) {Mn }n≥1 is a square integrable sequence with E [Mn+1 |Fn ] = 0 a.s., for n ≥ 0. (ii) E[kMn+1 k2 |Fn ] ≤ K 1 + kxn k2 , for n ≥ 0 and some constant K > 0. Without loss of generality assume that the same constant, K, works for both the point-wise boundedness condition of (A1) (see conditions (ii) of definition of Marchaud map in Section 2.1) and (A3).
For c ≥ 1 and x ∈ Rd , define hc (x) = {y | cy ∈ h(cx)}. For each x ∈ Rd , define h∞ (x) := Liminfc→∞ hc (x) i.e. the closure of the lower limit of {hc (x)}c≥1 .
(A4) h∞ (x) is non-empty for all x ∈ Rd . Further, the differential inclusion x(t) ˙ ∈ h∞ (x(t)) has the origin as an attracting set and B 1 (0) is a subset of its fundamental neighborhood. (A5) Let cn ≥ 1 be an increasing sequence of integers such that cn ↑ ∞ as n → ∞. Further, let xn → x and yn → y as n → ∞, such that yn ∈ hcn (xn ), ∀n, then y ∈ h∞ (x). Assumptions (A1) − (A3) are the same as in Bena¨ım [5]. However, the assumption on the stability of the iterates is replaced by (A4) and (A5). We show that (A4) and (A5) are sufficient conditions to ensure stability of iterates. We start by observing that hc and h∞ are Marchaud maps, where c ≥ 1. Further, we show that the constant associated with the point-wise boundedness property is K of (A1) and (A3). Proposition 1. h∞ and hc , c ≥ 1, are Marchaud maps.
Proof. Fix c ≥ 1 and x ∈ Rd . To prove that hc (x) is compact, we show that it is closed and bounded. For n ≥ 1, let yn ∈ hc (x) and let lim yn = y. It follows n→∞
that cyn ∈ h(cx) for each n ≥ 1 and lim cyn = cy. Since h(cx) is closed, we n→∞
have that cy ∈ h(cx) and y ∈ hc (x). If we show that hc is point-wise bounded then we can conclude that hc (x) is compact. To prove the aforementioned, let y ∈ hc (x), then cy ∈ h(cx). Since h satisfies (A1)(ii), we have that ckyk ≤ K (1 + kcxk) , hence 1 kyk ≤ K + kxk . c
Since c(≥ 1) and x is arbitrarily chosen, hc is point-wise bounded and the compactness of hc (x) follows. The set hc (x) = {z/c | z ∈ h(cx)} is convex since h(cx) is convex and hc (x) is obtained by scaling it by 1c . Next, we show that hc (x) is upper-semicontinuous. Let lim xn = x, lim n→∞
n→∞
yn = y and yn ∈ hc (xn ), ∀ n ≥ 1. We need to show that y ∈ hc (x). We have that cyn ∈ h(cxn ) for each n ≥ 1. Since lim cxn = cx and lim cyn = cy, we n→∞
n→∞
conclude that cy ∈ h(cx) since h is assumed to be upper-semicontinuous. 5
It is left to show that h∞ (x), x ∈ Rd is a Marchaud map. To prove that kzk ≤ K (1 + kxk) for all z ∈ h∞ (x), it is enough to prove that kyk ≤ K (1 + kxk) for all y ∈ Liminfc→∞ hc (x). Fix some y ∈ Liminfc→∞ hc (x) then there exist zn ∈ hn (x), n ≥ 1, such that lim ky − zn k = 0. We have that n→∞
kyk ≤ ky − zn k + kzn k. Since hc , c ≥ 1, is point-wise bounded (the constant associated is independent of c and equals K) the above inequality becomes kyk ≤ ky − zn k + K (1 + kxk). Letting n → ∞ in the above inequality, we obtain kyk ≤ K (1 + kxk). Recall that h∞ (x) = Liminfc→∞ hc (x), hence it is compact. Again, to show that h∞ (x) is convex, for each x ∈ Rd , we start by proving that Liminfc→∞ hc (x) is convex. Let u, v ∈ Liminfc→∞ hc (x) and 0 ≤ t ≤ 1. We need to show that tu + (1 − t)v ∈ Liminfc→∞ hc (x). Consider an arbitrary sequence {cn }n≥1 such that cn → ∞, then there exist un , vn ∈ hcn (x) such that kun − uk and kvn − vk → 0 as cn → ∞. Since hcn (x) is convex, it follows that tun + (1 − t)vn ∈ hcn (x), further lim ( tun + (1 − t)vn ) = tu + (1 − t)v.
cn →∞
Since we started with an arbitrary sequence cn → ∞, it follows that tu + (1 − t)v ∈ Liminfc→∞ hc (x). Now we can prove that h∞ (x) is convex. Let u, v ∈ h∞ (x). Then ∃ {un }n≥1 and {vn }n≥1 ⊆ Liminfc→∞ hc (x) such that un → u and vn → v as n → ∞. We need to show that tu + (1 − t)v ∈ h∞ (x), for 0 ≤ t ≤ 1. Since tun + (1 − t)vn ∈ Liminfc→∞ hc (x), the desired result is obtained by letting n → ∞ in tun + (1 − t)vn . Finally, we show that h∞ is upper-semicontinuous. Let lim xn = x, lim n→∞
n→∞
yn = y and yn ∈ h∞ (xn ), ∀ n ≥ 1. We need to show that y ∈ h∞ (x). Since yn ∈ h∞ (xn ), ∃ zn ∈ Liminfc→∞ hc (xn ) such that kzn − yn k < n1 . Since zn ∈ Liminfc→∞ hc (xn ), n ≥ 1, it follows that there exist cn such that for all c ≥ cn , d (zn , hc (xn )) < n1 . In particular, ∃ un ∈ hcn (xn ) such that kzn − un k < n1 . We choose the sequence {cn }n≥1 such that cn+1 > cn for each n ≥ 1. We now have the following: lim un = y, un ∈ hcn (xn ) ∀ n and lim xn = x. It n→∞
n→∞
follows directly from assumption (A5) that y ∈ h∞ (x).
3
Stability and convergence of stochastic recursive inclusions
We begin by providing a brief outline of our approach to prove the stability of a SRI under assumptions (A1) − (A5). First we divide the time line, [0, ∞), approximately into intervals of length T . We shall explain later how we choose and fix T . Then we construct a linearly interpolated trajectory from the given stochastic recursive inclusion; the construction is explained in the next paragraph. A sequence of ‘rescaled’ trajectories of length T is constructed as follows:
6
At the beginning of each T -length interval we observe the trajectory to see if it is outside the unit ball, if so we scale it back to the boundary of the unit ball. This scaling factor is then used to scale the ‘rest of the T -length trajectory’. To show that the iterates are bounded almost surely we need to show that the linearly interpolated trajectory does not ‘run off’ to infinity. To do so we assume that this is not true and show that there exists a subsequence of the rescaled T -length trajectories that has a solution to x(t) ˙ ∈ h∞ (x(t)) as a limit point in C([0, T ], Rd ). We choose and fix T such that any solution to x(t) ˙ ∈ h∞ (x(t)) with an initial value inside the unit ball is close to the origin at the end of time T . For example we could choose T = T (1/8). We then argue that the linearly interpolated trajectory is forced to make arbitrarily large ‘jumps’ within time T . The Gronwall inequality is then used to show that this is not possible. Once we prove stability of the recursion we invoke Theorem 3.6 & Lemma 3.8 from Bena¨ım, Hofbauer and Sorin [5] to conclude that the limit set is a closed, connected, internally chain transitive and invariant set associated with x(t) ˙ ∈ h∞ (x(t)). We construct the linearly interpolated trajectory x(t), for t ∈ [0, ∞), from Pn−1 the sequence {xn } as follows: Define t(0) := 0, t(n) := i=0 a(i). Let x(t(n)) := xn and for t ∈ (t(n), t(n + 1)), let t(n + 1) − t t − t(n) x(t) := x(t(n)) + x(t(n + 1)). t(n + 1) − t(n) t(n + 1) − t(n) We define a piecewise constant trajectory using the sequence {yn }n≥0 as follows: y(t) := yn for t ∈ [t(n), t(n + 1)), n ≥ 0. We know that the DI given by x(t) ˙ ∈ h∞ (x(t)) has the origin as an attractor set. Let us fix T := T (1/8), where T (1/8) is as defined in section 2.1. Then, ˙ ∈ kx(t)k < 81 , for all t ≥ T1/8 , where {x(t) : t ∈ [0, ∞)} is a solution to x(t) h∞ (x(t)) with an initial value inside the unit ball around the origin. Define T0 := 0 and Tn := min{t(m) : t(m) ≥ Tn−1 + T }, n ≥ 1. Observe that there exists a subsequence {m(n)}n≥0 of {n} such that Tn = t(m(n)) ∀ n ≥ 0. We construct the rescaled trajectory, x ˆ(t), t ≥ 0, as follows: Let t ∈ [Tn , Tn+1 ) x(t) for some n ≥ 0, then x ˆ(t) := r(n) , where r(n) = kx(Tn )k ∨ 1. Also, let − x ˆ(Tn+1 ) := lim xˆ(t), t ∈ [Tn , Tn+1 ). The corresponding ‘rescaled y itert↑Tn+1
y(t) and the rescaled martingale noise terms by ates’ are given by yˆ(t) := r(n) Mk+1 ˆ Mk+1 := r(n) , t(k) ∈ [Tn , Tn+1 ), n ≥ 0.
Consider the recursion at hand, i.e., x(t(k + 1)) = x(t(k)) + a(k) (y(t(k)) + Mk+1 ) , such that t(k), t(k + 1) ∈ [Tn , Tn+1 ). Multiplying both sides by 1/r(n) we get the rescaled recursion: ˆ k+1 . xˆ(t(k + 1)) = x ˆ(t(k)) + a(k) yˆ(t(k)) + M 7
Since y(t(k)) ∈h h (x(t(k))), it hr(n) (ˆ x(t(k))). It is worth i follows that yˆ(t(k)) ∈ ˆ k+1 k2 |Fk ≤ K 1 + kˆ noting that E kM x(t(k))k2 .
Characterizing limits, in C([0, T ], Rd ), of the rescaled trajectories
3.1
The first two lemmas prove that the rescaled martingale difference noise converges almost surely. Although they can be found in Borakar [8], we present them here with proofs for the sake of completeness and easy reference. Lemma 1. sup Ekˆ x(t)k2 < ∞. t∈[0,T ]
Proof. It is enough to show that sup m(n)≤k<m(n+1)
E kˆ x(t(k))k2 ≤ M,
for some M (> 0) that is independent of n. Recall that Tn = t(m(n)) and Tn+1 = t(m(n+1)). Let us fix n and k such that n ≥ 0 and m(n) ≤ k < m(n+1). Now consider ˆk . xˆ(t(k)) = x ˆ(t(k − 1)) + a(k − 1) yˆ(t(k − 1)) + M Unfolding the above recursion we get, x ˆ(t(k)) = x ˆ(t(m(n))) +
k−1 X
l=m(n)
ˆ l+1 . a(l) yˆ(t(l)) + M
Taking the expectation of the square of the norms on both sides of the above equation we get, k−1 X ˆ l+1 k2 . a(l) yˆ(t(l)) + M Ekˆ x(t(k))k2 = E kˆ x(t(m(n))) + l=m(n)
It follows from the Minkowski inequality that, E 1/2 kˆ x(t(k))k2 ≤ E 1/2 kˆ x(Tn )k2 +
k−1 X
l=m(n)
ˆ l+1 k2 . a(l) E 1/2 kˆ y (t(l))k2 + E 1/2 kM
From assumptions (A1) and (A2), it i for each m(n) ≤ l ≤ k − 1, h follows that ˆ l+1 k2 |Fl ≤ K 1 + kˆ x(t(l))k2 , respeckˆ y(t(l))k ≤ K (1 + kˆ x(t(l))k) and E kM
tively. We also observe that Tn+1 − Tn ≤ T + 1 (since supn a(n) ≤ 1). Using these observations we obtain the following set of inequalities: E
1/2
2
kˆ x(t(k))k ≤ 1 +
k−1 X
l=m(n)
√ 2 a(l) KE 1/2 (1 + kˆ x(t(l))k) + KE 1/2 1 + kˆ x(t(l))k2 ,
8
E
1/2
2
kˆ x(t(k))k ≤ 1 +
k−1 X
l=m(n)
√ a(l) K 1 + E 1/2 kˆ x(t(l))k2 + K 1 + E 1/2 kˆ x(t(l))k2 ,
k−1 i h X √ √ a(l)E 1/2 kˆ x(t(l))k2 . E 1/2 kˆ x(t(k))k2 ≤ 1 + (K + K)(T + 1) +(K+ K) l=m(n)
Applying the discrete version of the Gronwall inequality we get, i h √ √ E 1/2 kˆ x(t(k))k2 ≤ 1 + (K + K)(T + 1) e(K+ K)(T +1) . The claim follows by letting M =
h
1 + (K +
2 i √ √ K)(T + 1) e(K+ K)(T +1) .
Pn−1 ˆ Lemma 2. The rescaled sequence {ζˆn }n≥1 , where ζˆn = k=0 a(k)Mk+1 , is convergent almost surely. Proof. It is enough to prove that almost surely, ∞ X
k=0
Instead, we prove that " E
∞ X
k=0
i h ˆ k+1 k2 | Fk < ∞. E ka(k)M
i h ˆ k+1 k2 | Fk a(k) E kM 2
#
< ∞.
It follows as a consequence of assumption (A3) that, "∞ # ∞ h i X X ˆ k+1 k2 | Fk ≤ E a(k)2 K 1 + Ekˆ x(t(k))k2 . a(k)2 E kM k=0
k=0
The claim now follows from assumption (A2) and Lemma 1. The rest of the lemmas are needed to prove the stability theorem, Theorem 1. We begin by showing that the rescaled trajectories are bounded almost surely. Lemma 3.
sup kˆ x(t)k < ∞ a.s.
t∈[0,∞)
Proof. Let A = {ω | {ζˆn (ω)}n≥1 converges}, be the set on which {ζˆn }n≥1 converges. It is enough to prove that kˆ x(t(m(n) + k))k < Kω , where Tn ≤ t(m(n) + k) < Tn+1 and Kω is a constant independent of n. The constant may however be dependent on ω (sample path), where ω ∈ A.
9
Let us consider the rescaled recursion: ˆ m(n)+k . x ˆ(t(m(n)+k)) = x ˆ(t(m(n)+k−1)) + a(m(n)+k−1) yˆ(t(m(n) + k − 1)) + M Unfolding the above recursion, we obtain x ˆ(t(m(n) + k)) = xˆ(Tn ) + k−1 X l=0
ˆ m(n)+l+1 . (4) a(m(n) + l) yˆ(t(m(n) + l)) + M
Since ζˆn , n ≥ 1, converges on A, there exists Mω < ∞, that may be sample Pk−1 ˆ m(n)+l+1 k ≤ Mw , where Mω is path dependent, such that k l=0 a(m(n) + l)M independent of n. Since kˆ x(Tn )k ≤ 1 we have, kˆ x(t(m(n) + k))k ≤ 1 +
k−1 X
a(m(n) + l)kˆ y(t(m(n) + l))k + Mω .
l=0
From (A1) it follows that, kˆ x(t(m(n) + k))k ≤ 1 + K
k−1 X l=0
a(m(n) + l) (1 + kˆ x(t(m(n) + l))k) + Mω .
Rearranging the terms in the above inequality and applying the discrete Gronwall inequality we get, kˆ x(t(m(n) + k))k ≤ (1 + Mω + (T + 1)K) eK(T +1) , a constant independent of both n and k. The rest of the proof follows in a straightforward manner. Note that in the proof of the above lemma we get a bound on kˆ x(t)k that is dependent on T . This is sufficient for our purposes, since we fix T to be T1/8 . Let xn (t), t ∈ [0, T ] be the solution (upto time T ) to x˙ n (t) = yˆ(Tn + t), with the initial condition xn (0) = x ˆ(Tn ). Clearly, we have n
x (t) = x ˆ(Tn ) +
Z
t
yˆ(Tn + z) dz.
(5)
0
The following two lemmas are inspired by ideas from Benaim, Hofbauer and Sorin [5] as well as Borkar [7]. In the lemma that follows we show that the limit sets of {xn (· ) | n ≥ 0} and {ˆ x(Tn +· ) | n ≥ 0} coincide. We seek limits in C([0, T ], Rd ). Lemma 4. lim
sup
n→∞ t∈[T ,T +T ] n n
kxn (t) − x ˆ(t)k = 0 a.s.
Proof. Let t ∈ [t(m(n) + k), t(m(n) + k + 1)) and t(m(n) + k + 1) ≤ Tn+1 . We first assume that t(m(n) + k + 1) < Tn+1 . We have the following: x ˆ(t) =
t(m(n) + k + 1) − t a(m(n) + k)
xˆ(t(m(n)+k))+
10
t − t(m(n) + k) a(m(n) + k)
x ˆ(t(m(n)+k+1)).
Substituting for x ˆ(t(m(n) + k + 1)) in the above equation we get: t − t(m(n) + k) t(m(n) + k + 1) − t xˆ(t(m(n) + k)) + x ˆ(t) = a(m(n) + k) a(m(n) + k) ˆ m(n)+k+1 , x ˆ(t(m(n) + k)) + a(m(n) + k) yˆ(t(m(n) + k)) + M
hence,
ˆ m(n)+k+1 . x ˆ(t) = x ˆ(t(m(n) + k)) + (t − t(m(n) + k)) yˆ(t(m(n) + k)) + M
Unfolding x ˆ(t(m(n) + k)) over k (see (4)) we get, x ˆ(t) = x ˆ(Tn ) +
k−1 X l=0
ˆ m(n)+l+1 + a(m(n) + l) yˆ(t(m(n) + l)) + M
ˆ m(n)+k+1 . (6) (t − t(m(n) + k)) yˆ(t(m(n) + k)) + M
Now, we consider xn (t), i.e., n
x (t) = x ˆ(Tn ) +
Z
t
yˆ(Tn + z) dz.
0
Splitting the above integral, we get Z k−1 X Z t(m(n)+l+1) n yˆ(z) dz + x (t) = xˆ(Tn ) + l=0
t
yˆ(z) dz.
t(m(n)+k)
t(m(n)+l)
Thus, xn (t) = x ˆ(Tn ) +
k−1 X
a(m(n) + l)ˆ y(t(m(n) + l))+
l=0
(t − t(m(n) + k)) yˆ(t(m(n) + k)). (7)
From (6) and (7), it follows that
k−1
X
n ˆ m(n)+l+1 ˆ m(n)+k+1 a(m(n) + l)M kx (t)−ˆ x(t)k ≤
+ (t − t(m(n) + k)) M
,
l=0
and hence,
kxn (t) − x ˆ(t)k ≤ kζˆm(n)+k − ζˆm(n) k + kζˆm(n)+k+1 − ζˆm(n)+k k. If t(m(n) + k + 1) = Tn+1 then in the proof we may replace x ˆ(t(m(n) + k + 1)) − with x ˆ(Tn+1 ). The arguments remain the same. Since ζˆn , n ≥ 1, converges almost surely, the desired result follows. The sets {xn (t), t ∈ [0, T ] | n ≥ 0} and {ˆ x(Tn + t), t ∈ [0, T ] | n ≥ 0} can be viewed as subsets of C([0, T ], Rd ). It can be shown that {xn (t), t ∈ [0, T ] | n ≥ 0} is equi-continuous and point-wise bounded. Thus from the Arzela-Ascoli theorem, {xn (t), t ∈ [0, T ] | n ≥ 0} is relatively compact. It follows from Lemma 4 that the set {ˆ x(Tn + t), t ∈ [0, T ] | n ≥ 0} is also relatively compact in C([0, T ], Rd ). 11
Lemma 5. Let r(n) ↑ ∞, then any limit point of {ˆ x(Tn + t), t ∈ [0, T ] : n ≥ 0} Rt is of the form x(t) = x(0) + 0 y(s) ds, where y : [0, T ] → Rd is a measurable function and y(t) ∈ h∞ (x(t)), t ∈ [0, T ]. Proof. We define the notation [t] := max{t(k) | t(k) ≤ t}, t ≥ 0. Let t ∈ [Tn , Tn+1 ), then yˆ(t) ∈ hr(n) (ˆ x([t])) and kˆ y (t)k ≤ K (1 + kˆ x([t])k) since hr(n) is a Marchaud map (K is the constant associated with the point-wise boundedness property). It follows from Lemma 3 that sup kˆ y(t)k < ∞ a.s. Using obsert∈[0,∞)
vations made earlier, we can deduce that there exists a sub-sequence of N, say d {l} ⊆ {n}, such that x ˆ(T ) → y(· ) l + t) → x(t) in C [0, T ], R and yˆ(m(l)+· d weakly in L2 [0, T ], R . From Lemma 4 it follows that xl (· ) → x(· ) in C [0, T ], Rd . Letting r(l) ↑ ∞ in Z t xl (t) = xl (0) + yˆ(t(m(l) + z)) dz, t ∈ [0, T ], 0
we get x(t) = x(0) + kx(0)k ≤ 1.
Rt 0
y(z)dz for t ∈ [0, T ]. Since kˆ x(Tn )k ≤ 1 we have
Since yˆ(Tl + · ) → y(· ) weakly in L2 [0, T ], Rd , there exists {l(k)} ⊆ {l} such that N 1 X yˆ(Tl(k) + · ) → y(· ) strongly in L2 [0, T ], Rd . N k=1
Further, there exists {N (m)} ⊆ {N } such that
N (m) X 1 yˆ(Tl(k) + · ) → y(· ) a.e. on [0, T ]. N (m) k=1
Let us fix t0 ∈ {t |
1 N (m)
PN (m) k=1
yˆ(Tl(k) + t) → y(t), t ∈ [0, T ]}, then
N (m) X 1 yˆ(Tl(k) + t0 ) = y(t0 ). N (m)→∞ N (m)
lim
k=1
Since h∞ (x(t0 )) is convex and compact, to show that y(t0 ) ∈ h∞ (x(t0 )) it is enough to prove that lim d yˆ(Tl(k) + t0 ), h∞ (x(t0 )) = 0. If not, ∃ ǫ > 0 and l(k)→∞ {n(k)} ⊆ {l(k)} such that d yˆ(Tn(k) + t0 ), h∞ (x(t0 )) > ǫ. Since {ˆ y(Tn(k) + t0 )}k≥1 is norm bounded, it follows that there is a convergent sub-sequence. For the sake of convenience we assume that lim yˆ(Tn(k) + t0 ) = y, for some k→∞
y ∈ Rd . Since yˆ(Tn(k) + t0 ) ∈ hr(n(k)) (ˆ x([Tn(k) + t0 ])) and lim x ˆ([Tn(k) + t0 ]) = k→∞
x(t0 ), it follows from assumption (A5) that y ∈ h∞ (x(t0 )). This leads to a contradiction.
Note that in the statement of Lemma 5 we can replace ‘r(n) ↑ ∞’ by ‘r(l) ↑ ∞’, where {r(l))} is a subsequence of {r(n)}. Specifically we can conclude that any limit point of {ˆ x(Tk + t), t ∈ [0, T ]}{k}⊆{n} in C([0, T ], Rd), conditioned on Rt r(k) ↑ ∞, is of the form x(t) = x(0) + 0 y(z) dz, where y(t) ∈ h∞ (x(t)) for t ∈ [0, T ]. It should be noted that y(· ) may be sample path dependent. 12
3.2
The Stability Theorem
We are now ready to prove the stability of a SRI given by (2) under the assumptions (A1) − (A5). If sup r(n) < ∞, then the iterates are stable and n
there is nothing to prove. If on the other hand sup r(n) = ∞, there exists n
{l} ⊆ {n} such that r(l) ↑ ∞. It follows from Lemma 5 that any limit point of Rt {ˆ x(Tl + t), t ∈ [0, T ] : {l} ⊆ {n}} is of the form x(t) = x(0) + 0 y(s) ds, where y(t) ∈ h∞ (x(t)) for t ∈ [0, T ]. From assumption (A4), we have that kxk < 1/8 (T = T (1/8)). Since the time intervals are roughly T apart, for large values of − − k < 14 , where x r(n) we can conclude that kˆ x Tn+1 ˆ(Tn+1 ) = limt↑t(m(n+1)) x ˆ(t), t ∈ [Tn , Tn+1 ). Theorem 1 (Stability Theorem for DI). Under assumptions (A1) − (A5), supkxn k < ∞ a.s. n
Proof. As explained earlier it is sufficient to consider the case when sup r(n) = n
∞. Let {l} ⊆ {n} such that r(l) ↑ ∞. Recall that Tl = t(m(l)) and that [Tl + T ] = max{t(k) | t(k) ≤ Tl + T }. We have kx(T )k < 18 since x(t) is a solution, up to time T , to the DI given by x(t) ˙ ∈ h∞ (x(t)) and we have fixed T = T (1/8). From Lemma 5 we conclude that there exists N such that all of the following happen: (i) m(l) ≥ N =⇒ kˆ x([Tl + T ])k < 41 . (ii) n ≥ N =⇒ a(n)
m ≥ N =⇒ kζˆn − ζˆm k < Mω . (iv) m(l) ≥ N =⇒ r(l) > 1. In the above Kω and Mω are as explained in Lemma 3. Let m(l) ≥ N and t(m(l + 1)) = t(m(l) + k + 1) for some k ≥ 0. Clearly from the manner in which the Tn sequence is defined, we have t(m(l)+k) = [Tl + T ]. − As defined earlier x ˆ(Tn+1 ) = limt↑t(m(n+1)) x ˆ(t), t ∈ [Tn , Tn+1 ) and n ≥ 0. Consider the equation − ˆ m(l)+k+1 . x ˆ(Tl+1 ) = x ˆ(t(m(l) + k)) + a(m(l) + k) yˆ(t(m(l) + k)) + M Taking norms on both sides we get,
− kˆ x(Tl+1 )k ≤ kˆ x(t(m(l) + k))k +
ˆ m(l)+k+1 k. a(m(l) + k)kˆ y(t(m(l) + k))k + a(m(l) + k)kM
From the way we have chosen N we conclude that: kˆ y(t(m(l) + k))k ≤ K (1 + kˆ x(t(m(l) + k)k) ≤ K (1 + Kω ) and that ˆ m(l)+k+1 k = kζˆm(l)+k+1 − ζˆm(l)+k k ≤ Mω . kM 13
Thus we have that, − kˆ x(Tl+1 )k ≤ kˆ x(t(m(l) + k))k + a(m(l) + k) (K(1 + Kω ) + Mω ) . − Finally we have that kˆ x(Tl+1 )k
0) then such an algorithm can be characterized by the following stochastic recursive inclusion: xn+1 = xn + a(n) (yn + Mn+1 ) , 14
(10)
where yn ∈ h(xn ) + B ǫ (0) is an estimate of h(xn ) and B ǫ (0) is the closed ball of radius ǫ around the origin. We define a new set-valued map called the approximate drift by H(x) := h(x) + B ǫ (0) for each x ∈ Rd . In the following discussion we assume that ǫ ≥ 0. When ǫ = 0, the approximate drift algorithm described by (10) is really the SRE given by (9). In this section we answer the following question: Suppose the recursion given by (9) is stable and convergent, what can we say about the approximate drift version described by (10)? We shall show that if the recursion (9) satisfies the assumptions of the Borkar-Meyn Theorem [8], then the corresponding approximate drift version given by ( 10) satisfies (A1)−(A5). We then invoke Theorem 2 to conclude that the iterates converge to a closed, connected, internally chain transitive and invariant set associated with x(t) ˙ ∈ h(x(t)) + B ǫ (0)(= H(x(t))). Before we proceed we recall the assumptions and the statement of the BorkarMeyn Theorem [8]. (BM1) (i) The function h : Rd → Rd is Lipschitz continuous, with Lipschitz constant L. There exists a function h∞ : Rd → Rd such that lim h(cx) = c c→∞
h∞ (x), for each x ∈ Rd . (ii) hc → h∞ uniformly on compacts, as c → ∞. (iii) The o.d.e. x(t) ˙ = h∞ (x(t)) has the origin as the unique globally asymptotically stable equilibrium.
P a(n) = ∞ and (BM2) {a(n)}n≥0 is a scalar sequence such that: a(n) ≥ 0, n≥0 P a(n)2 < ∞. Without loss of generality, we assume that sup a(n) ≤ 1. n
n≥0
(BM3) {Mn }n≥1 is a martingale difference sequence with respect to the filtration Fn := σ (x0 , M1 , . . . , Mn ), n ≥ 0. Thus, E [Mn+1 |Fn ] = 0 a.s., ∀ n≥ 0. {Mn } is also square integrable with E[kMn+1 k2 |Fn ] ≤ L 1 + kxn k2 , for some constant L > 0. Without loss of generality, assume that the same constant, L, works for both (BM 1)(i) and (BM 3). Theorem 3 (Borkar-Meyn Theorem). Suppose (BM1)-(BM3) hold. Then supkxn k < n
∞ almost surely. Further, the sequence {xn } converges almost surely to a (possibly sample path dependent) compact connected internally chain transitive invariant set of x(t) ˙ = h(x(t)). We assume that the recursion given by (9) satisfies assumptions (BM 1) − (BM 3). As explained earlier we have to prove that the approximate drift version given by (10) satisfies (A1) − (A5). We begin by proving that H(x) = h(x) + B ǫ (0) is a Marchaud map. Since B ǫ (0) is convex and compact for each x ∈ Rd , it follows that H(x) is convex and compact. Fix x ∈ Rd and y ∈ H(x), then kyk ≤ kh(x)k + ǫ and kyk ≤ kh(0)k + Lkx − 0k + ǫ since h is Lipschitz continuous with Lipschitz constant L. If we set K := (kh(0)k + ǫ) ∨ L, then we get kyk ≤ K (1 + kxk). This shows that H is point-wise bounded. We now show that H is upper-semicontinuous. Assume lim xn = x, lim yn = y and n→∞
n→∞
yn ∈ h(xn ) for each n ≥ 1. Since yn ∈ H(xn ), we conclude that yn = h(xn ) + zn 15
for some zn ∈ B ǫ (0), n ≥ 1. Since {yn }n≥1 and {h(xn )}n≥1 are convergent sequences, it follows that {zn }n≥1 is also convergent. Let z := lim zn . Since n→∞
B ǫ (0) is compact we have that z ∈ B ǫ (0). Taking limits on both sides of yn = h(xn ) + zn , we get y = h(x) + z. Thus y ∈ H(x).
It follows directly from (BM 2) and (BM 3) that assumptions (A2) and (A3) are satisfied by the approximate drift algorithm (10). Before showing that (10) satisfies (A4), we construct the following family of set-valued maps: Hc (x) := { h(cx) + yc | y ∈ B ǫ (0)}. i.e., Hc (x) = c d hc (x) + B ǫ/c (0) for each x ∈ R . We claim that the lower limit, H∞ (x) := Liminfc→∞Hc (x) = {h∞ (x)}. Since lim d(hc (x), H∞ (x)) ≤ lim kh∞ (x) − c→∞
c→∞
hc (x)k = 0 it follows that h∞ (x) ∈ H∞ (x). Now we show that h∞ (x) is the only element of H∞ (x). Let y ∈ H∞ (x), then ∃ zn ∈ Hn (x), n ≥ 1, such that lim zn = y. We have the following inequality: n→∞
ky − h∞ (x)k ≤ ky − zn k + kzn − hn (x)k + khn (x) − h∞ (x)k. Letting n → ∞ in the above inequality, we get ky − h∞ (x)k = 0, in other words H∞ (x) = {h∞ (x)}. The differential inclusion x(t) ˙ ∈ H∞ (x(t)) is really the o.d.e. x(t) ˙ = h∞ (x(t)). Since the origin is assumed to be the unique globally asymptotically stable equilibrium point of x(t) ˙ = h∞ (x(t)), it follows that the origin is also Lyapunov stable. This can be used to show that the origin is an attracting set and that B 1 (0) is its fundamental neighborhood. For a proof of this fact the reader is referred to Lemma 1 from Chapter 3 of Borkar [7]. Finally we show that (A5) is satisfied. Let cn ↑ ∞, lim xn = x, lim yn = n→∞
n→∞
y, and yn ∈ Hcn (xn ) for each n ≥ 1. We need to show that y ∈ H∞ (x) (y = h∞ (x)). Consider the following set of inequalities: ky − h∞ (x)k ≤ ky − yn k + kyn − hcn (xn )k + khcn (xn ) − h∞ (x)k,
ǫ + khcn (x) − h∞ (x)k + khcn (xn ) − hcn (x)k. cn The second inequality follows from the first since khcn (xn )−h∞ (x)k ≤ khcn (xn )− hcn (x)k + khcn (x) − h∞ (x)k and kyn − h∞ (x)k ≤ cǫn . Letting n → ∞ in the second inequality we get y = h∞ (x). Thus (A5) is satisfied. ky − h∞ (x)k ≤ ky − yn k +
Since assumptions (A1) − (A5) are satisfied the following is a direct corollary to Theorem 2. Corollary 1. If a SRE, given by (9), satisfies (BM 1) − (BM 3) then the corresponding approximate drift algorithm, given by (10), is stable almost surely. In addition, it converges to a closed, connected, invariant and internally chain transitive set of x(t) ˙ ∈ H(x(t)), where H(x) = h(x) + B ǫ (0). Since the result holds true even when ǫ = 0, it follows that the Borkar-Meyn Theorem emerges as a special case of Theorem 2. 16
4.2
Relaxation of assumption (BM1) of the Borkar-Meyn Theorem
The assumptions involved in the Borkar-Meyn Theorem [8] are listed in Section 4.1. One of the assumptions, (BM 1)(ii), requires that hc → h∞ uniformly on compacts. In this section we discuss how to dispense with assumption (BM 1)(ii). As discussed in Section 4.1 the Borkar-Meyn Theorem emerges as a special case of Theorem 2. This happens when ǫ = 0 and H(x) = {h(x)} for all x ∈ Rd . Hence the SRI given by (10) is same as the SRE given by (9). In the foregoing discussion H and h are as defined in Section 4.1. In the previous section we have shown that if (9) satisfies assumptions (BM 1) − (BM 3) then it also satisfies assumptions (A1)−(A5). In proving this we did not use (BM 1)(ii) i.e., hc → h∞ uniformly on compacts. Now, we discuss in brief how we work around using (BM 1)(ii) in proving the Borkar-Meyn Theorem. The notations used in this paragraph are consistent with those found in Chapter 3 of Borkar [7]. We list a few below. 1. φn (· , x) denotes the solution to x(t) ˙ ∈ hr(n) (x(t)) with initial value x. 2. φ∞ (· , x) denotes the solution to x(t) ˙ ∈ h∞ (x(t)) with initial value x. 3. xn (t), t ∈ [0, T ] denotes the solution to x˙ n (t) = hr(n) (ˆ x(Tn + t)) with initial value xn (0) = xˆ(Tn ). Then xn (t) = φn (t, xˆ(Tn )), t ∈ [0, T ]. For more details the reader is referred to Borkar and Meyn [8] or Chapter 3 of Borkar [7]. In proving the Borkar-Meyn Theorem as outlined in [8] (BM 1)(ii) is used to show that for large values of r(n), φn (t, xˆ(Tn )) is ‘close’ to φ∞ (t, xˆ(Tn )), t ∈ [0, T ]. Here we deviate from [8] in the definition of xn (t), t ∈ [0, T ]. In this paper, xn (· ) denotes the solution up to time T to x˙ n (t) = yˆ(Tn + t) = hr(n) (ˆ x([Tn + t])) with xn (0) = xˆ(Tn ), where [· ] is defined in Lemma 5. In other words, we have the following: xn (t) = xˆ(Tn ) +
k−1 X Z t(m(n)+l+1) l=0
t(m(n)+l)
yˆ(z) dz +
Z
t
yˆ(z) dz.
t(m(n)+k)
For t ∈ [tn , tn+1 ), yˆ(t) is a constant and equals yˆ(tn ). We get the following: n
x (t) = x ˆ(Tn ) +
k−1 X
a(m(n) + l)hr(n) (ˆ x([t(m(n) + l)])) +
l=0
(t − t(m(n) + k)) hr(n) (ˆ x([t(m(n) + k)])) .
If the Borkar-Meyn Theorem is proven along of lines of Section 3.2 i.e., Lemmas 1 - 5 and Theorem 1, then we essentially show the following: If r(n) ↑ ∞ then the T -length trajectories given by {xn (· )}n≥0 have φ∞ (x, t), t ∈ [0, T ], as the limit point in C([0, T ], Rd), where x ∈ B 1 (0). This is proven in Lemmas 4 and 5, the proofs of which do not require (BM 1)(ii).
17
5
Another Stability Theorem for Stochastic Recursive Inclusions
In (A4) we assumed that Liminfc→∞hc (x) is nonempty for all x ∈ Rd . In this section we shall develop a stability criterion for the case when we can no longer make such an assumption. In other words, we work with a modified version of assumption (A4).
5.1
A Modification of Assumption (A4)
Recall the following SRI : xn+1 = xn + a(n) [yn + Mn+1 ] , for n ≥ 0.
(11)
Since hc is point-wise bounded for each c ≥ 1, we have sup kyk ≤ K(1+kxk), y∈hc (x)
where x ∈ Rd (see Proposition 1). This implies that {yc }c≥1 , where yc ∈ hc (x), has at least one convergent subsequence. It follows from the definition of upperlimit of a sequence of sets (see Section 2.1) that Limsupc→∞ hc (x) is non-empty for every x ∈ Rd . It is worth noting that Liminfc→∞hc (x) ⊆ Limsupc→∞ hc (x) for every x ∈ Rd . Another important point to consider is that the lower-limits of sequences of sets are harder to compute than their upper-limits, see Aubin [2] for more details. Recall that hc (x) = {y | cy ∈ h(cx)}, where x ∈ Rd and c ≥ 1. Clearly the upper-limit, Limsupc→∞ hc (x) = {y | lim d(y, hc (x)) = 0} is nonempty for c→∞
every x ∈ Rd . For A ⊆ Rd , co(A) denotes the closure of the convex hull of A, i.e., the closure of the smallest convex set containing A. Define h∞ (x) := co ( Limsupc→∞ hc (x)) . Below we state the modification of assumption (A4) that we call (A6).
(A6) The differential inclusion x(t) ˙ ∈ h∞ (x(t)) has the origin as an attracting set and B 1 (0) is a subset of some fundamental neighborhood of the origin. Note that in (A4), h∞ (x) := Liminfc→∞ hc (x) while in (A6), h∞ (x) := co ( Limsupc→∞ hc (x)). In this section we shall work with this new definition of h∞ . Proposition 2. h∞ is a Marchaud map. Proof. From the definition of h∞ it follows that h∞ (x) is convex, compact for all x ∈ Rd and h∞ is point-wise bounded. It is left to prove that h∞ is an upper-semicontinuous map. Let xn → x, yn → y and yn ∈ h∞ (xn ), for all n ≥ 1. We need to show that y ∈ h∞ (x). We present a proof by contradiction. Since h∞ (x) is convex and compact, y ∈ / h∞ (x) implies that there exists a linear functional on Rd , say f , such that sup f (z) ≤ α − ǫ and f (y) ≥ α + ǫ, for some α ∈ R z∈h∞ (x)
18
and ǫ > 0. Since yn → y, there exists N > 0 such that for all n ≥ N , f (yn ) ≥ α + 2ǫ . In other words, h∞ (x) ∩ [f ≥ α + 2ǫ ] 6= φ for all n ≥ N . We use the notation [f ≥ a] to denote the set {x | f (x) ≥ a}. For the sake of convenience let us denote the set Limsupc→∞hc (x) by A(x), where x ∈ Rd . We claim that A(xn ) ∩ [f ≥ α + 2ǫ ] 6= φ for all n ≥ N . We prove this claim later, for now we assume that the claim is true and proceed. Pick zn ∈ A(xn )∩[f ≥ α+ 2ǫ ] for each n ≥ N . It can be shown that {zn }n≥N is norm bounded and hence contains a convergent subsequence, {zn(k) }k≥1 ⊆ {zn }n≥N . Let lim zn(k) = z. k→∞
Since zn(k) ∈ Limsupc→∞(hc (xn(k) )), ∃ cn(k) ∈ N such that kwn(k) − zn(k) k < 1 n(k) , where wn(k) ∈ hcn(k) (xn(k) ). We choose the sequence {cn(k) }k≥1 such that cn(k+1) > cn(k) for each k ≥ 1. We have the following: cn(k) ↑ ∞, xn(k) → x, wn(k) → z and wn(k) ∈ hcn(k) (xn(k) ), for all k ≥ 1. It follows from assumption (A5) that z ∈ h∞ (x). Since zn(k) → z and f (zn(k) ) ≥ α + 2ǫ for each k ≥ 1, we have that f (z) ≥ α + 2ǫ . This contradicts the earlier conclusion that sup f (z) ≤ α − ǫ. z∈h∞ (x)
It remains to prove that A(xn ) ∩ [f ≥ α + 2ǫ ] 6= φ for all n ≥ N . If this were not true, then ∃{m(k)}k≥1 ⊆ {n ≥ N } such that A(xm(k) ) ⊆ [f < α + 2ǫ ] for all k. It follows that h∞ (xm(k) ) = co(A(xm(k) )) ⊆ [f ≤ α + 2ǫ ] for each k ≥ 1. Since yn(k) → y, ∃N1 such that for all n(k) ≥ N1 , f (yn(k) ) ≥ α + 3ǫ 4 . This is a contradiction. We are now ready to state the second stability theorem for an SRI given by (11) under a modified set of assumptions. We retain assumptions (A1)−(A3), replace (A4) by (A6) and finally in (A5) we let h∞ (x) := co ( Limsupc→∞ hc (x)). We state the theorem under these updated set of assumptions. Theorem 4 (Stability Theorem for DI #2). Under assumptions (A1) − (A3), (A5) (with h∞ (x) := co(Limsupc→∞ hc (x))) and (A6), almost surely the sequence {xn }n≥0 generated by the stochastic recursive inclusion, given by (11) is bounded and converges to a closed, connected internally chain transitive invariant set of x(t) ˙ ∈ h(x(t)). Proof. The statements of Lemmas 1−5 hold true even when h∞ := co ( Limsupc→∞ hc (x)) and (A5) is interpreted as explained earlier. The stability of the iterates can be proven in an identical manner to the proof of Theorem 1. Next, we invoke Theorem 3.6 & Lemma 3.8 of Bena¨ım, Hofbauer and Sorin to conclude that the iterates converge to a closed, connected, internally chain transitive and invariant set of x(t) ˙ ∈ h(x(t)) [5].
5.2
The Borkar-Meyn Theorem Revisited
As explained in section 4.2, the assumptions involved in the Borkar-Meyn Theorem can be relaxed to omit (BM 1)(ii). As an immediate application of the second stability theorem (Theorem 4) we address the following question: If lim hc (x) does not exist for all x ∈ Rd , then what are the sufficient conditions c→∞ for the stability and convergence of the algorithm?
19
We take our cue from assumption (A6) in constructing the following replacement for (BM 1) that we call (BM 4). (BM4) (i) The function h : Rd → Rd is Lipschitz continuous, with Lipschitz constant L. Define the set-valued map, h∞ (x) := co (Limsupc→∞{hc (x)}), where x ∈ Rd . Note that Limsupc→∞{hc (x)} = {y | lim khc (x) − yk = 0}. c→∞
(ii) The differential inclusion x(t) ˙ ∈ h∞ (x(t)) has the origin as an attractor set and B 1 (0) is a subset of its fundamental neighborhood. Recall that the function Limsup is defined for a sequence of sets (see Section 2.1) and hc is a single valued map. For any x ∈ Rd and c ≥ 1, {hc (x)} is a set of cardinality one, hence Limsupc→∞{hc (x)} is well defined. Let us recall the following SRE: xn+1 = xn + a(n) [h(xn ) + Mn+1 ] , f or n ≥ 0.
(12)
Observe that Limsupc→∞ {hc (x)} = lim hc (x) when lim hc (x) exists for each c→∞
c→∞
x ∈ Rd , where Limsup is the upper-limit of a sequence of sets (see section 2.1). It can be shown that if a recursion given by (12) satisfies assumptions (BM 1)(i) and (BM 1)(iii) then it also satisfies (BM 4). Assumption (BM 4) unifies the two possible cases: when the limit of hc , as c → ∞, exists and when it does not.
We claim that a recursion given by (12), satisfying assumptions (BM 2), (BM 3) and (BM 4) will also satisfy (A1) − (A3), (A6) and (A5) (see section 5.1). From Theorem 4 it follows that the iterates are stable and converge to a closed, connected, internally chain transitive and invariant set of x(t) ˙ = h(x(t)). The following generalization of the Borkar-Meyn Theorem is a direct consequence of Theorem 4. Corollary 2 (Generalized Borkar-Meyn Theorem). Under assumptions (BM 2), (BM 3) and (BM 4), almost surely the sequence {xn }n≥0 generated by the stochastic recursive equation (12), is bounded and converges to a closed, connected, internally chain transitive and invariant set of x(t) ˙ = h(x(t)). Proof. Assumptions (A1) − (A3) and (A6) follow directly from (BM 2), (BM 3) and (BM 4). We show that (A5) is also satisfied. Let cn ↑ ∞, xn → x, yn → y and yn ∈ hcn (xn ) (here yn = hcn (xn )), ∀ n ≥ 1. It can be shown that khcn (xn ) − hcn (x)k ≤ Lkxn − xk. Hence we get that hcn (x) → y. In other words, lim khc (x) − yk = 0. Hence we have y ∈ h∞ (x). The claim now follows c→∞
from Theorem 4.
In Section 4.1 we showed that the approximate drift version of an algorithm is stable and convergent if the original algorithm satisfies (BM 1)(i), (iii), (BM 2) and (BM 3) (assumptions of the Borkar-Meyn Theorem). Similarly, we claim if a SRE given by (12) satisfies (BM 2), (BM 3) and (BM 4), then the corresponding approximate drift version satisfies (A1) − (A3), (A6) and (A5). The arguments involved are similar to those discussed in Section 4.1. We omit the proof to avoid repetition. It follows from Theorem 4 that the iterates are stable and 20
converge to a closed, connected, internally chain transitive and invariant set of x(t) ˙ ∈ h(x(t)) + B ǫ (0).
5.3
Relaxation of Assumptions (A4) and (A6)
In assumptions (A4) and (A6), the infinity system (x(t) ˙ ∈ h∞ (x(t))) is assumed to have the origin as an attracting set. Here, we discuss how we can relax this n+1 )k 1 assumption. In the proof of Theorem 1, we used kx(T kx(Tn )k < 2 to assert that the trajectory, x(· ), falls exponentially to within the unit ball. This helped us in proving the stability of the iterates. The same conclusion can be drawn n+1 )k even if we are able to prove that kx(T kx(Tn )k < δ, for some 0 < δ < 1. Once stability is established, ‘convergence’ of the algorithm follows, under appropriate assumptions, from Theorems 2 and 4. In this section we show that Theorems 2, 4 and Corollary 2 are valid even when the attracting set is ‘sufficiently’ close to the origin. We modify assumptions (A4), (A6) and (BM 4) as follows: In the aforementioned assumptions, we now assume that the associated differential inclusion, x(t) ˙ ∈ h∞ (x(t)), has an attracting set, A, such that sup kxk < 1. Further, we x∈A
assume that B 1 (0) is a subset of some fundamental neighborhood of A. In the remainder of this section we discuss in brief why the theorems hitherto stated still hold with this modification. Below, we state the modified version of assumption (A4) below. We call it (M A4). Assumptions (A6) and (BM 4) can be similarly modified to get (M A6) and (M BM 4), respectively. (MA4) h∞ (x) is nonempty ∀x ∈ Rd . The differential inclusion x(t) ˙ ∈ h∞ (x(t)) has an attracting set, A, such that sup kxk < 1. Further, it is assumed x∈A
that B 1 (0) is a subset of some fundamental neighborhood of A. In what follows it should be noted that the definition of h∞ changes with the assumptions being made. For example, when (M A4) is used we let h∞ (x) := Liminfc→∞hc (x); when (M A6) is used we let h∞ (x) := Limsupc→∞ hc (x) and when (BM 4) is used we let h∞ (x) := Limsupc→∞ {hc (x)}. Proposition 3. 1. Under assumptions (A1) − (A3), (M A4) and (A5), the statement of Theorem 2 is true. 2. Under assumptions (A1) − (A3), (M A6) and (A5), the statement of Theorem 4 is true. 3. Under assumptions (BM 2), (BM 3) and (M BM 4), the statement of Corollary 2 is true. Proof. We merely highlight the differences as the proofs essentially remain the same. Define δ1 := sup kxk and choose δ2 , δ3 and δ4 such that 0 ≤ δ1 < x∈A
δ2 < δ3 < δ4 < 1. Recall that T (δ) is the time beyond which any solution 21
to the differential inclusion given by x(t) ˙ ∈ h∞ (x(t)), with starting point in the fundamental neighborhood of A, remains within N δ (A). Note that h∞ is appropriately interpreted based on the theorem being proven. We define T := T (δ2 − δ1 ). The statements of Lemma 1 − Lemma 5 are true with the aforementioned definition of T . In the proof of Theorem 1, we choose N such that the following hold: ∀ m(l) ≥ N , kˆ x([Tl + T ])k < δ3 ; ∀ n ≥ N , a(n) < δ4 −δ3 ˆ ˆ [K(1+Kω )+Mω ] ; ∀ n > m ≥ N , kζn − ζm k < Mω and ∀ m(l) ≥ N , r(l) > 1. Note that Kω and Mω are as explained in Lemma 3. Using similar arguments − as before, we can conclude that kˆ x(Tl+1 )k < δ4 , where l is such that m(l) ≥ N . Finally we can conclude that, kx(Tl+1 )k kx(Tl )k
=
− kˆ x(Tl+1 )k kˆ x(Tl )k
< δ4 .
Since δ4 < 1, we can deduce that the trajectory falls exponentially till it hits the unit ball around the origin. The rest of the proof follows in a similar manner as before.
6
Conclusions
This paper presents an extension to the theorem of Borkar and Meyn that includes the case where the mean field is a set-valued map. Two different sets of assumptions are discussed which guarantee the ‘stability and convergence’ of a stochastic recursive inclusion. As an immediate application of Theorem 2, a solution to the ‘approximate drift problem’ is discussed. Further, the assumptions of the Borkar-Meyn Theorem are relaxed as a consequence of Theorem 2. As a corollary to Theorem 4, a generalization of the Borkar-Meyn Theorem is presented which includes the case when lim hc (x) does not exist for all x ∈ Rd . c→∞
An important future direction would be to extend these results to the case when the set-valued drift is governed by a Markov process in addition to the iterate sequence. For the case of stochastic approximations, such a situation has been considered in [ [7], Chapter 6], where the Markov ‘noise’ is tackled using the ‘natural timescale averaging’ properties of stochastic approximation.
References [1] J. Aubin and A. Cellina. Differential Inclusions: Set-Valued Maps and Viability Theory. Springer, 1984. [2] J. Aubin and H. Frankowska. Set-Valued Analysis. Birkh¨auser, 1990. [3] M. Bena¨ım. Dynamics of stochastic approximation algorithms. Seminaire de probabilites XXXIII, pages 1–68, 1999. [4] M. Bena¨ım. A dynamical system approach to stochastic approximations. SIAM J. Control Optim., 34(2):437–472, 1996. [5] M. Bena¨ım, J. Hofbauer, and S. Sorin. Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization, pages 328–348, 2005. 22
[6] S. Bhatnagar. The Borkar-Meyn theorem for asynchronous stochastic approximations. Systems & Control Letters, 60(7):472–478, 2011. [7] V. S. Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, 2008. [8] V. S. Borkar and S.P. Meyn. The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim, 38:447–469, 1999. [9] H. Kushner and G.G. Yin. Stochastic Approximation and Recursive Algorithms and Applications. Springer, 2003. [10] L. Ljung. Analysis of recursive stochastic algorithms. Automatic Control, IEEE Transactions on, 22(4):551–575, 1977.
23
arXiv:1502.01953v2 [cs.SY] 2 Mar 2015
Arunselvan Ramaswamy
1
and Shalabh Bhatnagar
2
1
[email protected] 2 [email protected] 1,2 Department of Computer Science and Automation, Indian Institute of Science, Bangalore - 560012, India. March 3, 2015 Abstract In this paper the stability theorem of Borkar and Meyn is extended to include the case when the mean field is a set-valued map. Two different sets of sufficient conditions are presented that guarantee the stability and convergence of stochastic recursive inclusions. Our work builds on the works of Bena¨ım, Hofbauer and Sorin as well as Borkar and Meyn. As a corollary to one of the main theorems, a natural generalization of the Borkar and Meyn Theorem follows. In addition, the original theorem of Borkar and Meyn is shown to hold under slightly relaxed assumptions. Finally, as an application to one of the main theorems we discuss a solution to the ‘approximate drift problem’.
1
Introduction
Consider the following recursion in Rd (d ≥ 1): xn+1 = xn + a(n) [h(xn ) + Mn+1 ] , f or n ≥ 0, where
(i) h : Rd → Rd is a Lipschitz continuous function.
(ii) a(n) P > 0, for all n, is the step-size sequence satisfying ∞ and n=0 a(n)2 < ∞.
P∞
n=0
(1)
a(n) = ∞
(iii) Mn , n ≥ 0, is a sequence of martingale difference terms. The stochastic recursion given by (1) is often referred to as a stochastic recursive equation (SRE). A powerful method to analyze the limiting behavior of (1) is the ODE (Ordinary Differential Equation) method. Here the limiting behavior of the algorithm is described in terms of the asymptotics of the solution to the ODE x(t) ˙ = h(x(t)). This method was introduced by Ljung [10] in 1977. For a detailed exposition on the subject and a survey of results, the reader is referred to Kushner and Yin [9] as well as Borkar [7]. 1
In 1996, Benaim [4] showed that the asymptotic behavior of a stochastic recursive equation can be studied by analyzing the asymptotic behavior of the associated o.d.e.. However no assumptions were made on the dynamics of the o.d.e. Specifically, he developed sufficient conditions which guarantee that limit sets of the continuously interpolated stochastic iterates are compact, connected, internally chain transitive and invariant sets of the associated o.d.e. The results found in [4] are generalized in [3]. The assumptions made in [4] are sometimes referred to as the ‘classical assumptions’. One of the key assumptions used by Bena¨ım to prove this convergence theorem is the almost sure boundedness of the iterates. In other words the iterates are stable. In 1999, Borkar and Meyn [8] developed sufficient conditions which guarantee both the stability and convergence of stochastic recursive equations. These assumptions were consistent with those developed in [4]. In this paper we refer to the main result of Borkar and Meyn colloquially as the Borkar-Meyn Theorem. In the same paper [8], several applications to problems from reinforcement learning have also been discussed. An extention to the Borkar-Meyn result for asynchronous stochastic iterates was given by Bhatnagar [6]. In 2005, Bena¨ım, Hofbauer and Sorin [5] showed that the dynamical systems approach can be extended to the situation where the mean field is a set-valued map. In other words, they considered algorithms with iterates of the form: xn+1 = xn + a(n) [yn + Mn+1 ] , f or n ≥ 0, where (2) (i) yn ∈ h(xn ) and h : Rd → {subsets of Rd } is a Marchaud map. For a definition of Marchaud maps the reader is referred to section 2.1. P∞ (ii) a(n) P > 0, for all n, is the step-size sequence satisfying n=0 a(n) = ∞ ∞ and n=0 a(n)2 < ∞.
(iii) Mn , n ≥ 0, is a sequence of martingale difference terms.
Such iterates are referred to as stochastic recursive inclusions (SRI). An SRE given by (1) can be seen as a special case of an SRI given by (2). Also, a differential equation can be seen as a special case of a differential inclusion where the set {h(x)} has cardinality one for all x ∈ Rd . The main aim of this paper is to extend the original Borkar-Meyn theorem to the case of stochastic recursive inclusions. We present two overlapping yet different sets of assumptions, in Sections 2.2 and 5.1 respectively, that guarantee the stability and convergence of a SRI given by (2). We present several interesting consequences to the main results (Theorems 2 and 4). Specifically, in Section 5.2 we show that the Borkar-Meyn theorem can be naturally extended to include the case where the infinity system is more generally allowed to be the set of accumulation points. We present a unified set of assumptions that takes care of both the original theorem and the aforementioned extension. While doing so we show how the assumptions of the Borkar-Meyn theorem could be relaxed.
2
The organization of this paper is as follows: In section 2.1, we discuss the definitions and notations used in this paper. In section 2.2, we discuss the assumptions under which the iterates given by (2) are stable and converge to a closed, connected, internally chain transitive and invariant set. A preliminary result is also presented in this section. In section 3.1, we present auxiliary results needed to prove the main result. In section 3.2, a stability theorem (Theorem 2) for stochastic recursive inclusions is proved under the assumptions outlined in section 2.2. Theorem 2 is one of two stability theorems presented in this paper. In section 4 we discuss applications of Theorem 2. Specifically, in section 4.1 a solution to the problem of ‘approximate drift’ is discussed. For more details on the ‘approximate drift’ problem the reader is referred to Borkar [7]. In section 4.2, the Borkar-Meyn Theorem is proved under a relaxed set of assumptions. In section 5.1, another set of assumptions that guarantee the stability and convergence to a closed, connected, internally chain transitive and invariant set of a stochastic recursive inclusion is discussed. In the same section we provide a brief outline of the proof of stability under these set of assumptions. This yields our second stability result for SRI : Theorem 4. In section 5.2, as an application of Theorem 4, the Borkar-Meyn Theorem [8] is generalized to when one the key requirements does not hold. This result is summarized in Corollary 2. Finally, in section 5.3 we discuss how the assumptions described in sections 2.2, 5.1 could be relaxed.
2
Preliminaries and Assumptions
2.1
Definitions and Notations
The definitions and notations used in this paper are similar to those in Bena¨ım et. al. [5], Aubin et. al. [1], [2] and Borkar [7]. In this section, we present a few for easy reference. A set-valued map h : Rn → {subsets of Rm } is called a Marchaud map if it satisfies the following properties: (i) For each x ∈ Rn , h(x) is convex and compact. (ii) (point-wise boundedness) For each x ∈ Rn , sup kwk < K (1 + kxk) for w∈h(x)
some K > 0. (iii) h is an upper-semicontinuous map. We say that h is upper-semicontinuous, if given sequences {xn }n≥1 (in Rn ) and {yn }n≥1 (in Rm ) with xn → x, yn → y and yn ∈ h(xn ), n ≥ 1, implies that y ∈ h(x). In other words the graph of h, {(x, y) : y ∈ h(x), x ∈ Rn }, is closed in Rn × Rm . Let H be a Marchaud map on Rd . The differential inclusion (DI) given by x˙ ∈ H(x)
(3)
is guaranteed to have at least one solution that is absolutely P continuous. The reader is referred to [1] for more details. We say that x ∈ if x is an absolutely continuous map that satisfies (3). The set-valued semiflow Φ associated with 3
(3) is defined on [0, +∞) × Rd as: Φt (x) = {x(t) | x ∈ B × M ⊂ [0, +∞) × Rk and define [ ΦB (M ) = Φt (x).
P
, x(0) = x}. Let
t∈B, x∈M
T Let M ⊆ Rd , the ω − limit set be defined by ωΦ (M ) = t≥0 Φ[t,+∞) (M ). T Similarly the limit set of a solution x is given by L(x) = t≥0 x([t, +∞)). M ⊆ Rd is invariant if for every x ∈ MPthere exists a trajectory, x, entirely in M with x(0) = x. In other words, x ∈ with x(t) ∈ M , for all t ≥ 0. Internally Chain Transitive Set : M ⊂ Rd is said to be internally chain transitive if M is compact and for every x, y ∈ M , ǫ > 0 and T > 0 we have the following: There exist Φ1 , . . . , Φn that are n solutions to the differential inclusion x(t) ˙ ∈ h(x(t)), a sequence x1 (= x), . . . , xn+1 (= y) ⊂ M and n real numbers t1 , t2 , . . . , tn greater than T such that: Φiti (xi ) ∈ N ǫ (xi+1 ) and Φi[0,ti ] (xi ) ⊂ M for 1 ≤ i ≤ n. The sequence (x1 (= x), . . . , xn+1 (= y)) is called an (ǫ, T ) chain in M from x to y. Let x ∈ Rd and A ⊆ Rd , then d(x, A) := inf{ka − yk | y ∈ A}. We define the δ-open neighborhood of A by N δ (A) := {x | d(x, A) < δ}. The δ-closed neighborhood of A is defined by N δ (A) := {x | d(x, A) ≤ δ}. A ⊆ Rd is an attracting set if it is compact and there exists a neighborhood U such that for any ǫ > 0, ∃ T (ǫ) ≥ 0 such that Φ[T (ǫ),+∞) (U ) ⊂ N ǫ (A). Such a U is called the fundamental neighborhood of A. In addition to being compact if the attracting set is also invariant then it is called an attractor. The basin of attraction of A is given by B(A) = {x | ωΦ (x) ⊂ A}. It is called Lyapunov stable if for all δ > 0, ∃ ǫ > 0 such that Φ[0,+∞) (N ǫ (A)) ⊆ N δ (A). We use T (ǫ) and Tǫ interchangeably to denote the dependence of T on ǫ. The open ball of radius r around 0 is represented by Br (0), while the closed ball is represented by B r (0). We define the lower and upper limits of sequences of sets. Let {Kn }n≥1 be a sequence of sets in Rd . 1. The lower limit of {Kn }n≥1 is given by, Liminfn→∞Kn := {x | lim d(x, Kn ) = 0}. n→∞
2. The upper-limit of {Kn }n≥1 given by, Limsupn→∞Kn := {y | lim d(y, Kn ) = 0}. We may interpret that the n→∞
upper-limit collects the accumulation points of {Kn }n≥1 .
2.2
The assumptions
Recall that we have the following recursion in Rd : xn+1 = xn + a(n) [yn + Mn+1 ], where yn ∈ h(xn ). We state our assumptions below:
(A1) h : Rd → {subsets of Rd } is a Marchaud map. P a(n) = ∞ and (A2) {a(n)}n≥0 is a scalar sequence such that: a(n) ≥ 0 ∀n, n≥0 P a(n)2 < ∞. Without loss of generality we let sup a(n) ≤ 1. n
n≥0
4
(A3)
{Mn }n≥1 is a martingale difference sequence with respect to the filtration Fn := σ (x0 , M1 , . . . , Mn ), n ≥ 0. (i) {Mn }n≥1 is a square integrable sequence with E [Mn+1 |Fn ] = 0 a.s., for n ≥ 0. (ii) E[kMn+1 k2 |Fn ] ≤ K 1 + kxn k2 , for n ≥ 0 and some constant K > 0. Without loss of generality assume that the same constant, K, works for both the point-wise boundedness condition of (A1) (see conditions (ii) of definition of Marchaud map in Section 2.1) and (A3).
For c ≥ 1 and x ∈ Rd , define hc (x) = {y | cy ∈ h(cx)}. For each x ∈ Rd , define h∞ (x) := Liminfc→∞ hc (x) i.e. the closure of the lower limit of {hc (x)}c≥1 .
(A4) h∞ (x) is non-empty for all x ∈ Rd . Further, the differential inclusion x(t) ˙ ∈ h∞ (x(t)) has the origin as an attracting set and B 1 (0) is a subset of its fundamental neighborhood. (A5) Let cn ≥ 1 be an increasing sequence of integers such that cn ↑ ∞ as n → ∞. Further, let xn → x and yn → y as n → ∞, such that yn ∈ hcn (xn ), ∀n, then y ∈ h∞ (x). Assumptions (A1) − (A3) are the same as in Bena¨ım [5]. However, the assumption on the stability of the iterates is replaced by (A4) and (A5). We show that (A4) and (A5) are sufficient conditions to ensure stability of iterates. We start by observing that hc and h∞ are Marchaud maps, where c ≥ 1. Further, we show that the constant associated with the point-wise boundedness property is K of (A1) and (A3). Proposition 1. h∞ and hc , c ≥ 1, are Marchaud maps.
Proof. Fix c ≥ 1 and x ∈ Rd . To prove that hc (x) is compact, we show that it is closed and bounded. For n ≥ 1, let yn ∈ hc (x) and let lim yn = y. It follows n→∞
that cyn ∈ h(cx) for each n ≥ 1 and lim cyn = cy. Since h(cx) is closed, we n→∞
have that cy ∈ h(cx) and y ∈ hc (x). If we show that hc is point-wise bounded then we can conclude that hc (x) is compact. To prove the aforementioned, let y ∈ hc (x), then cy ∈ h(cx). Since h satisfies (A1)(ii), we have that ckyk ≤ K (1 + kcxk) , hence 1 kyk ≤ K + kxk . c
Since c(≥ 1) and x is arbitrarily chosen, hc is point-wise bounded and the compactness of hc (x) follows. The set hc (x) = {z/c | z ∈ h(cx)} is convex since h(cx) is convex and hc (x) is obtained by scaling it by 1c . Next, we show that hc (x) is upper-semicontinuous. Let lim xn = x, lim n→∞
n→∞
yn = y and yn ∈ hc (xn ), ∀ n ≥ 1. We need to show that y ∈ hc (x). We have that cyn ∈ h(cxn ) for each n ≥ 1. Since lim cxn = cx and lim cyn = cy, we n→∞
n→∞
conclude that cy ∈ h(cx) since h is assumed to be upper-semicontinuous. 5
It is left to show that h∞ (x), x ∈ Rd is a Marchaud map. To prove that kzk ≤ K (1 + kxk) for all z ∈ h∞ (x), it is enough to prove that kyk ≤ K (1 + kxk) for all y ∈ Liminfc→∞ hc (x). Fix some y ∈ Liminfc→∞ hc (x) then there exist zn ∈ hn (x), n ≥ 1, such that lim ky − zn k = 0. We have that n→∞
kyk ≤ ky − zn k + kzn k. Since hc , c ≥ 1, is point-wise bounded (the constant associated is independent of c and equals K) the above inequality becomes kyk ≤ ky − zn k + K (1 + kxk). Letting n → ∞ in the above inequality, we obtain kyk ≤ K (1 + kxk). Recall that h∞ (x) = Liminfc→∞ hc (x), hence it is compact. Again, to show that h∞ (x) is convex, for each x ∈ Rd , we start by proving that Liminfc→∞ hc (x) is convex. Let u, v ∈ Liminfc→∞ hc (x) and 0 ≤ t ≤ 1. We need to show that tu + (1 − t)v ∈ Liminfc→∞ hc (x). Consider an arbitrary sequence {cn }n≥1 such that cn → ∞, then there exist un , vn ∈ hcn (x) such that kun − uk and kvn − vk → 0 as cn → ∞. Since hcn (x) is convex, it follows that tun + (1 − t)vn ∈ hcn (x), further lim ( tun + (1 − t)vn ) = tu + (1 − t)v.
cn →∞
Since we started with an arbitrary sequence cn → ∞, it follows that tu + (1 − t)v ∈ Liminfc→∞ hc (x). Now we can prove that h∞ (x) is convex. Let u, v ∈ h∞ (x). Then ∃ {un }n≥1 and {vn }n≥1 ⊆ Liminfc→∞ hc (x) such that un → u and vn → v as n → ∞. We need to show that tu + (1 − t)v ∈ h∞ (x), for 0 ≤ t ≤ 1. Since tun + (1 − t)vn ∈ Liminfc→∞ hc (x), the desired result is obtained by letting n → ∞ in tun + (1 − t)vn . Finally, we show that h∞ is upper-semicontinuous. Let lim xn = x, lim n→∞
n→∞
yn = y and yn ∈ h∞ (xn ), ∀ n ≥ 1. We need to show that y ∈ h∞ (x). Since yn ∈ h∞ (xn ), ∃ zn ∈ Liminfc→∞ hc (xn ) such that kzn − yn k < n1 . Since zn ∈ Liminfc→∞ hc (xn ), n ≥ 1, it follows that there exist cn such that for all c ≥ cn , d (zn , hc (xn )) < n1 . In particular, ∃ un ∈ hcn (xn ) such that kzn − un k < n1 . We choose the sequence {cn }n≥1 such that cn+1 > cn for each n ≥ 1. We now have the following: lim un = y, un ∈ hcn (xn ) ∀ n and lim xn = x. It n→∞
n→∞
follows directly from assumption (A5) that y ∈ h∞ (x).
3
Stability and convergence of stochastic recursive inclusions
We begin by providing a brief outline of our approach to prove the stability of a SRI under assumptions (A1) − (A5). First we divide the time line, [0, ∞), approximately into intervals of length T . We shall explain later how we choose and fix T . Then we construct a linearly interpolated trajectory from the given stochastic recursive inclusion; the construction is explained in the next paragraph. A sequence of ‘rescaled’ trajectories of length T is constructed as follows:
6
At the beginning of each T -length interval we observe the trajectory to see if it is outside the unit ball, if so we scale it back to the boundary of the unit ball. This scaling factor is then used to scale the ‘rest of the T -length trajectory’. To show that the iterates are bounded almost surely we need to show that the linearly interpolated trajectory does not ‘run off’ to infinity. To do so we assume that this is not true and show that there exists a subsequence of the rescaled T -length trajectories that has a solution to x(t) ˙ ∈ h∞ (x(t)) as a limit point in C([0, T ], Rd ). We choose and fix T such that any solution to x(t) ˙ ∈ h∞ (x(t)) with an initial value inside the unit ball is close to the origin at the end of time T . For example we could choose T = T (1/8). We then argue that the linearly interpolated trajectory is forced to make arbitrarily large ‘jumps’ within time T . The Gronwall inequality is then used to show that this is not possible. Once we prove stability of the recursion we invoke Theorem 3.6 & Lemma 3.8 from Bena¨ım, Hofbauer and Sorin [5] to conclude that the limit set is a closed, connected, internally chain transitive and invariant set associated with x(t) ˙ ∈ h∞ (x(t)). We construct the linearly interpolated trajectory x(t), for t ∈ [0, ∞), from Pn−1 the sequence {xn } as follows: Define t(0) := 0, t(n) := i=0 a(i). Let x(t(n)) := xn and for t ∈ (t(n), t(n + 1)), let t(n + 1) − t t − t(n) x(t) := x(t(n)) + x(t(n + 1)). t(n + 1) − t(n) t(n + 1) − t(n) We define a piecewise constant trajectory using the sequence {yn }n≥0 as follows: y(t) := yn for t ∈ [t(n), t(n + 1)), n ≥ 0. We know that the DI given by x(t) ˙ ∈ h∞ (x(t)) has the origin as an attractor set. Let us fix T := T (1/8), where T (1/8) is as defined in section 2.1. Then, ˙ ∈ kx(t)k < 81 , for all t ≥ T1/8 , where {x(t) : t ∈ [0, ∞)} is a solution to x(t) h∞ (x(t)) with an initial value inside the unit ball around the origin. Define T0 := 0 and Tn := min{t(m) : t(m) ≥ Tn−1 + T }, n ≥ 1. Observe that there exists a subsequence {m(n)}n≥0 of {n} such that Tn = t(m(n)) ∀ n ≥ 0. We construct the rescaled trajectory, x ˆ(t), t ≥ 0, as follows: Let t ∈ [Tn , Tn+1 ) x(t) for some n ≥ 0, then x ˆ(t) := r(n) , where r(n) = kx(Tn )k ∨ 1. Also, let − x ˆ(Tn+1 ) := lim xˆ(t), t ∈ [Tn , Tn+1 ). The corresponding ‘rescaled y itert↑Tn+1
y(t) and the rescaled martingale noise terms by ates’ are given by yˆ(t) := r(n) Mk+1 ˆ Mk+1 := r(n) , t(k) ∈ [Tn , Tn+1 ), n ≥ 0.
Consider the recursion at hand, i.e., x(t(k + 1)) = x(t(k)) + a(k) (y(t(k)) + Mk+1 ) , such that t(k), t(k + 1) ∈ [Tn , Tn+1 ). Multiplying both sides by 1/r(n) we get the rescaled recursion: ˆ k+1 . xˆ(t(k + 1)) = x ˆ(t(k)) + a(k) yˆ(t(k)) + M 7
Since y(t(k)) ∈h h (x(t(k))), it hr(n) (ˆ x(t(k))). It is worth i follows that yˆ(t(k)) ∈ ˆ k+1 k2 |Fk ≤ K 1 + kˆ noting that E kM x(t(k))k2 .
Characterizing limits, in C([0, T ], Rd ), of the rescaled trajectories
3.1
The first two lemmas prove that the rescaled martingale difference noise converges almost surely. Although they can be found in Borakar [8], we present them here with proofs for the sake of completeness and easy reference. Lemma 1. sup Ekˆ x(t)k2 < ∞. t∈[0,T ]
Proof. It is enough to show that sup m(n)≤k<m(n+1)
E kˆ x(t(k))k2 ≤ M,
for some M (> 0) that is independent of n. Recall that Tn = t(m(n)) and Tn+1 = t(m(n+1)). Let us fix n and k such that n ≥ 0 and m(n) ≤ k < m(n+1). Now consider ˆk . xˆ(t(k)) = x ˆ(t(k − 1)) + a(k − 1) yˆ(t(k − 1)) + M Unfolding the above recursion we get, x ˆ(t(k)) = x ˆ(t(m(n))) +
k−1 X
l=m(n)
ˆ l+1 . a(l) yˆ(t(l)) + M
Taking the expectation of the square of the norms on both sides of the above equation we get, k−1 X ˆ l+1 k2 . a(l) yˆ(t(l)) + M Ekˆ x(t(k))k2 = E kˆ x(t(m(n))) + l=m(n)
It follows from the Minkowski inequality that, E 1/2 kˆ x(t(k))k2 ≤ E 1/2 kˆ x(Tn )k2 +
k−1 X
l=m(n)
ˆ l+1 k2 . a(l) E 1/2 kˆ y (t(l))k2 + E 1/2 kM
From assumptions (A1) and (A2), it i for each m(n) ≤ l ≤ k − 1, h follows that ˆ l+1 k2 |Fl ≤ K 1 + kˆ x(t(l))k2 , respeckˆ y(t(l))k ≤ K (1 + kˆ x(t(l))k) and E kM
tively. We also observe that Tn+1 − Tn ≤ T + 1 (since supn a(n) ≤ 1). Using these observations we obtain the following set of inequalities: E
1/2
2
kˆ x(t(k))k ≤ 1 +
k−1 X
l=m(n)
√ 2 a(l) KE 1/2 (1 + kˆ x(t(l))k) + KE 1/2 1 + kˆ x(t(l))k2 ,
8
E
1/2
2
kˆ x(t(k))k ≤ 1 +
k−1 X
l=m(n)
√ a(l) K 1 + E 1/2 kˆ x(t(l))k2 + K 1 + E 1/2 kˆ x(t(l))k2 ,
k−1 i h X √ √ a(l)E 1/2 kˆ x(t(l))k2 . E 1/2 kˆ x(t(k))k2 ≤ 1 + (K + K)(T + 1) +(K+ K) l=m(n)
Applying the discrete version of the Gronwall inequality we get, i h √ √ E 1/2 kˆ x(t(k))k2 ≤ 1 + (K + K)(T + 1) e(K+ K)(T +1) . The claim follows by letting M =
h
1 + (K +
2 i √ √ K)(T + 1) e(K+ K)(T +1) .
Pn−1 ˆ Lemma 2. The rescaled sequence {ζˆn }n≥1 , where ζˆn = k=0 a(k)Mk+1 , is convergent almost surely. Proof. It is enough to prove that almost surely, ∞ X
k=0
Instead, we prove that " E
∞ X
k=0
i h ˆ k+1 k2 | Fk < ∞. E ka(k)M
i h ˆ k+1 k2 | Fk a(k) E kM 2
#
< ∞.
It follows as a consequence of assumption (A3) that, "∞ # ∞ h i X X ˆ k+1 k2 | Fk ≤ E a(k)2 K 1 + Ekˆ x(t(k))k2 . a(k)2 E kM k=0
k=0
The claim now follows from assumption (A2) and Lemma 1. The rest of the lemmas are needed to prove the stability theorem, Theorem 1. We begin by showing that the rescaled trajectories are bounded almost surely. Lemma 3.
sup kˆ x(t)k < ∞ a.s.
t∈[0,∞)
Proof. Let A = {ω | {ζˆn (ω)}n≥1 converges}, be the set on which {ζˆn }n≥1 converges. It is enough to prove that kˆ x(t(m(n) + k))k < Kω , where Tn ≤ t(m(n) + k) < Tn+1 and Kω is a constant independent of n. The constant may however be dependent on ω (sample path), where ω ∈ A.
9
Let us consider the rescaled recursion: ˆ m(n)+k . x ˆ(t(m(n)+k)) = x ˆ(t(m(n)+k−1)) + a(m(n)+k−1) yˆ(t(m(n) + k − 1)) + M Unfolding the above recursion, we obtain x ˆ(t(m(n) + k)) = xˆ(Tn ) + k−1 X l=0
ˆ m(n)+l+1 . (4) a(m(n) + l) yˆ(t(m(n) + l)) + M
Since ζˆn , n ≥ 1, converges on A, there exists Mω < ∞, that may be sample Pk−1 ˆ m(n)+l+1 k ≤ Mw , where Mω is path dependent, such that k l=0 a(m(n) + l)M independent of n. Since kˆ x(Tn )k ≤ 1 we have, kˆ x(t(m(n) + k))k ≤ 1 +
k−1 X
a(m(n) + l)kˆ y(t(m(n) + l))k + Mω .
l=0
From (A1) it follows that, kˆ x(t(m(n) + k))k ≤ 1 + K
k−1 X l=0
a(m(n) + l) (1 + kˆ x(t(m(n) + l))k) + Mω .
Rearranging the terms in the above inequality and applying the discrete Gronwall inequality we get, kˆ x(t(m(n) + k))k ≤ (1 + Mω + (T + 1)K) eK(T +1) , a constant independent of both n and k. The rest of the proof follows in a straightforward manner. Note that in the proof of the above lemma we get a bound on kˆ x(t)k that is dependent on T . This is sufficient for our purposes, since we fix T to be T1/8 . Let xn (t), t ∈ [0, T ] be the solution (upto time T ) to x˙ n (t) = yˆ(Tn + t), with the initial condition xn (0) = x ˆ(Tn ). Clearly, we have n
x (t) = x ˆ(Tn ) +
Z
t
yˆ(Tn + z) dz.
(5)
0
The following two lemmas are inspired by ideas from Benaim, Hofbauer and Sorin [5] as well as Borkar [7]. In the lemma that follows we show that the limit sets of {xn (· ) | n ≥ 0} and {ˆ x(Tn +· ) | n ≥ 0} coincide. We seek limits in C([0, T ], Rd ). Lemma 4. lim
sup
n→∞ t∈[T ,T +T ] n n
kxn (t) − x ˆ(t)k = 0 a.s.
Proof. Let t ∈ [t(m(n) + k), t(m(n) + k + 1)) and t(m(n) + k + 1) ≤ Tn+1 . We first assume that t(m(n) + k + 1) < Tn+1 . We have the following: x ˆ(t) =
t(m(n) + k + 1) − t a(m(n) + k)
xˆ(t(m(n)+k))+
10
t − t(m(n) + k) a(m(n) + k)
x ˆ(t(m(n)+k+1)).
Substituting for x ˆ(t(m(n) + k + 1)) in the above equation we get: t − t(m(n) + k) t(m(n) + k + 1) − t xˆ(t(m(n) + k)) + x ˆ(t) = a(m(n) + k) a(m(n) + k) ˆ m(n)+k+1 , x ˆ(t(m(n) + k)) + a(m(n) + k) yˆ(t(m(n) + k)) + M
hence,
ˆ m(n)+k+1 . x ˆ(t) = x ˆ(t(m(n) + k)) + (t − t(m(n) + k)) yˆ(t(m(n) + k)) + M
Unfolding x ˆ(t(m(n) + k)) over k (see (4)) we get, x ˆ(t) = x ˆ(Tn ) +
k−1 X l=0
ˆ m(n)+l+1 + a(m(n) + l) yˆ(t(m(n) + l)) + M
ˆ m(n)+k+1 . (6) (t − t(m(n) + k)) yˆ(t(m(n) + k)) + M
Now, we consider xn (t), i.e., n
x (t) = x ˆ(Tn ) +
Z
t
yˆ(Tn + z) dz.
0
Splitting the above integral, we get Z k−1 X Z t(m(n)+l+1) n yˆ(z) dz + x (t) = xˆ(Tn ) + l=0
t
yˆ(z) dz.
t(m(n)+k)
t(m(n)+l)
Thus, xn (t) = x ˆ(Tn ) +
k−1 X
a(m(n) + l)ˆ y(t(m(n) + l))+
l=0
(t − t(m(n) + k)) yˆ(t(m(n) + k)). (7)
From (6) and (7), it follows that
k−1
X
n ˆ m(n)+l+1 ˆ m(n)+k+1 a(m(n) + l)M kx (t)−ˆ x(t)k ≤
+ (t − t(m(n) + k)) M
,
l=0
and hence,
kxn (t) − x ˆ(t)k ≤ kζˆm(n)+k − ζˆm(n) k + kζˆm(n)+k+1 − ζˆm(n)+k k. If t(m(n) + k + 1) = Tn+1 then in the proof we may replace x ˆ(t(m(n) + k + 1)) − with x ˆ(Tn+1 ). The arguments remain the same. Since ζˆn , n ≥ 1, converges almost surely, the desired result follows. The sets {xn (t), t ∈ [0, T ] | n ≥ 0} and {ˆ x(Tn + t), t ∈ [0, T ] | n ≥ 0} can be viewed as subsets of C([0, T ], Rd ). It can be shown that {xn (t), t ∈ [0, T ] | n ≥ 0} is equi-continuous and point-wise bounded. Thus from the Arzela-Ascoli theorem, {xn (t), t ∈ [0, T ] | n ≥ 0} is relatively compact. It follows from Lemma 4 that the set {ˆ x(Tn + t), t ∈ [0, T ] | n ≥ 0} is also relatively compact in C([0, T ], Rd ). 11
Lemma 5. Let r(n) ↑ ∞, then any limit point of {ˆ x(Tn + t), t ∈ [0, T ] : n ≥ 0} Rt is of the form x(t) = x(0) + 0 y(s) ds, where y : [0, T ] → Rd is a measurable function and y(t) ∈ h∞ (x(t)), t ∈ [0, T ]. Proof. We define the notation [t] := max{t(k) | t(k) ≤ t}, t ≥ 0. Let t ∈ [Tn , Tn+1 ), then yˆ(t) ∈ hr(n) (ˆ x([t])) and kˆ y (t)k ≤ K (1 + kˆ x([t])k) since hr(n) is a Marchaud map (K is the constant associated with the point-wise boundedness property). It follows from Lemma 3 that sup kˆ y(t)k < ∞ a.s. Using obsert∈[0,∞)
vations made earlier, we can deduce that there exists a sub-sequence of N, say d {l} ⊆ {n}, such that x ˆ(T ) → y(· ) l + t) → x(t) in C [0, T ], R and yˆ(m(l)+· d weakly in L2 [0, T ], R . From Lemma 4 it follows that xl (· ) → x(· ) in C [0, T ], Rd . Letting r(l) ↑ ∞ in Z t xl (t) = xl (0) + yˆ(t(m(l) + z)) dz, t ∈ [0, T ], 0
we get x(t) = x(0) + kx(0)k ≤ 1.
Rt 0
y(z)dz for t ∈ [0, T ]. Since kˆ x(Tn )k ≤ 1 we have
Since yˆ(Tl + · ) → y(· ) weakly in L2 [0, T ], Rd , there exists {l(k)} ⊆ {l} such that N 1 X yˆ(Tl(k) + · ) → y(· ) strongly in L2 [0, T ], Rd . N k=1
Further, there exists {N (m)} ⊆ {N } such that
N (m) X 1 yˆ(Tl(k) + · ) → y(· ) a.e. on [0, T ]. N (m) k=1
Let us fix t0 ∈ {t |
1 N (m)
PN (m) k=1
yˆ(Tl(k) + t) → y(t), t ∈ [0, T ]}, then
N (m) X 1 yˆ(Tl(k) + t0 ) = y(t0 ). N (m)→∞ N (m)
lim
k=1
Since h∞ (x(t0 )) is convex and compact, to show that y(t0 ) ∈ h∞ (x(t0 )) it is enough to prove that lim d yˆ(Tl(k) + t0 ), h∞ (x(t0 )) = 0. If not, ∃ ǫ > 0 and l(k)→∞ {n(k)} ⊆ {l(k)} such that d yˆ(Tn(k) + t0 ), h∞ (x(t0 )) > ǫ. Since {ˆ y(Tn(k) + t0 )}k≥1 is norm bounded, it follows that there is a convergent sub-sequence. For the sake of convenience we assume that lim yˆ(Tn(k) + t0 ) = y, for some k→∞
y ∈ Rd . Since yˆ(Tn(k) + t0 ) ∈ hr(n(k)) (ˆ x([Tn(k) + t0 ])) and lim x ˆ([Tn(k) + t0 ]) = k→∞
x(t0 ), it follows from assumption (A5) that y ∈ h∞ (x(t0 )). This leads to a contradiction.
Note that in the statement of Lemma 5 we can replace ‘r(n) ↑ ∞’ by ‘r(l) ↑ ∞’, where {r(l))} is a subsequence of {r(n)}. Specifically we can conclude that any limit point of {ˆ x(Tk + t), t ∈ [0, T ]}{k}⊆{n} in C([0, T ], Rd), conditioned on Rt r(k) ↑ ∞, is of the form x(t) = x(0) + 0 y(z) dz, where y(t) ∈ h∞ (x(t)) for t ∈ [0, T ]. It should be noted that y(· ) may be sample path dependent. 12
3.2
The Stability Theorem
We are now ready to prove the stability of a SRI given by (2) under the assumptions (A1) − (A5). If sup r(n) < ∞, then the iterates are stable and n
there is nothing to prove. If on the other hand sup r(n) = ∞, there exists n
{l} ⊆ {n} such that r(l) ↑ ∞. It follows from Lemma 5 that any limit point of Rt {ˆ x(Tl + t), t ∈ [0, T ] : {l} ⊆ {n}} is of the form x(t) = x(0) + 0 y(s) ds, where y(t) ∈ h∞ (x(t)) for t ∈ [0, T ]. From assumption (A4), we have that kxk < 1/8 (T = T (1/8)). Since the time intervals are roughly T apart, for large values of − − k < 14 , where x r(n) we can conclude that kˆ x Tn+1 ˆ(Tn+1 ) = limt↑t(m(n+1)) x ˆ(t), t ∈ [Tn , Tn+1 ). Theorem 1 (Stability Theorem for DI). Under assumptions (A1) − (A5), supkxn k < ∞ a.s. n
Proof. As explained earlier it is sufficient to consider the case when sup r(n) = n
∞. Let {l} ⊆ {n} such that r(l) ↑ ∞. Recall that Tl = t(m(l)) and that [Tl + T ] = max{t(k) | t(k) ≤ Tl + T }. We have kx(T )k < 18 since x(t) is a solution, up to time T , to the DI given by x(t) ˙ ∈ h∞ (x(t)) and we have fixed T = T (1/8). From Lemma 5 we conclude that there exists N such that all of the following happen: (i) m(l) ≥ N =⇒ kˆ x([Tl + T ])k < 41 . (ii) n ≥ N =⇒ a(n)
m ≥ N =⇒ kζˆn − ζˆm k < Mω . (iv) m(l) ≥ N =⇒ r(l) > 1. In the above Kω and Mω are as explained in Lemma 3. Let m(l) ≥ N and t(m(l + 1)) = t(m(l) + k + 1) for some k ≥ 0. Clearly from the manner in which the Tn sequence is defined, we have t(m(l)+k) = [Tl + T ]. − As defined earlier x ˆ(Tn+1 ) = limt↑t(m(n+1)) x ˆ(t), t ∈ [Tn , Tn+1 ) and n ≥ 0. Consider the equation − ˆ m(l)+k+1 . x ˆ(Tl+1 ) = x ˆ(t(m(l) + k)) + a(m(l) + k) yˆ(t(m(l) + k)) + M Taking norms on both sides we get,
− kˆ x(Tl+1 )k ≤ kˆ x(t(m(l) + k))k +
ˆ m(l)+k+1 k. a(m(l) + k)kˆ y(t(m(l) + k))k + a(m(l) + k)kM
From the way we have chosen N we conclude that: kˆ y(t(m(l) + k))k ≤ K (1 + kˆ x(t(m(l) + k)k) ≤ K (1 + Kω ) and that ˆ m(l)+k+1 k = kζˆm(l)+k+1 − ζˆm(l)+k k ≤ Mω . kM 13
Thus we have that, − kˆ x(Tl+1 )k ≤ kˆ x(t(m(l) + k))k + a(m(l) + k) (K(1 + Kω ) + Mω ) . − Finally we have that kˆ x(Tl+1 )k
0) then such an algorithm can be characterized by the following stochastic recursive inclusion: xn+1 = xn + a(n) (yn + Mn+1 ) , 14
(10)
where yn ∈ h(xn ) + B ǫ (0) is an estimate of h(xn ) and B ǫ (0) is the closed ball of radius ǫ around the origin. We define a new set-valued map called the approximate drift by H(x) := h(x) + B ǫ (0) for each x ∈ Rd . In the following discussion we assume that ǫ ≥ 0. When ǫ = 0, the approximate drift algorithm described by (10) is really the SRE given by (9). In this section we answer the following question: Suppose the recursion given by (9) is stable and convergent, what can we say about the approximate drift version described by (10)? We shall show that if the recursion (9) satisfies the assumptions of the Borkar-Meyn Theorem [8], then the corresponding approximate drift version given by ( 10) satisfies (A1)−(A5). We then invoke Theorem 2 to conclude that the iterates converge to a closed, connected, internally chain transitive and invariant set associated with x(t) ˙ ∈ h(x(t)) + B ǫ (0)(= H(x(t))). Before we proceed we recall the assumptions and the statement of the BorkarMeyn Theorem [8]. (BM1) (i) The function h : Rd → Rd is Lipschitz continuous, with Lipschitz constant L. There exists a function h∞ : Rd → Rd such that lim h(cx) = c c→∞
h∞ (x), for each x ∈ Rd . (ii) hc → h∞ uniformly on compacts, as c → ∞. (iii) The o.d.e. x(t) ˙ = h∞ (x(t)) has the origin as the unique globally asymptotically stable equilibrium.
P a(n) = ∞ and (BM2) {a(n)}n≥0 is a scalar sequence such that: a(n) ≥ 0, n≥0 P a(n)2 < ∞. Without loss of generality, we assume that sup a(n) ≤ 1. n
n≥0
(BM3) {Mn }n≥1 is a martingale difference sequence with respect to the filtration Fn := σ (x0 , M1 , . . . , Mn ), n ≥ 0. Thus, E [Mn+1 |Fn ] = 0 a.s., ∀ n≥ 0. {Mn } is also square integrable with E[kMn+1 k2 |Fn ] ≤ L 1 + kxn k2 , for some constant L > 0. Without loss of generality, assume that the same constant, L, works for both (BM 1)(i) and (BM 3). Theorem 3 (Borkar-Meyn Theorem). Suppose (BM1)-(BM3) hold. Then supkxn k < n
∞ almost surely. Further, the sequence {xn } converges almost surely to a (possibly sample path dependent) compact connected internally chain transitive invariant set of x(t) ˙ = h(x(t)). We assume that the recursion given by (9) satisfies assumptions (BM 1) − (BM 3). As explained earlier we have to prove that the approximate drift version given by (10) satisfies (A1) − (A5). We begin by proving that H(x) = h(x) + B ǫ (0) is a Marchaud map. Since B ǫ (0) is convex and compact for each x ∈ Rd , it follows that H(x) is convex and compact. Fix x ∈ Rd and y ∈ H(x), then kyk ≤ kh(x)k + ǫ and kyk ≤ kh(0)k + Lkx − 0k + ǫ since h is Lipschitz continuous with Lipschitz constant L. If we set K := (kh(0)k + ǫ) ∨ L, then we get kyk ≤ K (1 + kxk). This shows that H is point-wise bounded. We now show that H is upper-semicontinuous. Assume lim xn = x, lim yn = y and n→∞
n→∞
yn ∈ h(xn ) for each n ≥ 1. Since yn ∈ H(xn ), we conclude that yn = h(xn ) + zn 15
for some zn ∈ B ǫ (0), n ≥ 1. Since {yn }n≥1 and {h(xn )}n≥1 are convergent sequences, it follows that {zn }n≥1 is also convergent. Let z := lim zn . Since n→∞
B ǫ (0) is compact we have that z ∈ B ǫ (0). Taking limits on both sides of yn = h(xn ) + zn , we get y = h(x) + z. Thus y ∈ H(x).
It follows directly from (BM 2) and (BM 3) that assumptions (A2) and (A3) are satisfied by the approximate drift algorithm (10). Before showing that (10) satisfies (A4), we construct the following family of set-valued maps: Hc (x) := { h(cx) + yc | y ∈ B ǫ (0)}. i.e., Hc (x) = c d hc (x) + B ǫ/c (0) for each x ∈ R . We claim that the lower limit, H∞ (x) := Liminfc→∞Hc (x) = {h∞ (x)}. Since lim d(hc (x), H∞ (x)) ≤ lim kh∞ (x) − c→∞
c→∞
hc (x)k = 0 it follows that h∞ (x) ∈ H∞ (x). Now we show that h∞ (x) is the only element of H∞ (x). Let y ∈ H∞ (x), then ∃ zn ∈ Hn (x), n ≥ 1, such that lim zn = y. We have the following inequality: n→∞
ky − h∞ (x)k ≤ ky − zn k + kzn − hn (x)k + khn (x) − h∞ (x)k. Letting n → ∞ in the above inequality, we get ky − h∞ (x)k = 0, in other words H∞ (x) = {h∞ (x)}. The differential inclusion x(t) ˙ ∈ H∞ (x(t)) is really the o.d.e. x(t) ˙ = h∞ (x(t)). Since the origin is assumed to be the unique globally asymptotically stable equilibrium point of x(t) ˙ = h∞ (x(t)), it follows that the origin is also Lyapunov stable. This can be used to show that the origin is an attracting set and that B 1 (0) is its fundamental neighborhood. For a proof of this fact the reader is referred to Lemma 1 from Chapter 3 of Borkar [7]. Finally we show that (A5) is satisfied. Let cn ↑ ∞, lim xn = x, lim yn = n→∞
n→∞
y, and yn ∈ Hcn (xn ) for each n ≥ 1. We need to show that y ∈ H∞ (x) (y = h∞ (x)). Consider the following set of inequalities: ky − h∞ (x)k ≤ ky − yn k + kyn − hcn (xn )k + khcn (xn ) − h∞ (x)k,
ǫ + khcn (x) − h∞ (x)k + khcn (xn ) − hcn (x)k. cn The second inequality follows from the first since khcn (xn )−h∞ (x)k ≤ khcn (xn )− hcn (x)k + khcn (x) − h∞ (x)k and kyn − h∞ (x)k ≤ cǫn . Letting n → ∞ in the second inequality we get y = h∞ (x). Thus (A5) is satisfied. ky − h∞ (x)k ≤ ky − yn k +
Since assumptions (A1) − (A5) are satisfied the following is a direct corollary to Theorem 2. Corollary 1. If a SRE, given by (9), satisfies (BM 1) − (BM 3) then the corresponding approximate drift algorithm, given by (10), is stable almost surely. In addition, it converges to a closed, connected, invariant and internally chain transitive set of x(t) ˙ ∈ H(x(t)), where H(x) = h(x) + B ǫ (0). Since the result holds true even when ǫ = 0, it follows that the Borkar-Meyn Theorem emerges as a special case of Theorem 2. 16
4.2
Relaxation of assumption (BM1) of the Borkar-Meyn Theorem
The assumptions involved in the Borkar-Meyn Theorem [8] are listed in Section 4.1. One of the assumptions, (BM 1)(ii), requires that hc → h∞ uniformly on compacts. In this section we discuss how to dispense with assumption (BM 1)(ii). As discussed in Section 4.1 the Borkar-Meyn Theorem emerges as a special case of Theorem 2. This happens when ǫ = 0 and H(x) = {h(x)} for all x ∈ Rd . Hence the SRI given by (10) is same as the SRE given by (9). In the foregoing discussion H and h are as defined in Section 4.1. In the previous section we have shown that if (9) satisfies assumptions (BM 1) − (BM 3) then it also satisfies assumptions (A1)−(A5). In proving this we did not use (BM 1)(ii) i.e., hc → h∞ uniformly on compacts. Now, we discuss in brief how we work around using (BM 1)(ii) in proving the Borkar-Meyn Theorem. The notations used in this paragraph are consistent with those found in Chapter 3 of Borkar [7]. We list a few below. 1. φn (· , x) denotes the solution to x(t) ˙ ∈ hr(n) (x(t)) with initial value x. 2. φ∞ (· , x) denotes the solution to x(t) ˙ ∈ h∞ (x(t)) with initial value x. 3. xn (t), t ∈ [0, T ] denotes the solution to x˙ n (t) = hr(n) (ˆ x(Tn + t)) with initial value xn (0) = xˆ(Tn ). Then xn (t) = φn (t, xˆ(Tn )), t ∈ [0, T ]. For more details the reader is referred to Borkar and Meyn [8] or Chapter 3 of Borkar [7]. In proving the Borkar-Meyn Theorem as outlined in [8] (BM 1)(ii) is used to show that for large values of r(n), φn (t, xˆ(Tn )) is ‘close’ to φ∞ (t, xˆ(Tn )), t ∈ [0, T ]. Here we deviate from [8] in the definition of xn (t), t ∈ [0, T ]. In this paper, xn (· ) denotes the solution up to time T to x˙ n (t) = yˆ(Tn + t) = hr(n) (ˆ x([Tn + t])) with xn (0) = xˆ(Tn ), where [· ] is defined in Lemma 5. In other words, we have the following: xn (t) = xˆ(Tn ) +
k−1 X Z t(m(n)+l+1) l=0
t(m(n)+l)
yˆ(z) dz +
Z
t
yˆ(z) dz.
t(m(n)+k)
For t ∈ [tn , tn+1 ), yˆ(t) is a constant and equals yˆ(tn ). We get the following: n
x (t) = x ˆ(Tn ) +
k−1 X
a(m(n) + l)hr(n) (ˆ x([t(m(n) + l)])) +
l=0
(t − t(m(n) + k)) hr(n) (ˆ x([t(m(n) + k)])) .
If the Borkar-Meyn Theorem is proven along of lines of Section 3.2 i.e., Lemmas 1 - 5 and Theorem 1, then we essentially show the following: If r(n) ↑ ∞ then the T -length trajectories given by {xn (· )}n≥0 have φ∞ (x, t), t ∈ [0, T ], as the limit point in C([0, T ], Rd), where x ∈ B 1 (0). This is proven in Lemmas 4 and 5, the proofs of which do not require (BM 1)(ii).
17
5
Another Stability Theorem for Stochastic Recursive Inclusions
In (A4) we assumed that Liminfc→∞hc (x) is nonempty for all x ∈ Rd . In this section we shall develop a stability criterion for the case when we can no longer make such an assumption. In other words, we work with a modified version of assumption (A4).
5.1
A Modification of Assumption (A4)
Recall the following SRI : xn+1 = xn + a(n) [yn + Mn+1 ] , for n ≥ 0.
(11)
Since hc is point-wise bounded for each c ≥ 1, we have sup kyk ≤ K(1+kxk), y∈hc (x)
where x ∈ Rd (see Proposition 1). This implies that {yc }c≥1 , where yc ∈ hc (x), has at least one convergent subsequence. It follows from the definition of upperlimit of a sequence of sets (see Section 2.1) that Limsupc→∞ hc (x) is non-empty for every x ∈ Rd . It is worth noting that Liminfc→∞hc (x) ⊆ Limsupc→∞ hc (x) for every x ∈ Rd . Another important point to consider is that the lower-limits of sequences of sets are harder to compute than their upper-limits, see Aubin [2] for more details. Recall that hc (x) = {y | cy ∈ h(cx)}, where x ∈ Rd and c ≥ 1. Clearly the upper-limit, Limsupc→∞ hc (x) = {y | lim d(y, hc (x)) = 0} is nonempty for c→∞
every x ∈ Rd . For A ⊆ Rd , co(A) denotes the closure of the convex hull of A, i.e., the closure of the smallest convex set containing A. Define h∞ (x) := co ( Limsupc→∞ hc (x)) . Below we state the modification of assumption (A4) that we call (A6).
(A6) The differential inclusion x(t) ˙ ∈ h∞ (x(t)) has the origin as an attracting set and B 1 (0) is a subset of some fundamental neighborhood of the origin. Note that in (A4), h∞ (x) := Liminfc→∞ hc (x) while in (A6), h∞ (x) := co ( Limsupc→∞ hc (x)). In this section we shall work with this new definition of h∞ . Proposition 2. h∞ is a Marchaud map. Proof. From the definition of h∞ it follows that h∞ (x) is convex, compact for all x ∈ Rd and h∞ is point-wise bounded. It is left to prove that h∞ is an upper-semicontinuous map. Let xn → x, yn → y and yn ∈ h∞ (xn ), for all n ≥ 1. We need to show that y ∈ h∞ (x). We present a proof by contradiction. Since h∞ (x) is convex and compact, y ∈ / h∞ (x) implies that there exists a linear functional on Rd , say f , such that sup f (z) ≤ α − ǫ and f (y) ≥ α + ǫ, for some α ∈ R z∈h∞ (x)
18
and ǫ > 0. Since yn → y, there exists N > 0 such that for all n ≥ N , f (yn ) ≥ α + 2ǫ . In other words, h∞ (x) ∩ [f ≥ α + 2ǫ ] 6= φ for all n ≥ N . We use the notation [f ≥ a] to denote the set {x | f (x) ≥ a}. For the sake of convenience let us denote the set Limsupc→∞hc (x) by A(x), where x ∈ Rd . We claim that A(xn ) ∩ [f ≥ α + 2ǫ ] 6= φ for all n ≥ N . We prove this claim later, for now we assume that the claim is true and proceed. Pick zn ∈ A(xn )∩[f ≥ α+ 2ǫ ] for each n ≥ N . It can be shown that {zn }n≥N is norm bounded and hence contains a convergent subsequence, {zn(k) }k≥1 ⊆ {zn }n≥N . Let lim zn(k) = z. k→∞
Since zn(k) ∈ Limsupc→∞(hc (xn(k) )), ∃ cn(k) ∈ N such that kwn(k) − zn(k) k < 1 n(k) , where wn(k) ∈ hcn(k) (xn(k) ). We choose the sequence {cn(k) }k≥1 such that cn(k+1) > cn(k) for each k ≥ 1. We have the following: cn(k) ↑ ∞, xn(k) → x, wn(k) → z and wn(k) ∈ hcn(k) (xn(k) ), for all k ≥ 1. It follows from assumption (A5) that z ∈ h∞ (x). Since zn(k) → z and f (zn(k) ) ≥ α + 2ǫ for each k ≥ 1, we have that f (z) ≥ α + 2ǫ . This contradicts the earlier conclusion that sup f (z) ≤ α − ǫ. z∈h∞ (x)
It remains to prove that A(xn ) ∩ [f ≥ α + 2ǫ ] 6= φ for all n ≥ N . If this were not true, then ∃{m(k)}k≥1 ⊆ {n ≥ N } such that A(xm(k) ) ⊆ [f < α + 2ǫ ] for all k. It follows that h∞ (xm(k) ) = co(A(xm(k) )) ⊆ [f ≤ α + 2ǫ ] for each k ≥ 1. Since yn(k) → y, ∃N1 such that for all n(k) ≥ N1 , f (yn(k) ) ≥ α + 3ǫ 4 . This is a contradiction. We are now ready to state the second stability theorem for an SRI given by (11) under a modified set of assumptions. We retain assumptions (A1)−(A3), replace (A4) by (A6) and finally in (A5) we let h∞ (x) := co ( Limsupc→∞ hc (x)). We state the theorem under these updated set of assumptions. Theorem 4 (Stability Theorem for DI #2). Under assumptions (A1) − (A3), (A5) (with h∞ (x) := co(Limsupc→∞ hc (x))) and (A6), almost surely the sequence {xn }n≥0 generated by the stochastic recursive inclusion, given by (11) is bounded and converges to a closed, connected internally chain transitive invariant set of x(t) ˙ ∈ h(x(t)). Proof. The statements of Lemmas 1−5 hold true even when h∞ := co ( Limsupc→∞ hc (x)) and (A5) is interpreted as explained earlier. The stability of the iterates can be proven in an identical manner to the proof of Theorem 1. Next, we invoke Theorem 3.6 & Lemma 3.8 of Bena¨ım, Hofbauer and Sorin to conclude that the iterates converge to a closed, connected, internally chain transitive and invariant set of x(t) ˙ ∈ h(x(t)) [5].
5.2
The Borkar-Meyn Theorem Revisited
As explained in section 4.2, the assumptions involved in the Borkar-Meyn Theorem can be relaxed to omit (BM 1)(ii). As an immediate application of the second stability theorem (Theorem 4) we address the following question: If lim hc (x) does not exist for all x ∈ Rd , then what are the sufficient conditions c→∞ for the stability and convergence of the algorithm?
19
We take our cue from assumption (A6) in constructing the following replacement for (BM 1) that we call (BM 4). (BM4) (i) The function h : Rd → Rd is Lipschitz continuous, with Lipschitz constant L. Define the set-valued map, h∞ (x) := co (Limsupc→∞{hc (x)}), where x ∈ Rd . Note that Limsupc→∞{hc (x)} = {y | lim khc (x) − yk = 0}. c→∞
(ii) The differential inclusion x(t) ˙ ∈ h∞ (x(t)) has the origin as an attractor set and B 1 (0) is a subset of its fundamental neighborhood. Recall that the function Limsup is defined for a sequence of sets (see Section 2.1) and hc is a single valued map. For any x ∈ Rd and c ≥ 1, {hc (x)} is a set of cardinality one, hence Limsupc→∞{hc (x)} is well defined. Let us recall the following SRE: xn+1 = xn + a(n) [h(xn ) + Mn+1 ] , f or n ≥ 0.
(12)
Observe that Limsupc→∞ {hc (x)} = lim hc (x) when lim hc (x) exists for each c→∞
c→∞
x ∈ Rd , where Limsup is the upper-limit of a sequence of sets (see section 2.1). It can be shown that if a recursion given by (12) satisfies assumptions (BM 1)(i) and (BM 1)(iii) then it also satisfies (BM 4). Assumption (BM 4) unifies the two possible cases: when the limit of hc , as c → ∞, exists and when it does not.
We claim that a recursion given by (12), satisfying assumptions (BM 2), (BM 3) and (BM 4) will also satisfy (A1) − (A3), (A6) and (A5) (see section 5.1). From Theorem 4 it follows that the iterates are stable and converge to a closed, connected, internally chain transitive and invariant set of x(t) ˙ = h(x(t)). The following generalization of the Borkar-Meyn Theorem is a direct consequence of Theorem 4. Corollary 2 (Generalized Borkar-Meyn Theorem). Under assumptions (BM 2), (BM 3) and (BM 4), almost surely the sequence {xn }n≥0 generated by the stochastic recursive equation (12), is bounded and converges to a closed, connected, internally chain transitive and invariant set of x(t) ˙ = h(x(t)). Proof. Assumptions (A1) − (A3) and (A6) follow directly from (BM 2), (BM 3) and (BM 4). We show that (A5) is also satisfied. Let cn ↑ ∞, xn → x, yn → y and yn ∈ hcn (xn ) (here yn = hcn (xn )), ∀ n ≥ 1. It can be shown that khcn (xn ) − hcn (x)k ≤ Lkxn − xk. Hence we get that hcn (x) → y. In other words, lim khc (x) − yk = 0. Hence we have y ∈ h∞ (x). The claim now follows c→∞
from Theorem 4.
In Section 4.1 we showed that the approximate drift version of an algorithm is stable and convergent if the original algorithm satisfies (BM 1)(i), (iii), (BM 2) and (BM 3) (assumptions of the Borkar-Meyn Theorem). Similarly, we claim if a SRE given by (12) satisfies (BM 2), (BM 3) and (BM 4), then the corresponding approximate drift version satisfies (A1) − (A3), (A6) and (A5). The arguments involved are similar to those discussed in Section 4.1. We omit the proof to avoid repetition. It follows from Theorem 4 that the iterates are stable and 20
converge to a closed, connected, internally chain transitive and invariant set of x(t) ˙ ∈ h(x(t)) + B ǫ (0).
5.3
Relaxation of Assumptions (A4) and (A6)
In assumptions (A4) and (A6), the infinity system (x(t) ˙ ∈ h∞ (x(t))) is assumed to have the origin as an attracting set. Here, we discuss how we can relax this n+1 )k 1 assumption. In the proof of Theorem 1, we used kx(T kx(Tn )k < 2 to assert that the trajectory, x(· ), falls exponentially to within the unit ball. This helped us in proving the stability of the iterates. The same conclusion can be drawn n+1 )k even if we are able to prove that kx(T kx(Tn )k < δ, for some 0 < δ < 1. Once stability is established, ‘convergence’ of the algorithm follows, under appropriate assumptions, from Theorems 2 and 4. In this section we show that Theorems 2, 4 and Corollary 2 are valid even when the attracting set is ‘sufficiently’ close to the origin. We modify assumptions (A4), (A6) and (BM 4) as follows: In the aforementioned assumptions, we now assume that the associated differential inclusion, x(t) ˙ ∈ h∞ (x(t)), has an attracting set, A, such that sup kxk < 1. Further, we x∈A
assume that B 1 (0) is a subset of some fundamental neighborhood of A. In the remainder of this section we discuss in brief why the theorems hitherto stated still hold with this modification. Below, we state the modified version of assumption (A4) below. We call it (M A4). Assumptions (A6) and (BM 4) can be similarly modified to get (M A6) and (M BM 4), respectively. (MA4) h∞ (x) is nonempty ∀x ∈ Rd . The differential inclusion x(t) ˙ ∈ h∞ (x(t)) has an attracting set, A, such that sup kxk < 1. Further, it is assumed x∈A
that B 1 (0) is a subset of some fundamental neighborhood of A. In what follows it should be noted that the definition of h∞ changes with the assumptions being made. For example, when (M A4) is used we let h∞ (x) := Liminfc→∞hc (x); when (M A6) is used we let h∞ (x) := Limsupc→∞ hc (x) and when (BM 4) is used we let h∞ (x) := Limsupc→∞ {hc (x)}. Proposition 3. 1. Under assumptions (A1) − (A3), (M A4) and (A5), the statement of Theorem 2 is true. 2. Under assumptions (A1) − (A3), (M A6) and (A5), the statement of Theorem 4 is true. 3. Under assumptions (BM 2), (BM 3) and (M BM 4), the statement of Corollary 2 is true. Proof. We merely highlight the differences as the proofs essentially remain the same. Define δ1 := sup kxk and choose δ2 , δ3 and δ4 such that 0 ≤ δ1 < x∈A
δ2 < δ3 < δ4 < 1. Recall that T (δ) is the time beyond which any solution 21
to the differential inclusion given by x(t) ˙ ∈ h∞ (x(t)), with starting point in the fundamental neighborhood of A, remains within N δ (A). Note that h∞ is appropriately interpreted based on the theorem being proven. We define T := T (δ2 − δ1 ). The statements of Lemma 1 − Lemma 5 are true with the aforementioned definition of T . In the proof of Theorem 1, we choose N such that the following hold: ∀ m(l) ≥ N , kˆ x([Tl + T ])k < δ3 ; ∀ n ≥ N , a(n) < δ4 −δ3 ˆ ˆ [K(1+Kω )+Mω ] ; ∀ n > m ≥ N , kζn − ζm k < Mω and ∀ m(l) ≥ N , r(l) > 1. Note that Kω and Mω are as explained in Lemma 3. Using similar arguments − as before, we can conclude that kˆ x(Tl+1 )k < δ4 , where l is such that m(l) ≥ N . Finally we can conclude that, kx(Tl+1 )k kx(Tl )k
=
− kˆ x(Tl+1 )k kˆ x(Tl )k
< δ4 .
Since δ4 < 1, we can deduce that the trajectory falls exponentially till it hits the unit ball around the origin. The rest of the proof follows in a similar manner as before.
6
Conclusions
This paper presents an extension to the theorem of Borkar and Meyn that includes the case where the mean field is a set-valued map. Two different sets of assumptions are discussed which guarantee the ‘stability and convergence’ of a stochastic recursive inclusion. As an immediate application of Theorem 2, a solution to the ‘approximate drift problem’ is discussed. Further, the assumptions of the Borkar-Meyn Theorem are relaxed as a consequence of Theorem 2. As a corollary to Theorem 4, a generalization of the Borkar-Meyn Theorem is presented which includes the case when lim hc (x) does not exist for all x ∈ Rd . c→∞
An important future direction would be to extend these results to the case when the set-valued drift is governed by a Markov process in addition to the iterate sequence. For the case of stochastic approximations, such a situation has been considered in [ [7], Chapter 6], where the Markov ‘noise’ is tackled using the ‘natural timescale averaging’ properties of stochastic approximation.
References [1] J. Aubin and A. Cellina. Differential Inclusions: Set-Valued Maps and Viability Theory. Springer, 1984. [2] J. Aubin and H. Frankowska. Set-Valued Analysis. Birkh¨auser, 1990. [3] M. Bena¨ım. Dynamics of stochastic approximation algorithms. Seminaire de probabilites XXXIII, pages 1–68, 1999. [4] M. Bena¨ım. A dynamical system approach to stochastic approximations. SIAM J. Control Optim., 34(2):437–472, 1996. [5] M. Bena¨ım, J. Hofbauer, and S. Sorin. Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization, pages 328–348, 2005. 22
[6] S. Bhatnagar. The Borkar-Meyn theorem for asynchronous stochastic approximations. Systems & Control Letters, 60(7):472–478, 2011. [7] V. S. Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, 2008. [8] V. S. Borkar and S.P. Meyn. The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim, 38:447–469, 1999. [9] H. Kushner and G.G. Yin. Stochastic Approximation and Recursive Algorithms and Applications. Springer, 2003. [10] L. Ljung. Analysis of recursive stochastic algorithms. Automatic Control, IEEE Transactions on, 22(4):551–575, 1977.
23
