Error bounds for linear matrix inequalities
Error bounds for linear matrix inequalities Jos F. Sturm
Communications Research Laboratory, McMaster University, Hamilton, Canada. Supported by Netherlands Organization for Scienti c Research (NWO). E-mail: [email protected]
May 11, 1998 Abstract
For iterative sequences that converge to the solution set of a linear matrix inequality, we show that the distance of the iterates to the solution set is at most O(2?d ). The nonnegative integer d is the so{called degree of singularity of the linear matrix inequality, and denotes the amount of constraint violation in the iterate. For infeasible linear matrix inequalities, we show that the minimal norm of {approximate primal solutions is at least 1=O(1=(2d ?1) ), and the minimal norm of {approximate Farkas{ type dual solutions is at most O(1=2d ?1 ). As an application of these error bounds, we show that for any bounded sequence of {approximate solutions to a semi-de nite programming problem, the distance to the 2?k optimal solution set is at most O( ), where k is the degree of singularity of the optimal solution set. Keywords: semi-de nite programming, error bounds, linear matrix inequality, regularized duality. AMS subject classi cation: 90C31, 65G99, 15A42.
1 Introduction Linear matrix inequalities play an important role in system and control theory, see the book by Boyd et al. [3]. Recently, considerable progress has been made in optimization over linear matrix inequalities, i.e. semi-de nite programming, see [1, 6, 8, 9, 16, 19, 18, 23, 25] and the references cited therein. We study the linear matrix inequality (LMI)
X 2B+A X 0;
(1)
where X 0 means positive semi-de niteness, B is a given (real) symmetric matrix and A is a linear subspace of symmetric matrices. 1
The LMI (1) is in conic form, see e.g. [17, 23]. Since we leave complete freedom as to the formulation of A, it is in general not dicult to t a given LMI into conic form. Consider for instance a linear matrix inequality
F0 +
m X j =1
yj Fj 0;
where F0 ; F1 ; : : : ; Fm are given symmetric matrices. This is a conic form LMI (1) with B = F0 and A is the span of fF1 ; F2 ; : : : ; Fm g. Recently developed interior point codes for semi-de nite programming make it possible to solve LMIs numerically. Such algorithms generate sequences of increasingly good approximate solutions, provided that the LMI is solvable. For a discussion of interior point methods for semi-de nite programming, see e.g. [8, 23]. A typical way to measure the quality of an approximate solution, is by evaluating its constraint violation. For instance, if we denote the smallest eigenvalue of an approximate solution X~ by min (X~ ), then we may say that X~ violates the constraint `X 0' by an amount of [?min(X~ )]+ , where the operator []+ yields the positive part. In fact, [?min(X~ )]+ is the distance, measured in the matrix 2{norm, of the approximate solution X~ to the cone of positive semi-de nite matrices. The matrix 2{norm is a convenient measure for the amount by which the positive semi-de niteness constraint is violated, but other matrix norms can in principle be used as well. Similarly, we say that X~ violates the constraint `X 2 B + A' by an amount ~ B + A), where dist(; ) denotes the distance function (for a given of dist(X; norm). The total amount of constraint violation in X~ , i.e. ~ B + A) + [?min (X~ )]+ ; dist(X; (2) is called the backward error of X~ with respect to the LMI (1). The backward error indicates how much we should perturb the data of the problem, such that X~ is an exact solution to the perturbed problem. However, the backward error does not (immediately) tell us the distance from X~ to the solution set of the original LMI; this distance is called the forward error of X~ . Without knowing any exact solution, there is no straightforward way to estimate the forward error. For linear inequality and equation systems however, the forward error and backward error are of the same order of magnitude, see Ho man [7]. The equivalence between forward and backward errors holds also true for systems that are described by convex quadratic inequalities, if a Slater condition holds, see Luo and Luo [12]. In these cases, we have a relation of the form forwarderror = O(backwarderror); which is called a Lipschitzian error bound. For systems of convex quadratic inequalities without Slater's condition, an error bound of the form forwarderror = O((backwarderror)1=2 ) d
2
(3)
was obtained by Wang and Pang [26]. They also showed that d n + 1, where n is the dimension of the problem. Error bounds for systems with a nonconvex quadratic inequality are given in Luo and Sturm [14], and references cited therein. An error bound of the form (3) is called a Holderian error bound. A Holderian error bound has been demonstrated for analytic inequality and equation systems, if the size of the approximate solutions is bounded by a xed constant, see Luo and Pang [13]. However, there are no known positive lower bounds on the exponent , except in the linear and quadratic cases that are mentioned above, or when a Slater condition holds [4], For a comprehensive survey of error bounds, we refer to Pang [20]. Some issues on error bounds for LMIs and semi-de nite programming were recently addressed by Deng and Hu [4], Goldfarb and Scheinberg [5], Luo, Sturm and Zhang [16] and Sturm and Zhang [24]. Deng and Hu [4] derived upper bounds on the Lipschitz constant (or condition number) for LMIs, if Slater's condition holds. Luo Sturm and Zhang [16] and Sturm and Zhang [24] prove some Lipschitzian type error bounds for central solutions for semi-de nite programs under strict complementarity. Goldfarb and Scheinberg [5] prove Lipschitz continuity of the optimal value function for semi-de nite programs. In this paper, we show for LMIs in n n matrices, that (3) holds for a certain d 2 f0; 1; 2; : : :; n ? 1g, the so{called degree of singularity, provided that the size of the approximate solutions is bounded. We interpret the degree of singularity in the context of Ramana{type regularized duality. It is basically the number of elementary regularizations that are needed to obtain a fully regularized dual. Under Slater's constraint quali cation, the irregularity level d is zero. (Notice that this is also true for convex quadratic systems, see Wang and Pang [26].) The degree of singularity of the optimal solution set of a semi-de nite programming problem is at most one, if strict complementarity holds. The concept of singularity degrees thus embeds the Slater and strict complementarity conditions in a hierarchy of singularity for LMIs. This paper is organized as follows. In Section 2, we introduce the concept of regularized backward errors, which is closely related to the concept of minimal cones [2]. In this section, we also show that there is a close connection between the regularized backward error and the forward error. We will then estimate in Section 3 how the regularized backward error depends on the usual backward error. In Section 4, we apply the error bound for LMIs to semi-de nite programming problems. The paper is concluded in Section 5. Notation. Let S nn denote the space of n n real symmetric matrices. The cone of all positive semi-de nite matrices in S nn is denoted by S+nn , and we write X 0 if and only if X 2 S+nn . The interior of S+nn is the set of nn , and we write X 0 if and only if X 2 S nn . positive de nite matrices S++ ++ We let N := n(n + 1)=2 denote the dimension of the real linear space S nn . The standard inner product for two symmetric matrices X and Y is X Y = tr XY . The matrix norm kX k2 is the operator 2-norm that is associated with 3
the Euclidean norm for vectors, namely kX k2 = maxfkXyk2 j kyk2 = 1g: For symmetric matrices, kX k2 is the eigenvalue of X that has the largest absolute value.
2 The regularized backward error
Let A denote the smallest linear subspace containing B + A, i.e. A = fX 2 S nn j X + tB 2 A for some t 2 0g, such that
X () + Y () 2 fX 2 B + A j XU = 0; XN = 0g; kY ()k = O(): Using also the fact that XB is positive de nite, it follows that X () + Y () + ()X 0 8 > 0; with
() := [?min (X ())](X+ +) kY ()k2 : min
B
(13)
Notice that () = O(). Since X 2 A, there must exist t 2 < such that X ? tB 2 A. Let > 0 be such that t () > ?1 for all 2 (0; ]. Then 1 nn 1 + t () (X () + Y () + ()X ) 2 (B + A) \ S+ for 0 < ; and hence dist 1 + 1t () X (); (B + A) \ S+nn = O(); 7
for 0 < . Q.E.D. n n Under Slater's condition, i.e. if (B + A) \ S++ 6= ;, Lemma 3 generalizes Ho man's error bound [7] for systems of linear inequalities and equations to LMIs. Notice in particular that no boundedness assumptions are made, i.e. the error bound holds globally over S nn . However, the lemma requires a scaling factor 1 + (), which is not needed in case of linear inequalities and equations. The following example shows that this scaling factor is essential in the case of LMIs.
Example 1 Consider the LMI in S 22 with
11 x12 B = 00 01 ; A = X = xx12 0
x11 ; x12 2 < ;
i.e. we want to nd nd x11 and x12 such that x11 jx12 j2 . This LMI obviously has positive de nite solutions (the identity matrix for instance). Therefore, the regularized backward error is identical to the usual backward error. The approximate solution
2 3 X () := 1=(1=+ ) 11+=
has backward error > 0. However, X () + Y () 2 (B + A) \ S+22 if and only if y22 () = ?; y11 () (1 1+ ) + 2 y12 () + jy12 ()j2 ; which shows that the distance of X () to (B + A) \ S+22 is bounded from below by a positive constant as # 0. However, we have X ()=(1+ ) 2 (B + A) \S+22 , which agrees with the statement of Lemma 3.
Below are more remarks on the regularized error bound of Lemma 3.
Remark 1 Lemma 3 states that the mere existence of fX () j > 0g satisfying (12) for all > 0 implies that (B + A) \ S+nn = 6 ;, even though X () is not necessarily bounded for # 0. In the case of weak infeasibility, i.e. if dist(B + A; S+nn ) = 0; (B + A) \ S+nn = ;; we can therefore conclude that if X () satis es (7) then
lim#inf kXN ()k + kXU ()k > 0: 0
Remark 2 If X (1); X (2); : : : is a bounded sequence with dist(X (k) ; B + A) ! 0 and [?min(X (k) )]+ ! 0 for k ! 1; 8
then also kXU(k) k + kXN(k)k ! 0, as follows from Lemma 1. Letting
k := dist(X (k) ; B + A) + [?min(X (k) )]+ + kXU(k)k + kXN(k) k; it follows from Lemma 3 and the boundedness of the sequence fX (k) j k = 1; 2; : : :g that dist(X (k) ; (B + A) \ S+nn ) = O(k ):
3 Regularization steps In order to bound the regularized backward error (11) in terms of the original backward error (2), we use a sequence of regularization steps. In the preceding, we have partitioned n n matrices according to the structure of X , given by (9). In this section, we will also partition n n matrices into blocks, but with respect to a possibly di erent eigenvector basis; the sizes of the blocks can be di erent as well. We will denote the blocks by the subscripts 11 , 12 and 22 , i.e. 11 X12 : X= X XT X 12
22
We will also encounter the dual cone of a face of S+nn , viz.
face S+nn ;
0 0 0 I
= fZ j Z11 X11 0 for all X11 0g =
Obviously, we have
Z11 Z12 Z12T Z22
Z11 0 :
Z11 Z12 Z 0 : = Z12T Z22 11 In the following, we will allow the possibility that X = X11 , i.e. X12 are X22 are non-existent. For this case, we use the convention that kX12k = kX22k = 0. Lemma 4 Let A be a linear subspace of S nn , and suppose that fX () j 0 < 1g is such that dist(X (); A) ; kX12()k + kX22()k ; min(X ()) ?; for all 0 < 1. Let relint face S+nn ;
0 0 0 I
Z 2 relint A? \ face S+nn ; 00 I0 It holds that
9
:
Z11 0 if and only if
A \ face S+nn ; 00 I0
Z11 = 0 if and only if
A \ relint face S+nn ; 00 I0
= f0g:
6= ;:
For the remaining case that 0 6= Z11 6 0, let Q1 ; Q2 be an orthogonal matrix such that Z11 Q1 = 0, QT2 Z11 Q2 0. Then p kQT2 X ()Q2k = O(); kX ()Q2k = O( kX ()k):
Proof. The rst two cases, i.e. Z11 = 0 or Z11 0, are immediate applications
of (6). It remains to consider the case that Z11 is a nonzero but singular, positive semi-de nite matrix. Since dist(X (); A) , there must exist Y (), such that X () + Y () 2 A; kY ()k ; (14) for all > 0. This implies that Z ?(X ()+ Y ()) because Z 2 A? , and therefore X ( ) 0 11 Z11 X11 () = Z 0 0 ? X () ? Y ()
Y ( ) X ( ) + Y ( ) 11 12 12
kZ kF (X12 () + Y12 ())T X22 () + Y22 ()
; F where we used the Cauchy-Schwartz inequality. Recall now that kX12()k = O(); kX22 ()k = O(); kY ()k = O(); so that we further obtain Z11 X11 () = O(): (15) Since Z11 is positive semi-de nite and min(X ()) ?, we have 0 tr (QT2 Z11 Q2 )(QT2 X11 ()Q2 + I ) = tr Z11 (X11 () + I ) = O() where we used Z11 Q1 = 0 in the rst identity, and (15) in the last identity. Recalling that QT2 Z22 Q2 0, it easily follows from the above relation that kQT2X11 ()Q2 k = O(): (16) Finally, since min(X ()) ?, we know that X11 () + I is positive semide nite, and hence kQT1 X11 ()Q2 k2 = O((kQT1 X11 ()Q1 k + )(kQT2 X11 ()Q2 k + )) = O((kX11 ()k + )); (17) 10
where we used (16). This completes the proof. Q.E.D. n n For a given linear subspace A S , we de ne the level of singularity d(A) by recursively applying the construction of Lemma 4. This procedure is outlined below: Procedure 1 De nition of the level of singularity of a linear subspace A S nn . Step 1 Let Z (0) 2 relint (A? \ S+nn ). If Z (0) = 0 or Z (0) 0 then d(A) = 0. Otherwise, proceed with Step 2. Step 2 Let Q1 ; Q2 be such that Z (0) Q1 = 0 and QT2 Z (0) Q2 0. Set d = 1 and
11 X12 A1 = X = X X12T X22
Step 3 Let
Q1 ; Q2
T 1 X Q QT2 2 A :
: relint A?d \ face S+nn ; 00 I0 (d) = 0 then set d(A) = d. Otherwise, proceed with Step 4. If Z11
Z (d) 2
(d) Q = 0 and QT Z (d) Q 0, and de ne Step 4 Let Q1 ; Q2 be such that Z11 1 2 11 2
Q 1 = Q01 ; Q 2 = Q02 I0 : Let
11 X12 Ad+1 = X = X X T X22 12
T1 Q 1; Q 2 X Q Q T2 2 Ad :
Set d = d + 1 and return to Step 3.
In the above procedure, we start with the full dimensional cone S+nn , and in the rst iteration we determine a face of this cone. Next, we arrive at a face of this face, and so on. We claim that this procedure nally arrives at the minimal cone. To see this, notice that at any given step d = 0; 1; : : :; d(A) above, we perform a regularization step as described in Lemma 4. Recall from (5) that d(A) = 0 and Z (0) 0 if and only if A \ S+nn = f0g, and this case has already (d(A)) = 0. It is easily been treated in Section 2. In any other case, we have Z11 n n seen from Lemma 4 that if X 2 A \ S+ , then X11 Q2 = 0. This means that all nonzeros of X must be contained in the ( nal) 11 block for Ad(A) . On the (d(A)) = 0 in the above procedure, it follows from (6) that other hand, since Z11 there exists X~ 2 A \ S+nn such that X~11 0 and X~ 12 = 0, X~ 22 = 0. Since we just showed that X12 = 0 and X22 = 0 for all X 2 A \ S+nn , we must have X~ 2 relint (A \ S+nn ). Hence, the face in the nal iteration is the minimal 11
cone. For A = A + Img b, we may therefore take X = X~ and XB = X~11 , see (9). By applying a basis transformation if necessary, we may assume without loss of generality that there is a (d(A) + 1) (d(A) + 1) block partition, such that for k = 1; 2; : : : ; d(A), 8 > > > > < > > > > :
Z :=2Z (d(A)?k) Z=4
0
0 Z (k + 1; k + 1)
Z (k + 1; k + 1) 0:
Z (1 : k; k + 2 : d(A) + 1) Z (k + 1; k + 2 : d(A) + 1) Z (k + 2 : d(A) + 1; k + 2 : d(A) + 1)
3 5
(18)
Above, we used a Matlab{type1 notation, thus 1 : k means 1; 2; : : : ; k, and Z (i; j ) denotes the block on the ith row and j th column in the (d(A) + 1) (d(A) + 1) block partition. Since Z is symmetric, we only speci ed the upper block triangular part of Z . The relation between the (d(A) + 1) (d(A) + 1) partition in (18) and the 2 2 partition in iteration d = d(A) ? k of Procedure 1 is that Z11 = Z (1 : k + 1; 1 : k + 1): The minimal cone is the set of matrices X for which
X (1; 1) 0; X (i; j ) = 0 for all (i; j ) 6= (1; 1): In iteration d = d(A) ? k of Procedure 1, we arrive at the face where X (1 : k + 1; 1 + k + 1) 0; X (i; j ) = 0 if min(i; j ) > k + 1; which indeed includes the minimal cone. Remark that the 3rd row and column in the 3 3 block form of (18) are non-existent for k = d(A), i.e. for d = d(A) = 0. Based on Lemma 4, we can now estimate the regularized backward error. Lemma 5 Let A = A + Img b, and X 2 relint (A \ S+nn). Suppose without loss of generality that X is of the form (9). If d(A) > 1 and fX () j 0 < 1g is such that for all 0 < 1, dist(X (); B + A) ; min(X ()) ?; then
kXU ()k = O( kX ()k1? ); kXN ()k = O(2 kX ()k1?2 ); with = 2?d(A) , where d(A) is the degree of singularity of A.
Proof. Applying Lemma 4 in iteration d = 0 of Procedure 1, we have that 1
MATLAB is a registered trademark of The MathWorks, Inc.
12
p
kX(1 : d(A); d(A) + 1)k = O( kXk); kX(d(A) + 1; d(A) + 1)k = O(); where we used X as a synonym for X (). Suppose now that in iteration d 2 f0; : : :; d(A) ? 2g, we have 8 < kX(1 : k; k + 1 : d(A) + 1)k = O( kXk1? ) (19) : kX (k + 1 : d(A) + 1; k + 1; d(A ) + 1)k = O(2 kXk1?2 ); where
k = d(A) ? d; = 2?(d+1):
It then follows from Lemma 4 that (19) also holds for k0 = k ? 1 and 0 = =2. By induction. we obtain that (19) holds for d = d(A) ? 1, k = 1 and = 2?(d+1) = . Since X(1; 1) = XB (), the lemma follows. Q.E.D. We arrive now at the main result of this paper, namely an error bound for LMIs. Theorem 1 Let A = A + Img b. If fX () j 0 < 1g is such that kX ()k is bounded and dist(X (); B + A) and min (X ()) ? for all > 0; then
dist(X (); (B + A) \ S+nn ) = O(2
?d(A)
):
Proof. For the case that d(A) > 0, the theorem follows by combining Lemma 3 with Lemma 5. If d(A) = 0, there are two cases, either A \ S+nn = f0g or nn = A \ S++ 6 ;. In the former case, we have kX ()k = O(), and hence the error bound holds, see Section 2. In the latter case, we have that X = XB 0, and the error bound follows from Lemma 3. Q.E.D. An LMI is said to be weakly infeasible if 1. there is no solution to the LMI, i.e. (B + A) \ S+nn = ;, but 2. dist(B + A; S+nn ) = 0. For weakly infeasible LMIs, there exist approximate solutions with arbitrarily small constraint violations. However, the following theorem provides a lower bound on the size of such approximate solutions to weakly infeasible LMIs. Theorem 2 Let A = A + Img b and suppose that If fX () j > 0g is such that
(B + A) \ S+nn = ;:
dist(X (); B + A) and min (X ()) ? for all > 0; 13
then, for small enough, we have X () 6= 0 and 1 = O(1=(2d(A) ?1) ): kX ()k
Proof. Suppose to the contrary that there exists a sequence 1; 2; : : : with d(A) k ! 0 and kX (k )k = o(?k 1=(2 ?1) ). Applying Lemma 5, it follows that kXU (k )k + kXN (k )k = O(k2?d(A) kX (k )k1?2?d(A) ) = o(1): Together with Lemma 3, we obtain that (B + A) \ S+nn = 6 ;, a contradiction. Q.E.D.
There is an extension of Farkas' lemma from linear inequalities to convex cones, which states that dist(B + A; K) > 0 () 9Z 2 A? \ K : B Z < 0: (20) where K S nn is a convex cone, and K is the associated dual cone. See e.g. Lemma 2.5 in [23]. If dist(B + A; S+nn ) > 0, then we say that the LMI is strongly infeasible. Relation (20) states that strong infeasibility can be demonstrated by a matrix Z 2 A? \ S+nn with B Z < 0, and such Z is called a dual improving direction. For weakly infeasible LMIs, infeasibility cannot be demonstrated by a dual improving direction. However, an LMI is infeasible if and only if there exist approximate dual improving directions with arbitrarily small constraint violations. See e.g. Lemma 2.6 in [23]. The next theorem gives an upper bound for the minimal norm of such approximate dual improving directions in the case of infeasibility. Theorem 3 Let A = A + Img b. If (B + A) \ S+nn = ; then there exist fY () j > 0g such that for all 0 < 1, it holds that dist(Y (); A? ) = O(); B Y () < ?1 + O(); min (Y ()) ?; and kY ()k = O(1?2d(A) ):
Proof. Let X 2 relint (A \ S+nn ), and suppose without loss of generality that X is of the form (9). Using the same 2 2 partition as in (9), it follows from Lemma 3 that
dist B + A;
face S+nn ;
0 0 0 I
> 0:
Applying (20), it thus follows that there exists a matrix Y (0) such that B Y (0) < ?1; Y (0) 2 A? \ face S+nn ; 00 I0 : 14
(21)
Partitioning Y (0) , we have
Y (0) =
"
#
YB(0) YU(0) ; Y (0) 0: B (YU(0) )T YN(0)
(22)
We shall now construct fY (k) j k = 0; 1: : : : ; d(A)g such that 8 > > < > > :
kY (k) k = O(1?2k )
Y (k) (1 : k + 1; 1 : k + 1) 0 B Y (k) < ?1 + O() dist(Y (k) ; A? ) = O();
(23)
for 0 < 1. Remark from (21){(22) that (23) holds for k = 0. We will construct Y (k) for k 2 f1; 2; : : :; d(A) ? 1g in such a way that it satis es (23), provided that Y (k?1) satis es (23). We can then use induction. Let Yt := Y (k?1) + I + tZ (d(A)?k) : Since Z (d(A)?k) 2 A? = A? \ Ker bT , we immediately obtain from (23) that dist(Yt ; A? ) = O();
B Yt < ?1 + O();
(24)
irrespective of t. Furthermore, since Y (k?1) (1 : k; 1 : k) 0, it follows that Yt (1 : k +1; 1 : k +1) is positive semi-de nite if and only if the Schur{complement
Yt (k + 1; k + 1) ? Yt (1 : k; k + 1)T Yt (1 : k; 1 : k)?1 Yt (1 : k; k + 1) is positive semi-de nite. From (18) and the de nition of Yt , we have Yt (1 : k; k + 1) = Y (k?1) (1 : k; k + 1); and hence
Yt (k + 1; k + 1) ? Yt (1 : k; k + 1)T Yt (1 : k; 1 : k)?1 Yt (1 : k; k + 1) t Z (d(A)?k) (k + 1; k + 1) + Y (k?1) (k + 1; k + 1) ? 1 kY (k?1) k22 I:
Thus, Yt (1 : k + 1; 1 : k + 1) is positive semi-de nite if we choose t as (k?1) (k?1) k2 =) k 2 t = kY (d(kA2)?+k()kY = O(1?2 ); min (Z (k + 1; k + 1))
where we used that kY (k?1) k = O(1?2k?1 ). Setting Y (k) = Yt , we obtain (23). The theorem follows by letting
Y () = Y (d(A)) :
Q.E.D. 15
We remark from the proof of Theorem 3 that the matrices Y (0) and Z (k) , k = 0; 1; : : : ; d(A) ? 1, provide a nite certi cate of the infeasibility of the LMI. Together, these matrices form essentially a solution to the regularized Farkas{ type dual of Ramana [22], see also [10, 15]. Thus, the degree of singularity is the minimal number of layers that are needed in the perfect dual of Ramana. As discussed in the introduction, it is easy to calculate the backward error of an approximate solution. However, the error bound for the forward error of an LMI, as given in Theorem 1, does not only involve the backward error, but also the degree of singularity. We will now provide some easily computable upper bounds on the degree of singularity. Lemma 6 For the degree of singularity d(A) of a linear subspace A S nn , it holds that d(A) minfn ? 1; dim A; dim A? g:
Proof. If d(A) > 0 then A \ S+nn 6= f0g, by de nition of d(A). For this case, we have de ned the (d(A) + 1) (d(A) + 1) block partition (18), where each of the d(A) + 1 diagonal blocks is at least of size 1 1. Thus, d(A) n ? 1: Furthermore, Lemma 4 de nes a matrix Z (k) 2 A? , for each regularization step k 2 f0; 1; 2; : : :; d(A) ? 1g, and it is easily veri ed that these matrices are mutually independent. Therefore,
d(A) dim A? : Finally, using the (d(A) + 1) (d(A) + 1) block partition (18), we claim that there exists X (k) 2 A with 2 3 8 X (k) (1 : k; 1 : k) X (k) (1 : k; k + 1) 0 > > < X (k) = 4 X (k) (1 : k; k + 1)T 0 05 > > :
0
X (k) (1 : k; 1 : k) 0:
0
0
Namely, if such X (k) does not exist, then by (6), there must exist Z 2 A? such that 8 Z (1 : k; 1 : k) 0 > < Z (1 : k + 1; 1 : k + 1) = 0 Z (k; k) > :
0 6= Z (1 : k; 1 : k) 0;
and this contradicts the fact that Z (d(A)?k) (1 : k + 1; 1 : k + 1) is of maximal rank, see its de nition in Lemma 4. Again, it is easy to see that the matrices X (k) 2 A, k = 1; 2; : : : ; d(A), are mutually independent, and hence d(A) dim A. Q.E.D. 16
The bounds of Theorem 1 and Theorem 2 quickly become inattractive as the singularity degree increases. However, the next two examples show that these bounds can be tight. This means that problems with a large degree of singularity can be very hard to solve numerically.
Example 2 Consider the LMI 8
:99999, whereas X^1;25 = 0 for any solution X^ of the LMI.
Example 3 Extending Example 2 with the restriction `X1;n = 1', we obtain a (weakly) infeasible LMI: 8
0g with constraint violation and kX ()k = O(1=1=(2n?1?1) ). Namely, we let 8 X11 () = n= with := 2n?1 ? 1 > > < X22 () = X > 1;n () = 1? > ? : Xk+1;k+1 () = X1;k () = with = 2n?1?k ? 1 = 2n?1 ? 1 ; where k 2 f2; 3; : : : ; n ? 1g.
This example shows that (in)feasibility can be hard to detect. Namely, for
n = 10 and a backward error = 10?8 we have kX (10?8)k2 < 11, which is not unusually large; yet, the problem is infeasible.
17
4 Application to semi-de nite programming Error bounds for LMIs can be applied to semi-de nite optimization models as well. A standard form semi-de nite program is (P) minfC X j X 2 (B + A) \ S+nn g; where B and C are given symmetric matrices. Associated with this optimization problem is a dual problem, viz. (D) minfB Z j Z 2 (C + A? ) \ S+nn g: An obvious property of the primal{dual pair (P) and (D) is the weak duality relation. Namely, if X 2 (B + A) \ S+nn and Z 2 (C + A? ) \ S+nn , then 0 X Z = C X + B Z ? B C:
(25)
Clearly, if X Z = 0, then X and Z must be optimal solutions to (P) and (D) respectively; we say then that (X; Z ) is a pair of complementary solutions. In general, such a pair may not exist, even if both (P) and (D) are feasible. (We say that (P) is feasible if (B + A) \S+nn 6= ; and (D) is feasible if (C + A? ) \S+nn 6= ;.) A sucient condition for the existence of a complementary solution pair is that (P) and (D) are feasible and satisfy the primal{dual Slater condition, in which case d(A + Img b) = d(A? + Img c) = 0. Based on (25), we can formulate the set of complementary solutions as the LMI 8 C X +BZ =BC < X 2 B + A; Z 2 C + A? : X 0; Z 0: In principle, we can apply our error bound results for LMIs directly to the above system. But, tighter bounds can be obtained by exploring its special structure. Consider a bounded trajectory of approximate primal and dual solutions f(X (); Z ()) j > 0g, satisfying 8 < :
dist(X (); B + A) ; dist(Z (); C + A? ) min (X ()) ?; min (Z ()) ? X () Z () :
(26)
C ) be a complementary solution pair, i.e. Let (B; B C = 0; B 2 (B + A) \ S+nn ; C 2 (C + A? ) \ S+nn : Such a pair must exist, since in particular any cluster point of f(X (); Z () j > 0g for # 0 is a complementary solution pair. Notice that B + A = B + A and similarly C + A? = C + A? , from which we easily derive that X Z = C X + B Z; 18
for feasible solutions X and Z , and [C X ()]+ = O(); [B Z ()]+ = O(); for (X (); Z ()) satisfying (26). This means that X () has an O() constraint violation with respect to the LMI 8 +A < X2B C X 0 (27) : X 0: Notice that (27) describes the set of optimal solutions to (P). Letting A := Img b + (A \ Ker cT ); (28) the Theorems 1 and 2 are applicable to the LMI (27) and hence to the semide nite program (P). Speci cally, given a bounded trajectory fX (); Z () j > 0g satisfying (26), we know that the distance from X () to the set of optimal solutions to (P) is O(2?d(A) ), where d(A) is the degree of singularity of the linear subspace de ned in (28). Since B C = 0, we can move the parentheses in de nition (28) to get A = ( Img b + A) \ Ker cT ; from which we get A? = Img c + (A? \ Ker bT ): Noticing the primal{dual symmetry, we conclude that the distance from Z () ? to the set of optimal solutions to (D) is O(2?d(A ) ), where d(A? ) is the degree of singularity of A? .
5 Concluding remarks Theorem 1 provides a Holderian error bound for LMIs. For weakly infeasible LMIs, we have derived relations between backward errors and the size of approximate solutions, see Theorems 2 and 3. In Section 4, we applied the error bound of Theorem 1 to semi-de nite programming problems (SDPs). If the SDP has a strictly complementary solution, then its degree of singularity can be at most 1, and the bound becomes p forward error = O( backward error): For this case, Luo, Sturm and Zhang [16] obtained a Lipschitzian error bound if the approximate solutions (X (); Z ()) are restricted to the central path. The sensitivity of central solutions with respect to perturbations in the right{hand side was studied by Sturm and Zhang [24]. Acknowledgment. Tom Luo's comments on an earlier version of this paper have resulted in substantial improvements in the presentation. 19
References [1] F. Alizadeh, J.A. Haeberly, and M. Overton. Primal{dual interior{point methods for semide nite programming: convergence rates, stability and numerical results. Technical Report 721, Computer Science Department, New York University, New York, 1996. [2] J.M. Borwein and H. Wolkowicz. Regularizing the abstract convex program. Journal of Mathematical Analysis and Applications, 83:495{530, 1981. [3] S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan. Linear matrix inequalities in system and control theory, volume 15 of Studies in Applied Mathematics. SIAM, Philadelphia, PA, 1994. [4] S. Deng and H. Hu. Computable error bounds for semide nite programming. Technical report, Department of Mathematical Sciences, Northern Illinois University, DeKalb, Illinois, USA, 1996. [5] D. Goldfarb and K. Scheinberg. On parametric semide nite programming. Technical report, Columbia University, Department of IEOR, New York, USA, 1997. [6] C. Helmberg, F. Rendl, R.J. Vanderbei, and H. Wolkowicz. An interior{ point method for semide nite programming. SIAM Journal on Optimization, 6:342{361, 1996. [7] A.J. Ho man. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards, 49:263{265, 1952. [8] E. de Klerk. Interior Point Methods for Semide nite Programming. PhD thesis, Delft University of Technology, Faculty of Technical Mathematics and Informatics, Delft, The Netherlands, 1997. [9] M. Kojima, S. Shindoh, and S. Hara. Interior{point methods for the monotone semide nite linear complementarity problem in symmetric matrices. SIAM Journal on Optimization, 7(1):86{125, 1997. [10] K.O. Kortanek and Q. Zhang. Perfect duality in semi{in nite and semide nite programming. Technical report, Department of Management Sciences, University of Iowa, Iowa City, Iowa, USA, 1997. [11] A.S. Lewis. Eigenvalue{constrained faces. Linear Algebra and its Applications, 269:159{181, 1998. [12] X.D. Luo and Z.-Q. Luo. Extensions of Ho man's error bound to polynomial systems. SIAM Journal on Optimization, 4:383{392, 1994. [13] Z.-Q. Luo and J.-S. Pang. Error bounds for analytic systems and their applications. Mathematical Programming, 67:1{28, 1994. 20
[14] Z.-Q. Luo and J.F. Sturm. Error bounds for quadratic systems. Technical report, Communications Research Laboratory, McMaster University, Hamilton, Ontario, Canada, 1998. [15] Z.-Q. Luo, J.F. Sturm, and S. Zhang. Duality results for conic convex programming. Technical Report 9719/A, Econometric Institute, Erasmus University Rotterdam, Rotterdam, The Netherlands, 1997. [16] Z.-Q. Luo, J.F. Sturm, and S. Zhang. Superlinear convergence of a symmetric primal{dual path following algorithm for semide nite programming. SIAM Journal on Optimization, 8(1):59{81, 1998. [17] Y. Nesterov and A. Nemirovsky. Interior point polynomial methods in convex programming, volume 13 of Studies in Applied Mathematics. SIAM, Philadelphia, 1994. [18] Y. Nesterov and M.J. Todd. Self{scaled barriers and interior{point methods for convex programming. Mathematics of Operations Research, 22(1):1{42, 1997. [19] Y. Nesterov and M.J. Todd. Primal{dual interior{point methods for self{ scaled cones. SIAM Journal on Optimization, 8:324{364, 1998. [20] J.S. Pang. Error bounds in mathematical programming. Mathematical Programming, 97:299{332, 1998. [21] G. Pataki. On the rank of extreme matrices in semide nite programs and the multiplicity of optimal eigenvalues. Technical report, Department of IE/OR, Columbia University, New York, NY, USA, 1997. To appear in Mathematics of Operations Research. [22] M.V. Ramana. An exact duality theory for semide nite programming and its complexity implications. Mathematical Programming, 77(2):129{162, 1997. [23] J.F. Sturm. Primal{Dual Interior Point Approach to Semide nite Programming, volume 156 of Tinbergen Institute Research Series. Thesis Publishers, Amsterdam, The Netherlands, 1997. [24] J.F. Sturm and S. Zhang. On sensitivity of central solutions in semide nite programming. Technical Report 9813/A, Econometric Institute, Erasmus University Rotterdam, Rotterdam, The Netherlands, 1998. [25] L. Vandenberghe and S. Boyd. Semide nite programming. SIAM Review, 38:49{95, 1996. [26] T. Wang and J.S. Pang. Global error bounds for convex quadratic inequality systems. Optimization, 31:1{12, 1994.
21
Communications Research Laboratory, McMaster University, Hamilton, Canada. Supported by Netherlands Organization for Scienti c Research (NWO). E-mail: [email protected]
May 11, 1998 Abstract
For iterative sequences that converge to the solution set of a linear matrix inequality, we show that the distance of the iterates to the solution set is at most O(2?d ). The nonnegative integer d is the so{called degree of singularity of the linear matrix inequality, and denotes the amount of constraint violation in the iterate. For infeasible linear matrix inequalities, we show that the minimal norm of {approximate primal solutions is at least 1=O(1=(2d ?1) ), and the minimal norm of {approximate Farkas{ type dual solutions is at most O(1=2d ?1 ). As an application of these error bounds, we show that for any bounded sequence of {approximate solutions to a semi-de nite programming problem, the distance to the 2?k optimal solution set is at most O( ), where k is the degree of singularity of the optimal solution set. Keywords: semi-de nite programming, error bounds, linear matrix inequality, regularized duality. AMS subject classi cation: 90C31, 65G99, 15A42.
1 Introduction Linear matrix inequalities play an important role in system and control theory, see the book by Boyd et al. [3]. Recently, considerable progress has been made in optimization over linear matrix inequalities, i.e. semi-de nite programming, see [1, 6, 8, 9, 16, 19, 18, 23, 25] and the references cited therein. We study the linear matrix inequality (LMI)
X 2B+A X 0;
(1)
where X 0 means positive semi-de niteness, B is a given (real) symmetric matrix and A is a linear subspace of symmetric matrices. 1
The LMI (1) is in conic form, see e.g. [17, 23]. Since we leave complete freedom as to the formulation of A, it is in general not dicult to t a given LMI into conic form. Consider for instance a linear matrix inequality
F0 +
m X j =1
yj Fj 0;
where F0 ; F1 ; : : : ; Fm are given symmetric matrices. This is a conic form LMI (1) with B = F0 and A is the span of fF1 ; F2 ; : : : ; Fm g. Recently developed interior point codes for semi-de nite programming make it possible to solve LMIs numerically. Such algorithms generate sequences of increasingly good approximate solutions, provided that the LMI is solvable. For a discussion of interior point methods for semi-de nite programming, see e.g. [8, 23]. A typical way to measure the quality of an approximate solution, is by evaluating its constraint violation. For instance, if we denote the smallest eigenvalue of an approximate solution X~ by min (X~ ), then we may say that X~ violates the constraint `X 0' by an amount of [?min(X~ )]+ , where the operator []+ yields the positive part. In fact, [?min(X~ )]+ is the distance, measured in the matrix 2{norm, of the approximate solution X~ to the cone of positive semi-de nite matrices. The matrix 2{norm is a convenient measure for the amount by which the positive semi-de niteness constraint is violated, but other matrix norms can in principle be used as well. Similarly, we say that X~ violates the constraint `X 2 B + A' by an amount ~ B + A), where dist(; ) denotes the distance function (for a given of dist(X; norm). The total amount of constraint violation in X~ , i.e. ~ B + A) + [?min (X~ )]+ ; dist(X; (2) is called the backward error of X~ with respect to the LMI (1). The backward error indicates how much we should perturb the data of the problem, such that X~ is an exact solution to the perturbed problem. However, the backward error does not (immediately) tell us the distance from X~ to the solution set of the original LMI; this distance is called the forward error of X~ . Without knowing any exact solution, there is no straightforward way to estimate the forward error. For linear inequality and equation systems however, the forward error and backward error are of the same order of magnitude, see Ho man [7]. The equivalence between forward and backward errors holds also true for systems that are described by convex quadratic inequalities, if a Slater condition holds, see Luo and Luo [12]. In these cases, we have a relation of the form forwarderror = O(backwarderror); which is called a Lipschitzian error bound. For systems of convex quadratic inequalities without Slater's condition, an error bound of the form forwarderror = O((backwarderror)1=2 ) d
2
(3)
was obtained by Wang and Pang [26]. They also showed that d n + 1, where n is the dimension of the problem. Error bounds for systems with a nonconvex quadratic inequality are given in Luo and Sturm [14], and references cited therein. An error bound of the form (3) is called a Holderian error bound. A Holderian error bound has been demonstrated for analytic inequality and equation systems, if the size of the approximate solutions is bounded by a xed constant, see Luo and Pang [13]. However, there are no known positive lower bounds on the exponent , except in the linear and quadratic cases that are mentioned above, or when a Slater condition holds [4], For a comprehensive survey of error bounds, we refer to Pang [20]. Some issues on error bounds for LMIs and semi-de nite programming were recently addressed by Deng and Hu [4], Goldfarb and Scheinberg [5], Luo, Sturm and Zhang [16] and Sturm and Zhang [24]. Deng and Hu [4] derived upper bounds on the Lipschitz constant (or condition number) for LMIs, if Slater's condition holds. Luo Sturm and Zhang [16] and Sturm and Zhang [24] prove some Lipschitzian type error bounds for central solutions for semi-de nite programs under strict complementarity. Goldfarb and Scheinberg [5] prove Lipschitz continuity of the optimal value function for semi-de nite programs. In this paper, we show for LMIs in n n matrices, that (3) holds for a certain d 2 f0; 1; 2; : : :; n ? 1g, the so{called degree of singularity, provided that the size of the approximate solutions is bounded. We interpret the degree of singularity in the context of Ramana{type regularized duality. It is basically the number of elementary regularizations that are needed to obtain a fully regularized dual. Under Slater's constraint quali cation, the irregularity level d is zero. (Notice that this is also true for convex quadratic systems, see Wang and Pang [26].) The degree of singularity of the optimal solution set of a semi-de nite programming problem is at most one, if strict complementarity holds. The concept of singularity degrees thus embeds the Slater and strict complementarity conditions in a hierarchy of singularity for LMIs. This paper is organized as follows. In Section 2, we introduce the concept of regularized backward errors, which is closely related to the concept of minimal cones [2]. In this section, we also show that there is a close connection between the regularized backward error and the forward error. We will then estimate in Section 3 how the regularized backward error depends on the usual backward error. In Section 4, we apply the error bound for LMIs to semi-de nite programming problems. The paper is concluded in Section 5. Notation. Let S nn denote the space of n n real symmetric matrices. The cone of all positive semi-de nite matrices in S nn is denoted by S+nn , and we write X 0 if and only if X 2 S+nn . The interior of S+nn is the set of nn , and we write X 0 if and only if X 2 S nn . positive de nite matrices S++ ++ We let N := n(n + 1)=2 denote the dimension of the real linear space S nn . The standard inner product for two symmetric matrices X and Y is X Y = tr XY . The matrix norm kX k2 is the operator 2-norm that is associated with 3
the Euclidean norm for vectors, namely kX k2 = maxfkXyk2 j kyk2 = 1g: For symmetric matrices, kX k2 is the eigenvalue of X that has the largest absolute value.
2 The regularized backward error
Let A denote the smallest linear subspace containing B + A, i.e. A = fX 2 S nn j X + tB 2 A for some t 2 0g, such that
X () + Y () 2 fX 2 B + A j XU = 0; XN = 0g; kY ()k = O(): Using also the fact that XB is positive de nite, it follows that X () + Y () + ()X 0 8 > 0; with
() := [?min (X ())](X+ +) kY ()k2 : min
B
(13)
Notice that () = O(). Since X 2 A, there must exist t 2 < such that X ? tB 2 A. Let > 0 be such that t () > ?1 for all 2 (0; ]. Then 1 nn 1 + t () (X () + Y () + ()X ) 2 (B + A) \ S+ for 0 < ; and hence dist 1 + 1t () X (); (B + A) \ S+nn = O(); 7
for 0 < . Q.E.D. n n Under Slater's condition, i.e. if (B + A) \ S++ 6= ;, Lemma 3 generalizes Ho man's error bound [7] for systems of linear inequalities and equations to LMIs. Notice in particular that no boundedness assumptions are made, i.e. the error bound holds globally over S nn . However, the lemma requires a scaling factor 1 + (), which is not needed in case of linear inequalities and equations. The following example shows that this scaling factor is essential in the case of LMIs.
Example 1 Consider the LMI in S 22 with
11 x12 B = 00 01 ; A = X = xx12 0
x11 ; x12 2 < ;
i.e. we want to nd nd x11 and x12 such that x11 jx12 j2 . This LMI obviously has positive de nite solutions (the identity matrix for instance). Therefore, the regularized backward error is identical to the usual backward error. The approximate solution
2 3 X () := 1=(1=+ ) 11+=
has backward error > 0. However, X () + Y () 2 (B + A) \ S+22 if and only if y22 () = ?; y11 () (1 1+ ) + 2 y12 () + jy12 ()j2 ; which shows that the distance of X () to (B + A) \ S+22 is bounded from below by a positive constant as # 0. However, we have X ()=(1+ ) 2 (B + A) \S+22 , which agrees with the statement of Lemma 3.
Below are more remarks on the regularized error bound of Lemma 3.
Remark 1 Lemma 3 states that the mere existence of fX () j > 0g satisfying (12) for all > 0 implies that (B + A) \ S+nn = 6 ;, even though X () is not necessarily bounded for # 0. In the case of weak infeasibility, i.e. if dist(B + A; S+nn ) = 0; (B + A) \ S+nn = ;; we can therefore conclude that if X () satis es (7) then
lim#inf kXN ()k + kXU ()k > 0: 0
Remark 2 If X (1); X (2); : : : is a bounded sequence with dist(X (k) ; B + A) ! 0 and [?min(X (k) )]+ ! 0 for k ! 1; 8
then also kXU(k) k + kXN(k)k ! 0, as follows from Lemma 1. Letting
k := dist(X (k) ; B + A) + [?min(X (k) )]+ + kXU(k)k + kXN(k) k; it follows from Lemma 3 and the boundedness of the sequence fX (k) j k = 1; 2; : : :g that dist(X (k) ; (B + A) \ S+nn ) = O(k ):
3 Regularization steps In order to bound the regularized backward error (11) in terms of the original backward error (2), we use a sequence of regularization steps. In the preceding, we have partitioned n n matrices according to the structure of X , given by (9). In this section, we will also partition n n matrices into blocks, but with respect to a possibly di erent eigenvector basis; the sizes of the blocks can be di erent as well. We will denote the blocks by the subscripts 11 , 12 and 22 , i.e. 11 X12 : X= X XT X 12
22
We will also encounter the dual cone of a face of S+nn , viz.
face S+nn ;
0 0 0 I
= fZ j Z11 X11 0 for all X11 0g =
Obviously, we have
Z11 Z12 Z12T Z22
Z11 0 :
Z11 Z12 Z 0 : = Z12T Z22 11 In the following, we will allow the possibility that X = X11 , i.e. X12 are X22 are non-existent. For this case, we use the convention that kX12k = kX22k = 0. Lemma 4 Let A be a linear subspace of S nn , and suppose that fX () j 0 < 1g is such that dist(X (); A) ; kX12()k + kX22()k ; min(X ()) ?; for all 0 < 1. Let relint face S+nn ;
0 0 0 I
Z 2 relint A? \ face S+nn ; 00 I0 It holds that
9
:
Z11 0 if and only if
A \ face S+nn ; 00 I0
Z11 = 0 if and only if
A \ relint face S+nn ; 00 I0
= f0g:
6= ;:
For the remaining case that 0 6= Z11 6 0, let Q1 ; Q2 be an orthogonal matrix such that Z11 Q1 = 0, QT2 Z11 Q2 0. Then p kQT2 X ()Q2k = O(); kX ()Q2k = O( kX ()k):
Proof. The rst two cases, i.e. Z11 = 0 or Z11 0, are immediate applications
of (6). It remains to consider the case that Z11 is a nonzero but singular, positive semi-de nite matrix. Since dist(X (); A) , there must exist Y (), such that X () + Y () 2 A; kY ()k ; (14) for all > 0. This implies that Z ?(X ()+ Y ()) because Z 2 A? , and therefore X ( ) 0 11 Z11 X11 () = Z 0 0 ? X () ? Y ()
Y ( ) X ( ) + Y ( ) 11 12 12
kZ kF (X12 () + Y12 ())T X22 () + Y22 ()
; F where we used the Cauchy-Schwartz inequality. Recall now that kX12()k = O(); kX22 ()k = O(); kY ()k = O(); so that we further obtain Z11 X11 () = O(): (15) Since Z11 is positive semi-de nite and min(X ()) ?, we have 0 tr (QT2 Z11 Q2 )(QT2 X11 ()Q2 + I ) = tr Z11 (X11 () + I ) = O() where we used Z11 Q1 = 0 in the rst identity, and (15) in the last identity. Recalling that QT2 Z22 Q2 0, it easily follows from the above relation that kQT2X11 ()Q2 k = O(): (16) Finally, since min(X ()) ?, we know that X11 () + I is positive semide nite, and hence kQT1 X11 ()Q2 k2 = O((kQT1 X11 ()Q1 k + )(kQT2 X11 ()Q2 k + )) = O((kX11 ()k + )); (17) 10
where we used (16). This completes the proof. Q.E.D. n n For a given linear subspace A S , we de ne the level of singularity d(A) by recursively applying the construction of Lemma 4. This procedure is outlined below: Procedure 1 De nition of the level of singularity of a linear subspace A S nn . Step 1 Let Z (0) 2 relint (A? \ S+nn ). If Z (0) = 0 or Z (0) 0 then d(A) = 0. Otherwise, proceed with Step 2. Step 2 Let Q1 ; Q2 be such that Z (0) Q1 = 0 and QT2 Z (0) Q2 0. Set d = 1 and
11 X12 A1 = X = X X12T X22
Step 3 Let
Q1 ; Q2
T 1 X Q QT2 2 A :
: relint A?d \ face S+nn ; 00 I0 (d) = 0 then set d(A) = d. Otherwise, proceed with Step 4. If Z11
Z (d) 2
(d) Q = 0 and QT Z (d) Q 0, and de ne Step 4 Let Q1 ; Q2 be such that Z11 1 2 11 2
Q 1 = Q01 ; Q 2 = Q02 I0 : Let
11 X12 Ad+1 = X = X X T X22 12
T1 Q 1; Q 2 X Q Q T2 2 Ad :
Set d = d + 1 and return to Step 3.
In the above procedure, we start with the full dimensional cone S+nn , and in the rst iteration we determine a face of this cone. Next, we arrive at a face of this face, and so on. We claim that this procedure nally arrives at the minimal cone. To see this, notice that at any given step d = 0; 1; : : :; d(A) above, we perform a regularization step as described in Lemma 4. Recall from (5) that d(A) = 0 and Z (0) 0 if and only if A \ S+nn = f0g, and this case has already (d(A)) = 0. It is easily been treated in Section 2. In any other case, we have Z11 n n seen from Lemma 4 that if X 2 A \ S+ , then X11 Q2 = 0. This means that all nonzeros of X must be contained in the ( nal) 11 block for Ad(A) . On the (d(A)) = 0 in the above procedure, it follows from (6) that other hand, since Z11 there exists X~ 2 A \ S+nn such that X~11 0 and X~ 12 = 0, X~ 22 = 0. Since we just showed that X12 = 0 and X22 = 0 for all X 2 A \ S+nn , we must have X~ 2 relint (A \ S+nn ). Hence, the face in the nal iteration is the minimal 11
cone. For A = A + Img b, we may therefore take X = X~ and XB = X~11 , see (9). By applying a basis transformation if necessary, we may assume without loss of generality that there is a (d(A) + 1) (d(A) + 1) block partition, such that for k = 1; 2; : : : ; d(A), 8 > > > > < > > > > :
Z :=2Z (d(A)?k) Z=4
0
0 Z (k + 1; k + 1)
Z (k + 1; k + 1) 0:
Z (1 : k; k + 2 : d(A) + 1) Z (k + 1; k + 2 : d(A) + 1) Z (k + 2 : d(A) + 1; k + 2 : d(A) + 1)
3 5
(18)
Above, we used a Matlab{type1 notation, thus 1 : k means 1; 2; : : : ; k, and Z (i; j ) denotes the block on the ith row and j th column in the (d(A) + 1) (d(A) + 1) block partition. Since Z is symmetric, we only speci ed the upper block triangular part of Z . The relation between the (d(A) + 1) (d(A) + 1) partition in (18) and the 2 2 partition in iteration d = d(A) ? k of Procedure 1 is that Z11 = Z (1 : k + 1; 1 : k + 1): The minimal cone is the set of matrices X for which
X (1; 1) 0; X (i; j ) = 0 for all (i; j ) 6= (1; 1): In iteration d = d(A) ? k of Procedure 1, we arrive at the face where X (1 : k + 1; 1 + k + 1) 0; X (i; j ) = 0 if min(i; j ) > k + 1; which indeed includes the minimal cone. Remark that the 3rd row and column in the 3 3 block form of (18) are non-existent for k = d(A), i.e. for d = d(A) = 0. Based on Lemma 4, we can now estimate the regularized backward error. Lemma 5 Let A = A + Img b, and X 2 relint (A \ S+nn). Suppose without loss of generality that X is of the form (9). If d(A) > 1 and fX () j 0 < 1g is such that for all 0 < 1, dist(X (); B + A) ; min(X ()) ?; then
kXU ()k = O( kX ()k1? ); kXN ()k = O(2 kX ()k1?2 ); with = 2?d(A) , where d(A) is the degree of singularity of A.
Proof. Applying Lemma 4 in iteration d = 0 of Procedure 1, we have that 1
MATLAB is a registered trademark of The MathWorks, Inc.
12
p
kX(1 : d(A); d(A) + 1)k = O( kXk); kX(d(A) + 1; d(A) + 1)k = O(); where we used X as a synonym for X (). Suppose now that in iteration d 2 f0; : : :; d(A) ? 2g, we have 8 < kX(1 : k; k + 1 : d(A) + 1)k = O( kXk1? ) (19) : kX (k + 1 : d(A) + 1; k + 1; d(A ) + 1)k = O(2 kXk1?2 ); where
k = d(A) ? d; = 2?(d+1):
It then follows from Lemma 4 that (19) also holds for k0 = k ? 1 and 0 = =2. By induction. we obtain that (19) holds for d = d(A) ? 1, k = 1 and = 2?(d+1) = . Since X(1; 1) = XB (), the lemma follows. Q.E.D. We arrive now at the main result of this paper, namely an error bound for LMIs. Theorem 1 Let A = A + Img b. If fX () j 0 < 1g is such that kX ()k is bounded and dist(X (); B + A) and min (X ()) ? for all > 0; then
dist(X (); (B + A) \ S+nn ) = O(2
?d(A)
):
Proof. For the case that d(A) > 0, the theorem follows by combining Lemma 3 with Lemma 5. If d(A) = 0, there are two cases, either A \ S+nn = f0g or nn = A \ S++ 6 ;. In the former case, we have kX ()k = O(), and hence the error bound holds, see Section 2. In the latter case, we have that X = XB 0, and the error bound follows from Lemma 3. Q.E.D. An LMI is said to be weakly infeasible if 1. there is no solution to the LMI, i.e. (B + A) \ S+nn = ;, but 2. dist(B + A; S+nn ) = 0. For weakly infeasible LMIs, there exist approximate solutions with arbitrarily small constraint violations. However, the following theorem provides a lower bound on the size of such approximate solutions to weakly infeasible LMIs. Theorem 2 Let A = A + Img b and suppose that If fX () j > 0g is such that
(B + A) \ S+nn = ;:
dist(X (); B + A) and min (X ()) ? for all > 0; 13
then, for small enough, we have X () 6= 0 and 1 = O(1=(2d(A) ?1) ): kX ()k
Proof. Suppose to the contrary that there exists a sequence 1; 2; : : : with d(A) k ! 0 and kX (k )k = o(?k 1=(2 ?1) ). Applying Lemma 5, it follows that kXU (k )k + kXN (k )k = O(k2?d(A) kX (k )k1?2?d(A) ) = o(1): Together with Lemma 3, we obtain that (B + A) \ S+nn = 6 ;, a contradiction. Q.E.D.
There is an extension of Farkas' lemma from linear inequalities to convex cones, which states that dist(B + A; K) > 0 () 9Z 2 A? \ K : B Z < 0: (20) where K S nn is a convex cone, and K is the associated dual cone. See e.g. Lemma 2.5 in [23]. If dist(B + A; S+nn ) > 0, then we say that the LMI is strongly infeasible. Relation (20) states that strong infeasibility can be demonstrated by a matrix Z 2 A? \ S+nn with B Z < 0, and such Z is called a dual improving direction. For weakly infeasible LMIs, infeasibility cannot be demonstrated by a dual improving direction. However, an LMI is infeasible if and only if there exist approximate dual improving directions with arbitrarily small constraint violations. See e.g. Lemma 2.6 in [23]. The next theorem gives an upper bound for the minimal norm of such approximate dual improving directions in the case of infeasibility. Theorem 3 Let A = A + Img b. If (B + A) \ S+nn = ; then there exist fY () j > 0g such that for all 0 < 1, it holds that dist(Y (); A? ) = O(); B Y () < ?1 + O(); min (Y ()) ?; and kY ()k = O(1?2d(A) ):
Proof. Let X 2 relint (A \ S+nn ), and suppose without loss of generality that X is of the form (9). Using the same 2 2 partition as in (9), it follows from Lemma 3 that
dist B + A;
face S+nn ;
0 0 0 I
> 0:
Applying (20), it thus follows that there exists a matrix Y (0) such that B Y (0) < ?1; Y (0) 2 A? \ face S+nn ; 00 I0 : 14
(21)
Partitioning Y (0) , we have
Y (0) =
"
#
YB(0) YU(0) ; Y (0) 0: B (YU(0) )T YN(0)
(22)
We shall now construct fY (k) j k = 0; 1: : : : ; d(A)g such that 8 > > < > > :
kY (k) k = O(1?2k )
Y (k) (1 : k + 1; 1 : k + 1) 0 B Y (k) < ?1 + O() dist(Y (k) ; A? ) = O();
(23)
for 0 < 1. Remark from (21){(22) that (23) holds for k = 0. We will construct Y (k) for k 2 f1; 2; : : :; d(A) ? 1g in such a way that it satis es (23), provided that Y (k?1) satis es (23). We can then use induction. Let Yt := Y (k?1) + I + tZ (d(A)?k) : Since Z (d(A)?k) 2 A? = A? \ Ker bT , we immediately obtain from (23) that dist(Yt ; A? ) = O();
B Yt < ?1 + O();
(24)
irrespective of t. Furthermore, since Y (k?1) (1 : k; 1 : k) 0, it follows that Yt (1 : k +1; 1 : k +1) is positive semi-de nite if and only if the Schur{complement
Yt (k + 1; k + 1) ? Yt (1 : k; k + 1)T Yt (1 : k; 1 : k)?1 Yt (1 : k; k + 1) is positive semi-de nite. From (18) and the de nition of Yt , we have Yt (1 : k; k + 1) = Y (k?1) (1 : k; k + 1); and hence
Yt (k + 1; k + 1) ? Yt (1 : k; k + 1)T Yt (1 : k; 1 : k)?1 Yt (1 : k; k + 1) t Z (d(A)?k) (k + 1; k + 1) + Y (k?1) (k + 1; k + 1) ? 1 kY (k?1) k22 I:
Thus, Yt (1 : k + 1; 1 : k + 1) is positive semi-de nite if we choose t as (k?1) (k?1) k2 =) k 2 t = kY (d(kA2)?+k()kY = O(1?2 ); min (Z (k + 1; k + 1))
where we used that kY (k?1) k = O(1?2k?1 ). Setting Y (k) = Yt , we obtain (23). The theorem follows by letting
Y () = Y (d(A)) :
Q.E.D. 15
We remark from the proof of Theorem 3 that the matrices Y (0) and Z (k) , k = 0; 1; : : : ; d(A) ? 1, provide a nite certi cate of the infeasibility of the LMI. Together, these matrices form essentially a solution to the regularized Farkas{ type dual of Ramana [22], see also [10, 15]. Thus, the degree of singularity is the minimal number of layers that are needed in the perfect dual of Ramana. As discussed in the introduction, it is easy to calculate the backward error of an approximate solution. However, the error bound for the forward error of an LMI, as given in Theorem 1, does not only involve the backward error, but also the degree of singularity. We will now provide some easily computable upper bounds on the degree of singularity. Lemma 6 For the degree of singularity d(A) of a linear subspace A S nn , it holds that d(A) minfn ? 1; dim A; dim A? g:
Proof. If d(A) > 0 then A \ S+nn 6= f0g, by de nition of d(A). For this case, we have de ned the (d(A) + 1) (d(A) + 1) block partition (18), where each of the d(A) + 1 diagonal blocks is at least of size 1 1. Thus, d(A) n ? 1: Furthermore, Lemma 4 de nes a matrix Z (k) 2 A? , for each regularization step k 2 f0; 1; 2; : : :; d(A) ? 1g, and it is easily veri ed that these matrices are mutually independent. Therefore,
d(A) dim A? : Finally, using the (d(A) + 1) (d(A) + 1) block partition (18), we claim that there exists X (k) 2 A with 2 3 8 X (k) (1 : k; 1 : k) X (k) (1 : k; k + 1) 0 > > < X (k) = 4 X (k) (1 : k; k + 1)T 0 05 > > :
0
X (k) (1 : k; 1 : k) 0:
0
0
Namely, if such X (k) does not exist, then by (6), there must exist Z 2 A? such that 8 Z (1 : k; 1 : k) 0 > < Z (1 : k + 1; 1 : k + 1) = 0 Z (k; k) > :
0 6= Z (1 : k; 1 : k) 0;
and this contradicts the fact that Z (d(A)?k) (1 : k + 1; 1 : k + 1) is of maximal rank, see its de nition in Lemma 4. Again, it is easy to see that the matrices X (k) 2 A, k = 1; 2; : : : ; d(A), are mutually independent, and hence d(A) dim A. Q.E.D. 16
The bounds of Theorem 1 and Theorem 2 quickly become inattractive as the singularity degree increases. However, the next two examples show that these bounds can be tight. This means that problems with a large degree of singularity can be very hard to solve numerically.
Example 2 Consider the LMI 8
:99999, whereas X^1;25 = 0 for any solution X^ of the LMI.
Example 3 Extending Example 2 with the restriction `X1;n = 1', we obtain a (weakly) infeasible LMI: 8
0g with constraint violation and kX ()k = O(1=1=(2n?1?1) ). Namely, we let 8 X11 () = n= with := 2n?1 ? 1 > > < X22 () = X > 1;n () = 1? > ? : Xk+1;k+1 () = X1;k () = with = 2n?1?k ? 1 = 2n?1 ? 1 ; where k 2 f2; 3; : : : ; n ? 1g.
This example shows that (in)feasibility can be hard to detect. Namely, for
n = 10 and a backward error = 10?8 we have kX (10?8)k2 < 11, which is not unusually large; yet, the problem is infeasible.
17
4 Application to semi-de nite programming Error bounds for LMIs can be applied to semi-de nite optimization models as well. A standard form semi-de nite program is (P) minfC X j X 2 (B + A) \ S+nn g; where B and C are given symmetric matrices. Associated with this optimization problem is a dual problem, viz. (D) minfB Z j Z 2 (C + A? ) \ S+nn g: An obvious property of the primal{dual pair (P) and (D) is the weak duality relation. Namely, if X 2 (B + A) \ S+nn and Z 2 (C + A? ) \ S+nn , then 0 X Z = C X + B Z ? B C:
(25)
Clearly, if X Z = 0, then X and Z must be optimal solutions to (P) and (D) respectively; we say then that (X; Z ) is a pair of complementary solutions. In general, such a pair may not exist, even if both (P) and (D) are feasible. (We say that (P) is feasible if (B + A) \S+nn 6= ; and (D) is feasible if (C + A? ) \S+nn 6= ;.) A sucient condition for the existence of a complementary solution pair is that (P) and (D) are feasible and satisfy the primal{dual Slater condition, in which case d(A + Img b) = d(A? + Img c) = 0. Based on (25), we can formulate the set of complementary solutions as the LMI 8 C X +BZ =BC < X 2 B + A; Z 2 C + A? : X 0; Z 0: In principle, we can apply our error bound results for LMIs directly to the above system. But, tighter bounds can be obtained by exploring its special structure. Consider a bounded trajectory of approximate primal and dual solutions f(X (); Z ()) j > 0g, satisfying 8 < :
dist(X (); B + A) ; dist(Z (); C + A? ) min (X ()) ?; min (Z ()) ? X () Z () :
(26)
C ) be a complementary solution pair, i.e. Let (B; B C = 0; B 2 (B + A) \ S+nn ; C 2 (C + A? ) \ S+nn : Such a pair must exist, since in particular any cluster point of f(X (); Z () j > 0g for # 0 is a complementary solution pair. Notice that B + A = B + A and similarly C + A? = C + A? , from which we easily derive that X Z = C X + B Z; 18
for feasible solutions X and Z , and [C X ()]+ = O(); [B Z ()]+ = O(); for (X (); Z ()) satisfying (26). This means that X () has an O() constraint violation with respect to the LMI 8 +A < X2B C X 0 (27) : X 0: Notice that (27) describes the set of optimal solutions to (P). Letting A := Img b + (A \ Ker cT ); (28) the Theorems 1 and 2 are applicable to the LMI (27) and hence to the semide nite program (P). Speci cally, given a bounded trajectory fX (); Z () j > 0g satisfying (26), we know that the distance from X () to the set of optimal solutions to (P) is O(2?d(A) ), where d(A) is the degree of singularity of the linear subspace de ned in (28). Since B C = 0, we can move the parentheses in de nition (28) to get A = ( Img b + A) \ Ker cT ; from which we get A? = Img c + (A? \ Ker bT ): Noticing the primal{dual symmetry, we conclude that the distance from Z () ? to the set of optimal solutions to (D) is O(2?d(A ) ), where d(A? ) is the degree of singularity of A? .
5 Concluding remarks Theorem 1 provides a Holderian error bound for LMIs. For weakly infeasible LMIs, we have derived relations between backward errors and the size of approximate solutions, see Theorems 2 and 3. In Section 4, we applied the error bound of Theorem 1 to semi-de nite programming problems (SDPs). If the SDP has a strictly complementary solution, then its degree of singularity can be at most 1, and the bound becomes p forward error = O( backward error): For this case, Luo, Sturm and Zhang [16] obtained a Lipschitzian error bound if the approximate solutions (X (); Z ()) are restricted to the central path. The sensitivity of central solutions with respect to perturbations in the right{hand side was studied by Sturm and Zhang [24]. Acknowledgment. Tom Luo's comments on an earlier version of this paper have resulted in substantial improvements in the presentation. 19
References [1] F. Alizadeh, J.A. Haeberly, and M. Overton. Primal{dual interior{point methods for semide nite programming: convergence rates, stability and numerical results. Technical Report 721, Computer Science Department, New York University, New York, 1996. [2] J.M. Borwein and H. Wolkowicz. Regularizing the abstract convex program. Journal of Mathematical Analysis and Applications, 83:495{530, 1981. [3] S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan. Linear matrix inequalities in system and control theory, volume 15 of Studies in Applied Mathematics. SIAM, Philadelphia, PA, 1994. [4] S. Deng and H. Hu. Computable error bounds for semide nite programming. Technical report, Department of Mathematical Sciences, Northern Illinois University, DeKalb, Illinois, USA, 1996. [5] D. Goldfarb and K. Scheinberg. On parametric semide nite programming. Technical report, Columbia University, Department of IEOR, New York, USA, 1997. [6] C. Helmberg, F. Rendl, R.J. Vanderbei, and H. Wolkowicz. An interior{ point method for semide nite programming. SIAM Journal on Optimization, 6:342{361, 1996. [7] A.J. Ho man. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards, 49:263{265, 1952. [8] E. de Klerk. Interior Point Methods for Semide nite Programming. PhD thesis, Delft University of Technology, Faculty of Technical Mathematics and Informatics, Delft, The Netherlands, 1997. [9] M. Kojima, S. Shindoh, and S. Hara. Interior{point methods for the monotone semide nite linear complementarity problem in symmetric matrices. SIAM Journal on Optimization, 7(1):86{125, 1997. [10] K.O. Kortanek and Q. Zhang. Perfect duality in semi{in nite and semide nite programming. Technical report, Department of Management Sciences, University of Iowa, Iowa City, Iowa, USA, 1997. [11] A.S. Lewis. Eigenvalue{constrained faces. Linear Algebra and its Applications, 269:159{181, 1998. [12] X.D. Luo and Z.-Q. Luo. Extensions of Ho man's error bound to polynomial systems. SIAM Journal on Optimization, 4:383{392, 1994. [13] Z.-Q. Luo and J.-S. Pang. Error bounds for analytic systems and their applications. Mathematical Programming, 67:1{28, 1994. 20
[14] Z.-Q. Luo and J.F. Sturm. Error bounds for quadratic systems. Technical report, Communications Research Laboratory, McMaster University, Hamilton, Ontario, Canada, 1998. [15] Z.-Q. Luo, J.F. Sturm, and S. Zhang. Duality results for conic convex programming. Technical Report 9719/A, Econometric Institute, Erasmus University Rotterdam, Rotterdam, The Netherlands, 1997. [16] Z.-Q. Luo, J.F. Sturm, and S. Zhang. Superlinear convergence of a symmetric primal{dual path following algorithm for semide nite programming. SIAM Journal on Optimization, 8(1):59{81, 1998. [17] Y. Nesterov and A. Nemirovsky. Interior point polynomial methods in convex programming, volume 13 of Studies in Applied Mathematics. SIAM, Philadelphia, 1994. [18] Y. Nesterov and M.J. Todd. Self{scaled barriers and interior{point methods for convex programming. Mathematics of Operations Research, 22(1):1{42, 1997. [19] Y. Nesterov and M.J. Todd. Primal{dual interior{point methods for self{ scaled cones. SIAM Journal on Optimization, 8:324{364, 1998. [20] J.S. Pang. Error bounds in mathematical programming. Mathematical Programming, 97:299{332, 1998. [21] G. Pataki. On the rank of extreme matrices in semide nite programs and the multiplicity of optimal eigenvalues. Technical report, Department of IE/OR, Columbia University, New York, NY, USA, 1997. To appear in Mathematics of Operations Research. [22] M.V. Ramana. An exact duality theory for semide nite programming and its complexity implications. Mathematical Programming, 77(2):129{162, 1997. [23] J.F. Sturm. Primal{Dual Interior Point Approach to Semide nite Programming, volume 156 of Tinbergen Institute Research Series. Thesis Publishers, Amsterdam, The Netherlands, 1997. [24] J.F. Sturm and S. Zhang. On sensitivity of central solutions in semide nite programming. Technical Report 9813/A, Econometric Institute, Erasmus University Rotterdam, Rotterdam, The Netherlands, 1998. [25] L. Vandenberghe and S. Boyd. Semide nite programming. SIAM Review, 38:49{95, 1996. [26] T. Wang and J.S. Pang. Global error bounds for convex quadratic inequality systems. Optimization, 31:1{12, 1994.
21
