Chapter 7 Iterative Techniques in Matrix Algebra
Updated December 5, 2003
Chapter 7 Iterative Techniques in Matrix Algebra In the previous chapter, we were introduced to techniques solve the matrix equation A x = b that involved algorithms that produced the solution (or an approximation to the solution) in a finite number of steps. These were called direct methods. In this chapter we study techniques that produce a sequence of approximations 8xn < which hopefully converge to the solution. The analogous situation existed when we discussed methods to a solve nonlinear equation. A trivial example is the solution of a x2 + b x + c = 0. A direct method is the quadratic formula; an iterative method is Newton's Method.
7.1 Norms of Vectors and Matrices Suppose x is a vector in !n . We are all familiar with the Euclidean length of the vector: »» x »»E = "######## x2i . It may ⁄ni=1####### surprise you, but there are other ways to characterize its "length." Definition: Suppose x œ !n . A vector norm for !n is a function, »» ÿ »» : !n ö @0, ¶L such that (i) »» x »» ¥ 0 for all x œ !n
(ii) »» x »» = 0 if and only if x = 0
(iiii) »» a x »» ¥ » a » »» x »» for all a œ !
(iv) »» x + y »» § »» x »» + »» y »» for all x, y œ !n
n Example: »» x »»E = »» x »»2 = "######## x2i is a vector norm. ⁄i=1#######
Example: »» x »»¶ = max 8xi : i = 1, 2, …, n< is a vector norm. Example: »» x »»1 = ⁄ni=1 » xi » is a vector norm.
Chapter7.nb
2
Example: »» x »» p = H⁄ni=1 » xi » p L
1ê p
is a vector norm for any positive integer p.
1 y jij z jj 0 zzz jj zz Example: Let x = jj z . Compute »» x »»E , »» x »»¶ , and »» x »»1 . jj -2 zzz zz jj k 5 { »» x »»E =
"################################ ############# è!!!!!! 12 + 02 + H-2L2 + 52 = 30
»» x »»¶ = max 8 » 1 », » 0 », » -2 », » 5 »< = 5 »» x »»1 = » 1 » + » 0 » + » -2 » + » 5 » = 8
We can prove other properties of the vector norm. Theorem: Let »» ÿ »» be any vector norm. Then » »» x »» - »» y »» » § »» x - y »». We now define the notation of convergence with respect to a vector norm. Definition: Let 8xn < be a sequence of vectors. We say that the sequence of vectors converges to a vector x with respect to the vector norm »» ÿ »» if and only if given e > 0 , there exists and N œ "+ such that if n ¥ N , then »» x - xn »» < e.
ij 1 ê n2 yz ij 0 yz jj zz j z Example: The sequence of vectors 8xn < = 9jjjj 1 + 1 ê n zzzz= converges to the vector x = jjjj 1 zzzz in the norm »» ÿ »»2 . jj 2 n2 -1 zz k1{ ÅÅÅÅÅÅÅÅ { k ÅÅÅÅÅÅÅÅ n 2 +1
In place of the definition for convergence, there is an easier way of establishing convergence. x ij x1 yz jij n1 zyz j z j z j z Theorem: Let xn = jj ª zz . Then 8xn < converges to x = jjjj ª zzzz in the »» ÿ »»¶ vector norm if and only if limnض xn j = x j , j z j z k xk { k xk { j = 1, 2, …, k . That is, each component in the sequence of vectors converge to the corresponding component of x .
Question: If a sequence of vectors converges in any vector norm, then does it converge in any other vector norm?
Chapter7.nb
3
Answer: The answer is yes. The reason is the following theorem. Theorem: Suppose 8xn < is a sequence of vectors that converges to a vector x in a vector norm »» ÿ »» . Then the sequence converges to x in any vector norm.
Remark: This observation can be quite useful, because it is often easier to show convergence in the »» ÿ »»¶ norm, but not the »» ÿ »»2 which is often the choice of norms to work with.
There is some interesting relationships between different vector norms. ####### Theorem: If »» x »»1 = ⁄ki=1 xi , »» x »»2 = "######## ⁄ki=1 x2i , and »» x »»• = max 8 » xi » : i = 1, 2, …, k 0D := t@kD = ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ; Dot@v@kD, A.v@kDD Dot@r@kD, r@kDD s@k_ ê; k > 0D := s@kD = ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ; Dot@r@k - 1D, r@k - 1DD v@k_ ê; k > 1D := v@kD = r@k - 1D + s@k - 1D * v@k - 1D; x@k_ ê; k > 0D := x@kD = x@k - 1D + t@kD * v@kD; Do@ If@Sqrt@Dot@r@iD, r@iDDD > e, sol = x@iD, 8sol = x@iD, savei = i, Break@D
Chapter 7 Iterative Techniques in Matrix Algebra In the previous chapter, we were introduced to techniques solve the matrix equation A x = b that involved algorithms that produced the solution (or an approximation to the solution) in a finite number of steps. These were called direct methods. In this chapter we study techniques that produce a sequence of approximations 8xn < which hopefully converge to the solution. The analogous situation existed when we discussed methods to a solve nonlinear equation. A trivial example is the solution of a x2 + b x + c = 0. A direct method is the quadratic formula; an iterative method is Newton's Method.
7.1 Norms of Vectors and Matrices Suppose x is a vector in !n . We are all familiar with the Euclidean length of the vector: »» x »»E = "######## x2i . It may ⁄ni=1####### surprise you, but there are other ways to characterize its "length." Definition: Suppose x œ !n . A vector norm for !n is a function, »» ÿ »» : !n ö @0, ¶L such that (i) »» x »» ¥ 0 for all x œ !n
(ii) »» x »» = 0 if and only if x = 0
(iiii) »» a x »» ¥ » a » »» x »» for all a œ !
(iv) »» x + y »» § »» x »» + »» y »» for all x, y œ !n
n Example: »» x »»E = »» x »»2 = "######## x2i is a vector norm. ⁄i=1#######
Example: »» x »»¶ = max 8xi : i = 1, 2, …, n< is a vector norm. Example: »» x »»1 = ⁄ni=1 » xi » is a vector norm.
Chapter7.nb
2
Example: »» x »» p = H⁄ni=1 » xi » p L
1ê p
is a vector norm for any positive integer p.
1 y jij z jj 0 zzz jj zz Example: Let x = jj z . Compute »» x »»E , »» x »»¶ , and »» x »»1 . jj -2 zzz zz jj k 5 { »» x »»E =
"################################ ############# è!!!!!! 12 + 02 + H-2L2 + 52 = 30
»» x »»¶ = max 8 » 1 », » 0 », » -2 », » 5 »< = 5 »» x »»1 = » 1 » + » 0 » + » -2 » + » 5 » = 8
We can prove other properties of the vector norm. Theorem: Let »» ÿ »» be any vector norm. Then » »» x »» - »» y »» » § »» x - y »». We now define the notation of convergence with respect to a vector norm. Definition: Let 8xn < be a sequence of vectors. We say that the sequence of vectors converges to a vector x with respect to the vector norm »» ÿ »» if and only if given e > 0 , there exists and N œ "+ such that if n ¥ N , then »» x - xn »» < e.
ij 1 ê n2 yz ij 0 yz jj zz j z Example: The sequence of vectors 8xn < = 9jjjj 1 + 1 ê n zzzz= converges to the vector x = jjjj 1 zzzz in the norm »» ÿ »»2 . jj 2 n2 -1 zz k1{ ÅÅÅÅÅÅÅÅ { k ÅÅÅÅÅÅÅÅ n 2 +1
In place of the definition for convergence, there is an easier way of establishing convergence. x ij x1 yz jij n1 zyz j z j z j z Theorem: Let xn = jj ª zz . Then 8xn < converges to x = jjjj ª zzzz in the »» ÿ »»¶ vector norm if and only if limnض xn j = x j , j z j z k xk { k xk { j = 1, 2, …, k . That is, each component in the sequence of vectors converge to the corresponding component of x .
Question: If a sequence of vectors converges in any vector norm, then does it converge in any other vector norm?
Chapter7.nb
3
Answer: The answer is yes. The reason is the following theorem. Theorem: Suppose 8xn < is a sequence of vectors that converges to a vector x in a vector norm »» ÿ »» . Then the sequence converges to x in any vector norm.
Remark: This observation can be quite useful, because it is often easier to show convergence in the »» ÿ »»¶ norm, but not the »» ÿ »»2 which is often the choice of norms to work with.
There is some interesting relationships between different vector norms. ####### Theorem: If »» x »»1 = ⁄ki=1 xi , »» x »»2 = "######## ⁄ki=1 x2i , and »» x »»• = max 8 » xi » : i = 1, 2, …, k 0D := t@kD = ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ; Dot@v@kD, A.v@kDD Dot@r@kD, r@kDD s@k_ ê; k > 0D := s@kD = ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ; Dot@r@k - 1D, r@k - 1DD v@k_ ê; k > 1D := v@kD = r@k - 1D + s@k - 1D * v@k - 1D; x@k_ ê; k > 0D := x@kD = x@k - 1D + t@kD * v@kD; Do@ If@Sqrt@Dot@r@iD, r@iDDD > e, sol = x@iD, 8sol = x@iD, savei = i, Break@D
