Functions of Matrices - Semantic Scholar
Functions of Matrices: Theory and Computation Nicholas J. Higham March 2008
MIMS EPrint: 2008.39
Manchester Institute for Mathematical Sciences School of Mathematics The University of Manchester
Reports available from: And by contacting:
http://www.manchester.ac.uk/mims/eprints The MIMS Secretary School of Mathematics The University of Manchester Manchester, M13 9PL, UK
ISSN 1749-9097
Contents
List of Figures
xiii
List of Tables
xv
Preface
xvii
1 Theory of Matrix Functions 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Definitions of f (A) . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Jordan Canonical Form . . . . . . . . . . . . . . . . . . . 1.2.2 Polynomial Interpolation . . . . . . . . . . . . . . . . . . 1.2.3 Cauchy Integral Theorem . . . . . . . . . . . . . . . . . . 1.2.4 Equivalence of Definitions . . . . . . . . . . . . . . . . . 1.2.5 Example: Function of Identity Plus Rank-1 Matrix . . . 1.2.6 Example: Function of Discrete Fourier Transform Matrix 1.3 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Nonprimary Matrix Functions . . . . . . . . . . . . . . . . . . . 1.5 Existence of (Real) Matrix Square Roots and Logarithms . . . . 1.6 Classification of Matrix Square Roots and Logarithms . . . . . . 1.7 Principal Square Root and Logarithm . . . . . . . . . . . . . . . 1.8 f (AB) and f (BA) . . . . . . . . . . . . . . . . . . . . . . . . . . 1.9 Miscellany . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10 A Brief History of Matrix Functions . . . . . . . . . . . . . . . . 1.11 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
1 1 2 2 4 7 8 8 10 10 14 16 17 20 21 23 26 27 29
2 Applications 2.1 Differential Equations . . . . . . . . . . . . . . . . . . . . . 2.1.1 Exponential Integrators . . . . . . . . . . . . . . . . 2.2 Nuclear Magnetic Resonance . . . . . . . . . . . . . . . . . 2.3 Markov Models . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Control Theory . . . . . . . . . . . . . . . . . . . . . . . . 2.5 The Nonsymmetric Eigenvalue Problem . . . . . . . . . . . 2.6 Orthogonalization and the Orthogonal Procrustes Problem 2.7 Theoretical Particle Physics . . . . . . . . . . . . . . . . . 2.8 Other Matrix Functions . . . . . . . . . . . . . . . . . . . . 2.9 Nonlinear Matrix Equations . . . . . . . . . . . . . . . . . 2.10 Geometric Mean . . . . . . . . . . . . . . . . . . . . . . . . 2.11 Pseudospectra . . . . . . . . . . . . . . . . . . . . . . . . . 2.12 Algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
35 35 36 37 37 39 41 42 43 44 44 46 47 47
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
vii
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
viii
Contents 2.13 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.14 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.14.1 Boundary Value Problems . . . . . . . . . . . . . . . . . . . 2.14.2 Semidefinite Programming . . . . . . . . . . . . . . . . . . . 2.14.3 Matrix Sector Function . . . . . . . . . . . . . . . . . . . . . 2.14.4 Matrix Disk Function . . . . . . . . . . . . . . . . . . . . . . 2.14.5 The Average Eye in Optics . . . . . . . . . . . . . . . . . . . 2.14.6 Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . 2.14.7 Bregman Divergences . . . . . . . . . . . . . . . . . . . . . . 2.14.8 Structured Matrix Interpolation . . . . . . . . . . . . . . . . 2.14.9 The Lambert W Function and Delay Differential Equations 2.15 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 Conditioning 3.1 Condition Numbers . . . . . . . . . . . . . . . . . 3.2 Properties of the Fr´echet Derivative . . . . . . . . 3.3 Bounding the Condition Number . . . . . . . . . 3.4 Computing or Estimating the Condition Number 3.5 Notes and References . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
4 Techniques for General Functions 4.1 Matrix Powers . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Polynomial Evaluation . . . . . . . . . . . . . . . . . . . 4.3 Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Rational Approximation . . . . . . . . . . . . . . . . . . 4.4.1 Best L∞ Approximation . . . . . . . . . . . . . . 4.4.2 Pad´e Approximation . . . . . . . . . . . . . . . . 4.4.3 Evaluating Rational Functions . . . . . . . . . . . 4.5 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . 4.6 Schur Decomposition and Triangular Matrices . . . . . . 4.7 Block Diagonalization . . . . . . . . . . . . . . . . . . . . 4.8 Interpolating Polynomial and Characteristic Polynomial 4.9 Matrix Iterations . . . . . . . . . . . . . . . . . . . . . . 4.9.1 Order of Convergence . . . . . . . . . . . . . . . . 4.9.2 Termination Criteria . . . . . . . . . . . . . . . . 4.9.3 Convergence . . . . . . . . . . . . . . . . . . . . . 4.9.4 Numerical Stability . . . . . . . . . . . . . . . . . 4.10 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . 4.11 Bounds for kf (A)k . . . . . . . . . . . . . . . . . . . . . 4.12 Notes and References . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
71 . 71 . 72 . 76 . 78 . 79 . 79 . 80 . 81 . 84 . 89 . 89 . 91 . 91 . 92 . 93 . 95 . 99 . 102 . 104 . 105
5 Matrix Sign Function 5.1 Sensitivity and Conditioning . 5.2 Schur Method . . . . . . . . . 5.3 Newton’s Method . . . . . . . 5.4 The Pad´e Family of Iterations 5.5 Scaling the Newton Iteration .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . . .
55 55 57 63 64 69 70
. . . . . .
. . . . .
. . . . . .
48 48 48 48 48 49 50 50 50 50 51 51 52
. . . . . .
. . . . .
. . . . . .
. . . . . . . . . . . . .
. . . . .
. . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
107 109 112 113 115 119
ix
Contents 5.6 5.7 5.8 5.9 5.10
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
121 123 125 128 129 131
6 Matrix Square Root 6.1 Sensitivity and Conditioning . . . . . . . . . 6.2 Schur Method . . . . . . . . . . . . . . . . . 6.3 Newton’s Method and Its Variants . . . . . . 6.4 Stability and Limiting Accuracy . . . . . . . 6.4.1 Newton Iteration . . . . . . . . . . . 6.4.2 DB Iterations . . . . . . . . . . . . . 6.4.3 CR Iteration . . . . . . . . . . . . . . 6.4.4 IN Iteration . . . . . . . . . . . . . . 6.4.5 Summary . . . . . . . . . . . . . . . . 6.5 Scaling the Newton Iteration . . . . . . . . . 6.6 Numerical Experiments . . . . . . . . . . . . 6.7 Iterations via the Matrix Sign Function . . . 6.8 Special Matrices . . . . . . . . . . . . . . . . 6.8.1 Binomial Iteration . . . . . . . . . . . 6.8.2 Modified Newton Iterations . . . . . 6.8.3 M-Matrices and H-Matrices . . . . . 6.8.4 Hermitian Positive Definite Matrices 6.9 Computing Small-Normed Square Roots . . 6.10 Comparison of Methods . . . . . . . . . . . . 6.11 Involutory Matrices . . . . . . . . . . . . . . 6.12 Notes and References . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
133 133 135 139 144 144 145 146 146 147 147 148 152 154 154 157 159 161 162 164 165 166 168
7 Matrix pth Root 7.1 Theory . . . . . . . . . . 7.2 Schur Method . . . . . . 7.3 Newton’s Method . . . . 7.4 Inverse Newton Method . 7.5 Schur–Newton Algorithm 7.6 Matrix Sign Method . . . 7.7 Notes and References . . Problems . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
173 173 175 177 181 184 186 187 189
Polar Decomposition Approximation Properties . . . . . . . . . . . . . Sensitivity and Conditioning . . . . . . . . . . . . Newton’s Method . . . . . . . . . . . . . . . . . . Obtaining Iterations via the Matrix Sign Function The Pad´e Family of Methods . . . . . . . . . . . . Scaling the Newton Iteration . . . . . . . . . . . . Terminating the Iterations . . . . . . . . . . . . . Numerical Stability and Choice of H . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
193 197 199 202 202 203 205 207 209
8 The 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8
Terminating the Iterations . . . . . . . Numerical Stability of Sign Iterations . Numerical Experiments and Algorithm Best L∞ Approximation . . . . . . . . Notes and References . . . . . . . . . . Problems . . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . .
. . . . . . . .
. . . . . .
. . . . . . . .
. . . . . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
x
Contents 8.9 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 8.10 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
9 Schur–Parlett Algorithm 9.1 Evaluating Functions of the Atomic Blocks . . 9.2 Evaluating the Upper Triangular Part of f (T ) 9.3 Reordering and Blocking the Schur Form . . . 9.4 Schur–Parlett Algorithm for f (A) . . . . . . . 9.5 Preprocessing . . . . . . . . . . . . . . . . . . 9.6 Notes and References . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
221 221 225 226 228 230 231 231
10 Matrix Exponential 10.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . 10.2 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Scaling and Squaring Method . . . . . . . . . . . . . . . 10.4 Schur Algorithms . . . . . . . . . . . . . . . . . . . . . . 10.4.1 Newton Divided Difference Interpolation . . . . . 10.4.2 Schur–Fr´echet Algorithm . . . . . . . . . . . . . . 10.4.3 Schur–Parlett Algorithm . . . . . . . . . . . . . . 10.5 Numerical Experiment . . . . . . . . . . . . . . . . . . . 10.6 Evaluating the Fr´echet Derivative and Its Norm . . . . . 10.6.1 Quadrature . . . . . . . . . . . . . . . . . . . . . 10.6.2 The Kronecker Formulae . . . . . . . . . . . . . . 10.6.3 Computing and Estimating the Norm . . . . . . . 10.7 Miscellany . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.1 Hermitian Matrices and Best L∞ Approximation 10.7.2 Essentially Nonnegative Matrices . . . . . . . . . 10.7.3 Preprocessing . . . . . . . . . . . . . . . . . . . . 10.7.4 The ψ Functions . . . . . . . . . . . . . . . . . . . 10.8 Notes and References . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
233 233 238 241 250 250 251 251 252 253 254 256 258 259 259 260 261 261 262 265
11 Matrix Logarithm 11.1 Basic Properties . . . . . . . . . . . . . . . . . . . 11.2 Conditioning . . . . . . . . . . . . . . . . . . . . . 11.3 Series Expansions . . . . . . . . . . . . . . . . . . 11.4 Pad´e Approximation . . . . . . . . . . . . . . . . 11.5 Inverse Scaling and Squaring Method . . . . . . . 11.5.1 Schur Decomposition: Triangular Matrices 11.5.2 Full Matrices . . . . . . . . . . . . . . . . 11.6 Schur Algorithms . . . . . . . . . . . . . . . . . . 11.6.1 Schur–Fr´echet Algorithm . . . . . . . . . . 11.6.2 Schur–Parlett Algorithm . . . . . . . . . . 11.7 Numerical Experiment . . . . . . . . . . . . . . . 11.8 Evaluating the Fr´echet Derivative . . . . . . . . . 11.9 Notes and References . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
269 269 272 273 274 275 276 278 279 279 279 280 281 283 284
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . . . . . . . . .
. . . . . . .
. . . . . . . . . . . . . .
. . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
xi
Contents 12 Matrix Cosine and Sine 12.1 Basic Properties . . . . . . . . . . . . . . . . 12.2 Conditioning . . . . . . . . . . . . . . . . . . 12.3 Pad´e Approximation of Cosine . . . . . . . . 12.4 Double Angle Algorithm for Cosine . . . . . 12.5 Numerical Experiment . . . . . . . . . . . . 12.6 Double Angle Algorithm for Sine and Cosine 12.6.1 Preprocessing . . . . . . . . . . . . . 12.7 Notes and References . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
287 287 289 290 290 295 296 299 299 300
13 Function of Matrix Times Vector: f (A)b 13.1 Representation via Polynomial Interpolation 13.2 Krylov Subspace Methods . . . . . . . . . . 13.2.1 The Arnoldi Process . . . . . . . . . 13.2.2 Arnoldi Approximation of f (A)b . . . 13.2.3 Lanczos Biorthogonalization . . . . . 13.3 Quadrature . . . . . . . . . . . . . . . . . . . 13.3.1 On the Real Line . . . . . . . . . . . 13.3.2 Contour Integration . . . . . . . . . . 13.4 Differential Equations . . . . . . . . . . . . . 13.5 Other Methods . . . . . . . . . . . . . . . . 13.6 Notes and References . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
301 301 302 302 304 306 306 306 307 308 309 309 310
14 Miscellany 14.1 Structured Matrices . . . . . . . . . . . . . . . . . . . 14.1.1 Algebras and Groups . . . . . . . . . . . . . . 14.1.2 Monotone Functions . . . . . . . . . . . . . . 14.1.3 Other Structures . . . . . . . . . . . . . . . . 14.1.4 Data Sparse Representations . . . . . . . . . . 14.1.5 Computing Structured f (A) for Structured A 14.2 Exponential Decay of Functions of Banded Matrices . 14.3 Approximating Entries of Matrix Functions . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
313 313 313 315 315 316 316 317 318
A Notation B Background: Definitions and Useful Facts B.1 Basic Notation . . . . . . . . . . . . . . . . . B.2 Eigenvalues and Jordan Canonical Form . . B.3 Invariant Subspaces . . . . . . . . . . . . . . B.4 Special Classes of Matrices . . . . . . . . . . B.5 Matrix Factorizations and Decompositions . B.6 Pseudoinverse and Orthogonality . . . . . . B.6.1 Pseudoinverse . . . . . . . . . . . . . B.6.2 Projector and Orthogonal Projector . B.6.3 Partial Isometry . . . . . . . . . . . . B.7 Norms . . . . . . . . . . . . . . . . . . . . . B.8 Matrix Sequences and Series . . . . . . . . . B.9 Perturbation Expansions for Matrix Inverse
319 . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
. . . . . . . . . . . .
321 321 321 323 323 324 325 325 326 326 326 328 328
xii
Contents B.10 B.11 B.12 B.13 B.14 B.15 B.16
Sherman–Morrison–Woodbury Formula Nonnegative Matrices . . . . . . . . . . Positive (Semi)definite Ordering . . . . Kronecker Product and Sum . . . . . . Sylvester Equation . . . . . . . . . . . Floating Point Arithmetic . . . . . . . Divided Differences . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
329 329 330 331 331 331 332 334
C Operation Counts
335
D Matrix Function Toolbox
339
E Solutions to Problems
343
Bibliography
379
Index
415
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
MIMS EPrint: 2008.39
Manchester Institute for Mathematical Sciences School of Mathematics The University of Manchester
Reports available from: And by contacting:
http://www.manchester.ac.uk/mims/eprints The MIMS Secretary School of Mathematics The University of Manchester Manchester, M13 9PL, UK
ISSN 1749-9097
Contents
List of Figures
xiii
List of Tables
xv
Preface
xvii
1 Theory of Matrix Functions 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Definitions of f (A) . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Jordan Canonical Form . . . . . . . . . . . . . . . . . . . 1.2.2 Polynomial Interpolation . . . . . . . . . . . . . . . . . . 1.2.3 Cauchy Integral Theorem . . . . . . . . . . . . . . . . . . 1.2.4 Equivalence of Definitions . . . . . . . . . . . . . . . . . 1.2.5 Example: Function of Identity Plus Rank-1 Matrix . . . 1.2.6 Example: Function of Discrete Fourier Transform Matrix 1.3 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Nonprimary Matrix Functions . . . . . . . . . . . . . . . . . . . 1.5 Existence of (Real) Matrix Square Roots and Logarithms . . . . 1.6 Classification of Matrix Square Roots and Logarithms . . . . . . 1.7 Principal Square Root and Logarithm . . . . . . . . . . . . . . . 1.8 f (AB) and f (BA) . . . . . . . . . . . . . . . . . . . . . . . . . . 1.9 Miscellany . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10 A Brief History of Matrix Functions . . . . . . . . . . . . . . . . 1.11 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
1 1 2 2 4 7 8 8 10 10 14 16 17 20 21 23 26 27 29
2 Applications 2.1 Differential Equations . . . . . . . . . . . . . . . . . . . . . 2.1.1 Exponential Integrators . . . . . . . . . . . . . . . . 2.2 Nuclear Magnetic Resonance . . . . . . . . . . . . . . . . . 2.3 Markov Models . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Control Theory . . . . . . . . . . . . . . . . . . . . . . . . 2.5 The Nonsymmetric Eigenvalue Problem . . . . . . . . . . . 2.6 Orthogonalization and the Orthogonal Procrustes Problem 2.7 Theoretical Particle Physics . . . . . . . . . . . . . . . . . 2.8 Other Matrix Functions . . . . . . . . . . . . . . . . . . . . 2.9 Nonlinear Matrix Equations . . . . . . . . . . . . . . . . . 2.10 Geometric Mean . . . . . . . . . . . . . . . . . . . . . . . . 2.11 Pseudospectra . . . . . . . . . . . . . . . . . . . . . . . . . 2.12 Algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
35 35 36 37 37 39 41 42 43 44 44 46 47 47
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
vii
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
viii
Contents 2.13 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.14 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.14.1 Boundary Value Problems . . . . . . . . . . . . . . . . . . . 2.14.2 Semidefinite Programming . . . . . . . . . . . . . . . . . . . 2.14.3 Matrix Sector Function . . . . . . . . . . . . . . . . . . . . . 2.14.4 Matrix Disk Function . . . . . . . . . . . . . . . . . . . . . . 2.14.5 The Average Eye in Optics . . . . . . . . . . . . . . . . . . . 2.14.6 Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . 2.14.7 Bregman Divergences . . . . . . . . . . . . . . . . . . . . . . 2.14.8 Structured Matrix Interpolation . . . . . . . . . . . . . . . . 2.14.9 The Lambert W Function and Delay Differential Equations 2.15 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 Conditioning 3.1 Condition Numbers . . . . . . . . . . . . . . . . . 3.2 Properties of the Fr´echet Derivative . . . . . . . . 3.3 Bounding the Condition Number . . . . . . . . . 3.4 Computing or Estimating the Condition Number 3.5 Notes and References . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
4 Techniques for General Functions 4.1 Matrix Powers . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Polynomial Evaluation . . . . . . . . . . . . . . . . . . . 4.3 Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Rational Approximation . . . . . . . . . . . . . . . . . . 4.4.1 Best L∞ Approximation . . . . . . . . . . . . . . 4.4.2 Pad´e Approximation . . . . . . . . . . . . . . . . 4.4.3 Evaluating Rational Functions . . . . . . . . . . . 4.5 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . 4.6 Schur Decomposition and Triangular Matrices . . . . . . 4.7 Block Diagonalization . . . . . . . . . . . . . . . . . . . . 4.8 Interpolating Polynomial and Characteristic Polynomial 4.9 Matrix Iterations . . . . . . . . . . . . . . . . . . . . . . 4.9.1 Order of Convergence . . . . . . . . . . . . . . . . 4.9.2 Termination Criteria . . . . . . . . . . . . . . . . 4.9.3 Convergence . . . . . . . . . . . . . . . . . . . . . 4.9.4 Numerical Stability . . . . . . . . . . . . . . . . . 4.10 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . 4.11 Bounds for kf (A)k . . . . . . . . . . . . . . . . . . . . . 4.12 Notes and References . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
71 . 71 . 72 . 76 . 78 . 79 . 79 . 80 . 81 . 84 . 89 . 89 . 91 . 91 . 92 . 93 . 95 . 99 . 102 . 104 . 105
5 Matrix Sign Function 5.1 Sensitivity and Conditioning . 5.2 Schur Method . . . . . . . . . 5.3 Newton’s Method . . . . . . . 5.4 The Pad´e Family of Iterations 5.5 Scaling the Newton Iteration .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . . .
55 55 57 63 64 69 70
. . . . . .
. . . . .
. . . . . .
48 48 48 48 48 49 50 50 50 50 51 51 52
. . . . . .
. . . . .
. . . . . .
. . . . . . . . . . . . .
. . . . .
. . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
107 109 112 113 115 119
ix
Contents 5.6 5.7 5.8 5.9 5.10
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
121 123 125 128 129 131
6 Matrix Square Root 6.1 Sensitivity and Conditioning . . . . . . . . . 6.2 Schur Method . . . . . . . . . . . . . . . . . 6.3 Newton’s Method and Its Variants . . . . . . 6.4 Stability and Limiting Accuracy . . . . . . . 6.4.1 Newton Iteration . . . . . . . . . . . 6.4.2 DB Iterations . . . . . . . . . . . . . 6.4.3 CR Iteration . . . . . . . . . . . . . . 6.4.4 IN Iteration . . . . . . . . . . . . . . 6.4.5 Summary . . . . . . . . . . . . . . . . 6.5 Scaling the Newton Iteration . . . . . . . . . 6.6 Numerical Experiments . . . . . . . . . . . . 6.7 Iterations via the Matrix Sign Function . . . 6.8 Special Matrices . . . . . . . . . . . . . . . . 6.8.1 Binomial Iteration . . . . . . . . . . . 6.8.2 Modified Newton Iterations . . . . . 6.8.3 M-Matrices and H-Matrices . . . . . 6.8.4 Hermitian Positive Definite Matrices 6.9 Computing Small-Normed Square Roots . . 6.10 Comparison of Methods . . . . . . . . . . . . 6.11 Involutory Matrices . . . . . . . . . . . . . . 6.12 Notes and References . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
133 133 135 139 144 144 145 146 146 147 147 148 152 154 154 157 159 161 162 164 165 166 168
7 Matrix pth Root 7.1 Theory . . . . . . . . . . 7.2 Schur Method . . . . . . 7.3 Newton’s Method . . . . 7.4 Inverse Newton Method . 7.5 Schur–Newton Algorithm 7.6 Matrix Sign Method . . . 7.7 Notes and References . . Problems . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
173 173 175 177 181 184 186 187 189
Polar Decomposition Approximation Properties . . . . . . . . . . . . . Sensitivity and Conditioning . . . . . . . . . . . . Newton’s Method . . . . . . . . . . . . . . . . . . Obtaining Iterations via the Matrix Sign Function The Pad´e Family of Methods . . . . . . . . . . . . Scaling the Newton Iteration . . . . . . . . . . . . Terminating the Iterations . . . . . . . . . . . . . Numerical Stability and Choice of H . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
193 197 199 202 202 203 205 207 209
8 The 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8
Terminating the Iterations . . . . . . . Numerical Stability of Sign Iterations . Numerical Experiments and Algorithm Best L∞ Approximation . . . . . . . . Notes and References . . . . . . . . . . Problems . . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . .
. . . . . . . .
. . . . . .
. . . . . . . .
. . . . . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
x
Contents 8.9 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 8.10 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
9 Schur–Parlett Algorithm 9.1 Evaluating Functions of the Atomic Blocks . . 9.2 Evaluating the Upper Triangular Part of f (T ) 9.3 Reordering and Blocking the Schur Form . . . 9.4 Schur–Parlett Algorithm for f (A) . . . . . . . 9.5 Preprocessing . . . . . . . . . . . . . . . . . . 9.6 Notes and References . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
221 221 225 226 228 230 231 231
10 Matrix Exponential 10.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . 10.2 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Scaling and Squaring Method . . . . . . . . . . . . . . . 10.4 Schur Algorithms . . . . . . . . . . . . . . . . . . . . . . 10.4.1 Newton Divided Difference Interpolation . . . . . 10.4.2 Schur–Fr´echet Algorithm . . . . . . . . . . . . . . 10.4.3 Schur–Parlett Algorithm . . . . . . . . . . . . . . 10.5 Numerical Experiment . . . . . . . . . . . . . . . . . . . 10.6 Evaluating the Fr´echet Derivative and Its Norm . . . . . 10.6.1 Quadrature . . . . . . . . . . . . . . . . . . . . . 10.6.2 The Kronecker Formulae . . . . . . . . . . . . . . 10.6.3 Computing and Estimating the Norm . . . . . . . 10.7 Miscellany . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.1 Hermitian Matrices and Best L∞ Approximation 10.7.2 Essentially Nonnegative Matrices . . . . . . . . . 10.7.3 Preprocessing . . . . . . . . . . . . . . . . . . . . 10.7.4 The ψ Functions . . . . . . . . . . . . . . . . . . . 10.8 Notes and References . . . . . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
233 233 238 241 250 250 251 251 252 253 254 256 258 259 259 260 261 261 262 265
11 Matrix Logarithm 11.1 Basic Properties . . . . . . . . . . . . . . . . . . . 11.2 Conditioning . . . . . . . . . . . . . . . . . . . . . 11.3 Series Expansions . . . . . . . . . . . . . . . . . . 11.4 Pad´e Approximation . . . . . . . . . . . . . . . . 11.5 Inverse Scaling and Squaring Method . . . . . . . 11.5.1 Schur Decomposition: Triangular Matrices 11.5.2 Full Matrices . . . . . . . . . . . . . . . . 11.6 Schur Algorithms . . . . . . . . . . . . . . . . . . 11.6.1 Schur–Fr´echet Algorithm . . . . . . . . . . 11.6.2 Schur–Parlett Algorithm . . . . . . . . . . 11.7 Numerical Experiment . . . . . . . . . . . . . . . 11.8 Evaluating the Fr´echet Derivative . . . . . . . . . 11.9 Notes and References . . . . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
269 269 272 273 274 275 276 278 279 279 279 280 281 283 284
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . . . . . . . . .
. . . . . . .
. . . . . . . . . . . . . .
. . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
xi
Contents 12 Matrix Cosine and Sine 12.1 Basic Properties . . . . . . . . . . . . . . . . 12.2 Conditioning . . . . . . . . . . . . . . . . . . 12.3 Pad´e Approximation of Cosine . . . . . . . . 12.4 Double Angle Algorithm for Cosine . . . . . 12.5 Numerical Experiment . . . . . . . . . . . . 12.6 Double Angle Algorithm for Sine and Cosine 12.6.1 Preprocessing . . . . . . . . . . . . . 12.7 Notes and References . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
287 287 289 290 290 295 296 299 299 300
13 Function of Matrix Times Vector: f (A)b 13.1 Representation via Polynomial Interpolation 13.2 Krylov Subspace Methods . . . . . . . . . . 13.2.1 The Arnoldi Process . . . . . . . . . 13.2.2 Arnoldi Approximation of f (A)b . . . 13.2.3 Lanczos Biorthogonalization . . . . . 13.3 Quadrature . . . . . . . . . . . . . . . . . . . 13.3.1 On the Real Line . . . . . . . . . . . 13.3.2 Contour Integration . . . . . . . . . . 13.4 Differential Equations . . . . . . . . . . . . . 13.5 Other Methods . . . . . . . . . . . . . . . . 13.6 Notes and References . . . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
301 301 302 302 304 306 306 306 307 308 309 309 310
14 Miscellany 14.1 Structured Matrices . . . . . . . . . . . . . . . . . . . 14.1.1 Algebras and Groups . . . . . . . . . . . . . . 14.1.2 Monotone Functions . . . . . . . . . . . . . . 14.1.3 Other Structures . . . . . . . . . . . . . . . . 14.1.4 Data Sparse Representations . . . . . . . . . . 14.1.5 Computing Structured f (A) for Structured A 14.2 Exponential Decay of Functions of Banded Matrices . 14.3 Approximating Entries of Matrix Functions . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
313 313 313 315 315 316 316 317 318
A Notation B Background: Definitions and Useful Facts B.1 Basic Notation . . . . . . . . . . . . . . . . . B.2 Eigenvalues and Jordan Canonical Form . . B.3 Invariant Subspaces . . . . . . . . . . . . . . B.4 Special Classes of Matrices . . . . . . . . . . B.5 Matrix Factorizations and Decompositions . B.6 Pseudoinverse and Orthogonality . . . . . . B.6.1 Pseudoinverse . . . . . . . . . . . . . B.6.2 Projector and Orthogonal Projector . B.6.3 Partial Isometry . . . . . . . . . . . . B.7 Norms . . . . . . . . . . . . . . . . . . . . . B.8 Matrix Sequences and Series . . . . . . . . . B.9 Perturbation Expansions for Matrix Inverse
319 . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
. . . . . . . . . . . .
321 321 321 323 323 324 325 325 326 326 326 328 328
xii
Contents B.10 B.11 B.12 B.13 B.14 B.15 B.16
Sherman–Morrison–Woodbury Formula Nonnegative Matrices . . . . . . . . . . Positive (Semi)definite Ordering . . . . Kronecker Product and Sum . . . . . . Sylvester Equation . . . . . . . . . . . Floating Point Arithmetic . . . . . . . Divided Differences . . . . . . . . . . . Problems . . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
329 329 330 331 331 331 332 334
C Operation Counts
335
D Matrix Function Toolbox
339
E Solutions to Problems
343
Bibliography
379
Index
415
Copyright ©2008 by the Society for Industrial and Applied Mathematics. This electronic version is for personal use and may not be duplicated or distributed. From "Functions of Matrices: Theory and Computation" by Nicholas J. Higham. This book is available for purchase at www.siam.org/catalog.
