A Brief Introduction to R
Page 3
A Brief Introduction to R > 1+1 [1] 2 > 2^3 # Two to the power 3 [1] 8 > 1:30 [1] 1 2 3 4 5 [26] 26 27 28 29 30
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
> gamma(.5)^2 [1] 3.141593
# Gamma(1/2) = Sqrt(Pi)
> x y x+y [1] 3 > z z [1] 3 > x z # No dynamic updating; it's not a spreadsheet [1] 3 > x+y [1] 3 4 5 6 7 8 > y = 1 + 2*x # Another way to do assignment > cbind(x,y) x y [1,] 1 3 [2,] 2 5 [3,] 3 7 [4,] 4 9 [5,] 5 11 [6,] 6 13 > z 4] > z [1] 11 13
# z gets y such that x > 4
> # If you put an array of integers inside the brackets, you get those > # elements, in the order indicated. > y[c(6,5,4,3,2,1)] # y in opposite order [1] 13 11 9 7 5 3 > y[c(2,2,2,3,4)] # Repeats are okay [1] 5 5 5 7 9 > y[7] # There is no seventh element. NA is the missing value code [1] NA
Page 1 of 6
Page 4
> # Computing probabilities, etc. > > pnorm(0) # Area less than zero for a standard normal [1] 0.5 > > pnorm(160,mean=100,sd=15) # IQ of 160 [1] 0.9999683 > > pcauchy(4) [1] 0.9220209 > > dnorm(0) # height of the curve [1] 0.3989423 > > dpois(0,lambda=3) # P(Y=0) for Y ~ Poisson(3) [1] 0.04978707 > > qnorm(0.975) # z value with P(Z > qf(0.975,df1=6,df2=122) # Critical value for F, not in any table [1] 2.513606 > > CriticalValue = qchisq(0.95,df=1:8) > df=1:8; cbind(df,CriticalValue) df CriticalValue [1,] 1 3.841459 [2,] 2 5.991465 [3,] 3 7.814728 [4,] 4 9.487729 [5,] 5 11.070498 [6,] 6 12.591587 [7,] 7 14.067140 [8,] 8 15.507313
Page 2 of 6
Page 5
> # data() lists available data sets > trees Girth Height Volume 1 8.3 70 10.3 2 8.6 65 10.3 3 8.8 63 10.2 4 10.5 72 16.4 5 10.7 81 18.8 6 10.8 83 19.7 7 11.0 66 15.6 8 11.0 75 18.2 9 11.1 80 22.6 10 11.2 75 19.9 11 11.3 79 24.2 12 11.4 76 21.0 13 11.4 76 21.4 14 11.7 69 21.3 15 12.0 75 19.1 16 12.9 74 22.2 17 12.9 85 33.8 18 13.3 86 27.4 19 13.7 71 25.7 20 13.8 64 24.9 21 14.0 78 34.5 22 14.2 80 31.7 23 14.5 74 36.3 24 16.0 72 38.3 25 16.3 77 42.6 26 17.3 81 55.4 27 17.5 82 55.7 28 17.9 80 58.3 29 18.0 80 51.5 30 18.0 80 51.0 31 20.6 87 77.0 > mean(trees$Height); var(trees$Height) [1] 76 [1] 40.6 > cor(trees) Girth Height Volume Girth 1.0000000 0.5192801 0.9671194 Height 0.5192801 1.0000000 0.5982497 Volume 0.9671194 0.5982497 1.0000000
Page 3 of 6
Page 6
> plot(trees$Girth,trees$Volume)
Page 4 of 6
Page 7
> pairs(trees) # Can also give it a matrix, in which the columns are variables
Page 5 of 6
Page 8
Going further on your own Look at the free mini-book An introduction to R by Venables et al. A copy is available at http://www.utstat.toronto.edu/~brunner/help/R-intro.pdf. Or, try > library(help="stats")
AIC ARMAacf ARMAtoMA Beta Binomial Box.test C Cauchy Chisquare Exponential FDist GammaDist Geometric HoltWinters
Akaike's An Information Criterion Compute Theoretical ACF for an ARMA Process Convert ARMA Process to Infinite MA Process The Beta Distribution The Binomial Distribution Box-Pierce and Ljung-Box Tests Sets Contrasts for a Factor The Cauchy Distribution The (non-central) Chi-Squared Distribution The Exponential Distribution The F Distribution The Gamma Distribution The Geometric Distribution Holt-Winters Filtering
. . . list continues!
To see details on one of these packages, type something like > help(t.test)
Generally, R's help assumes knowledge of the topic, but they often give references. For a quicker introduction to many statistical topics, try the Wikipedia: http://en.wikipedia.org
Or you could ask me, but I might have to look in the Wikipedia.
This document is licensed under a Creative Commons Attribution - ShareAlike 3.0 Unported License: http://creativecommons.org/licenses/by-sa/3.0/deed.en_US. Use any part of it as you like and share the result freely.
Page 6 of 6
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 9
R
More Linear Algebra1 STA 302: Fall 2013
1
See Chapter 2 of Linear models in statistics for more detail. This slide show is an open-source document. See last slide for copyright information. 1 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 10
R
Overview
1
Things you already know
2
Spectral decomposition
3
Positive definite matrices
4
Square root matrices
5
R
2 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 11
R
You already know about Matrices A = [aij ] Matrix addition and subtraction A + B = [aij + bij ] Scalar multiplication aB = [a bij ] ! # " Matrix multiplication AB = aik bkj k
Inverse A−1 A = AA−1 = I Transpose A" = [aji ]
Symmetric matrices A = A" Determinants Linear independence
3 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 12
R
Linear independence
X be an n × p matrix of constants.The columns of X are said to be linearly dependent if there exists v "= 0 with Xv = 0. We will say that the columns of X are linearly independent if Xv = 0 implies v = 0.
For example, show that A−1 exists implies that the columns of A are linearly independent. Av = 0 ⇒ A−1 Av = A−1 0 ⇒ v = 0
4 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 13
R
How to show A−1% = A%−1
Suppose B = A−1 , meaning AB = BA = I. Must show two things: B" A" = I and A" B" = I. AB = I ⇒ B" A" = I" = I BA = I ⇒ A" B" = I" = I !
5 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 14
R
Extras You may not know about these, but we may use them occasionally
Trace Rank Partitioned matrices
6 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 15
R
Trace of a square matrix
Sum of diagonal elements Obvious: tr(A + B) = tr(A) + tr(B) Not obvious: tr(AB) = tr(BA)
7 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 16
R
Rank
Row rank is the number of linearly independent rows Column rank is the number of linearly independent columns Rank of a matrix is the minimum of row rank and column rank rank(AB) = min (rank(A), rank(B))
8 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 17
R
Partitioned matrix
A matrix of matrices
$
A B C D
%
Row by column (matrix) multiplication works, provided the matrices are the right sizes.
9 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 18
R
Eigenvalues and eigenvectors
Let A = [ai,j ] be an n × n matrix, so that the following applies to square matrices. A is said to have an eigenvalue λ and (non-zero) eigenvector x corresponding to λ if Ax = λx.
Eigenvalues are the λ values that solve the determinantal equation |A − λI| = 0. The determinant is the product of the eigenvalues: &n |A| = i=1 λi
10 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 19
R
Spectral decomposition of symmetric matrices The Spectral decomposition theorem says that every square and symmetric matrix A = [ai,j ] may be written A = CDC" , where the columns of C (which may also be denoted x1 , . . . , xn ) are the eigenvectors of A, and the diagonal matrix D contains the corresponding eigenvalues. λ1 0 · · · 0 0 λ2 · · · 0 D= . .. . . .. . . . . . 0
0
···
λn
The eigenvectors may be chosen to be orthonormal, so that C is an orthogonal matrix. That is, CC" = C" C = I. 11 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 20
R
Positive definite matrices
The n × n matrix A is said to be positive definite if y" Ay > 0 for all n × 1 vectors y "= 0. It is called non-negative definite (or sometimes positive semi-definite) if y" Ay ≥ 0.
12 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 21
R
Example: Show X% X non-negative definite
Let X be an n × p matrix of real constants and y be p × 1. Then Z = Xy is n × 1, and y" (X" X) y = (Xy)" (Xy) = Z" Z n " = Zi2 ≥ 0 i=1
13 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 22
R
Some properties of symmetric positive definite matrices Variance-covariance matrices are often assumed positive definite.
For a symmetric matrix, Positive definite ⇓
All eigenvalues positive ⇓
Inverse exists ⇔ Columns (rows) linearly independent If a real symmetric matrix is also non-negative definite, as a variance-covariance matrix must be, Inverse exists ⇒ Positive definite
14 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 23
R
Showing Positive definite ⇒ Eigenvalues positive For example
Let A be square and symmetric as well as positive definite. Spectral decomposition says A = CDC" . Using y" Ay > 0, let y be an eigenvector, say the third one. Because eigenvectors are orthonormal, y" Ay = y" CDC" y
= ( 0 0 1 ··· = λ3
0 )
λ1 0 · · · 0 λ2 · · · .. .. . . . . . 0 0 ···
0 0 .. . λn
0 0 1 .. . 0
> 0 15 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 24
R
Inverse of a diagonal matrix Suppose D = [di,j ] is a diagonal matrix with elements. It is easy to verify that d1,1 0 · · · 0 1/d1,1 0 0 d2,2 · · · 0 0 1/d 2,2 .. .. .. .. .. . . . . . . . . 0
0
···
dn,n
And 1/d1,1 0 ··· 0 1/d 2,2 · · · .. .. .. . . . 0 0 ···
0
0 0 .. . 1/dn,n
0
non-zero diagonal ··· ··· .. .
0 0 .. .
···
1/dn,n
d1,1 0 · · · 0 d2,2 · · · .. .. .. . . . 0 0 ···
0 0 .. . dn,n
=I
=I 16 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 25
R
Showing Eigenvalues positive ⇒ Inverse exists
For a symmetric, positive definite matrix
Let A be symmetric and positive definite. Then A = CDC" and its eigenvalues are positive. Let B = CD−1 C" Showing B = A−1 : AB = CDC" CD−1 C" = I BA = CD−1 C" CDC" = I So
A−1 = CD−1C% 17 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 26
R
Square root matrices For symmetric, non-negative definite matrices
Define
√
D1/2 =
So that
D1/2 D1/2
λ1 0 .. .
=
√
=
λ1 0 .. .
√0 λ2 .. .
··· ··· .. .
0
···
0
√0 λ2 .. .
··· ··· .. .
0
···
0 λ1 0 .. .
0 λ2 .. .
··· ··· .. .
0
0
···
0 0 .. .
λn
0 0 .. √. λn
0 0 .. √. λn
√
λ1 0 .. . 0
√0 λ2 .. .
··· ··· .. .
0
···
0 0 .. √. λn
=D 18 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 27
R
For a non-negative definite, symmetric matrix A
Define
A1/2 = CD1/2C% So that A1/2 A1/2 = CD1/2 C" CD1/2 C" = CD1/2 I D1/2 C" = CD1/2 D1/2 C" = CDC" = A
19 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 28
R
The square root of the inverse is the inverse of the square root Let A be symmetric and positive definite, with A = CDC" . Let B = CD−1/2 C" . What is D−1/2 ? 3 41/2 Show B = A−1
BB = CD−1/2 C" CD−1/2 C" B = CD−1 C" = A−1
3 4−1 Show B = A1/2
A1/2 B = CD1/2 C" CD−1/2 C" = I
BA1/2 = CD−1/2 C" CD1/2 C" = I Just write
A−1/2 = CD−1/2C% 20 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 29
R
Matrix calculation with R > is.matrix(3)
# Is the number 3 a 1x1 matrix?
[1] FALSE
> treecorr = cor(trees); treecorr Girth Height Volume Girth 1.0000000 0.5192801 0.9671194 Height 0.5192801 1.0000000 0.5982497 Volume 0.9671194 0.5982497 1.0000000
> is.matrix(treecorr) [1] TRUE 21 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 30
R
Creating matrices Bind rows into a matrix > # Bind rows > A = rbind( + + [1,] [2,] [3,]
of a matrix together c(3, 2, 6,8), c(2,10,-7,4), c(6, 6, 9,1) ); A
[,1] [,2] [,3] [,4] 3 2 6 8 2 10 -7 4 6 6 9 1
> # Transpose > t(A) [1,] [2,] [3,] [4,]
[,1] [,2] [,3] 3 2 6 2 10 6 6 -7 9 8 4 1 22 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 31
R
Matrix multiplication Remember, A is 3 × 4
> # U = AA’ (3x3), V = A’A (4x4) > U = A %*% t(A) > V = t(A) %*% A; V
[1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81
23 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 32
R
Determinants
> # U = AA’ (3x3), V = A’A (4x4) > # So rank(V) cannot exceed 3 and det(V)=0 > det(U); det(V) [1] 1490273 [1] -3.622862e-09
Inverse of U exists, but inverse of V does not.
24 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 33
R
Inverses The solve function is for solving systems of linear equations like Mx = b. Just typing solve(M) gives M−1 . > solve(U) [,1] [,2] [,3] [1,] 0.0173505123 -8.508508e-04 -1.029342e-02 [2,] -0.0008508508 5.997559e-03 2.013054e-06 [3,] -0.0102934160 2.013054e-06 1.264265e-02 > solve(V) Error in solve.default(V) : system is computationally singular: reciprocal condition number = 6.64193e-18 25 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 34
R
Eigenvalues and eigenvectors
> eigen(U) $values [1] 234.01162 162.89294
39.09544
$vectors [,1] [,2] [,3] [1,] -0.6025375 0.1592598 0.78203893 [2,] -0.2964610 -0.9544379 -0.03404605 [3,] -0.7409854 0.2523581 -0.62229894
26 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 35
R
V should have one zero eigenvalue
> eigen(V) $values [1] 2.340116e+02
1.628929e+02
3.909544e+01 -1.012719e-14
$vectors [1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] -0.4475551 0.006507269 -0.2328249 0.863391352 -0.5632053 -0.604226296 -0.4014589 -0.395652773 -0.5366171 0.776297432 -0.1071763 -0.312917928 -0.4410627 -0.179528649 0.8792818 0.009829883
27 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Spectral decomposition V = CDC%
Square root matrices
Page 36
R
> eigenV = eigen(V) > C = eigenV$vectors; D = diag(eigenV$values); D [,1] [,2] [,3] [,4] [1,] 234.0116 0.0000 0.00000 0.000000e+00 [2,] 0.0000 162.8929 0.00000 0.000000e+00 [3,] 0.0000 0.0000 39.09544 0.000000e+00 [4,] 0.0000 0.0000 0.00000 -1.012719e-14
> # C is an orthoganal matrix > C %*% t(C) [,1] [1,] 1.000000e+00 [2,] 5.551115e-17 [3,] 0.000000e+00 [4,] -3.989864e-17
[,2] 5.551115e-17 1.000000e+00 2.636780e-16 3.556183e-17
[,3] [,4] 0.000000e+00 -3.989864e-17 2.636780e-16 3.556183e-17 1.000000e+00 2.558717e-16 2.558717e-16 1.000000e+00 28 / 32
Things you already know
Spectral decomposition
Verify V = CDC%
> V;
C %*% D %*% t(C)
[1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81
[1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81
Positive definite matrices
Square root matrices
Page 37
R
29 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 38
R
Square root matrix V1/2 = CD1/2 C% > sqrtV = C %*% sqrt(D) %*% t(C) Warning message: In sqrt(D) : NaNs produced > # Multiply to get V > sqrtV %*% sqrtV; V [1,] [2,] [3,] [4,] [1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN [,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81 30 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 39
R
What happened? > D; sqrt(D) [,1] [,2] [,3] [,4] [1,] 234.0116 0.0000 0.00000 0.000000e+00 [2,] 0.0000 162.8929 0.00000 0.000000e+00 [3,] 0.0000 0.0000 39.09544 0.000000e+00 [4,] 0.0000 0.0000 0.00000 -1.012719e-14 [,1] [,2] [,3] [,4] [1,] 15.29744 0.00000 0.000000 0 [2,] 0.00000 12.76295 0.000000 0 [3,] 0.00000 0.00000 6.252635 0 [4,] 0.00000 0.00000 0.000000 NaN
Warning message: In sqrt(D) : NaNs produced 31 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 40
R
Copyright Information
This slide show was prepared by Jerry Brunner, Department of Statistical Sciences, University of Toronto. It is licensed under a Creative Commons Attribution - ShareAlike 3.0 Unported License. Use any part of it as you like and share the result freely. The LATEX source code is available from the course website: http://www.utstat.toronto.edu/∼ brunner/oldclass/302f13
32 / 32
A Brief Introduction to R > 1+1 [1] 2 > 2^3 # Two to the power 3 [1] 8 > 1:30 [1] 1 2 3 4 5 [26] 26 27 28 29 30
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
> gamma(.5)^2 [1] 3.141593
# Gamma(1/2) = Sqrt(Pi)
> x y x+y [1] 3 > z z [1] 3 > x z # No dynamic updating; it's not a spreadsheet [1] 3 > x+y [1] 3 4 5 6 7 8 > y = 1 + 2*x # Another way to do assignment > cbind(x,y) x y [1,] 1 3 [2,] 2 5 [3,] 3 7 [4,] 4 9 [5,] 5 11 [6,] 6 13 > z 4] > z [1] 11 13
# z gets y such that x > 4
> # If you put an array of integers inside the brackets, you get those > # elements, in the order indicated. > y[c(6,5,4,3,2,1)] # y in opposite order [1] 13 11 9 7 5 3 > y[c(2,2,2,3,4)] # Repeats are okay [1] 5 5 5 7 9 > y[7] # There is no seventh element. NA is the missing value code [1] NA
Page 1 of 6
Page 4
> # Computing probabilities, etc. > > pnorm(0) # Area less than zero for a standard normal [1] 0.5 > > pnorm(160,mean=100,sd=15) # IQ of 160 [1] 0.9999683 > > pcauchy(4) [1] 0.9220209 > > dnorm(0) # height of the curve [1] 0.3989423 > > dpois(0,lambda=3) # P(Y=0) for Y ~ Poisson(3) [1] 0.04978707 > > qnorm(0.975) # z value with P(Z > qf(0.975,df1=6,df2=122) # Critical value for F, not in any table [1] 2.513606 > > CriticalValue = qchisq(0.95,df=1:8) > df=1:8; cbind(df,CriticalValue) df CriticalValue [1,] 1 3.841459 [2,] 2 5.991465 [3,] 3 7.814728 [4,] 4 9.487729 [5,] 5 11.070498 [6,] 6 12.591587 [7,] 7 14.067140 [8,] 8 15.507313
Page 2 of 6
Page 5
> # data() lists available data sets > trees Girth Height Volume 1 8.3 70 10.3 2 8.6 65 10.3 3 8.8 63 10.2 4 10.5 72 16.4 5 10.7 81 18.8 6 10.8 83 19.7 7 11.0 66 15.6 8 11.0 75 18.2 9 11.1 80 22.6 10 11.2 75 19.9 11 11.3 79 24.2 12 11.4 76 21.0 13 11.4 76 21.4 14 11.7 69 21.3 15 12.0 75 19.1 16 12.9 74 22.2 17 12.9 85 33.8 18 13.3 86 27.4 19 13.7 71 25.7 20 13.8 64 24.9 21 14.0 78 34.5 22 14.2 80 31.7 23 14.5 74 36.3 24 16.0 72 38.3 25 16.3 77 42.6 26 17.3 81 55.4 27 17.5 82 55.7 28 17.9 80 58.3 29 18.0 80 51.5 30 18.0 80 51.0 31 20.6 87 77.0 > mean(trees$Height); var(trees$Height) [1] 76 [1] 40.6 > cor(trees) Girth Height Volume Girth 1.0000000 0.5192801 0.9671194 Height 0.5192801 1.0000000 0.5982497 Volume 0.9671194 0.5982497 1.0000000
Page 3 of 6
Page 6
> plot(trees$Girth,trees$Volume)
Page 4 of 6
Page 7
> pairs(trees) # Can also give it a matrix, in which the columns are variables
Page 5 of 6
Page 8
Going further on your own Look at the free mini-book An introduction to R by Venables et al. A copy is available at http://www.utstat.toronto.edu/~brunner/help/R-intro.pdf. Or, try > library(help="stats")
AIC ARMAacf ARMAtoMA Beta Binomial Box.test C Cauchy Chisquare Exponential FDist GammaDist Geometric HoltWinters
Akaike's An Information Criterion Compute Theoretical ACF for an ARMA Process Convert ARMA Process to Infinite MA Process The Beta Distribution The Binomial Distribution Box-Pierce and Ljung-Box Tests Sets Contrasts for a Factor The Cauchy Distribution The (non-central) Chi-Squared Distribution The Exponential Distribution The F Distribution The Gamma Distribution The Geometric Distribution Holt-Winters Filtering
. . . list continues!
To see details on one of these packages, type something like > help(t.test)
Generally, R's help assumes knowledge of the topic, but they often give references. For a quicker introduction to many statistical topics, try the Wikipedia: http://en.wikipedia.org
Or you could ask me, but I might have to look in the Wikipedia.
This document is licensed under a Creative Commons Attribution - ShareAlike 3.0 Unported License: http://creativecommons.org/licenses/by-sa/3.0/deed.en_US. Use any part of it as you like and share the result freely.
Page 6 of 6
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 9
R
More Linear Algebra1 STA 302: Fall 2013
1
See Chapter 2 of Linear models in statistics for more detail. This slide show is an open-source document. See last slide for copyright information. 1 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 10
R
Overview
1
Things you already know
2
Spectral decomposition
3
Positive definite matrices
4
Square root matrices
5
R
2 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 11
R
You already know about Matrices A = [aij ] Matrix addition and subtraction A + B = [aij + bij ] Scalar multiplication aB = [a bij ] ! # " Matrix multiplication AB = aik bkj k
Inverse A−1 A = AA−1 = I Transpose A" = [aji ]
Symmetric matrices A = A" Determinants Linear independence
3 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 12
R
Linear independence
X be an n × p matrix of constants.The columns of X are said to be linearly dependent if there exists v "= 0 with Xv = 0. We will say that the columns of X are linearly independent if Xv = 0 implies v = 0.
For example, show that A−1 exists implies that the columns of A are linearly independent. Av = 0 ⇒ A−1 Av = A−1 0 ⇒ v = 0
4 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 13
R
How to show A−1% = A%−1
Suppose B = A−1 , meaning AB = BA = I. Must show two things: B" A" = I and A" B" = I. AB = I ⇒ B" A" = I" = I BA = I ⇒ A" B" = I" = I !
5 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 14
R
Extras You may not know about these, but we may use them occasionally
Trace Rank Partitioned matrices
6 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 15
R
Trace of a square matrix
Sum of diagonal elements Obvious: tr(A + B) = tr(A) + tr(B) Not obvious: tr(AB) = tr(BA)
7 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 16
R
Rank
Row rank is the number of linearly independent rows Column rank is the number of linearly independent columns Rank of a matrix is the minimum of row rank and column rank rank(AB) = min (rank(A), rank(B))
8 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 17
R
Partitioned matrix
A matrix of matrices
$
A B C D
%
Row by column (matrix) multiplication works, provided the matrices are the right sizes.
9 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 18
R
Eigenvalues and eigenvectors
Let A = [ai,j ] be an n × n matrix, so that the following applies to square matrices. A is said to have an eigenvalue λ and (non-zero) eigenvector x corresponding to λ if Ax = λx.
Eigenvalues are the λ values that solve the determinantal equation |A − λI| = 0. The determinant is the product of the eigenvalues: &n |A| = i=1 λi
10 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 19
R
Spectral decomposition of symmetric matrices The Spectral decomposition theorem says that every square and symmetric matrix A = [ai,j ] may be written A = CDC" , where the columns of C (which may also be denoted x1 , . . . , xn ) are the eigenvectors of A, and the diagonal matrix D contains the corresponding eigenvalues. λ1 0 · · · 0 0 λ2 · · · 0 D= . .. . . .. . . . . . 0
0
···
λn
The eigenvectors may be chosen to be orthonormal, so that C is an orthogonal matrix. That is, CC" = C" C = I. 11 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 20
R
Positive definite matrices
The n × n matrix A is said to be positive definite if y" Ay > 0 for all n × 1 vectors y "= 0. It is called non-negative definite (or sometimes positive semi-definite) if y" Ay ≥ 0.
12 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 21
R
Example: Show X% X non-negative definite
Let X be an n × p matrix of real constants and y be p × 1. Then Z = Xy is n × 1, and y" (X" X) y = (Xy)" (Xy) = Z" Z n " = Zi2 ≥ 0 i=1
13 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 22
R
Some properties of symmetric positive definite matrices Variance-covariance matrices are often assumed positive definite.
For a symmetric matrix, Positive definite ⇓
All eigenvalues positive ⇓
Inverse exists ⇔ Columns (rows) linearly independent If a real symmetric matrix is also non-negative definite, as a variance-covariance matrix must be, Inverse exists ⇒ Positive definite
14 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 23
R
Showing Positive definite ⇒ Eigenvalues positive For example
Let A be square and symmetric as well as positive definite. Spectral decomposition says A = CDC" . Using y" Ay > 0, let y be an eigenvector, say the third one. Because eigenvectors are orthonormal, y" Ay = y" CDC" y
= ( 0 0 1 ··· = λ3
0 )
λ1 0 · · · 0 λ2 · · · .. .. . . . . . 0 0 ···
0 0 .. . λn
0 0 1 .. . 0
> 0 15 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 24
R
Inverse of a diagonal matrix Suppose D = [di,j ] is a diagonal matrix with elements. It is easy to verify that d1,1 0 · · · 0 1/d1,1 0 0 d2,2 · · · 0 0 1/d 2,2 .. .. .. .. .. . . . . . . . . 0
0
···
dn,n
And 1/d1,1 0 ··· 0 1/d 2,2 · · · .. .. .. . . . 0 0 ···
0
0 0 .. . 1/dn,n
0
non-zero diagonal ··· ··· .. .
0 0 .. .
···
1/dn,n
d1,1 0 · · · 0 d2,2 · · · .. .. .. . . . 0 0 ···
0 0 .. . dn,n
=I
=I 16 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 25
R
Showing Eigenvalues positive ⇒ Inverse exists
For a symmetric, positive definite matrix
Let A be symmetric and positive definite. Then A = CDC" and its eigenvalues are positive. Let B = CD−1 C" Showing B = A−1 : AB = CDC" CD−1 C" = I BA = CD−1 C" CDC" = I So
A−1 = CD−1C% 17 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 26
R
Square root matrices For symmetric, non-negative definite matrices
Define
√
D1/2 =
So that
D1/2 D1/2
λ1 0 .. .
=
√
=
λ1 0 .. .
√0 λ2 .. .
··· ··· .. .
0
···
0
√0 λ2 .. .
··· ··· .. .
0
···
0 λ1 0 .. .
0 λ2 .. .
··· ··· .. .
0
0
···
0 0 .. .
λn
0 0 .. √. λn
0 0 .. √. λn
√
λ1 0 .. . 0
√0 λ2 .. .
··· ··· .. .
0
···
0 0 .. √. λn
=D 18 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 27
R
For a non-negative definite, symmetric matrix A
Define
A1/2 = CD1/2C% So that A1/2 A1/2 = CD1/2 C" CD1/2 C" = CD1/2 I D1/2 C" = CD1/2 D1/2 C" = CDC" = A
19 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 28
R
The square root of the inverse is the inverse of the square root Let A be symmetric and positive definite, with A = CDC" . Let B = CD−1/2 C" . What is D−1/2 ? 3 41/2 Show B = A−1
BB = CD−1/2 C" CD−1/2 C" B = CD−1 C" = A−1
3 4−1 Show B = A1/2
A1/2 B = CD1/2 C" CD−1/2 C" = I
BA1/2 = CD−1/2 C" CD1/2 C" = I Just write
A−1/2 = CD−1/2C% 20 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 29
R
Matrix calculation with R > is.matrix(3)
# Is the number 3 a 1x1 matrix?
[1] FALSE
> treecorr = cor(trees); treecorr Girth Height Volume Girth 1.0000000 0.5192801 0.9671194 Height 0.5192801 1.0000000 0.5982497 Volume 0.9671194 0.5982497 1.0000000
> is.matrix(treecorr) [1] TRUE 21 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 30
R
Creating matrices Bind rows into a matrix > # Bind rows > A = rbind( + + [1,] [2,] [3,]
of a matrix together c(3, 2, 6,8), c(2,10,-7,4), c(6, 6, 9,1) ); A
[,1] [,2] [,3] [,4] 3 2 6 8 2 10 -7 4 6 6 9 1
> # Transpose > t(A) [1,] [2,] [3,] [4,]
[,1] [,2] [,3] 3 2 6 2 10 6 6 -7 9 8 4 1 22 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 31
R
Matrix multiplication Remember, A is 3 × 4
> # U = AA’ (3x3), V = A’A (4x4) > U = A %*% t(A) > V = t(A) %*% A; V
[1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81
23 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 32
R
Determinants
> # U = AA’ (3x3), V = A’A (4x4) > # So rank(V) cannot exceed 3 and det(V)=0 > det(U); det(V) [1] 1490273 [1] -3.622862e-09
Inverse of U exists, but inverse of V does not.
24 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 33
R
Inverses The solve function is for solving systems of linear equations like Mx = b. Just typing solve(M) gives M−1 . > solve(U) [,1] [,2] [,3] [1,] 0.0173505123 -8.508508e-04 -1.029342e-02 [2,] -0.0008508508 5.997559e-03 2.013054e-06 [3,] -0.0102934160 2.013054e-06 1.264265e-02 > solve(V) Error in solve.default(V) : system is computationally singular: reciprocal condition number = 6.64193e-18 25 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 34
R
Eigenvalues and eigenvectors
> eigen(U) $values [1] 234.01162 162.89294
39.09544
$vectors [,1] [,2] [,3] [1,] -0.6025375 0.1592598 0.78203893 [2,] -0.2964610 -0.9544379 -0.03404605 [3,] -0.7409854 0.2523581 -0.62229894
26 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 35
R
V should have one zero eigenvalue
> eigen(V) $values [1] 2.340116e+02
1.628929e+02
3.909544e+01 -1.012719e-14
$vectors [1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] -0.4475551 0.006507269 -0.2328249 0.863391352 -0.5632053 -0.604226296 -0.4014589 -0.395652773 -0.5366171 0.776297432 -0.1071763 -0.312917928 -0.4410627 -0.179528649 0.8792818 0.009829883
27 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Spectral decomposition V = CDC%
Square root matrices
Page 36
R
> eigenV = eigen(V) > C = eigenV$vectors; D = diag(eigenV$values); D [,1] [,2] [,3] [,4] [1,] 234.0116 0.0000 0.00000 0.000000e+00 [2,] 0.0000 162.8929 0.00000 0.000000e+00 [3,] 0.0000 0.0000 39.09544 0.000000e+00 [4,] 0.0000 0.0000 0.00000 -1.012719e-14
> # C is an orthoganal matrix > C %*% t(C) [,1] [1,] 1.000000e+00 [2,] 5.551115e-17 [3,] 0.000000e+00 [4,] -3.989864e-17
[,2] 5.551115e-17 1.000000e+00 2.636780e-16 3.556183e-17
[,3] [,4] 0.000000e+00 -3.989864e-17 2.636780e-16 3.556183e-17 1.000000e+00 2.558717e-16 2.558717e-16 1.000000e+00 28 / 32
Things you already know
Spectral decomposition
Verify V = CDC%
> V;
C %*% D %*% t(C)
[1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81
[1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81
Positive definite matrices
Square root matrices
Page 37
R
29 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 38
R
Square root matrix V1/2 = CD1/2 C% > sqrtV = C %*% sqrt(D) %*% t(C) Warning message: In sqrt(D) : NaNs produced > # Multiply to get V > sqrtV %*% sqrtV; V [1,] [2,] [3,] [4,] [1,] [2,] [3,] [4,]
[,1] [,2] [,3] [,4] NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN [,1] [,2] [,3] [,4] 49 62 58 38 62 140 -4 62 58 -4 166 29 38 62 29 81 30 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 39
R
What happened? > D; sqrt(D) [,1] [,2] [,3] [,4] [1,] 234.0116 0.0000 0.00000 0.000000e+00 [2,] 0.0000 162.8929 0.00000 0.000000e+00 [3,] 0.0000 0.0000 39.09544 0.000000e+00 [4,] 0.0000 0.0000 0.00000 -1.012719e-14 [,1] [,2] [,3] [,4] [1,] 15.29744 0.00000 0.000000 0 [2,] 0.00000 12.76295 0.000000 0 [3,] 0.00000 0.00000 6.252635 0 [4,] 0.00000 0.00000 0.000000 NaN
Warning message: In sqrt(D) : NaNs produced 31 / 32
Things you already know
Spectral decomposition
Positive definite matrices
Square root matrices
Page 40
R
Copyright Information
This slide show was prepared by Jerry Brunner, Department of Statistical Sciences, University of Toronto. It is licensed under a Creative Commons Attribution - ShareAlike 3.0 Unported License. Use any part of it as you like and share the result freely. The LATEX source code is available from the course website: http://www.utstat.toronto.edu/∼ brunner/oldclass/302f13
32 / 32
