Finding corners
CS4495/6495 Introduction to Computer Vision 4A-L2 Finding corners
Feature points
Characteristics of good features
Repeatability/Precision • The same feature can be found in several images
despite geometric and photometric transformations
Characteristics of good features
Saliency/Matchability • Each feature has a distinctive description
Characteristics of good features
Compactness and efficiency • Many fewer features than image pixels
Characteristics of good features
Locality • A feature occupies a relatively small area of the
image; robust to clutter and occlusion
Corner Detection: Basic Idea
“flat” region: no change in all directions
“edge”: no change along the edge direction
“corner”: significant change in all directions with small shift Source: A. Efros
Finding Corners • Key property: in the region around a corner, image
gradient has two or more dominant directions
Finding Corners C. Harris and M. Stephens. "A Combined Corner and Edge Detector,” Proceedings of the 4th Alvey Vision Conference: 1988
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
Window function
Shifted intensity
Window function w(x,y) =
Intensity
or
1 in window, 0 outside
Gaussian Source: R. Szeliski
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
x,y
I(x, y)
E(u, v)
E(3,2) E(0,0)
2
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
We want to find out how this function behaves for small shifts (u,v near 0,0)
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0) (local quadratic approximation for small u,v):
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
F ( x ) F ( 0 ) x·
d F (0 ) dx
1 2
2
x · 2
d F (0 ) dx
2
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
F ( x ) F ( 0 ) x·
d F (0 ) dx
1 2
2
x · 2
d F (0 ) dx
2
E u (0, 0 ) E uu (0 , 0 ) 1 E (u , v ) E (0 , 0 ) [u v ] [u v ] 2 E v (0, 0 ) E uv (0 , 0 )
E uv (0 , 0 ) E vv ( 0 , 0 )
u v
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
E (0, 0 ) E ( 0 , 0 ) uv
vv
u v
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
Need these derivatives…
E (0, 0 ) E ( 0 , 0 ) uv
vv
u v
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E u (u , v ) 2 w ( x , y ) I ( x u , y v ) I ( x , y ) I x ( x u , y v ) x,y
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E uu (u , v ) 2 w ( x , y )I x ( x u , y v ) I x ( x u , y v ) x,y
2 w ( x , y ) I ( x u , y v ) I ( x , y ) I xx ( x u , y v ) x,y
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E uv (u , v ) 2 w ( x , y )I y ( x u , y v ) I x ( x u , y v ) x,y
2 w ( x , y ) I ( x u , y v ) I ( x , y ) I xy ( x u , y v ) x,y
Second-order Taylor expansion of E(u,v) about (0,0): E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
E (0, 0 ) E ( 0 , 0 ) uv
vv
E (u , v ) 2 w ( x , y ) I ( x u , y v ) I ( x , y ) I ( x u , y v ) u
x
x,y
E (u , v ) 2 w ( x , y )I ( x u , y v ) I ( x u , y v ) uu
x
x
x,y
2 w( x, y) I (x u , y v) I (x, y) I (x u , y v) xx
x,y
E (u , v ) 2 w ( x , y )I ( x u , y v ) I ( x u , y v ) uv
y
x
x,y
2 w ( x, y ) I ( x u , y v) I ( x, y ) I ( x u , y v) xy
x,y
u v
Evaluate E and its derivatives at (0,0): =0 E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
E (0, 0 ) E ( 0 , 0 ) uv
vv
E (0, 0 ) 2 w ( x, y ) I ( x, y ) I ( x, y ) I ( x, y ) u
=0
x,y
x
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uu
x
x
x,y
=0
2 w( x, y) I (x, y) I (x, y) I (x, y) xx
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uv
y
x,y
x
=0
2 w( x, y) I (x, y) I (x, y) I (x, y) xy
x,y
u v
Second-order Taylor expansion of E(u,v) about (0,0): 1 E (0, 0 ) E (0, 0 ) E (u , v ) E (0 , 0 ) [u v ] [u v ] E ( 0 , 0 ) E ( 0 , 0 ) 2
E (0, 0 ) 0
u
uu
v
uv
E (0, 0 ) u E ( 0 , 0 ) v uv
vv
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uu
x
x
x,y
E (0, 0 ) 0 u
E (0, 0 ) 0 v
E
vv
( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) y
y
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uv
x
x,y
y
Second-order Taylor expansion of E(u,v) about (0,0): w(x, y)I (x, y) E (u , v ) [u v ] w(x, y)I (x, y)I (x, y) 2
x
x,y
x
y
x,y
E (0, 0 ) 0
w(x, y)I (x, y)I (x, y) u v w(x, y)I (x, y) x
y
x,y
2
y
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uu
x
x
x,y
E (0, 0 ) 0 u
E (0, 0 ) 0 v
E
vv
( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) y
y
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uv
x
x,y
y
Corner Detection: Mathematics The quadratic approximation simplifies to
E (u , v ) [u v ] M
u v
where M is a second moment matrix computed from image derivatives: Ix w(x, y) I x I y 2
M
x,y
IxIy 2 Iy
The second moment matrix M: Ix w(x, y) I x I y 2
M
x,y
IxIy 2 Iy
Can be written (without the weight): I I x x M I x I y
IxIy IyIy
Ix I y
Ix
Iy
Each product is a rank 1 2x2
I ( I )
T
Interpreting the second moment matrix The surface E(u,v) is locally approximated by a quadratic form.
E (u , v ) [u v ] M
u v
I M w(x, y) I I
I I I
2
x
x,y
x
x
y
2
y
y
Interpreting the second moment matrix Consider a constant “slice” of E(u, v): I x u 2 I x I yu v I y v 2
2
2
2
k
This is the equation of an ellipse.
[u v ] M
u const v
Interpreting the second moment matrix First, consider the axis-aligned case where gradients are either horizontal or vertical
I M w(x, y) I I
2
x
x,y
x
I I I 0 x
y
1
2
y
y
If either λ is close to 0, then this is not a corner, so look for locations where both are large.
0 2
Interpreting the second moment matrix Diagonalization of M: M R
1
0
1
0 R 2
The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R
Interpreting the second moment matrix direction of the fastest change (max)-1/2 (min)-1/2
direction of the slowest change
Interpreting the eigenvalues Classification of image points using eigenvalues of M:
2
1 and 2 are small; E is almost constant in all directions
“Edge” 2 >> 1
“Flat” region
“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions “Edge” 1 >> 2
1
Harris corner response function R d e t( M ) tr a c e ( M ) ( ) 2
1
α: constant (0.04 to 0.06)
“Edge” R0
|R| small “Flat” region
“Edge” R large 1, large 2
Harris Detector: Algorithm 1. 2. 3. 4.
5.
Compute Gaussian derivatives at each pixel Compute second moment matrix M in a Gaussian window around each pixel Compute corner response function R Threshold R Find local maxima of response function (nonmaximum suppression) C. Harris and M. Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
Harris Detector: Workflow
Harris Detector: Workflow Compute corner response R
Harris Detector: Workflow Find points with large corner response: R>threshold
Harris Detector: Workflow Take only the points of local maxima of R
Harris Detector: Workflow
Other corners: Shi-Tomasi ’94: “Cornerness” = min (λ1, λ2) Find local maximums cvGoodFeaturesToTrack(...) Reportedly better for region undergoing affine deformations
Other corners: • Brown, M., Szeliski, R., and Winder, S. (2005):
det M tr M • There are others…
0 1 0 1
Feature points
Characteristics of good features
Repeatability/Precision • The same feature can be found in several images
despite geometric and photometric transformations
Characteristics of good features
Saliency/Matchability • Each feature has a distinctive description
Characteristics of good features
Compactness and efficiency • Many fewer features than image pixels
Characteristics of good features
Locality • A feature occupies a relatively small area of the
image; robust to clutter and occlusion
Corner Detection: Basic Idea
“flat” region: no change in all directions
“edge”: no change along the edge direction
“corner”: significant change in all directions with small shift Source: A. Efros
Finding Corners • Key property: in the region around a corner, image
gradient has two or more dominant directions
Finding Corners C. Harris and M. Stephens. "A Combined Corner and Edge Detector,” Proceedings of the 4th Alvey Vision Conference: 1988
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
Window function
Shifted intensity
Window function w(x,y) =
Intensity
or
1 in window, 0 outside
Gaussian Source: R. Szeliski
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
x,y
I(x, y)
E(u, v)
E(3,2) E(0,0)
2
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
We want to find out how this function behaves for small shifts (u,v near 0,0)
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0) (local quadratic approximation for small u,v):
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
F ( x ) F ( 0 ) x·
d F (0 ) dx
1 2
2
x · 2
d F (0 ) dx
2
Corner Detection: Mathematics Change in appearance for the shift [u,v]: E (u , v )
w(x, y) I (x u , y v) I (x, y)
2
x,y
F ( x ) F ( 0 ) x·
d F (0 ) dx
1 2
2
x · 2
d F (0 ) dx
2
E u (0, 0 ) E uu (0 , 0 ) 1 E (u , v ) E (0 , 0 ) [u v ] [u v ] 2 E v (0, 0 ) E uv (0 , 0 )
E uv (0 , 0 ) E vv ( 0 , 0 )
u v
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
E (0, 0 ) E ( 0 , 0 ) uv
vv
u v
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
Need these derivatives…
E (0, 0 ) E ( 0 , 0 ) uv
vv
u v
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E u (u , v ) 2 w ( x , y ) I ( x u , y v ) I ( x , y ) I x ( x u , y v ) x,y
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E uu (u , v ) 2 w ( x , y )I x ( x u , y v ) I x ( x u , y v ) x,y
2 w ( x , y ) I ( x u , y v ) I ( x , y ) I xx ( x u , y v ) x,y
E (u , v )
w( x, y) I ( x u , y v) I ( x, y )
2
x,y
Second-order Taylor expansion of E(u,v) about (0,0): E uv (u , v ) 2 w ( x , y )I y ( x u , y v ) I x ( x u , y v ) x,y
2 w ( x , y ) I ( x u , y v ) I ( x , y ) I xy ( x u , y v ) x,y
Second-order Taylor expansion of E(u,v) about (0,0): E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
E (0, 0 ) E ( 0 , 0 ) uv
vv
E (u , v ) 2 w ( x , y ) I ( x u , y v ) I ( x , y ) I ( x u , y v ) u
x
x,y
E (u , v ) 2 w ( x , y )I ( x u , y v ) I ( x u , y v ) uu
x
x
x,y
2 w( x, y) I (x u , y v) I (x, y) I (x u , y v) xx
x,y
E (u , v ) 2 w ( x , y )I ( x u , y v ) I ( x u , y v ) uv
y
x
x,y
2 w ( x, y ) I ( x u , y v) I ( x, y ) I ( x u , y v) xy
x,y
u v
Evaluate E and its derivatives at (0,0): =0 E (u , v ) E (0 , 0 ) [u v ]
E (0, 0 ) 1 E (0, 0 ) [u v ] E ( 0 , 0 ) 2 E ( 0 , 0 ) u
uu
v
uv
E (0, 0 ) E ( 0 , 0 ) uv
vv
E (0, 0 ) 2 w ( x, y ) I ( x, y ) I ( x, y ) I ( x, y ) u
=0
x,y
x
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uu
x
x
x,y
=0
2 w( x, y) I (x, y) I (x, y) I (x, y) xx
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uv
y
x,y
x
=0
2 w( x, y) I (x, y) I (x, y) I (x, y) xy
x,y
u v
Second-order Taylor expansion of E(u,v) about (0,0): 1 E (0, 0 ) E (0, 0 ) E (u , v ) E (0 , 0 ) [u v ] [u v ] E ( 0 , 0 ) E ( 0 , 0 ) 2
E (0, 0 ) 0
u
uu
v
uv
E (0, 0 ) u E ( 0 , 0 ) v uv
vv
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uu
x
x
x,y
E (0, 0 ) 0 u
E (0, 0 ) 0 v
E
vv
( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) y
y
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uv
x
x,y
y
Second-order Taylor expansion of E(u,v) about (0,0): w(x, y)I (x, y) E (u , v ) [u v ] w(x, y)I (x, y)I (x, y) 2
x
x,y
x
y
x,y
E (0, 0 ) 0
w(x, y)I (x, y)I (x, y) u v w(x, y)I (x, y) x
y
x,y
2
y
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uu
x
x
x,y
E (0, 0 ) 0 u
E (0, 0 ) 0 v
E
vv
( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) y
y
x,y
E ( 0 , 0 ) 2 w ( x , y )I ( x , y ) I ( x , y ) uv
x
x,y
y
Corner Detection: Mathematics The quadratic approximation simplifies to
E (u , v ) [u v ] M
u v
where M is a second moment matrix computed from image derivatives: Ix w(x, y) I x I y 2
M
x,y
IxIy 2 Iy
The second moment matrix M: Ix w(x, y) I x I y 2
M
x,y
IxIy 2 Iy
Can be written (without the weight): I I x x M I x I y
IxIy IyIy
Ix I y
Ix
Iy
Each product is a rank 1 2x2
I ( I )
T
Interpreting the second moment matrix The surface E(u,v) is locally approximated by a quadratic form.
E (u , v ) [u v ] M
u v
I M w(x, y) I I
I I I
2
x
x,y
x
x
y
2
y
y
Interpreting the second moment matrix Consider a constant “slice” of E(u, v): I x u 2 I x I yu v I y v 2
2
2
2
k
This is the equation of an ellipse.
[u v ] M
u const v
Interpreting the second moment matrix First, consider the axis-aligned case where gradients are either horizontal or vertical
I M w(x, y) I I
2
x
x,y
x
I I I 0 x
y
1
2
y
y
If either λ is close to 0, then this is not a corner, so look for locations where both are large.
0 2
Interpreting the second moment matrix Diagonalization of M: M R
1
0
1
0 R 2
The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R
Interpreting the second moment matrix direction of the fastest change (max)-1/2 (min)-1/2
direction of the slowest change
Interpreting the eigenvalues Classification of image points using eigenvalues of M:
2
1 and 2 are small; E is almost constant in all directions
“Edge” 2 >> 1
“Flat” region
“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions “Edge” 1 >> 2
1
Harris corner response function R d e t( M ) tr a c e ( M ) ( ) 2
1
α: constant (0.04 to 0.06)
“Edge” R0
|R| small “Flat” region
“Edge” R large 1, large 2
Harris Detector: Algorithm 1. 2. 3. 4.
5.
Compute Gaussian derivatives at each pixel Compute second moment matrix M in a Gaussian window around each pixel Compute corner response function R Threshold R Find local maxima of response function (nonmaximum suppression) C. Harris and M. Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
Harris Detector: Workflow
Harris Detector: Workflow Compute corner response R
Harris Detector: Workflow Find points with large corner response: R>threshold
Harris Detector: Workflow Take only the points of local maxima of R
Harris Detector: Workflow
Other corners: Shi-Tomasi ’94: “Cornerness” = min (λ1, λ2) Find local maximums cvGoodFeaturesToTrack(...) Reportedly better for region undergoing affine deformations
Other corners: • Brown, M., Szeliski, R., and Winder, S. (2005):
det M tr M • There are others…
0 1 0 1
