Projective Geometry and Camera Models
01/20/11
Projective Geometry and Camera Models Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem
Administrative Stuff • Office hours
– Derek: Wed 4-5pm + drop by – Ian: Mon 3-4pm, Thurs 3:30-4:30pm
• HW 1: out Monday
– Prob1: Geometry, today and Tues – Prob2: Lighting, next Thurs – Prob3: Filters, following week
• Next Thurs: I’m out, David Forsyth will cover
Last class: intro • Overview of vision, examples of state of art • Logistics
Next two classes: Single-view Geometry How tall is this woman? How high is the camera? What is the camera rotation? What is the focal length of the camera?
Which ball is closer?
Today’s class Mapping between image and world coordinates – Pinhole camera model – Projective geometry • Vanishing points and lines
– Projection matrix
Image formation
Let’s design a camera
– Idea 1: put a piece of film in front of an object – Do we get a reasonable image?
Slide source: Seitz
Pinhole camera
Idea 2: add a barrier to block off most of the rays – This reduces blurring – The opening known as the aperture Slide source: Seitz
Pinhole camera f
c
f = focal length c = center of the camera Figure from Forsyth
Camera obscura: the pre-camera • First idea: Mo-Ti, China (470BC to 390BC) • First built: Alhacen, Iraq/Egypt (965 to 1039AD)
Illustration of Camera Obscura
Freestanding camera obscura at UNC Chapel Hill Photo by Seth Ilys
Camera Obscura used for Tracing
Lens Based Camera Obscura, 1568
First Photograph Oldest surviving photograph – Took 8 hours on pewter plate
Joseph Niepce, 1826
Photograph of the first photograph
Stored at UT Austin
Niepce later teamed up with Daguerre, who eventually created Daguerrotypes
Dimensionality Reduction Machine (3D to 2D) 3D world
2D image
Point of observation
Figures © Stephen E. Palmer, 2002
Projection can be tricky…
Slide source: Seitz
Projection can be tricky…
Slide source: Seitz
Projective Geometry What is lost? • Length
Who is taller?
Which is closer?
Length is not preserved
A’ C’ B’
Figure by David Forsyth
Projective Geometry What is lost? • Length • Angles Parallel?
Perpendicular?
Projective Geometry What is preserved? • Straight lines are still straight
Vanishing points and lines Parallel lines in the world intersect in the image at a “vanishing point”
Vanishing points and lines
Vanishing Point Vanishing Line
o
Vanishing Point
o
Vanishing points and lines Vertical vanishing point (at infinity) Vanishing line
Vanishing point
Slide from Efros, Photo from Criminisi
Vanishing point
Vanishing points and lines
Photo from online Tate collection
Note on estimating vanishing points
Use multiple lines for better accuracy … but lines will not intersect at exactly the same point in practice One solution: take mean of intersecting pairs … bad idea! Instead, minimize angular differences
Vanishing objects
Projection: world coordinatesimage coordinates
.
Optical Center (u0, v0)
.
.
v
u
u p= v
f
. Camera Center (tx, ty, tz)
Z
Y
X P = Y Z
Homogeneous coordinates Conversion Converting to homogeneous coordinates
homogeneous image coordinates
homogeneous scene coordinates
Converting from homogeneous coordinates
Homogeneous coordinates Invariant to scaling x kx kx x kw w k y = ky ⇒ ky = y w kw kw w Homogeneous Coordinates
Cartesian Coordinates
Point in Cartesian is ray in Homogeneous
Basic geometry in homogeneous coordinates ai • Line equation: ax + by + c = 0 linei = bi ci • Append 1 to pixel coordinate to get ui homogeneous coordinate pi = vi 1
• Line given by cross product of two points
lineij = pi × p j
• Intersection of two lines given by cross product of the lines qij = linei × line j
Another problem solved by homogeneous coordinates
Intersection of parallel lines Cartesian: (Inf, Inf) Homogeneous: (1, 1, 0)
Cartesian: (Inf, Inf) Homogeneous: (1, 2, 0)
Projection matrix
Slide Credit: Saverese
R,T
jw kw Ow iw
x = K[R t ] X
x: Image Coordinates: (u,v,1) K: Intrinsic Matrix (3x3) R: Rotation (3x3) t: Translation (3x1) X: World Coordinates: (X,Y,Z,1)
Interlude: when have I used this stuff?
When have I used this stuff? Object Recognition (CVPR 2006)
When have I used this stuff? Single-view reconstruction (SIGGRAPH 2005)
When have I used this stuff? Getting spatial layout in indoor scenes (ICCV 2009)
When have I used this stuff? Inserting photographed objects into images (SIGGRAPH 2007)
Original
Created
When have I used this stuff? Inserting synthetic objects into images
Projection matrix
Intrinsic Assumptions Extrinsic Assumptions • No rotation • Unit aspect ratio • Optical center at (0,0) • No skew
x = K [I 0] X Slide Credit: Saverese
• Camera at (0,0,0) K
u f wv = 0 1 0
0 f 0
x 0 0 y 0 0 z 1 0 1
Remove assumption: known optical center Intrinsic Assumptions Extrinsic Assumptions • No rotation • Unit aspect ratio • No skew
x = K [I 0] X
• Camera at (0,0,0)
u f wv = 0 1 0
0 f
u0
0
1
v0
x 0 y 0 z 0 1
Remove assumption: square pixels Intrinsic Assumptions Extrinsic Assumptions • No skew
x = K [I 0] X
• No rotation • Camera at (0,0,0)
u α wv = 0 1 0
β
u0 v0
0
1
0
x 0 y 0 z 0 1
Remove assumption: non-skewed pixels Intrinsic Assumptions Extrinsic Assumptions • No rotation • Camera at (0,0,0)
x = K [I 0] X
u α wv = 0 1 0
s
u0
β
v0
0
1
Note: different books use different notation for parameters
x 0 y 0 z 0 1
Oriented and Translated Camera R jw
t
kw Ow iw
Allow camera translation Intrinsic Assumptions Extrinsic Assumptions • No rotation
x = K [I t ] X
u α wv = 0 1 0
0
β 0
x u0 1 0 0 t x y v0 0 1 0 t y z 1 0 0 1 t z 1
Slide Credit: Saverese
3D Rotation of Points
Rotation around the coordinate axes, counter-clockwise: 0 1 Rx (α ) = 0 cosα 0 sin α
p’
γ y
z
p
− sin α cosα 0 sin β 1 0 0 cos β
cos β R y ( β ) = 0 − sin β cos γ − sin γ Rz (γ ) = sin γ cos γ 0 0
0
0 0 1
Allow camera rotation x = K [R t ] X u α wv = 0 1 0
s
β 0
u0 r11 v0 r21 1 r31
r12 r22 r32
r13 r23 r33
x tx y ty z t z 1
Degrees of freedom x = K [R t ] X 5
u α wv = 0 1 0
6
s
β 0
u0 r11 v0 r21 1 r31
r12 r22 r32
r13 r23 r33
x tx y ty z t z 1
Vanishing Point = Projection from Infinity x xR x y p = K[R t ] ⇒ p = KR y ⇒ p = K y R z z R z 0 u f wv = 0 1 0
0 f 0
u0 xR v0 y R ⇒ 1 z R
fxR u= + u0 zR fyR v= + v0 zR
Orthographic Projection • Special case of perspective projection – Distance from the COP to the image plane is infinite Image
– Also called “parallel projection” – What’s the projection matrix?
World
x u 1 0 0 0 y w v = 0 1 0 0 z 1 0 0 0 1 1 Slide by Steve Seitz
Scaled Orthographic Projection • Special case of perspective projection – Object dimensions are small compared to distance to camera Image
World
u f – Also called “weak perspective” w v = 0 – What’s the projection matrix? 1 0
0 f 0
x 0 0 y 0 0 z 0 s 1 Slide by Steve Seitz
Suppose we have two 3D cubes on the ground facing the viewer, one near, one far. 1. What would they look like in perspective? 2. What would they look like in weak perspective?
Photo credit: GazetteLive.co.uk
Beyond Pinholes: Radial Distortion
Corrected Barrel Distortion Image from Martin Habbecke
Things to remember Vanishing line
• Vanishing points and vanishing lines • Pinhole camera model and camera projection matrix • Homogeneous coordinates
Vertical vanishing point (at infinity)
Vanishing point
Vanishing point
x = K [R t ] X
Next class • Applications of camera model and projective geometry – Recovering the camera intrinsic and extrinsic parameters from an image – Recovering size in the world – Projecting from one plane to another
Questions
What about focus, aperture, DOF, FOV, etc?
Adding a lens
“circle of confusion”
• A lens focuses light onto the film – There is a specific distance at which objects are “in focus” • other points project to a “circle of confusion” in the image
– Changing the shape of the lens changes this distance
Focal length, aperture, depth of field F focal point optical center (Center Of Projection)
A lens focuses parallel rays onto a single focal point – focal point at a distance f beyond the plane of the lens – Aperture of diameter D restricts the range of rays
Slide source: Seitz
The eye
• The human eye is a camera
– Iris - colored annulus with radial muscles – Pupil - the hole (aperture) whose size is controlled by the iris – What’s the “film”?
– photoreceptor cells (rods and cones) in the retina
Depth of field
Slide source: Seitz
f / 5.6
f / 32
Changing the aperture size or focal length affects depth of field Flower images from Wikipedia
http://en.wikipedia.org/wiki/Depth_of_field
Varying the aperture
Large aperture = small DOF
Slide from Efros
Small aperture = large DOF
Shrinking the aperture
• Why not make the aperture as small as possible? – Less light gets through – Diffraction effects Slide by Steve Seitz
Shrinking the aperture
Slide by Steve Seitz
Relation between field of view and focal length Field of view (angle width)
d fov = tan 2f −1
Film/Sensor Width
Focal length
Dolly Zoom or “Vertigo Effect” http://www.youtube.com/watch?v=Y48R6-iIYHs How is this done?
Zoom in while moving away
http://en.wikipedia.org/wiki/Focal_length
Projective Geometry and Camera Models Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem
Administrative Stuff • Office hours
– Derek: Wed 4-5pm + drop by – Ian: Mon 3-4pm, Thurs 3:30-4:30pm
• HW 1: out Monday
– Prob1: Geometry, today and Tues – Prob2: Lighting, next Thurs – Prob3: Filters, following week
• Next Thurs: I’m out, David Forsyth will cover
Last class: intro • Overview of vision, examples of state of art • Logistics
Next two classes: Single-view Geometry How tall is this woman? How high is the camera? What is the camera rotation? What is the focal length of the camera?
Which ball is closer?
Today’s class Mapping between image and world coordinates – Pinhole camera model – Projective geometry • Vanishing points and lines
– Projection matrix
Image formation
Let’s design a camera
– Idea 1: put a piece of film in front of an object – Do we get a reasonable image?
Slide source: Seitz
Pinhole camera
Idea 2: add a barrier to block off most of the rays – This reduces blurring – The opening known as the aperture Slide source: Seitz
Pinhole camera f
c
f = focal length c = center of the camera Figure from Forsyth
Camera obscura: the pre-camera • First idea: Mo-Ti, China (470BC to 390BC) • First built: Alhacen, Iraq/Egypt (965 to 1039AD)
Illustration of Camera Obscura
Freestanding camera obscura at UNC Chapel Hill Photo by Seth Ilys
Camera Obscura used for Tracing
Lens Based Camera Obscura, 1568
First Photograph Oldest surviving photograph – Took 8 hours on pewter plate
Joseph Niepce, 1826
Photograph of the first photograph
Stored at UT Austin
Niepce later teamed up with Daguerre, who eventually created Daguerrotypes
Dimensionality Reduction Machine (3D to 2D) 3D world
2D image
Point of observation
Figures © Stephen E. Palmer, 2002
Projection can be tricky…
Slide source: Seitz
Projection can be tricky…
Slide source: Seitz
Projective Geometry What is lost? • Length
Who is taller?
Which is closer?
Length is not preserved
A’ C’ B’
Figure by David Forsyth
Projective Geometry What is lost? • Length • Angles Parallel?
Perpendicular?
Projective Geometry What is preserved? • Straight lines are still straight
Vanishing points and lines Parallel lines in the world intersect in the image at a “vanishing point”
Vanishing points and lines
Vanishing Point Vanishing Line
o
Vanishing Point
o
Vanishing points and lines Vertical vanishing point (at infinity) Vanishing line
Vanishing point
Slide from Efros, Photo from Criminisi
Vanishing point
Vanishing points and lines
Photo from online Tate collection
Note on estimating vanishing points
Use multiple lines for better accuracy … but lines will not intersect at exactly the same point in practice One solution: take mean of intersecting pairs … bad idea! Instead, minimize angular differences
Vanishing objects
Projection: world coordinatesimage coordinates
.
Optical Center (u0, v0)
.
.
v
u
u p= v
f
. Camera Center (tx, ty, tz)
Z
Y
X P = Y Z
Homogeneous coordinates Conversion Converting to homogeneous coordinates
homogeneous image coordinates
homogeneous scene coordinates
Converting from homogeneous coordinates
Homogeneous coordinates Invariant to scaling x kx kx x kw w k y = ky ⇒ ky = y w kw kw w Homogeneous Coordinates
Cartesian Coordinates
Point in Cartesian is ray in Homogeneous
Basic geometry in homogeneous coordinates ai • Line equation: ax + by + c = 0 linei = bi ci • Append 1 to pixel coordinate to get ui homogeneous coordinate pi = vi 1
• Line given by cross product of two points
lineij = pi × p j
• Intersection of two lines given by cross product of the lines qij = linei × line j
Another problem solved by homogeneous coordinates
Intersection of parallel lines Cartesian: (Inf, Inf) Homogeneous: (1, 1, 0)
Cartesian: (Inf, Inf) Homogeneous: (1, 2, 0)
Projection matrix
Slide Credit: Saverese
R,T
jw kw Ow iw
x = K[R t ] X
x: Image Coordinates: (u,v,1) K: Intrinsic Matrix (3x3) R: Rotation (3x3) t: Translation (3x1) X: World Coordinates: (X,Y,Z,1)
Interlude: when have I used this stuff?
When have I used this stuff? Object Recognition (CVPR 2006)
When have I used this stuff? Single-view reconstruction (SIGGRAPH 2005)
When have I used this stuff? Getting spatial layout in indoor scenes (ICCV 2009)
When have I used this stuff? Inserting photographed objects into images (SIGGRAPH 2007)
Original
Created
When have I used this stuff? Inserting synthetic objects into images
Projection matrix
Intrinsic Assumptions Extrinsic Assumptions • No rotation • Unit aspect ratio • Optical center at (0,0) • No skew
x = K [I 0] X Slide Credit: Saverese
• Camera at (0,0,0) K
u f wv = 0 1 0
0 f 0
x 0 0 y 0 0 z 1 0 1
Remove assumption: known optical center Intrinsic Assumptions Extrinsic Assumptions • No rotation • Unit aspect ratio • No skew
x = K [I 0] X
• Camera at (0,0,0)
u f wv = 0 1 0
0 f
u0
0
1
v0
x 0 y 0 z 0 1
Remove assumption: square pixels Intrinsic Assumptions Extrinsic Assumptions • No skew
x = K [I 0] X
• No rotation • Camera at (0,0,0)
u α wv = 0 1 0
β
u0 v0
0
1
0
x 0 y 0 z 0 1
Remove assumption: non-skewed pixels Intrinsic Assumptions Extrinsic Assumptions • No rotation • Camera at (0,0,0)
x = K [I 0] X
u α wv = 0 1 0
s
u0
β
v0
0
1
Note: different books use different notation for parameters
x 0 y 0 z 0 1
Oriented and Translated Camera R jw
t
kw Ow iw
Allow camera translation Intrinsic Assumptions Extrinsic Assumptions • No rotation
x = K [I t ] X
u α wv = 0 1 0
0
β 0
x u0 1 0 0 t x y v0 0 1 0 t y z 1 0 0 1 t z 1
Slide Credit: Saverese
3D Rotation of Points
Rotation around the coordinate axes, counter-clockwise: 0 1 Rx (α ) = 0 cosα 0 sin α
p’
γ y
z
p
− sin α cosα 0 sin β 1 0 0 cos β
cos β R y ( β ) = 0 − sin β cos γ − sin γ Rz (γ ) = sin γ cos γ 0 0
0
0 0 1
Allow camera rotation x = K [R t ] X u α wv = 0 1 0
s
β 0
u0 r11 v0 r21 1 r31
r12 r22 r32
r13 r23 r33
x tx y ty z t z 1
Degrees of freedom x = K [R t ] X 5
u α wv = 0 1 0
6
s
β 0
u0 r11 v0 r21 1 r31
r12 r22 r32
r13 r23 r33
x tx y ty z t z 1
Vanishing Point = Projection from Infinity x xR x y p = K[R t ] ⇒ p = KR y ⇒ p = K y R z z R z 0 u f wv = 0 1 0
0 f 0
u0 xR v0 y R ⇒ 1 z R
fxR u= + u0 zR fyR v= + v0 zR
Orthographic Projection • Special case of perspective projection – Distance from the COP to the image plane is infinite Image
– Also called “parallel projection” – What’s the projection matrix?
World
x u 1 0 0 0 y w v = 0 1 0 0 z 1 0 0 0 1 1 Slide by Steve Seitz
Scaled Orthographic Projection • Special case of perspective projection – Object dimensions are small compared to distance to camera Image
World
u f – Also called “weak perspective” w v = 0 – What’s the projection matrix? 1 0
0 f 0
x 0 0 y 0 0 z 0 s 1 Slide by Steve Seitz
Suppose we have two 3D cubes on the ground facing the viewer, one near, one far. 1. What would they look like in perspective? 2. What would they look like in weak perspective?
Photo credit: GazetteLive.co.uk
Beyond Pinholes: Radial Distortion
Corrected Barrel Distortion Image from Martin Habbecke
Things to remember Vanishing line
• Vanishing points and vanishing lines • Pinhole camera model and camera projection matrix • Homogeneous coordinates
Vertical vanishing point (at infinity)
Vanishing point
Vanishing point
x = K [R t ] X
Next class • Applications of camera model and projective geometry – Recovering the camera intrinsic and extrinsic parameters from an image – Recovering size in the world – Projecting from one plane to another
Questions
What about focus, aperture, DOF, FOV, etc?
Adding a lens
“circle of confusion”
• A lens focuses light onto the film – There is a specific distance at which objects are “in focus” • other points project to a “circle of confusion” in the image
– Changing the shape of the lens changes this distance
Focal length, aperture, depth of field F focal point optical center (Center Of Projection)
A lens focuses parallel rays onto a single focal point – focal point at a distance f beyond the plane of the lens – Aperture of diameter D restricts the range of rays
Slide source: Seitz
The eye
• The human eye is a camera
– Iris - colored annulus with radial muscles – Pupil - the hole (aperture) whose size is controlled by the iris – What’s the “film”?
– photoreceptor cells (rods and cones) in the retina
Depth of field
Slide source: Seitz
f / 5.6
f / 32
Changing the aperture size or focal length affects depth of field Flower images from Wikipedia
http://en.wikipedia.org/wiki/Depth_of_field
Varying the aperture
Large aperture = small DOF
Slide from Efros
Small aperture = large DOF
Shrinking the aperture
• Why not make the aperture as small as possible? – Less light gets through – Diffraction effects Slide by Steve Seitz
Shrinking the aperture
Slide by Steve Seitz
Relation between field of view and focal length Field of view (angle width)
d fov = tan 2f −1
Film/Sensor Width
Focal length
Dolly Zoom or “Vertigo Effect” http://www.youtube.com/watch?v=Y48R6-iIYHs How is this done?
Zoom in while moving away
http://en.wikipedia.org/wiki/Focal_length
