Combining Scene and Auto-calibration Constraints
David Liebowitz and Andrew Zisserman Department of Engineering Science University of Oxford Oxford OX1 3PJ, UK dl,az @robots.ox.ac.uk
We present a simple approach to combining scene and auto-calibration constraints for the calibration of cameras from single views and stereo pairs. Calibration constraints are provided by imaged scene structure, such as vanishing points of orthogonal directions, or rectified planes. In addition, constraints are available from the nature of the cameras and the motion between views. We formulate these constraints in terms of the geometry of the imaged absolute conic and its relationship to pole-polar pairs and the imaged circular points of planes. Three significant advantages result: first, constraints from scene features, camera characteristics and auto-calibration constraints provide linear equations in the elements of the image of the absolute conic. This means that constraints may easily be combined, and their solution is straightforward. Second, the degeneracies that occur when constraints are not independent may be easily identified. Lastly, the constraints from scene planes and image planes may be treated uniformly. Examples of various cases of constraint combination and degeneracy as well as computational techniques are presented.
1 Introduction Auto- (or self-) calibration is the computation of camera internal calibration and/or metric properties of the scene from a set of uncalibrated images. The original autocalibration method based on Kruppa’s equations [16] was restricted to cameras with fixed internal parameters, and early work in this area maintained this restriction [1, 9, 10, 18]. However, with a sufficiently large number of views auto-calibration has now been demonstrated for cases where several of the internal parameters, for example the focal length and principal point, are allowed to vary [17, 23]. More classically, cameras may be calibrated from scene points with known coordinates [21] or from sets of vanishing points corresponding to directions which are orthogonal
This work was supported by the EU Esprit Project IMPROOFS. We
are grateful for discussions with Stefan Carlsson.
in the scene [12]. In particular, Caprile and Torre [2] described a method to calibrate a camera with known aspect ratio and skew from a single view of three vanishing points corresponding to orthogonal directions. We present here a method to combine image, scene and auto-calibration constraints for calibration of single views and multiple views. The method is formulated in terms of the geometry of the imaged absolute conic and the circular points of planes. There are three significant advantages of this formulation: first, it will be seen that both scene and auto-calibration constraints are linear equations. This means that constraints may easily be combined, and their solution is straightforward. Second, degeneracies, where there are insufficient independent constraints to determine a unique calibration, may be identified by simple geometric reasoning and the family of solutions parametrized. The method of Caprile and Torre [2], for example, is degenerate if one or more of the vanishing points is at infinity in the image. Third, the constraints from scene planes and multiple image planes are treated uniformly. This is a very pragmatic approach for the case of a small number of views where auto-calibration based on motion alone is likely to be ill-conditioned. It is well suited to the case of building architectural models from a small number of views [4] using only calibration information from the images themselves [3, 13]. The ubiquitous presence on buildings and other man-made structures of planes and lines in orthogonal directions means that such scenes often provide sufficient constraints to determine a unique calibration.
2 Notation and background Points on the image plane are represented by homogeneous 3 vectors with the Euclidean position on the plane. A line is also represented by a homogeneous column 3-vector, such that if a point lies on then . There are two points on the plane that are of special importance in what follows: the circular points are a complex conjugate point pair with the property that their position is invariant
, *) +' )" *!
"!$#
,
, #-.0/
%'&!(
to similarity transformations of the plane. In a metric coordinate frame on the plane the circular points’ position is , and it may be verified that the position is unchanged by a plane similarity transformation. homogeConics are represented by a symmetric neous matrix such that a point on the conic satisfies [20]
124357698:;/ !$#
=
ÿ ÿ
ÿ ÿ with an image size of +Ç £=ú3*/5ì+c
ÿ
|
ÿ
WU ¬£U U 'ª V U Wó¬zV U U 'V W F¬ U ª W ¬FU W ¬zV ª W ¬zV , , where yù and y] are the vanishing lines of two planes and is the angle between them. The angled plane in the
D
D
D
3 3 ì7=¿Ç ì
ûFügþ@ý ÿ*ÿ
pixels. The example also has an additional plane aligned at an angle to the parallel frontal planes. Having calibrated the camera, the orientation of this plane relative to others in the scene may be computed from
ü
,y
on and is collinear with the circular points. The two circular points therefore only provide one independent constraint on and a one parameter family of (natural) cameras remain. Suppose is ideal, then the principal point lies on the line through the two finite vanishing points and . In figure 5, for example, where one vanishing point is ideal, the ‘orthocentre’ of the triangle for the Caprile and Torre construction may be chosen anywhere on the line joining the finite vanishing points. Figure 6 similarly has a near ideal vertical vanishing point. In this case, however, additional constraints are available since the frontal plane of the building may be rectified from its vanishing line together with the imaged circles (since circles intersect the line at infinity in the circular points). The imaged circular points of this plane lie on one of the sides of the self-polar triangle. This configuration provides five independent constraints. The image is of size ! pixels, and the computed camera obtained by solving a linear system containing all the constraints is
ÿ
% " # $ $
#"
#
ÿ
$ $
ÿ
The computed aspect ratio is 1.05 and computed skew angle between axes is 88.15 . This (non-natural camera) solution is a result of obtaining a least-squares solution for from over-determined noisy measurements. Alternatively, the natural camera constraint may be enforced and a leastsquares solution obtained for the remaining four homogeneous parameters of .
|
|
example is computed to be at 45.75 to the frontal planes. The veridical value is 45.0 .
(a)
(b)
Figure 5: Calibration with one of three orthogonal vanishing points ideal and natural camera constraints. (a) Parallel lines defining the vanishing points. (b) The vertical vanishing point is at infinity. The resultant camera is defined up to a 1 parameter ambiguity; the principal point lies on the line joining the finite vanishing points.
(a)
(b)
Figure 4: Calibration from a triad of finite orthogonal vanishing points and natural camera constraints. (a) Parallel line sets defining the three vanishing points. (b) The self-polar triangle and orthocentre.
A natural camera and two finite, of three (orthogonal), vanishing points: One vertex of the self-polar triangle lies
A natural camera and one finite, of three (orthogonal), vanishing points: Two vertices of the self-polar triangle . Again is determined up to a one parameter lie on family of conics. The two ideal vanishing points are the vanishing points for directions parallel to the image plane. The third orthogonal direction must therefore be orthogonal to the image plane, parallel to the optical axis of the camera. The vanishing point of this axis is the principal point, which is therefore given by the finite vanishing point. The one
,y
|
when additional information, such as that the cameras are natural, is introduced. Two natural cameras: Consider the first camera. The two parameter family must contain the circular points , from the metric image plane of the first camera, and also the imaged circular points of the second cam. The era which are transferred to the first camera via transferred circular points lie on , the vanishing line of the second camera image plane as seen by the first camera. If neither this vanishing line nor the line at infinity of the image plane intersect any of the vertices of the triangle of vanishing points, seven independent constraints on are available. If either line passes through one of the vertices (as in the one view cases) six constraints are available. When both and intersect a vertex, five constraints are available, i.e. the ambiguity is still resolved.
435:698:'/ ! #
y
, y
(a)
,y
,y
(b)
|
, y
Figure 6: Calibration with one of three orthogonal vanishing near ideal, natural camera constraints and a rectified plane. (a) The vertical vanishing point is near ideal in this case, but the front plane may be rectified from the circles on the plane imaged as ellipses. (b) The rectified front plane.
parameter family of natural cameras is parametrized by the focal length. Additional orthogonal vanishing points: In general, an additional pair of orthogonal vanishing points adds a conjugacy constraint. If, however, one of the additional vanishing points is a vertex of the self-polar triangle, the other lies on a polar of the triangle and the conjugacy is already satisfied, so no further independent constraints are added. Such a vanishing point pair may be found in figure 4, defined by the lines on the angled plane, and figure 5, by the bounding lines of the roof.
Î
Ï'&
(
ÖÑÓ
ÓXÛÔ*) Û,+ ÚúÛØ
ÓX×Ú
We consider the case of two views with non-coincident camera centres for which the infinite homography mapping is known. This mapping can be computed from the corresponding of four or more vanishing points. However, we will focus here on the case that is computed from the fundamental matrix together with the correspondence of the vanishing points of the orthogonal triad. This computation is described in section 5. In the absence of any other information each camera calibration has five unknown degrees of freedom, and this is reduced to a two parameter family by using the orthogonal triad independently in each view as described in section 4.1. The family in one view can be mapped onto the family in the other by (5), but no additional constraints are gained by this. We will now examine how this ambiguity is reduced
y
y
(a)
(b)
Figure 7: A stereo pair imaged with the vertical vanishing point vertex of the self-polar triangle ideal, and rotation about the vertical axis.
The last case is common, for example when a building is imaged in a pair of views with the scene vertical axis parallel to the image plane and the rotation between views is about this axis. The vertical vanishing point is ideal, the line at infinity of the first image plane and the vanishing line of the second image plane both intersect the ideal vertex, as in figure 7. The cameras are computed for this image pair using the fact that both cameras are natural. Image sizes are - pixels and the computed cameras are
Ç
3`/5ì c =
û U ügýþ ÿ " ÿ
# # / . $ "% $
"
# 01 2# "
û V ügýþ@ÿ+ÿ
ÿ ÿ
ÿ
ÿ +ÿ
ÿ ÿ
Fixed camera: A camera with the same parameters in both views introduces the four constraints of (9). Added to the three constraints from the self-polar triangle, the general case provides seven constraints on . In the case that coincides with a vertex of the triangle of orthogonal vanishing points, the conjugacy and polarity are already satisfied and
|
the circular points lie on a polar, so only a single constraint is added, making four constraints on .
|
Vanishing points
Cameras
3 finite 1 ideal 2 ideal 3 finite
1 natural 1 natural 1 natural 2 natural
5 4 4 7
2 natural
6
2 natural
6
2 natural
5
, y # D ê4G 3 / 3 finite , y # D ê G/ 1 or 2 ideal , y # D ê G 3 / 1, or 2 ideal y # D ê G/ , # D ê A 3 / ,ó # D ê A/ , ,Ùê and ª
Independent constraints
2 fixed
7
2 fixed
4
T
Table 1: Constraints on for various combinations of cameras and motion given the vanishing points 65 of three orthogonal directions in the scene. The first three cases describe a single view of the scene where additional constraints are obtained from the knowledge that the camera is natural. Where two views are available, the vanishing line of the second view image plane seen in the first view is 87 . In the case of a fixed camera, 89 is the fixed line and 5 a side of the triangle of orthogonal vanishing points of
« ¬
W
W¬
W
5 Implementation The calibration methods we have described rely heavily on the computation of vanishing points. The points are usually computed from imaged parallel line segments, but due to ‘noise’ a set of such segments will generally not intersect in a unique point. Often the vanishing point is then computed by finding the closest point to all the measured lines[2]. However, this is not optimal. The maximum likelihood estimate (MLE) of the vanishing point and line segments is found by computing a set of lines that do intersect in a single point, and which minimise the sum of squared orthogonal distances from the endpoints of the measured line segments [14], as shown in figure 8. The minimization is computed using the Levenberg-Marquardt algorithm [19]. When dealing with two views, vanishing point estimation is important both in defining consistent orthogonal triads in each view and in computing the infinite homography. For each vanishing point corresponding over two views, the MLE is obtained by varying a point : in 3-space which
D D
projects to the vanishing points in the first and second views respectively [7]. The position of : is determined by minimizing a cost function of the sum of squared orthogonal distances from the end points of the measured (imaged parallel) lines to lines through in the first view, and through in the second. See figure 8. Three such 3-space points determine the plane at infinity , and thence .
D
D
x{y
y
A@A
V c