Vanishing Point Estimation by Spherical Gradient - Semantic Scholar
21st International Conference on Pattern Recognition (ICPR 2012) November 11-15, 2012. Tsukuba, Japan
Vanishing Point Estimation by Spherical Gradient Shigang Li and Hanchao Jia Graduate School of Engineering, Tottori University, Japan [email protected] points can be estimated by spherical gradient of edge points without extracting lines.
Abstract In this paper we propose a novel method of estimating vanishing point by spherical gradient. In contrast with the conventional methods in which vanishing point is estimated from lines, the proposed method does not necessarily extract lines, but employs the spherical gradient cues of edge points. Based on the observation that spherical gradient is aligned with the normal vector of the projection plane of space lines, the vanishing point is estimated directly from spherical gradient of edge points by the Hough Transform.
Π
G np
G l ng
Figure 1: A space line is projected as a great circle in
1. Introduction
spherical image. The normal vector
G np
of the
Π of a space line L is aligned with G the gradient vector ng of the edge points of the line l .
projection plane
Vanishing point is defined as the intersection of space parallel lines in image. To estimate vanishing points, the first step of the conventional methods is to extract lines from images. However, can we estimate vanishing points from image without extracting lines? Figure 1 shows a spherical camera which observes a space line L . In the spherical image, the space line L is projected as a part of a great circle l . Because the projection plane Π of the space line L is perpendicular to the spherical surface, its normal G G vector n p is aligned with the gradient vector ng of
In this paper we propose a novel method of estimating vanishing points based on the above observation. The feature of the proposed method, which distinguishes itself from other conventional ones, is that it does not necessarily extract lines from the image. Concretely, the Hough transform is used to estimate vanishing points in terms of the spherical gradient cues of edge points. The remainder of this paper is organized as follows. We introduce the related research in Section 2. In Section 3, the proposed method is presented. After showing the experimental results in Section 4, we will present our conclusions in Section 5.
the edge points of the line in spherical image. Because for a group of space parallel lines the normal vectors of their projection planes are co-planar, their vanishing point can be estimated as the vector perpendicular to the normal vetors. Because for a spherical image the G normal vector n p is aligned with the gradient vector
2. Related work
G ng of the edge points of the line, as shown in Figure 1,
Vanishing point is an important feature of manmade environments, and is widely used for scene analysis, camera calibration and camera pose estimation [1][2][3]. Because vanishing point is
the vanishing point of a group of parallel lines can be estimated directly from the gradient vector of the edge points of the lines. Thus, in spherical image vanishing
978-4-9906441-1-6 ©2012 IAPR
L
902
defined as the intersection of space parallel lines in image, line extraction from images is a necessary preprocessing for the conventional methods for vanishing point estimation. As recent studies, a J-Linkage algorithm is used to generate hypothetical classes of parallel lines, followed by EM to find the vanishing points [4]. In [5], an analytical method for computing the globally optimal estimates of orthogonal vanishing points in a “Manhattan world” with a calibrated camera is proposed. On the other hand, in omnidirectional image processing an intermediate spherical camera model is usually used to cope with the wide field of view [6]. A space line is projected as a great circle onto the sphere. A vanishing point is estimated as the intersection great circles. Nevertheless, the methods described in a perspective image can be simply extended to a spherical image. However, in a spherical image the gradient vector of the edge points of a line is aligned with the normal vector of the projection plane of the line, as described in Section 1. This observation makes it possible to estimate vanishing point directly from the gradient cues of edge points without line detection. In this paper we show the effectiveness of this approach. Note that because a perspective image can be transformed to a spherical image using the camera intrinsic parameters, the proposed method of is not limited to omnidirectional images. In this sense, estimating vanishing point from the gradient of edge points can be seen as the by-product of spherical image representation.
z
er eϕ
P
θ
r
eθ
O
y
ϕ P′
x
Figure 2: The curvilinear orthogonal coordinate system and the orthogonal coordinate system.
Using the camera intrinsic parameters, a pixel in an input image can be mapped onto a spherical image. The mapping relation is represented as follows.
θ = θ ( xi , y i ) ⎫ ⎬. ϕ = ϕ ( xi , y i ) ⎭
(2)
For the corresponding pixel between an input image and the spherical image, we have I ( xi , yi ) = I s (θ ( xi , yi ),ϕ ( xi , yi ) ) . (3) Using the differential chain rule,
∂I ∂I s ∂θ ∂I s + = ∂xi ∂θ ∂xi ∂ϕ ∂I ∂I s ∂θ ∂I s + = ∂y i ∂θ ∂yi ∂ϕ
3. Vanishing point estimation by spherical gradient
∂ϕ ⎫ ∂xi ⎪⎪ ∂ϕ ⎬ ⎪ ∂yi ⎪⎭
(5)
⎞ ⎛ ∂θ ⎟ ⎜ ⎟ = ⎜ ∂xi ⎟ ⎜ ∂θ ⎟ ⎜ ∂y ⎠ ⎝ i
∂ϕ ⎞⎛ ∂I s ⎞ ⎟⎜ ⎟ ∂xi ⎟⎜ ∂θ ⎟ . ∂ϕ ⎟⎜ ∂I s ⎟ ⎜ ⎟ ∂yi ⎟⎠⎝ ∂ϕ ⎠
(6)
⎞ ⎛ ∂θ ⎟ ⎜ ⎟ = ⎜ ∂xi ⎟ ⎜ ∂θ ⎟ ⎜ ⎠ ⎝ ∂yi
∂ϕ ⎞ ⎟ ∂xi ⎟ ∂ϕ ⎟ ∂yi ⎟⎠
That is,
In this section, we first give the algorithm of computing spherical gradient from an input image, and then present the method of estimating vanishing point by spherical gradient.
⎛ ∂I ⎜ ⎜ ∂xi ⎜ ∂I ⎜ ∂y ⎝ i
3.1. Spherical gradient computation
Thus,
Suppose an input image and its corresponding spherical image are I ( xi , y i ) and I s (θ ,ϕ ) ,
⎛ ∂I s ⎜ ⎜ ∂θ ⎜ ∂I s ⎜ ∂ϕ ⎝
respectively. At the curvilinear orthogonal coordinate system, as shown in Figure 2, the gradient of the spherical image on a unit sphere is computed as follows.
−1
⎛ ∂I ⎜ ⎜ ∂xi ⎜ ∂I ⎜ ∂y ⎝ i
⎞ ⎟ ⎟. ⎟ ⎟ ⎠
(7)
From equation (7), the derivatives of the spherical image can be computed from the derivatives of the input image. Then, the gradient of the spherical image at the curvilinear orthogonal coordinate system can be obtained in terms of equation (1). Finally, the gradient represented in equation (1) at the curvilinear
∂I (θ ,ϕ ) 1 ∂I s (θ ,ϕ ) ∇I s (θ , ϕ ) = s eθ + eϕ (1) ∂θ sin θ ∂ϕ
Therefore, we need to compute the derivatives of the spherical image in order to obtain the gradient.
903
G nv (θ v ,φv ) .
orthogonal coordinate system is transformed to threedimensional vector ∇I o ( x, y , z ) represented at the orthogonal coordinate system O-XYZ. Concretely, in this paper we use the Sobel operator to compute the derivatives of the input image.
4. Experiment At first, a simulation experiment is carried out to evaluate the proposed method quantitatively in comparison with the conventional line-based method. Then, the experimental results of the vanishing point estimation for real-world images are also presented to show the effectiveness of the proposed method.
3.2. Vanishing point estimation Suppose there are a group of parallel lines at space. Let the vector of the vanishing point in spherical G image be nv (θ v , φv ) . Because in a spherical image
4.1. Simulation experiment
the gradient vector of the edge points of a line is aligned with the normal vector of the projection plane of the line as mentioned in Section 1, given an edge point, PL , of the parallel lines, we have
Figure 3(a) shows a synthesized fisheye image of a stripe pattern with a hemispherical field of view. The spherical gradient of the spherical image is computed as described in subsection 3.1. Figure 3 (b) and (c) show the magnitude and direction of the computed gradient of the edge points, respectively; different colors indicate different magnitudes or directions of the computed spherical gradient. Note that in Figure 3(c) the edge points belonging to the same line have the same colors because the spherical gradient direction of these edge points is aligned with the normal vector of the projection plane of the line.
G G g (θ L ,φ L ) ⋅ nv (θ v ,φv ) = 0 , (8) G where g (θ L , φ L ) is the spherical gradient computed
in the orthogonal coordinate system of the spherical image for edge point PL . Because for any edge points belonging to the parallel lines equation (8) is satisfied, the Hough transform can be used to determine the vanishing point
X
Y (a)
(c)
(b)
(e)
(e)
Figure 3 (a) The synthesized fisheye image. (b) The image of the magnitude of the spherical gradient of the edge points. (c) The image of the direction of the spherical gradient of the edge points. (d) The Hough parameter space indicated in a hemisphere. (e) The Hough parameter space indicated by the data structure of [7].
904
corresponding to the vanishing point is indicated in red color in Figure 4(c). The detected vanishing point in the input image is indicated by a red dot in Figure 4(a).
Next, the image of the direction of the gradient of the edge points is used to determine the vanishing point by the Hough transform. Figure 3(d) shows the hemispherical Hough parameter space. The spherical Hough parameter space is represented by the SCVT (Spherical Centroidal Voronoi Tessellation) so as to obtain cells as uniform as possible [7]. The hemispherical parameter space represented by the data structure of [7] is shown in Figure 3(e). In both Figure 3 (d) and (e), the darker points have more votes. The found greatest peak corresponding to the vanishing point is indicated in red color in Figure 3(e). The ground truth of the location of the vanishing point on a unit sphere is (0, 1, 0). The estimates of the vanishing point by the proposed method (Gradientbased method) and the conventional method (Linebased method) are shown Table 1. The computational error is measured by the angle between the ground truth vector and the estimate vector. As shown in Table 1, although the proposed method does not extract lines, there is not a great difference between the proposed gradient-based method and the conventional line-based method in the computational accuracy.
5. Conclusions This paper proposes a novel method of estimating vanishing point by spherical gradient. The proposed method does not necessarily extract lines. This feature distinguishes itself from the other conventional methods. In this paper the Hough transform is used to detect vanishing point based on spherical gradient cues. How to improving the accuracy is our future work to do.
References [1] K. Daniilidis and J. Ernst, “Active intrinsic calibration using vanishing points”, Proc. of CVPR, pp.708-713, 1996. [2] J. Bazin, I. Kweon, C. Demonceaux and P. Vasseur, “A robust top-down approach for rotation estimation and vanishing points extraction by catadioptric vision in urban environment”, Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.346353, 2008. [3] S. Li and Y. Hai, “Estimating camera pose from Hpattern of parking lot”, Proc. of ICRA, pp.3954-3959, 2010. [4] T.P. Tardif, “non-iterative approach for fast and accurate vanishing point detection”, Proc. of ICCV, pp.1250-1257, 2009. [5] F. M. Mirzaei and S. I. Roumeliotis, “Optimal estimation of vanishing points in a Manhattan world”, Proc. of ICCV, pp.2454-2461, 2011. [6] K. Daniilidis and C. Geyer, “A unifying theory for central panoramic systems and practical implications”, ECCV, pp.445-461, 2000. [7] S. Li and Y. Hai, “A full-view spherical image format”, ICPR, pp.2337-2340, 2010. [8] S. Li, “full-view spherical image camera”, ICPR, pp.386390, 2006.
Table 1 Comparison of Computation Accuracy Estimate of Vanishing Point
Angle (Deg.)
Gradient-Based method
(-0.004786, 0.999973, 0.004941)
0.417284
Line-Based method
(-0.000559, 0.999995, -0.003057)
0.178074
4.1. Real-world experiment Figure 4(a) shows an image captured by a fisheye camera. The intrinsic parameters of the fisheye camera are calibrated by the method of [8]. The image of the direction of the spherical gradient of the edge points is shown in Figure 4(b), where different colors indicate different directions of the spherical gradient. The hemispherical parameter space represented by the data structure of [7] is shown in Figure 4(c); the darker points have more votes. The found greatest peak
(a)
(c)
(b)
Figure 4: (a) The image captured by a fisheye camera. (b) The image of the direction of the spherical gradient of the edge points. (c) The Hough parameter space indicated by the data structure of [7]. The detected vanishing point is indicated by a red dot in (a).
905
Vanishing Point Estimation by Spherical Gradient Shigang Li and Hanchao Jia Graduate School of Engineering, Tottori University, Japan [email protected] points can be estimated by spherical gradient of edge points without extracting lines.
Abstract In this paper we propose a novel method of estimating vanishing point by spherical gradient. In contrast with the conventional methods in which vanishing point is estimated from lines, the proposed method does not necessarily extract lines, but employs the spherical gradient cues of edge points. Based on the observation that spherical gradient is aligned with the normal vector of the projection plane of space lines, the vanishing point is estimated directly from spherical gradient of edge points by the Hough Transform.
Π
G np
G l ng
Figure 1: A space line is projected as a great circle in
1. Introduction
spherical image. The normal vector
G np
of the
Π of a space line L is aligned with G the gradient vector ng of the edge points of the line l .
projection plane
Vanishing point is defined as the intersection of space parallel lines in image. To estimate vanishing points, the first step of the conventional methods is to extract lines from images. However, can we estimate vanishing points from image without extracting lines? Figure 1 shows a spherical camera which observes a space line L . In the spherical image, the space line L is projected as a part of a great circle l . Because the projection plane Π of the space line L is perpendicular to the spherical surface, its normal G G vector n p is aligned with the gradient vector ng of
In this paper we propose a novel method of estimating vanishing points based on the above observation. The feature of the proposed method, which distinguishes itself from other conventional ones, is that it does not necessarily extract lines from the image. Concretely, the Hough transform is used to estimate vanishing points in terms of the spherical gradient cues of edge points. The remainder of this paper is organized as follows. We introduce the related research in Section 2. In Section 3, the proposed method is presented. After showing the experimental results in Section 4, we will present our conclusions in Section 5.
the edge points of the line in spherical image. Because for a group of space parallel lines the normal vectors of their projection planes are co-planar, their vanishing point can be estimated as the vector perpendicular to the normal vetors. Because for a spherical image the G normal vector n p is aligned with the gradient vector
2. Related work
G ng of the edge points of the line, as shown in Figure 1,
Vanishing point is an important feature of manmade environments, and is widely used for scene analysis, camera calibration and camera pose estimation [1][2][3]. Because vanishing point is
the vanishing point of a group of parallel lines can be estimated directly from the gradient vector of the edge points of the lines. Thus, in spherical image vanishing
978-4-9906441-1-6 ©2012 IAPR
L
902
defined as the intersection of space parallel lines in image, line extraction from images is a necessary preprocessing for the conventional methods for vanishing point estimation. As recent studies, a J-Linkage algorithm is used to generate hypothetical classes of parallel lines, followed by EM to find the vanishing points [4]. In [5], an analytical method for computing the globally optimal estimates of orthogonal vanishing points in a “Manhattan world” with a calibrated camera is proposed. On the other hand, in omnidirectional image processing an intermediate spherical camera model is usually used to cope with the wide field of view [6]. A space line is projected as a great circle onto the sphere. A vanishing point is estimated as the intersection great circles. Nevertheless, the methods described in a perspective image can be simply extended to a spherical image. However, in a spherical image the gradient vector of the edge points of a line is aligned with the normal vector of the projection plane of the line, as described in Section 1. This observation makes it possible to estimate vanishing point directly from the gradient cues of edge points without line detection. In this paper we show the effectiveness of this approach. Note that because a perspective image can be transformed to a spherical image using the camera intrinsic parameters, the proposed method of is not limited to omnidirectional images. In this sense, estimating vanishing point from the gradient of edge points can be seen as the by-product of spherical image representation.
z
er eϕ
P
θ
r
eθ
O
y
ϕ P′
x
Figure 2: The curvilinear orthogonal coordinate system and the orthogonal coordinate system.
Using the camera intrinsic parameters, a pixel in an input image can be mapped onto a spherical image. The mapping relation is represented as follows.
θ = θ ( xi , y i ) ⎫ ⎬. ϕ = ϕ ( xi , y i ) ⎭
(2)
For the corresponding pixel between an input image and the spherical image, we have I ( xi , yi ) = I s (θ ( xi , yi ),ϕ ( xi , yi ) ) . (3) Using the differential chain rule,
∂I ∂I s ∂θ ∂I s + = ∂xi ∂θ ∂xi ∂ϕ ∂I ∂I s ∂θ ∂I s + = ∂y i ∂θ ∂yi ∂ϕ
3. Vanishing point estimation by spherical gradient
∂ϕ ⎫ ∂xi ⎪⎪ ∂ϕ ⎬ ⎪ ∂yi ⎪⎭
(5)
⎞ ⎛ ∂θ ⎟ ⎜ ⎟ = ⎜ ∂xi ⎟ ⎜ ∂θ ⎟ ⎜ ∂y ⎠ ⎝ i
∂ϕ ⎞⎛ ∂I s ⎞ ⎟⎜ ⎟ ∂xi ⎟⎜ ∂θ ⎟ . ∂ϕ ⎟⎜ ∂I s ⎟ ⎜ ⎟ ∂yi ⎟⎠⎝ ∂ϕ ⎠
(6)
⎞ ⎛ ∂θ ⎟ ⎜ ⎟ = ⎜ ∂xi ⎟ ⎜ ∂θ ⎟ ⎜ ⎠ ⎝ ∂yi
∂ϕ ⎞ ⎟ ∂xi ⎟ ∂ϕ ⎟ ∂yi ⎟⎠
That is,
In this section, we first give the algorithm of computing spherical gradient from an input image, and then present the method of estimating vanishing point by spherical gradient.
⎛ ∂I ⎜ ⎜ ∂xi ⎜ ∂I ⎜ ∂y ⎝ i
3.1. Spherical gradient computation
Thus,
Suppose an input image and its corresponding spherical image are I ( xi , y i ) and I s (θ ,ϕ ) ,
⎛ ∂I s ⎜ ⎜ ∂θ ⎜ ∂I s ⎜ ∂ϕ ⎝
respectively. At the curvilinear orthogonal coordinate system, as shown in Figure 2, the gradient of the spherical image on a unit sphere is computed as follows.
−1
⎛ ∂I ⎜ ⎜ ∂xi ⎜ ∂I ⎜ ∂y ⎝ i
⎞ ⎟ ⎟. ⎟ ⎟ ⎠
(7)
From equation (7), the derivatives of the spherical image can be computed from the derivatives of the input image. Then, the gradient of the spherical image at the curvilinear orthogonal coordinate system can be obtained in terms of equation (1). Finally, the gradient represented in equation (1) at the curvilinear
∂I (θ ,ϕ ) 1 ∂I s (θ ,ϕ ) ∇I s (θ , ϕ ) = s eθ + eϕ (1) ∂θ sin θ ∂ϕ
Therefore, we need to compute the derivatives of the spherical image in order to obtain the gradient.
903
G nv (θ v ,φv ) .
orthogonal coordinate system is transformed to threedimensional vector ∇I o ( x, y , z ) represented at the orthogonal coordinate system O-XYZ. Concretely, in this paper we use the Sobel operator to compute the derivatives of the input image.
4. Experiment At first, a simulation experiment is carried out to evaluate the proposed method quantitatively in comparison with the conventional line-based method. Then, the experimental results of the vanishing point estimation for real-world images are also presented to show the effectiveness of the proposed method.
3.2. Vanishing point estimation Suppose there are a group of parallel lines at space. Let the vector of the vanishing point in spherical G image be nv (θ v , φv ) . Because in a spherical image
4.1. Simulation experiment
the gradient vector of the edge points of a line is aligned with the normal vector of the projection plane of the line as mentioned in Section 1, given an edge point, PL , of the parallel lines, we have
Figure 3(a) shows a synthesized fisheye image of a stripe pattern with a hemispherical field of view. The spherical gradient of the spherical image is computed as described in subsection 3.1. Figure 3 (b) and (c) show the magnitude and direction of the computed gradient of the edge points, respectively; different colors indicate different magnitudes or directions of the computed spherical gradient. Note that in Figure 3(c) the edge points belonging to the same line have the same colors because the spherical gradient direction of these edge points is aligned with the normal vector of the projection plane of the line.
G G g (θ L ,φ L ) ⋅ nv (θ v ,φv ) = 0 , (8) G where g (θ L , φ L ) is the spherical gradient computed
in the orthogonal coordinate system of the spherical image for edge point PL . Because for any edge points belonging to the parallel lines equation (8) is satisfied, the Hough transform can be used to determine the vanishing point
X
Y (a)
(c)
(b)
(e)
(e)
Figure 3 (a) The synthesized fisheye image. (b) The image of the magnitude of the spherical gradient of the edge points. (c) The image of the direction of the spherical gradient of the edge points. (d) The Hough parameter space indicated in a hemisphere. (e) The Hough parameter space indicated by the data structure of [7].
904
corresponding to the vanishing point is indicated in red color in Figure 4(c). The detected vanishing point in the input image is indicated by a red dot in Figure 4(a).
Next, the image of the direction of the gradient of the edge points is used to determine the vanishing point by the Hough transform. Figure 3(d) shows the hemispherical Hough parameter space. The spherical Hough parameter space is represented by the SCVT (Spherical Centroidal Voronoi Tessellation) so as to obtain cells as uniform as possible [7]. The hemispherical parameter space represented by the data structure of [7] is shown in Figure 3(e). In both Figure 3 (d) and (e), the darker points have more votes. The found greatest peak corresponding to the vanishing point is indicated in red color in Figure 3(e). The ground truth of the location of the vanishing point on a unit sphere is (0, 1, 0). The estimates of the vanishing point by the proposed method (Gradientbased method) and the conventional method (Linebased method) are shown Table 1. The computational error is measured by the angle between the ground truth vector and the estimate vector. As shown in Table 1, although the proposed method does not extract lines, there is not a great difference between the proposed gradient-based method and the conventional line-based method in the computational accuracy.
5. Conclusions This paper proposes a novel method of estimating vanishing point by spherical gradient. The proposed method does not necessarily extract lines. This feature distinguishes itself from the other conventional methods. In this paper the Hough transform is used to detect vanishing point based on spherical gradient cues. How to improving the accuracy is our future work to do.
References [1] K. Daniilidis and J. Ernst, “Active intrinsic calibration using vanishing points”, Proc. of CVPR, pp.708-713, 1996. [2] J. Bazin, I. Kweon, C. Demonceaux and P. Vasseur, “A robust top-down approach for rotation estimation and vanishing points extraction by catadioptric vision in urban environment”, Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.346353, 2008. [3] S. Li and Y. Hai, “Estimating camera pose from Hpattern of parking lot”, Proc. of ICRA, pp.3954-3959, 2010. [4] T.P. Tardif, “non-iterative approach for fast and accurate vanishing point detection”, Proc. of ICCV, pp.1250-1257, 2009. [5] F. M. Mirzaei and S. I. Roumeliotis, “Optimal estimation of vanishing points in a Manhattan world”, Proc. of ICCV, pp.2454-2461, 2011. [6] K. Daniilidis and C. Geyer, “A unifying theory for central panoramic systems and practical implications”, ECCV, pp.445-461, 2000. [7] S. Li and Y. Hai, “A full-view spherical image format”, ICPR, pp.2337-2340, 2010. [8] S. Li, “full-view spherical image camera”, ICPR, pp.386390, 2006.
Table 1 Comparison of Computation Accuracy Estimate of Vanishing Point
Angle (Deg.)
Gradient-Based method
(-0.004786, 0.999973, 0.004941)
0.417284
Line-Based method
(-0.000559, 0.999995, -0.003057)
0.178074
4.1. Real-world experiment Figure 4(a) shows an image captured by a fisheye camera. The intrinsic parameters of the fisheye camera are calibrated by the method of [8]. The image of the direction of the spherical gradient of the edge points is shown in Figure 4(b), where different colors indicate different directions of the spherical gradient. The hemispherical parameter space represented by the data structure of [7] is shown in Figure 4(c); the darker points have more votes. The found greatest peak
(a)
(c)
(b)
Figure 4: (a) The image captured by a fisheye camera. (b) The image of the direction of the spherical gradient of the edge points. (c) The Hough parameter space indicated by the data structure of [7]. The detected vanishing point is indicated by a red dot in (a).
905
