2D/3D Automatic Matching Technique for 3D Recovering of Free Form Objects Fakhreddine Ababsa, David Roussel, Malik Mallem, Jean-Yves Didier Laboratoire des Systèmes Complexes 40, Rue du Pelvoux CE1455-Courcouronnes 91020 Evry Cedex France ababsa, roussel, mallem,
[email protected] Abstract This paper presents a new method for an automatic matching between a free form 3D objects and a single image. This matching could be used for object recognition and/or object 3D recovering. This kind of problem has raised a tremendous amount of interest in research fields such as Computer Vision and more specifically in Augmented Reality. The original idea brought by our method is to use a photoclinometric framework (such as Shape from Shading) in order to extract normal vectors information from the image. Our interest is focused on the distribution of angles between surface normal vectors and incident light vector. This information is discriminate for free form objects without symmetries, and one can show that it could be used to match between the image and the model by using an aspect graph featuring this distribution.
1. Introduction The 3D free form objects recognition problem is probably one of the most studied in computer vision. Several methods have been developed these last years [1], they are generally based on a priori knowledge of the object model to identify. 3D models are generally built from geometric features like points, lines, curves, etc. Identification of an object in the scene from an image is obtained by matching 2D features extracted from the image with 3D features from the object model. Several matching algorithms have been elaborated in order to solve object recognition problem, one can cite for instance the prediction-verification method [2], geometric hashing [3], or geometric invariant indexing algorithm [4]. These methods are complex and noise sensitive because the geometric features used in these algorithms (points, lines, planes, etc.) tends to be mislocated. This paper contributes to the development of a robust matching algorithm inspired from geometric invariant indexing methods. The projective invariant that we constructed is based on the distribution of angles between
surface normal vectors and incident light vector, it is less noise sensitive and better adapted for free form object recognition problem. We used a photoclinometric framework to construct our projective invariant [5]. In section 2, we present the basic hypotheses that we have considered. In section 3, we describe the principles of the elaborated matching method while explaining how we construct our projective invariant. Section 4 presents the procedure developed for constructing an aspect graph of the 3D model used. In section 5, we will present the details of our automatic matching process. In section 6, we develop a 3D recovering algorithm to determine the current position of the object in the scene. And finally, we presents some results obtained on synthetic simulations.
2. Basic Hypothesis Studied objects are placed on a scene which defines the absolute coordinates (World Coordinates System : S0). The scene is illuminated by a single directional light source who’s orientation is already known in this Coordinates System. Image shooting is performed by a single camera who’s optical axis is collinear to the incident light direction (Figure 1). Y Light source S0 Z camera
X free form object
Figure 1. Experimental Setup Furthermore, we will consider in this study only objects with matte surfaces (microscopically rough surfaces). This kind of objects has a diffuse reflection scheme also known as Lambertian reflection [6]. The reflected light amount depends only on the angle between the surface normal and incident light direction. Besides, the free form objects used in this study are also modeled
1051-4651/02 $17.00 (c) 2002 IEEE
with a 3D mesh. We will focused our attention on a particular example provided by the “Venus” object.
plane, and finding the exact outline is performed by searching for vertices having a normal vector perpendicular to the reference normal vector.
3. Proposed Approach
P4
Our matching method is based on the following concept: Each gray level in the image is proportional to the angle between the light source vector and the surface normal vector [6]. We search the most illuminated areas on the image of a particular object (which we will call spots), these spots matches surface patches where surface normal vectors and incident light vectors are collinear. As we know the incident light vector, we also know the surface normal vectors in these spots explained in the world coordinate system. For each spot in the image, we compute the angles between the spot normal vector and the normal vectors directly around the spot. Such an angles distribution is then considered as a projective invariant of the diffuse reflection model. Each spot in the image have its own angle distribution which could be identified as well in the 3D model. In order to exploit this idea in a 2D/3D matching process, one can build an “aspect graph”1 of the 3D model featuring the angle distributions. Every aspect of the model in this graph is defined by a reference normal vector characterizing all the surface elements with the same normal vectors and of course the angle distributions around these surface elements. Using these angular distributions allows to run a search through the aspect graph in order to identify spots extracted from the image with spots (surface elements) defined for this orientation, hence the 2D/3D matching.
4. Constructing Aspect Graph As every aspect in the graph is characterized by a reference normal vector we should build a discrete spherical grid around the model. Such a grid should also be equally distributed in the angle space. This is why we choose to apply an ISEA (Icosahedral Snyder Equal Area) grid in order to build our discrete aspect graph. A few tests shows that a subdivision level of 4 generates 2562 nodes in the grid. For every reference normal vector and add them in a list of concerned vertices with respect to the reference normal vector. At the same time we store the angles between the concerned vertices normal vectors and all the vertices normal vector located around them. The “around” term is ruled by a profile that we call the “angular profile” and which is defined referring to the current outline of the object (Figure 2). The outline is obtained by projecting the 3D model along the current reference normal vector in a 1
Where “aspect” denotes the object’s orientation with respect to the camera.
S
P1
P2
P3
Figure 2. The angular profile Then, we search in the outline the closest point from the current concerned vertex (denoted P1) which will be the first point of a cross profile defined by the [SP1] [SP2] [SP3] and [SP4] segments. Moreover, we compute the relative position of the current vertex in the profile with the ratio R such that: PP (1) R= 1 2 P3 P4 All the previously defined data are stored in each node of the aspect graph by using the following arrays where N denotes the number of reference normal vectors described by the aspect graph and M denotes the number of vertices in the 3D model: 1. Normals(Nx3) : this array contains the 3D coordinates of the reference normal vectors describing all the aspects of the 3D model. 2. Vertices(1xNxJN) : Each cell of this array contains the list of the concerned array. And JN denotes the number of concerned vertices for each aspect. 3. Aspects(1xMxAM) : Each cell of this array contains the list of aspects where this vertex is concerned. AM denotes the number of aspects where this vertex is concerned. 4. Distrib(Mx4xP) : This array contains the angular distributions for each vertex in the mesh. P denotes the number of subdivisions on each segment of the cross profile. 5. Ratio(Mx1) : This array contains the value of the Ratio R for each vertex of the mesh.
5. 2D/3D Matching Method The Gray level image of the scene is also a particular aspect of the object that we have to match with one of the aspects of the “aspect graph” previously described. The matching process follows these steps: •= First, find the brightest areas (spots) in the image. We consider that brightest areas denotes the areas where surface normal vector and incident light vector are collinear.
1051-4651/02 $17.00 (c) 2002 IEEE
•= Compute the extracted spot’s barycenter. •= Compute the angular distributions around each of these barycenters by applying the relationship between the light (in this case the gray level in the image) and the angle described in section 2. The outline of the object is extracted from the image by using classical edge detection algorithms. Then for each spot’s barycenter we search for the closest point in the outline, and then build the cross profile in the same way we did on the concerned vertices of the 3D model. The normal vector angle’s value along the profile is computed by reversing the diffuse light equation :
æ ö θ=cos ç c (2) è cm where, cm is the maximal gray level in the image associated to the minimal angle, and c is the gray level of the profile point. Estimated angular distributions extracted from the image are stored in an array Dist_est(Lx4xP), where L denotes the number of extracted spots. As we did on the 3D model we computes the ratio R (defined in equation 1) on every spot and store the value in the array Rat_est(Lx1). At this point, we are now able to start the matching process which will match the spot’s barycenters with the concerned vertices in the model, and hence, identify the aspect in the graph corresponding to the image. First of all the extracted angular distributions and the R ratios are compared with those stored in the aspect graph. Only vertices having the same angular distributions and ratio are selected in the graph which leads to the selection of several aspect hypotheses. Plotting the selected normal vectors associated with the selected vertices brings out the conical distributions of theses vectors. Hence, the selected aspects are determined by the principal directions of these conical distributions. For each principal direction we choose the closest aspect in the graph, and then search for vertices matching the image’s spots. This matching process is performed by applying the following criterion: selected vertices must minimize the error ei between its own angular distribution and angular distribution of the (considered) image spot. We also define the quadratic mean E obtained for all the spots such that: E = 1k ⋅ L1
L i =1
ei
2
(3)
Where k denotes the number of normal vectors in the current conical distribution. Finally the chosen aspect should of course minimize error E. Identifying the desired aspect of the object and also matching image spots with the 3D model vertices is the core of our 2D/3D matching process. These information will therefore be used by the recovering algorithm in order
to determine the object’s current position in the scene with respect to the world coordinate system.
6. 3D Recovering Algorithm The main objective of 3D Recovering is to find the geometric transformation (namely translation + rotation) applied on object in the world coordinate system from one images shot from an unknown point of view. This process requires to match 2D features extracted from the image with 3D elements extracted from the model, and this is why the previous method was focused on the angular distributions located on the spots barycenters and 3D vertices. So we have to reformulate the 3D recovering problem in suitable way for our data sets.
Figure 3. Coordinates system
Let M i be a 3D point of the model in its initial position. ' M i be the point M i displaced by the model transformation we are looking for. ' M i points coordinates are given in (S0) by : ' ' M i / R = R×M i / R +T (4) 0
m
where R et T are respectively the desired rotation and translation. ' M i coordinates are already know within (SM), as they match exactly M i coordinates within (S0). So we could state that : '
M i / R = R×M i / R +T 0
(5)
0
projects into the image as mi (ui ,vi ) point defined in homogeneous coordinates by: ' Mi/ R 0
[s⋅ui
[
]
s⋅vi s ] T =C3x4 × M i / R '
1
0
T
(6)
where C3x4 is the global transformation defined by the camera calibration which encapsulate intrinsic parameters (optical projection parameters) and extrinsic parameters (camera position parameters). Then we have :
[s⋅ui
[
s⋅vi s ]T =C3x4 × R×M i / R
0
]
1T
(7)
In order to determine R and T, we have to minimize the following criterion :
1051-4651/02 $17.00 (c) 2002 IEEE
c=
n i =1
2
ˆ i −mi m
(8)
4
50
5 100 8
1
13
2
150
200 7 10
11
250
6 300
3 12 9
350
400 250
300
350
400
Figure 4. An intensity image of the “Venus”
Figure 5. Conical distributions
ˆ i are the spot’s barycenters extracted from the such that m image. n denotes the number of matching between image spots and 3D model vertices. The parameters vector V to estimate is constituted of the rotation and translation components. We use non linear optimization techniques to solve this problem.
7. Simulation results Algorithms defined above have been tested and validated on synthetic images of the “Venus” model. Figure 4 shows the initial image of the model where the brightest areas numbered from 1 to 13 (the spots) have been extracted and highlighted. Figure 5 shows that normal vectors are concentrated with three distinct conical distributions. The retained aspects are presented in the table 1. Aspect number 77 has the closest angle error, and is the best hypothesis for finding the object’s orientation in the image. Aspect
77 720 802
Orientation (radian) elevation azimuth 1.6347 -1.5312 1.5073 -1.4526 1.7699 -1.5292 Table 1. Retained aspects
Error E (degrees) 5.6630 20.8588 160.1544
Figure 6 shows the “Venus” object with respect to azimuth and elevation found in the 77th aspect, it also highlights the concerned vertices having similar normal vectors in this aspect. Once the 2D/3D matching is done, 2D and 3D matched points are feeding to the 3D recovering algorithm in order to find the Object’s transformation. Figure 7 shows the 3D model re-projected onto the test image after the transformation has been determined. Obtained results are very satisfying. Actually the recovering algorithm could recognize the closest aspect and match the 2D/3D points in 95 % of the test cases.
Figure 6. Venus representation Figure 7. 3D recovering in aspect N°77 of the model
Nevertheless, some ambiguous cases can occur when the retained aspects by the matching process are very similar.
8. Conclusion This paper proposed a method for automatic 2D/3D matching of free form objects. We have used photoclinometry framework to construct a projective invariant based on the distribution of angles between surface normal vectors and incident light vector. We also developed a 3D recovering algorithm which uses matching results to compute accurate position and orientation of the object in the scene. The simulation results are very satisfying and supportive of our approach. Our current work are focused on the improvement of accuracy and robustness of the elaborated method. Therefore, we are working on the improvement of the spot recognition algorithm while developing a classification method in order to recognize spots more quickly (currently the recognition is performed in one minute approximately).
References [1] I.Weiss, M. Ray, “Model-based recognition of 3D objects from single image”, IEEE trans. Pattern Analysis and Machine Intelligence, Vol. 23 no. 2, February 2001, pp. 116-128. [2] D. P. Huttenlocher, S. Ullman, “Recognizing solid objects by alignment with an image“, Int. Journ. Computer Vision, Vol. 5 no. 2, 1990, pp. 195-212. [3] Y. Lambdan, H. J. Wolfson, “Geometric hashing : a general and efficient model-based recognition scheme”, Proc. IEEE Conf. Computer Vision, 1988, pp. 238-289. [4] A. Califano, R. Mohan, “Multidimentional indexing for recognition visual shapes”, Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 1991, pp. 28-34. [5] B. K .P. Horn, “Height and gradient from shading”, Int. Journ. Computer Vision, Vol. 5 no. 1, 1990, pp.37-76. [6] J. D. Foley, A. Van Dam, S. K. Feiner, J. F. Hughes, “Computer graphics principle and practice“, AddisonWesley Publishing Company, 1990.
1051-4651/02 $17.00 (c) 2002 IEEE