Integration of Multiple Range and Intensity Image Pairs Using a ...
103 W. H. Gray, C. Dumont, M. A. Abidi, "Integration of Multiple Range and Intensity Image Pairs Using a Volumetric Method to Create Textured 3D Models," Proc. of SPIE Conf. on Machine Vision Applications in Industrial Inspection VII, Vol. 3966, pp. 94-104, San Jose, CA, January 2000.
Integration of Multiple Range and Intensity Image Pairs Using a Volumetric Method to Create Textured 3D Models W. Harvey Gray *a, Christophe Dumont b, Mongi A. Abidi a a b
IRIS Laboratory, University of Tennessee, Knoxville, TN, 37996 U.S.A.
Le2I laboratory, IUT Le Creusot, University of Burgundy, Le Creusot, 71200, France ABSTRACT
The reconstruction of highly detailed, 3D object models is a major goal of current research. Such models can be used in machine vision applications as well as for visualization purposes. The method presented here assumes that there are multiple range and intensity image pairs of an object, all registered to a global coordinate system. The individual range images are then used to create a surface mesh and the associated intensity images are applied to the surface mesh as a texture map. These multiple, textured, range meshes are then used to update a volume grid - based upon whether a location in the volume grid is known, unknown, or empty - using information that has the highest confidence for any given voxel. The updated volume grid can then be passed through a marching cubes algorithm with adaptive subdivisions to get a fully textured 3D model. The adaptive marching cubes algorithm takes into account additional information concerning edge weights and texture coordinates to give a smoother surface than that produced with standard marching cubes. Once complete, additional, registered intensity images can be applied to the surface of the object. Keywords: Volume grid, marching cubes, range scan, integration, texture map
1. INTRODUCTION 1.1. Background Simulation is a key component for efficient machine vision building. A quality control system by artificial vision aims at capturing information about an object of interest in order to decide whether or not the object is defective. A defect can be a dust particle located under the metallic layer of a cosmetic product [1] or a scratch on the surface of a metallic object [2][3]. A defect can also be a smooth modification of the object surface [4]. Defects are to be detected using specific sensors capturing visible and/or non-visible information about the object to be inspected. Sensors can be associated with a lighting system in some cases to improve defect detection. For other non-visible sensors, the use of supplementary equipment is not required: the thermal information taken from an infrared camera can suffice to reveal a defect on the surface of the object being inspected. Camera placement is a key issue for efficient defect detection. For any vision system, the placement of the camera(s) determines the effectiveness of the defect detection: i.e. a black and white camera associated with a lighting system can be set with respect to an object so that the defects clearly appear in the image [5]. Many applications of quality control tend to optimize the camera placement using a heuristic approach. This approach consists in approaching the optimal solution with an empirical method when the vision system is being built [3]. Although this approach does not rely on any theories, it allows construction of robust and effective vision systems. Other applications of quality control attempt to formalize or simulate the vision system in order to optimize its parameters. Hence, a lighting system can be optimally chosen and placed with respect to the camera in order to reveal defects that are located on the surface of metallic objects (see [1][2][5][6]). In the application described in [1][7][8], the lighting system is complex and requires to be characterized before its implementation. In this application a mathematical model of the
94
In Machine Vision Applications in Industrial Inspection VIII, Kenneth W. Tobin, Jr., Editor, Proceedings of SPIE Vol. 3966 (2000) - 0277-786X/00/$15.00
vision system is built and an optimal set of parameters is chosen according to an effectiveness function based on the intensity of a defect in an image. A good simulation relies on an accurate model of reality [1][9][10][11]. A model of the type of sensor to be used in the vision system is built. Simulation consists of building a virtual vision system that simulates the camera(s) taking images of objects according to the camera model. A complete system is built by integrating the camera model and the object to be inspected into a complete virtual vision system. A simulation mimics not only the image capture, but also the mechanical system that controls object rotation and support. Furthermore, it can synchronize actions between each other: for example the image capture is performed after receiving a signal emitted when the object being inspected has completed a full rotation. Simulation allows low cost and fast design of vision systems (see [1][9][10][11]). For complex vision systems involving heavy equipment, the cost of design is a critical issue. High performance of the vision system has to be proven before making a purchase of very expensive equipment or sensors such as a thermal camera. Simulation allows building of very complex system with a cost of development relying only on salaries and use of computers and software. Simulation provides engineers a means for easy modification of both the vision system and the algorithms controlling object inspection. The camera placement can be optimized off-line before construction of the prototype. The algorithms are studied and optimized to meet the needs and requirements necessary for future implementation and use of the inspection system. The goal of any imaging system is to provide users with quality data about an object. In this case the object is a tire with a specific thermal signature. The setup of such a system relies on the interrelation between several key thermal camera parameters and the positioning of the thermal camera relative to the tire in the system. The inclusion of 3D modeling of the tire, and the subsequent mapping of thermal data back onto the tire model should provide the user with a level of interaction not possible with 2D imaging alone. The theoretical information gained from such a simulation can be used to determine the best possible detection setup according to the users needs and resources. 1.2. System Design Methodology 1.2.1. Interrelated Parts The relationships between the parts of machine vision system must be taken into account for that system to be effective. The most common example in quality control is the heuristic approach where the positioning of sensors follows an empirical method. The users setting up the system take the relationships between the parts of the machine vision system into account. While this approach can be quite effective, it typically involves moving expensive and/or delicate equipment, something that should be avoided if at all possible. The accurate simulation of a machine vision system can provide a solution to this problem. Tire size, camera parameters, and camera positioning can all be used to determine an optimal placement of the camera(s) relative to the tire. This simulation will provide the users a system that is already optimized before it is set up. 1.2.2. Modeling of Parts Any simulation of a machine vision system is only as good as its individual models. To facilitate creating a realistic tire inspection system a high quality tire model is used and is shown in Figures 1 and 2.
95
Figure 1: Detailed Tire Model
Figure 2: Wireframe of Detailed Tire Model
The tire model is to be placed in a scene simulating an endurance drum (dynamometer) test machine. The dynamometer test machine consists of three major components: a steel drum to rotate the tire against, the tire mount, and a hydraulic ram to keep the tire in contact with the drum. The machine allows a tire to be rotated at a given speed and load for lengthy amounts of time. The modeling of the dynamometer setup was deemed to have secondary importance, as the thermal data from the tire is what the users desire. A coarse dynamometer model was developed in order to show the general orientation of the setup, and where a thermal camera could likely be mounted. This setup is shown in Figure 3.
Figure 3: Dynamometer Model
The model of the thermal camera used was a pinhole [12] model used in many applications when cameras focus at infinity. This model is used in this paper to model an infrared camera and its optical lenses. The camera has several internal parameters adjustable by the user, i. e. resolution, field of view, and aspect ratio. In addition the user can specify the position of the camera relative to the tire. 1.2.3. Thermal Data Acquisition As can be seen from Figure 1, there is no thermal data known about the tire model. In order to build a system to test thermal data acquisition it is imperative to have a model that actually reflects real world thermal data. To achieve this requirement, the tire models geometry was used, but the color was altered. Instead of retaining its uniform dark gray color, a program was written to encode the surface with a full range of colors in order to mimic a real world thermal signature. A sample tire model with pseudo thermal data is shown in Figure 4.
96
Figure 4: Tire Model with Thermal Information
2. 2D THERMAL INSPECTION RESULTS 2.1. 2D Imaging The initial goal of this project was to provide a way of capturing the thermal signature of a tire. To accomplish this goal a model of the tire model in Figure 4 was used in an Open Inventor scene. A camera with adjustable parameters was placed in the same scene, and various images were taken from the viewpoint of the camera. A sample representation of camera viewpoints is shown in Figure 6, and the images taken from those viewpoints are shown in Figures 7-9. Viewpoint 1
Viewpoint 2
Viewpoint 3 Figure 6: Orientation of Camera Viewpoints
Figure 7: Image from Viewpoint 1
Figure 8: Image from Viewpoint 2
Figure 9: Image from Viewpoint 3
97
As can be seen from the preceding figures, a high quality thermal image can be obtained from the theoretical tire model. These images can then be inspected individually by the user of the system. While this method has proven itself quite capable, the decision was made to incorporate the 3D model of the tire back into the display process. Each of the individual images generated from the thermal model can be mapped back to the original model using texture mapping. The program can decide which texture maps to use based upon what regions of the tire are being displayed. 2.2. 2.5D Imaging While the previous approach provides many good results, it also assumes several items that can have an effect on a real world system. The first is that a sufficiently detailed model of the tire is available to the users. The second assumption is that the tire being examined suffers from no manufacturing defects that may alter the geometry of the tire. The solution that has been found to combat these problems is to not only take thermal images at the various viewpoints, but also to take range images at the same viewpoints. The range information can be used to generate a 2.5D surface mesh of the tire that will provide information about the geometry of the tire. This information can then be used in conjunction with the thermal image taken from the same location. Figures 10 and 11 show a pair of range and thermal information taken from the same viewpoint (the surface mesh is rotated slightly to show its 2.5D nature).
Figure 10: Surface Mesh
Figure 11: Thermal Image
The thermal image can then be used as a texture map for the surface mesh rather than the original 3D model. This approach minimizes the amount of knowledge the user must have concerning the geometry of the tire such as tread patterns or physical defects. An example using the surface mesh from figure 10 and the thermal image from figure 11 is shown in figure 12.
98
Figure 12: Surface Mesh with Thermal Texture
Since the data in figure 12 is 2.5D it can be rotated and examined from many different angles, something not possible with 2D images alone. This approach also intrinsically ties the thermal information to a location on the tire, something that can be confusing if looking at 2D images alone.
3. 3D MESH INTEGRATION 3.1. Overview Multiple range scans can be merged together using a volumetric approach, or a variety of other techniques such as zippering. These combined range scans can form a reconstructed 3D tire model that is textured using multiple thermal images. The 3D tire model can be inspected from a variety of camera angles by an end user. 3.2. Volumetric Benefits There are several general reasons that a volumetric based approach was chosen to perform the mesh integration stage of this project. 3.2.1. Number of Triangles First was the general level of detail that the user receives from a volumetric based approach. The defining factor for the number of triangles in the output model is the resolution of the volume grid, not the number of scans integrated as in some other methods. The output model of a volume-based technique should have approximately the same number of triangles irregardless of how many range scans are used in the integration process. This characteristic is important for the end user as the volumetric approach should be able to yield output models easily manipulated by a wide variety of computer configurations. 3.2.2. Noise Reduction The second attribute that was of great interest was the ability to easily incorporate some basic noise reduction techniques into the volume grid. The most basic volume grid is a binary volume grid, where each voxel is on or off. As an extension to this general concept an accumulation type volume grid was used to mimic the results of a median filter. Each voxel is treated as an integer value rather than a binary value. The voxel is incremented whenever it lies on the surface of an object, it is decremented when the algorithm decides the voxel should be empty. All values that are positive are considered to be “full”, all other values are considered “empty” when rendering. This approach can help to eliminate noise from the actual input range scans.
99
3.3. Marching Cubes Methodology After the volume grid has been update it is time to run the Marching Cubes Algorithm (MCA). The MCA will run through all of the voxels in a volume grid, and draw triangles based upon the relationships of a voxel to its nearest neighbors. A standard MCA will draw triangles that have vertices that lie a fixed distance between a set of voxels. This method of drawing triangles provides a very rough looking surface since the only information used is taken from the volume grid itself. The MCA with adaptive subdivisions allows for a much smoother surface by providing an additional data structure to accommodate the point at which the vertex of a particular triangle should lie. Since this updated vertex lies on the surface of the object in question, a texture map taken of the original object may be applied to the updated surface. Figure 13 shows how adaptive subdivisions allow the triangle to be drawn at the “real” values of the surface of the object. The voxel in the bottom right is the only one filled in this particular case. The dots in between the voxels show the correct positioning of the surface. The triangle on the left is drawn with a standard MCA, the one on the right is a MCA with adaptive subdivisions.
Figure 13: Comparison between standard and adaptive subdivisions
3.4. 3D Results The initial inputs used are surface meshes generated from range scans with a thermal image used as a texture map. Two such inputs can be seen in Figures 14 and 15.
Figure 14: View 1
100
Figure 15: View 2
As can be seen from these images there is a fair amount of overlap between the two initial surface meshes. The general goal of the volumetric approach is to use all of the information possible to generate an output volume grid for the MCA to run through. From there we want to use the best texture information possible (as measured by the surface normal of the range scan relative to the camera position) to texture the surface of the final object. The volume grid after adding the first view can be seen in Figures 16, 17, and 18. The integration of the second view can be seen in Figures 19, 20, and 21.
Figure 16: After first view
Figure 17: Side view one
Figure 18: Top view one
Figure 19: After second view
Figure 20: Side view two
Figure 21: Top view two
As more views are added more of the unknown area is removed from the volume grid, and the final 3D model begins to take shape. The next set of results (Figures 22-27) show the integration of 8 sets of views of the tire model at varying resolutions. The wireframe models are included to give the reader an idea of the number of triangles in the final object.
Figure 22: 16 resolution with 8 views
Figure 23: 16 x 8 wireframe
101
Figure 24: 32 resolution with 8 views
Figure 25: 32 x 8 wireframe
Figure 26: 64 resolution with 8 views
Figure 27: 64 x 8 wireframe
These final 3D show an important aspect of increasing the resolution with this particular set of data. As the resolution increases the tread pattern of the original tire begins generating small inconsistencies on the output tire model. To the volume grid these tread patterns mimic noise to a certain extent. The volume grid does it’s best to eliminate the areas that appear as white, but in some cases more views are necessary to eliminate the noise through redundancy of information. As such Figures 28-33 show the same resolutions with double the number of integrated surface meshes.
102
Figure 28: 16 resolution with 16 views
Figure 29: 16 x 16 wireframe
Figure 30: 32 resolution with 16 views
Figure 31: 32 x 16 wireframe
Figure 32: 64 resolution with 16 views
Figure 33: 64 x 16 wireframe
103
As can be seen from this last set of figures the incorporation of more views eliminates most of the unknown information concerning the tread. As such it is important to have sufficient overlap of known areas to eliminate noise, or features such as the tread that are smaller than the size of a voxel..
4. CONCLUSION Simulation is an important aspect to consider when designing an inspection system based upon machine vision. A thermal tire inspection system that takes into account camera parameters and location has been described. Results from simulated thermal cameras have been shown. The benefits of incorporating thermal data with range data have been described and illustrated. A fully reconstructed 3D tire model with associated thermal textures has been shown. The methodology behind generating the 3D model from multiple range scans and intensity images has been described.
REFERENCES [1] D. Aluse, "Systeme de detection et de caracterisation de defauts d'aspects sur des surfaces parfaitement specu;aire et non planes: application au controle de produits destines a l'emballage cosmetique", Ph.D. dissertation, University of Burgundy, France, Nov. 1998 [2] C. Coulot, "Etude de l'eclairage de surfaces metalliques pour la vision artificielle: application au controle dimensionnel", Ph.D. dissertation, University of Burgundy, France, June 1997 [3] H. Jender, "Contrôle temps réel par vision artificielle de tubes métalliques en défilement continu", Ph.D. dissertation, University of Burgundy, France, 1993 [4] D. Perard, J. Beyerer, "Three-dimensional measurement of free-form surfaces with a structuredlighting reflection technique", SPIE conference on machine vision application in industrial inspection, Pittsburgh (USA), 1997, vol. 3204, pp. 74-80 [5] D. Aluze, C. Coulot, F. Meriaudeau, P. Gorria, C. Dumont, "Machine vision for the control of reflecting non plane surface", Journal of the machine vision association, Society of Manufacturing Engineers, 1998, Vol. 14, No 3, pp. 1-4. 1D-2D [6] C. Coulot, S. Kohler-Hermmerlin, C. Dumont, D. Aluze, B. Lamalle, "Simulation of lighting for an optimal inspection of metallic-objects", AIM'97, Tokyo (Japan), 16-20 June 97 [7] D. Aluze, C. Dumont, P. Gorria, M. A. Abidi, "Machine Vision for controlling reflective 3-D Objects", Society of Manufacturing Engineers, Conference on Applied Machine Vision, Nashville (USA), May 1998, pp 45-62 [8] D. Aluze, C. Coulot, F. Meriaudeau, P. Gorria, C. Dumont, "Machine vision for the control of reflecting non plane surfaces", SPIE, Pittsburgh (USA), Machine vision application in industrial inspection, 15-17 Oct 1997, pp 180-186 [9] L. M. Wong, C. Dumont and M. A. Abidi, "An Algorithm for Finding the Next Best View in Object Reconstruction", Photonics EAST, Intelligent System and Advanced Manufacturing 98, SPIE Conference on Sensor Fusion and Decentralized Control in Robotics Systems, Boston (USA), November 1998, 3523, pp 191-200 [10] L. M. Wong, C. Dumont, and M. A. Abidi, "Determining Optimal Sensor Poses in 3-D Object Inspection", Conference on Quality Control By Artificial Vision, Takamatsu (Japan), Nov. 98, ISBN: 4-921073-01-5, pp 371-377 [11] L. D. Han, M. Qureshi, M. A. Abidi, H. Gray, Truck rollover warning system simulations, accepted to ISATA, Vienna, Austria, 14-18 June 99. [12] R. C. Gonzales, P. Wintz, Digital image processing, Addison-Wesley Publishing Company, 1992, 2nd edition *Correspondence: Email: [email protected]; WWW: http://iristown.engr.utk.edu/; Telephone: (423)974-9737
104
Integration of Multiple Range and Intensity Image Pairs Using a Volumetric Method to Create Textured 3D Models W. Harvey Gray *a, Christophe Dumont b, Mongi A. Abidi a a b
IRIS Laboratory, University of Tennessee, Knoxville, TN, 37996 U.S.A.
Le2I laboratory, IUT Le Creusot, University of Burgundy, Le Creusot, 71200, France ABSTRACT
The reconstruction of highly detailed, 3D object models is a major goal of current research. Such models can be used in machine vision applications as well as for visualization purposes. The method presented here assumes that there are multiple range and intensity image pairs of an object, all registered to a global coordinate system. The individual range images are then used to create a surface mesh and the associated intensity images are applied to the surface mesh as a texture map. These multiple, textured, range meshes are then used to update a volume grid - based upon whether a location in the volume grid is known, unknown, or empty - using information that has the highest confidence for any given voxel. The updated volume grid can then be passed through a marching cubes algorithm with adaptive subdivisions to get a fully textured 3D model. The adaptive marching cubes algorithm takes into account additional information concerning edge weights and texture coordinates to give a smoother surface than that produced with standard marching cubes. Once complete, additional, registered intensity images can be applied to the surface of the object. Keywords: Volume grid, marching cubes, range scan, integration, texture map
1. INTRODUCTION 1.1. Background Simulation is a key component for efficient machine vision building. A quality control system by artificial vision aims at capturing information about an object of interest in order to decide whether or not the object is defective. A defect can be a dust particle located under the metallic layer of a cosmetic product [1] or a scratch on the surface of a metallic object [2][3]. A defect can also be a smooth modification of the object surface [4]. Defects are to be detected using specific sensors capturing visible and/or non-visible information about the object to be inspected. Sensors can be associated with a lighting system in some cases to improve defect detection. For other non-visible sensors, the use of supplementary equipment is not required: the thermal information taken from an infrared camera can suffice to reveal a defect on the surface of the object being inspected. Camera placement is a key issue for efficient defect detection. For any vision system, the placement of the camera(s) determines the effectiveness of the defect detection: i.e. a black and white camera associated with a lighting system can be set with respect to an object so that the defects clearly appear in the image [5]. Many applications of quality control tend to optimize the camera placement using a heuristic approach. This approach consists in approaching the optimal solution with an empirical method when the vision system is being built [3]. Although this approach does not rely on any theories, it allows construction of robust and effective vision systems. Other applications of quality control attempt to formalize or simulate the vision system in order to optimize its parameters. Hence, a lighting system can be optimally chosen and placed with respect to the camera in order to reveal defects that are located on the surface of metallic objects (see [1][2][5][6]). In the application described in [1][7][8], the lighting system is complex and requires to be characterized before its implementation. In this application a mathematical model of the
94
In Machine Vision Applications in Industrial Inspection VIII, Kenneth W. Tobin, Jr., Editor, Proceedings of SPIE Vol. 3966 (2000) - 0277-786X/00/$15.00
vision system is built and an optimal set of parameters is chosen according to an effectiveness function based on the intensity of a defect in an image. A good simulation relies on an accurate model of reality [1][9][10][11]. A model of the type of sensor to be used in the vision system is built. Simulation consists of building a virtual vision system that simulates the camera(s) taking images of objects according to the camera model. A complete system is built by integrating the camera model and the object to be inspected into a complete virtual vision system. A simulation mimics not only the image capture, but also the mechanical system that controls object rotation and support. Furthermore, it can synchronize actions between each other: for example the image capture is performed after receiving a signal emitted when the object being inspected has completed a full rotation. Simulation allows low cost and fast design of vision systems (see [1][9][10][11]). For complex vision systems involving heavy equipment, the cost of design is a critical issue. High performance of the vision system has to be proven before making a purchase of very expensive equipment or sensors such as a thermal camera. Simulation allows building of very complex system with a cost of development relying only on salaries and use of computers and software. Simulation provides engineers a means for easy modification of both the vision system and the algorithms controlling object inspection. The camera placement can be optimized off-line before construction of the prototype. The algorithms are studied and optimized to meet the needs and requirements necessary for future implementation and use of the inspection system. The goal of any imaging system is to provide users with quality data about an object. In this case the object is a tire with a specific thermal signature. The setup of such a system relies on the interrelation between several key thermal camera parameters and the positioning of the thermal camera relative to the tire in the system. The inclusion of 3D modeling of the tire, and the subsequent mapping of thermal data back onto the tire model should provide the user with a level of interaction not possible with 2D imaging alone. The theoretical information gained from such a simulation can be used to determine the best possible detection setup according to the users needs and resources. 1.2. System Design Methodology 1.2.1. Interrelated Parts The relationships between the parts of machine vision system must be taken into account for that system to be effective. The most common example in quality control is the heuristic approach where the positioning of sensors follows an empirical method. The users setting up the system take the relationships between the parts of the machine vision system into account. While this approach can be quite effective, it typically involves moving expensive and/or delicate equipment, something that should be avoided if at all possible. The accurate simulation of a machine vision system can provide a solution to this problem. Tire size, camera parameters, and camera positioning can all be used to determine an optimal placement of the camera(s) relative to the tire. This simulation will provide the users a system that is already optimized before it is set up. 1.2.2. Modeling of Parts Any simulation of a machine vision system is only as good as its individual models. To facilitate creating a realistic tire inspection system a high quality tire model is used and is shown in Figures 1 and 2.
95
Figure 1: Detailed Tire Model
Figure 2: Wireframe of Detailed Tire Model
The tire model is to be placed in a scene simulating an endurance drum (dynamometer) test machine. The dynamometer test machine consists of three major components: a steel drum to rotate the tire against, the tire mount, and a hydraulic ram to keep the tire in contact with the drum. The machine allows a tire to be rotated at a given speed and load for lengthy amounts of time. The modeling of the dynamometer setup was deemed to have secondary importance, as the thermal data from the tire is what the users desire. A coarse dynamometer model was developed in order to show the general orientation of the setup, and where a thermal camera could likely be mounted. This setup is shown in Figure 3.
Figure 3: Dynamometer Model
The model of the thermal camera used was a pinhole [12] model used in many applications when cameras focus at infinity. This model is used in this paper to model an infrared camera and its optical lenses. The camera has several internal parameters adjustable by the user, i. e. resolution, field of view, and aspect ratio. In addition the user can specify the position of the camera relative to the tire. 1.2.3. Thermal Data Acquisition As can be seen from Figure 1, there is no thermal data known about the tire model. In order to build a system to test thermal data acquisition it is imperative to have a model that actually reflects real world thermal data. To achieve this requirement, the tire models geometry was used, but the color was altered. Instead of retaining its uniform dark gray color, a program was written to encode the surface with a full range of colors in order to mimic a real world thermal signature. A sample tire model with pseudo thermal data is shown in Figure 4.
96
Figure 4: Tire Model with Thermal Information
2. 2D THERMAL INSPECTION RESULTS 2.1. 2D Imaging The initial goal of this project was to provide a way of capturing the thermal signature of a tire. To accomplish this goal a model of the tire model in Figure 4 was used in an Open Inventor scene. A camera with adjustable parameters was placed in the same scene, and various images were taken from the viewpoint of the camera. A sample representation of camera viewpoints is shown in Figure 6, and the images taken from those viewpoints are shown in Figures 7-9. Viewpoint 1
Viewpoint 2
Viewpoint 3 Figure 6: Orientation of Camera Viewpoints
Figure 7: Image from Viewpoint 1
Figure 8: Image from Viewpoint 2
Figure 9: Image from Viewpoint 3
97
As can be seen from the preceding figures, a high quality thermal image can be obtained from the theoretical tire model. These images can then be inspected individually by the user of the system. While this method has proven itself quite capable, the decision was made to incorporate the 3D model of the tire back into the display process. Each of the individual images generated from the thermal model can be mapped back to the original model using texture mapping. The program can decide which texture maps to use based upon what regions of the tire are being displayed. 2.2. 2.5D Imaging While the previous approach provides many good results, it also assumes several items that can have an effect on a real world system. The first is that a sufficiently detailed model of the tire is available to the users. The second assumption is that the tire being examined suffers from no manufacturing defects that may alter the geometry of the tire. The solution that has been found to combat these problems is to not only take thermal images at the various viewpoints, but also to take range images at the same viewpoints. The range information can be used to generate a 2.5D surface mesh of the tire that will provide information about the geometry of the tire. This information can then be used in conjunction with the thermal image taken from the same location. Figures 10 and 11 show a pair of range and thermal information taken from the same viewpoint (the surface mesh is rotated slightly to show its 2.5D nature).
Figure 10: Surface Mesh
Figure 11: Thermal Image
The thermal image can then be used as a texture map for the surface mesh rather than the original 3D model. This approach minimizes the amount of knowledge the user must have concerning the geometry of the tire such as tread patterns or physical defects. An example using the surface mesh from figure 10 and the thermal image from figure 11 is shown in figure 12.
98
Figure 12: Surface Mesh with Thermal Texture
Since the data in figure 12 is 2.5D it can be rotated and examined from many different angles, something not possible with 2D images alone. This approach also intrinsically ties the thermal information to a location on the tire, something that can be confusing if looking at 2D images alone.
3. 3D MESH INTEGRATION 3.1. Overview Multiple range scans can be merged together using a volumetric approach, or a variety of other techniques such as zippering. These combined range scans can form a reconstructed 3D tire model that is textured using multiple thermal images. The 3D tire model can be inspected from a variety of camera angles by an end user. 3.2. Volumetric Benefits There are several general reasons that a volumetric based approach was chosen to perform the mesh integration stage of this project. 3.2.1. Number of Triangles First was the general level of detail that the user receives from a volumetric based approach. The defining factor for the number of triangles in the output model is the resolution of the volume grid, not the number of scans integrated as in some other methods. The output model of a volume-based technique should have approximately the same number of triangles irregardless of how many range scans are used in the integration process. This characteristic is important for the end user as the volumetric approach should be able to yield output models easily manipulated by a wide variety of computer configurations. 3.2.2. Noise Reduction The second attribute that was of great interest was the ability to easily incorporate some basic noise reduction techniques into the volume grid. The most basic volume grid is a binary volume grid, where each voxel is on or off. As an extension to this general concept an accumulation type volume grid was used to mimic the results of a median filter. Each voxel is treated as an integer value rather than a binary value. The voxel is incremented whenever it lies on the surface of an object, it is decremented when the algorithm decides the voxel should be empty. All values that are positive are considered to be “full”, all other values are considered “empty” when rendering. This approach can help to eliminate noise from the actual input range scans.
99
3.3. Marching Cubes Methodology After the volume grid has been update it is time to run the Marching Cubes Algorithm (MCA). The MCA will run through all of the voxels in a volume grid, and draw triangles based upon the relationships of a voxel to its nearest neighbors. A standard MCA will draw triangles that have vertices that lie a fixed distance between a set of voxels. This method of drawing triangles provides a very rough looking surface since the only information used is taken from the volume grid itself. The MCA with adaptive subdivisions allows for a much smoother surface by providing an additional data structure to accommodate the point at which the vertex of a particular triangle should lie. Since this updated vertex lies on the surface of the object in question, a texture map taken of the original object may be applied to the updated surface. Figure 13 shows how adaptive subdivisions allow the triangle to be drawn at the “real” values of the surface of the object. The voxel in the bottom right is the only one filled in this particular case. The dots in between the voxels show the correct positioning of the surface. The triangle on the left is drawn with a standard MCA, the one on the right is a MCA with adaptive subdivisions.
Figure 13: Comparison between standard and adaptive subdivisions
3.4. 3D Results The initial inputs used are surface meshes generated from range scans with a thermal image used as a texture map. Two such inputs can be seen in Figures 14 and 15.
Figure 14: View 1
100
Figure 15: View 2
As can be seen from these images there is a fair amount of overlap between the two initial surface meshes. The general goal of the volumetric approach is to use all of the information possible to generate an output volume grid for the MCA to run through. From there we want to use the best texture information possible (as measured by the surface normal of the range scan relative to the camera position) to texture the surface of the final object. The volume grid after adding the first view can be seen in Figures 16, 17, and 18. The integration of the second view can be seen in Figures 19, 20, and 21.
Figure 16: After first view
Figure 17: Side view one
Figure 18: Top view one
Figure 19: After second view
Figure 20: Side view two
Figure 21: Top view two
As more views are added more of the unknown area is removed from the volume grid, and the final 3D model begins to take shape. The next set of results (Figures 22-27) show the integration of 8 sets of views of the tire model at varying resolutions. The wireframe models are included to give the reader an idea of the number of triangles in the final object.
Figure 22: 16 resolution with 8 views
Figure 23: 16 x 8 wireframe
101
Figure 24: 32 resolution with 8 views
Figure 25: 32 x 8 wireframe
Figure 26: 64 resolution with 8 views
Figure 27: 64 x 8 wireframe
These final 3D show an important aspect of increasing the resolution with this particular set of data. As the resolution increases the tread pattern of the original tire begins generating small inconsistencies on the output tire model. To the volume grid these tread patterns mimic noise to a certain extent. The volume grid does it’s best to eliminate the areas that appear as white, but in some cases more views are necessary to eliminate the noise through redundancy of information. As such Figures 28-33 show the same resolutions with double the number of integrated surface meshes.
102
Figure 28: 16 resolution with 16 views
Figure 29: 16 x 16 wireframe
Figure 30: 32 resolution with 16 views
Figure 31: 32 x 16 wireframe
Figure 32: 64 resolution with 16 views
Figure 33: 64 x 16 wireframe
103
As can be seen from this last set of figures the incorporation of more views eliminates most of the unknown information concerning the tread. As such it is important to have sufficient overlap of known areas to eliminate noise, or features such as the tread that are smaller than the size of a voxel..
4. CONCLUSION Simulation is an important aspect to consider when designing an inspection system based upon machine vision. A thermal tire inspection system that takes into account camera parameters and location has been described. Results from simulated thermal cameras have been shown. The benefits of incorporating thermal data with range data have been described and illustrated. A fully reconstructed 3D tire model with associated thermal textures has been shown. The methodology behind generating the 3D model from multiple range scans and intensity images has been described.
REFERENCES [1] D. Aluse, "Systeme de detection et de caracterisation de defauts d'aspects sur des surfaces parfaitement specu;aire et non planes: application au controle de produits destines a l'emballage cosmetique", Ph.D. dissertation, University of Burgundy, France, Nov. 1998 [2] C. Coulot, "Etude de l'eclairage de surfaces metalliques pour la vision artificielle: application au controle dimensionnel", Ph.D. dissertation, University of Burgundy, France, June 1997 [3] H. Jender, "Contrôle temps réel par vision artificielle de tubes métalliques en défilement continu", Ph.D. dissertation, University of Burgundy, France, 1993 [4] D. Perard, J. Beyerer, "Three-dimensional measurement of free-form surfaces with a structuredlighting reflection technique", SPIE conference on machine vision application in industrial inspection, Pittsburgh (USA), 1997, vol. 3204, pp. 74-80 [5] D. Aluze, C. Coulot, F. Meriaudeau, P. Gorria, C. Dumont, "Machine vision for the control of reflecting non plane surface", Journal of the machine vision association, Society of Manufacturing Engineers, 1998, Vol. 14, No 3, pp. 1-4. 1D-2D [6] C. Coulot, S. Kohler-Hermmerlin, C. Dumont, D. Aluze, B. Lamalle, "Simulation of lighting for an optimal inspection of metallic-objects", AIM'97, Tokyo (Japan), 16-20 June 97 [7] D. Aluze, C. Dumont, P. Gorria, M. A. Abidi, "Machine Vision for controlling reflective 3-D Objects", Society of Manufacturing Engineers, Conference on Applied Machine Vision, Nashville (USA), May 1998, pp 45-62 [8] D. Aluze, C. Coulot, F. Meriaudeau, P. Gorria, C. Dumont, "Machine vision for the control of reflecting non plane surfaces", SPIE, Pittsburgh (USA), Machine vision application in industrial inspection, 15-17 Oct 1997, pp 180-186 [9] L. M. Wong, C. Dumont and M. A. Abidi, "An Algorithm for Finding the Next Best View in Object Reconstruction", Photonics EAST, Intelligent System and Advanced Manufacturing 98, SPIE Conference on Sensor Fusion and Decentralized Control in Robotics Systems, Boston (USA), November 1998, 3523, pp 191-200 [10] L. M. Wong, C. Dumont, and M. A. Abidi, "Determining Optimal Sensor Poses in 3-D Object Inspection", Conference on Quality Control By Artificial Vision, Takamatsu (Japan), Nov. 98, ISBN: 4-921073-01-5, pp 371-377 [11] L. D. Han, M. Qureshi, M. A. Abidi, H. Gray, Truck rollover warning system simulations, accepted to ISATA, Vienna, Austria, 14-18 June 99. [12] R. C. Gonzales, P. Wintz, Digital image processing, Addison-Wesley Publishing Company, 1992, 2nd edition *Correspondence: Email: [email protected]; WWW: http://iristown.engr.utk.edu/; Telephone: (423)974-9737
104
