Autonomous Photogrammetric Network Design ... - Semantic Scholar
Autonomous Photogrammetric Network Design Using Genetic Algorithms Gustavo Olague Departamento de Ciencias de la Computaci´ on, Divisi´ on de F´ısica Aplicada, Centro de Investigaci´ on Cient´ıfica y de Educaci´ on Superior de Ensenada, B.C. Km. 107 carretera Tijuana-Ensenada, 22860, Ensenada, B.C. M´exico. [email protected]
Abstract. This work describes the use of genetic algorithms for automating the photogrammetric network design process. When planning a photogrammetric network, the cameras should be placed in order to satisfy a set of interrelated and competing constraints. Furthermore, when the object is three-dimensional a combinatorial problem occurs. Genetic algorithms are stochastic optimization techniques, which have proved useful at solving computationally difficult problems with high combinatorial aspects. EPOCA (an acronym for “Evolving POsitions of CAmeras”) has been developed using a three-dimensional CAD interface. EPOCA is a genetic based system that provides the attitude of each camera in the network, taking into account the imaging geometry, as well as several major constraints like visibility, convergence angle, and workspace constraint. EPOCA reproduces configurations reported in the photogrammetric literature. Moreover, the system can design networks for several adjoining planes and complex objects opening interesting new research avenues.
1
Introduction
Photogrammetric network design is the process of placing cameras in order to perform photogrammetric tasks. An important aspect of any close range photogrammetric system is to achieve an optimal spatial distribution of the cameras comprising the network. Planning an optimal photogrammetric network for some special purpose, such as for monitoring structural deformation or for determining the precise shape characteristics of an object demands special attention from the quality of the network design. Previous approaches to photogrammetric network design have attempted to identify the main stages in the process. Following the widely accepted classification scheme of Grafarend [1], network design has been divided into four design stages from which only the first three are used in close-range photogrammetry: 1. Zero Order Design (ZOD): This stage attempts to define an optimal datum in order to obtain accurate object point coordinates and exterior orientation parameters. E.J.W. Boers et al. (Eds.) EvoWorkshop 2001, LNCS 2037, pp. 353–363, 2001. c Springer-Verlag Berlin Heidelberg 2001
354
G. Olague
2. First Order Design (FOD): This stage involves defining an optimal imaging geometry, which in turn determines the accuracy of the system. 3. Second Order Design (SOD): This stage is concerned with adopting a suitable measurement precision for the image coordinates. It consists usually in taking multiple images from each camera station. 4. Third Order Design (TOD): This stage deals with the improvement of a network through the inclusion of additional points in a weak region. Photogrammetric measurement operations attempt to satisfy, in an optimal manner, several objectives such as precision, reliability and economy. The ZOD and SOD are greatly simplified in comparison to geodetic networks for which the four stages were originally developed. Indeed FOD, the design of network configuration or the sensor placement task needs to be comprehensively addressed for photogrammetric projects. This design stage must provide an optimal imaging geometry and convergence angle for each set of points placed over a complex object [2]. Photogrammetrists have acknowledged the degree of expertise needed to carry out a photogrammetric project. For example, Mason and Gr¨ un [3] developed a work called CONSENS that follows the expert system approach and uses multiple cameras in combination with optical triangulation. It outlines a method of overcoming the set of constraints and objectives presented in camera station placement. The method is based on the theory of generic networks, which constitutes compiled expertise, describing an ideal configuration of four camera stations that can be employed to provide a strong imaging geometry for the class of planar network design problems. Complex objects are divided into planes; each plane is evaluated through one of these networks and then connected with some additional cameras with the purpose of establishing just one common datum. However, the expert system approach has shown it unlikely that full automation of the network design process will be achieved, due in large part to human expert’s extensive use of common-sense reasoning [2]. On the other hand, the Grafarend classification just presented serves the photogrammetric user by identifying what set of tasks needs to be implemented in designing a network. Despite the progress that photogrammetrists have made in understanding this design problem, the photogrammetric measurement technique has rarely been applied by other than experienced photogrammetrists. Although its definition seems simple, it reaches a high complexity mainly due to the numerous constraints and design decisions that need to be made. Photogrammetric network design is also difficult to obtain due to the unknown number of configurations all having very similar accuracy, but with a very different imaging geometry. Consequently, photogrammetric network design in many machine vision applications is often conducted in a trial-and-error fashion or using heuristic reasoning strategies [4]. These strategies fail at solving the problem for the case of complex objects. Moreover, the main question, how to obtain an initial configuration with an optimal imaging geometry, is unsolved and left as the responsibility of the designer. The motivation of this research is to reduce the cost of vision system design and to equip autonomous inspection systems with photogrammetric net-
Autonomous Photogrammetric Network Design
355
work capabilities, e.g., measurement robots used in flexible manufacturing, see Figure 1.
Fig. 1. Photogrammetric network simulation of four robots, each camera is mounted on the robot’s hand, with the goal of measuring the box on the table.
Expert photogrammetrists regard simulation as a viable strategy to the problem of photogrammetric network design [2]. Computer simulation of close range photogrammetric networks has been successfully employed and, with the sophistication of computers, a considerable boost to interactive network design has been achieved. The process of photogrammetric network design optimization through computer simulation can follow a number of approaches. One traditional procedure is based on the stages ZOD, FOD and SOD. Given the criteria related to required triangulation precision, the initial step is to adopt a suitable observation and measuring scheme (FOD stage). This entails the selection of an appropriate camera format, focal length, and image measurement system, as well as a first approximation to suitable network geometry. Once this design stage is finished, the network is evaluated against the specified criteria. If the network fails to achieve the criteria, a new stage to diagnose and identify the problem is performed. FOD or SOD will be applied to produce the new solution. If both corrections are insufficient a completely new network will be proposed until a solution to the problem is achieved. In this way, network design is iterative in nature. The aim of this paper is to present a new simulation-based method for solving the most fundamental stage in network design. The problem
356
G. Olague
is set in terms of a global optimization design [5,6], which is capable of managing the problem using an adaptive strategy. It explores the solution space using both non-continuous optimization and combinatorial search. The approach then is to minimize the uncertainty of the three dimensional measurements using as a criterion the average variance of the 3D object points, presuming that the optimization satisfies a number of primary constraints. This paper is organized as follows: first the bundle adjustment, the mathematical model universally accepted by photogrammetrists, is reviewed in order to obtain a criterion useful to the optimization process. Then, a brief summary of the constraints on network design is presented. The problem of photogrammetric network design in terms of a stochastic global optimization is described together with implementation details about visibility and occlusion constraints related to the complexity of the search space. Finally, results are presented followed by a conclusion.
2
Photogrammetric Network Modeling
Brown originally developed the bundle method in a fully general form. Today, the bundle method is recognized as a critical factor in exploiting the mensuration potential of photogrammetry and is almost exclusively used in applications requiring high accuracy. The method accords simultaneous consideration to all sets (or bundles) of photogrammetric rays from all cameras. The bundle method is based on a mathematical camera model comprised of separate functional and stochastic models. The functional model describing the relationship between the desired and measured quantities consists of the well-known collinearity equations. The collinearity equations, derived from the perspective transformation, are based on the fundamental assumption that the perspective center, the ground point and its corresponding image point, all lie on a straight line. For each pair of image coordinates (xij , yij ) observed on each image, the following pair of equations is written: m11 (Xj − Xic ) + m12 (Yj − Yic ) + m13 (Zj − Zic ) xij = xp − f m31 (Xj − Xic ) + m32 (Yj − Yic ) + m33 (Zj − Zic ) yij = yp − f
m21 (Xj − Xic ) + m22 (Yj − Yic ) + m23 (Zj − Zic ) m31 (Xj − Xic ) + m32 (Yj − Yic ) + m33 (Zj − Zic )
(1) ,
where (xij , yij ) denote the coordinates of point j on photograph i, f and (xp , yp ) are the camera constant and image coordinates of the principal point of the sensor defining the sensor’s orientation, (Xj , Yj , Zj ) are the object space coordinates of the corresponding point feature, (Xic , Yic , Zic ) are the object space coordinates of the perspective center, and mkl are elements of an orthogonal matrix which defines the rotation between the image and object coordinate systems. This system of equations assumes that light rays travel in straight lines, that all rays entering a camera lens system pass through a single point and that the lens
Autonomous Photogrammetric Network Design
357
system is distortion-less or, as is usual in highly accurate measurement, that distortion has been cancelled out after having been estimated. Due to the nature of the measurement process, observations are accompanied by errors. Because of random errors, as evidenced by the small differences between observations of the same quantity, observations can be regarded as random variables and their effects described by means of a stochastic model. Equation 1 can be linearized through the first order development using the Taylor series. A functional model can be given as v = Ay − l C1 = σ02 P−1 where l, v and y are the vectors of observations, residuals and unknown parameters, respectively; A is the design or configuration matrix; C1 the covariance matrix of observations; P the weight matrix; and σ02 the variance factor. In situations where A is of full rank (i.e., where redundant or explicit minimal constraints are imposed), the parameter estimates y and the corresponding cofactor matrix Qy and covariance matrix Cy are obtained as
and
y = (AT PA)−1 AT PL = Qy AT PL ,
(2)
Cy = σ02 Qy .
(3)
The ultimate aim of any photogrammetric measurement is the determination of triangulated object point coordinates along with estimates for their precision. The bundle method is simplified by considering two groups of parameters in the vector y ˆ: y1 comprising exterior orientation (self-calibration parameters were not considered for simplicity), and y2 containing object coordinate corrections. Equation 2, then assumes the form −1 T T A1 PA1 AT A PL y1 1 PA2 , = T y2 AT AT PL 2 PA1 A2 PA2 and the cofactor matrix Qy can be written Q1 Q1,2 . Qy = Q2,1 Q2 The design optimization goal for precision is to achieve an optimal form of Q2 and therefore the covariance matrix of object point coordinates (Xj , Yj , Zj ), considering the applicable design constraints. The criterion used in the minimization process was the average variance along the covariance matrix σc2 σc2 =
σ02 (trace Q2 ) . 3n
Before dramatic improvements in computer processing power in recent years, a valid criticism of designing close range networks by simulation was the computation time required for a bundle adjustment after each design-iteration even for
358
G. Olague
relatively small networks. As shown in [7], the covariance matrix can be obtained through the equation −1 + K] , Q2 = σ02 [(AT 2 PA2 )
where and
K = MQ1 MT , −1 T A2 PA1 . M = (AT 2 PA2 )
In this way, the determination of Q2 using this approach represents a rigorous approach that is termed Total Error Propagation (TEP.) On the other hand, it has been demonstrated [8] that for a wide range of convergent photogrammetric networks, K = 0. This consideration is non-rigorous in that it implicitly assumes that exterior orientation parameters exhibit no dispersion and is called Limited Error Propagation (LEP). The perspective parameters are assumed to be error free and the variances in object point coordinates arise solely from the propagation of random errors in the image coordinate measurements. What is remarkable from a network design standpoint is that for strong networks (convergent networks) LEP is sufficiently accurate compared to TEP, causing considerable computation savings.
3
Constraints on Network Design
The problem of photogrammetric network design (PND) must deal with a series of constraints in order to propose an optimal camera distribution. The accuracy of the system is related to the imaging geometry (main objective in PND) as well as the convergence angle of each camera with respect to each object surface. In order to answer the most basic question of a favorable imaging geometry (FOD or the configuration problem) we must distinguish among the several constraints limiting the search space. Mason [9] has proposed a set of constraints and objectives that we separate into two parts: 3.1
Main Objective and Primary Constraints
Considering the constraints limiting the search space we identify the following main objective and three constraints due to the characteristics of the FOD problem: – Contribution to intersection angles or the imaging geometry. Within a camera placement system the main objective is to know the contribution of each camera with respect to the others. Two fundamental questions need to be answered: how many cameras will be needed and where should they be placed. However, before answering the first question we need to answer the second one. Once we know where to place a given number of cameras, it is a trivial matter to decide on the number.
Autonomous Photogrammetric Network Design
359
– Convergence angle. The reliability of image measurements from directions close to coplanar are difficult and even impossible to obtain. The minimum allowable incidence angle is dependent on the type of feature, its geometry and material. The accuracy of the measurement with respect to the convergence angle is a function of the viewing direction and the surface normal at the feature. In the case of circular targets the minimum convergence angle is about 20 to 30 degrees for the kind of retro-reflective targets that are normally used. – Working space constraint. The workspace in which the photogrammetric survey is conducted can impose restrictions on the selection of an ideal imaging geometry. This constraint includes the walls of the room, any obstructions in the working environment, and the workspace of the robot where the camera could be mounted. – Visibility. This constraint is related to the problem of obstructions in the environment. Viewpoints affected by occlusions caused by other objects in the workspace, or the object itself, should be avoided if possible. A ray tracing technique (POV-RAY, a free software package) was used in order to obtain visibility information of an object from different viewpoints. We created a database that was then used into our optimization process. 3.2
Secondary Constraints
Optical constraints such as field of view, depth of field, resolution, and image scale will not be taken into account when estimating a favorable imaging geometry. PND is mainly a function of the imaging geometry, as well as the convergence angle. Optical constraints lack significant importance once the camera observes the entire object. In this way, an optimal distance of the camera to the object can be defined a priori in order to measure the different object points. Thus, for the purpose here all object points appear within the field-of-view, in focus, at a given resolution and depth of field. In addition, in order to compute the exterior orientation parameters photogrammetrists affirm that the total number of points is irrelevant once a sufficient number of points are used during the simulation.
4
The Multi-cellular Genetic Algorithm
The multi-cellular genetic algorithm (MGA) then proceeds as follows: 1. An initial random population of N convergent networks that satisfy the environment constraints is chosen and is represented by (αn , βn ), coded into a binary string representation. 2. Next, we evaluate each network, and store the corresponding maximum value of the diagonal of ΛPn for each tree structure. This corresponds to the fitness value which says how good the network is, compared with other solutions in the population P (t).
360
G. Olague
eval( Λ P_N)
eval( Λ P_1)
Cam_1
α _1
β _1
Cam_M
Cam_2
α _2
β _2
α _M
Cam_1
β _M
α _1
β_1
Cam_M
Cam_2
α _2
β _2
α _M
β _M
9 bits 101001010 0 --> 511
Fig. 2. The multi-cellular genetic algorithm is represented by a tree structure composed of a main node where the evaluation process is stored and several leaves corresponding to each camera. All cameras are codified in two parameters (α, β), which correspond to the cells of an artificial being. As network evaluation uses only the cells that satisfy the visibility constraints a combinatorial problem is then involved.
3. Then, we select a population of “good” networks by tournament selection: two networks are selected from P (t) and are compared selecting the best individual according to its fitness, yielding the population P (t + 1). 4. From this population, we recombine the binary strings (αn , βn ) for each camera using the following operations: – Crossover, with a probability1 Pc = 0.7. This operation was implemented using one-cut-point2 . Let the two parents be: αx = [αx1 αx2 αx3 αx4 αx5 αx6 αx7 αx8 αx9 ] , αy = [αy1 αy2 αy3 αy4 αy5 αy6 αy7 αy8 αy9 ] . If they are crossed after the random kth position = 4, the resulting offspring are: αx0 = [αx1 αx2 αx3 αx4 αy5 αy6 αy7 αy8 αy9 ] , αy0 = [αy1 αy2 αy3 αy4 αx5 αx6 αx7 αx8 αx9 ] . – Mutation, with a probability Pm = 0.005. This operation alters one or more genes. Assume that the αy5 = 1 gene of the chromosome αx0 is selected for a mutation. Since the gene is 1, it would be flipped into 0. These operations yield a new population, which we copy into P (t). 5. Steps 2,3 and 4 are repeated until the optimization criterion stabilizes. 1 2
The threshold values associated to Pc and Pm were adopted from to the literature. Due to the classification of the MGA this operation works like a multiple-cut-point.
Autonomous Photogrammetric Network Design
361
Finally, this algorithm minimizes the maximum average variance along the covariance matrix σc2 : (4) f itness = min (max σc2 ) . i=1...N
Thereby, the camera placement Mi relative to the world coordinate frame is optimized. Geometrically, each ΛPi represents a hyper-ellipsoid, which changes its orientation and size as each sensor placement Mi does. Thus, an optimal placement solution is proposed, where the combined uncertainty of all points is minimal.
a) 6 cameras over a plane.
b) Similar to Mason [3].
c) Fraser’s [10] configuration.
d) Multi-robot system.
Configurations produced by our evolutionary system. Fig. 3. Configurations reported in the literature b) and c) were reproduced by EPOCA. Figure a) improves upon Fraser’s configuration due to SOD operation, which is automatically generated. Moreover, EPOCA can be used in the case of complex objects, as can be appreciated from Figure d).
362
5
G. Olague
Examples and Conclusion
We have run a series of experiments to test the validity of our approach. We present select results in Figure 3, which show four configurations designed by EPOCA. The cameras are looking at a set of targets represented by their error ellipsoids aligned in one or two planes, as well as over a complex object. These configurations are a product of our evolutionary system. In fact, within a stochastic optimization process we cannot make conclusions from just one trial. Each configuration presented is the product of about 50 independent runs. Figure 3c illustrates a solution with four cameras looking at a planar surface. This solution is not the standard one used by the expert photogrammetrists: a photogrammetrist usually puts the four cameras at four-corners of a cube whose center contains the targets to be measured. In fact, Fraser [10] has already discussed our configuration; he noticed that this configuration is not atypical. Our experiments confirm Fraser’s statement, hence the equivalence between both configurations [9]. Acknowledgements. This work was supported by CONACyT under project 35267-A. Prof. Roger Mohr contributed to the development of EPOCA with many useful ideas, criticisms and suggestions. I am also grateful to Dr. Scott Mason and Dr. Marc Schoenauer for his helpful comments and interest.
References 1. Grafarend, E.W., 1974, Optimization of Geodetic Networks, Bollettino di Geodesia e Scienze Affini, 33(4):351-406. 2. Fraser, C.S., 1996, Network Design, Close-Range Photogrammetry and Machine Vision, K.B. Atkinson, editor, Whittles Publishing, Chapter 9, pp. 256-281. 3. Mason, S.O. and A. Gr¨ un, 1995, Automatic Sensor Placement for Accurate Dimensional Inspection, Computer Vision and Image Understanding, 61(3):454-467. 4. Mason, S.O., 1997, Heuristic Reasoning Strategy for Automated Sensor Placement, Photogrammetric Engineering & Remote Sensing, 63(9):1093-1102, September. 5. Olague, G., 1998, Planification du placement de cam´eras pour des mesures 3D de pr´ecision, PhD Thesis, Institut National Polytechnique de Grenoble, France. ftp://ftp.imag.fr/pub/Mediatheque.IMAG/theses/98-Olague.Gustavo/notice-francais.html
6. Olague, G. and R. Mohr, 1998, Optimal Camera Placement to Obtain Accurate 3D Point Positions, In Proceedings of the 14th International Conference on Pattern Recognition, Vol. 1, pages 8-10. 7. Brown, D.C., 1980, Application of Close-Range Photogrammetry to Measurements of Structures in Orbit, Vol. 1 and 2, GSI Technical Report No. 80-012, Melbourne. 8. Fraser, C.S., 1987, Limiting Error Propagation in Network Design,Photogrammetric Engineering & Remote Sensing, 53(5):487-493, May. 9. Mason, S.O., 1994, Expert System-Based Design of Photogrammetric Networks, PhD Thesis, Institut f¨ ur Geod´ asie und Photogrammetrie, Zurich. 10. Fraser, C.S., 1982, Optimization of Precision in Close-Range Photogrammetry, Photogrammetric Engineering & Remote Sensing, 48(4):561-570, April.
Autonomous Photogrammetric Network Design
363
11. Olague, G., 2000, Design and Simulation of Photogrammetric Networks using Genetic Algorithms, In American Society for Photogrammetry and Remote Sensing 2000, Annual Conference Proceedings, 12 pages, Washington DC, USA, Copyright.
Abstract. This work describes the use of genetic algorithms for automating the photogrammetric network design process. When planning a photogrammetric network, the cameras should be placed in order to satisfy a set of interrelated and competing constraints. Furthermore, when the object is three-dimensional a combinatorial problem occurs. Genetic algorithms are stochastic optimization techniques, which have proved useful at solving computationally difficult problems with high combinatorial aspects. EPOCA (an acronym for “Evolving POsitions of CAmeras”) has been developed using a three-dimensional CAD interface. EPOCA is a genetic based system that provides the attitude of each camera in the network, taking into account the imaging geometry, as well as several major constraints like visibility, convergence angle, and workspace constraint. EPOCA reproduces configurations reported in the photogrammetric literature. Moreover, the system can design networks for several adjoining planes and complex objects opening interesting new research avenues.
1
Introduction
Photogrammetric network design is the process of placing cameras in order to perform photogrammetric tasks. An important aspect of any close range photogrammetric system is to achieve an optimal spatial distribution of the cameras comprising the network. Planning an optimal photogrammetric network for some special purpose, such as for monitoring structural deformation or for determining the precise shape characteristics of an object demands special attention from the quality of the network design. Previous approaches to photogrammetric network design have attempted to identify the main stages in the process. Following the widely accepted classification scheme of Grafarend [1], network design has been divided into four design stages from which only the first three are used in close-range photogrammetry: 1. Zero Order Design (ZOD): This stage attempts to define an optimal datum in order to obtain accurate object point coordinates and exterior orientation parameters. E.J.W. Boers et al. (Eds.) EvoWorkshop 2001, LNCS 2037, pp. 353–363, 2001. c Springer-Verlag Berlin Heidelberg 2001
354
G. Olague
2. First Order Design (FOD): This stage involves defining an optimal imaging geometry, which in turn determines the accuracy of the system. 3. Second Order Design (SOD): This stage is concerned with adopting a suitable measurement precision for the image coordinates. It consists usually in taking multiple images from each camera station. 4. Third Order Design (TOD): This stage deals with the improvement of a network through the inclusion of additional points in a weak region. Photogrammetric measurement operations attempt to satisfy, in an optimal manner, several objectives such as precision, reliability and economy. The ZOD and SOD are greatly simplified in comparison to geodetic networks for which the four stages were originally developed. Indeed FOD, the design of network configuration or the sensor placement task needs to be comprehensively addressed for photogrammetric projects. This design stage must provide an optimal imaging geometry and convergence angle for each set of points placed over a complex object [2]. Photogrammetrists have acknowledged the degree of expertise needed to carry out a photogrammetric project. For example, Mason and Gr¨ un [3] developed a work called CONSENS that follows the expert system approach and uses multiple cameras in combination with optical triangulation. It outlines a method of overcoming the set of constraints and objectives presented in camera station placement. The method is based on the theory of generic networks, which constitutes compiled expertise, describing an ideal configuration of four camera stations that can be employed to provide a strong imaging geometry for the class of planar network design problems. Complex objects are divided into planes; each plane is evaluated through one of these networks and then connected with some additional cameras with the purpose of establishing just one common datum. However, the expert system approach has shown it unlikely that full automation of the network design process will be achieved, due in large part to human expert’s extensive use of common-sense reasoning [2]. On the other hand, the Grafarend classification just presented serves the photogrammetric user by identifying what set of tasks needs to be implemented in designing a network. Despite the progress that photogrammetrists have made in understanding this design problem, the photogrammetric measurement technique has rarely been applied by other than experienced photogrammetrists. Although its definition seems simple, it reaches a high complexity mainly due to the numerous constraints and design decisions that need to be made. Photogrammetric network design is also difficult to obtain due to the unknown number of configurations all having very similar accuracy, but with a very different imaging geometry. Consequently, photogrammetric network design in many machine vision applications is often conducted in a trial-and-error fashion or using heuristic reasoning strategies [4]. These strategies fail at solving the problem for the case of complex objects. Moreover, the main question, how to obtain an initial configuration with an optimal imaging geometry, is unsolved and left as the responsibility of the designer. The motivation of this research is to reduce the cost of vision system design and to equip autonomous inspection systems with photogrammetric net-
Autonomous Photogrammetric Network Design
355
work capabilities, e.g., measurement robots used in flexible manufacturing, see Figure 1.
Fig. 1. Photogrammetric network simulation of four robots, each camera is mounted on the robot’s hand, with the goal of measuring the box on the table.
Expert photogrammetrists regard simulation as a viable strategy to the problem of photogrammetric network design [2]. Computer simulation of close range photogrammetric networks has been successfully employed and, with the sophistication of computers, a considerable boost to interactive network design has been achieved. The process of photogrammetric network design optimization through computer simulation can follow a number of approaches. One traditional procedure is based on the stages ZOD, FOD and SOD. Given the criteria related to required triangulation precision, the initial step is to adopt a suitable observation and measuring scheme (FOD stage). This entails the selection of an appropriate camera format, focal length, and image measurement system, as well as a first approximation to suitable network geometry. Once this design stage is finished, the network is evaluated against the specified criteria. If the network fails to achieve the criteria, a new stage to diagnose and identify the problem is performed. FOD or SOD will be applied to produce the new solution. If both corrections are insufficient a completely new network will be proposed until a solution to the problem is achieved. In this way, network design is iterative in nature. The aim of this paper is to present a new simulation-based method for solving the most fundamental stage in network design. The problem
356
G. Olague
is set in terms of a global optimization design [5,6], which is capable of managing the problem using an adaptive strategy. It explores the solution space using both non-continuous optimization and combinatorial search. The approach then is to minimize the uncertainty of the three dimensional measurements using as a criterion the average variance of the 3D object points, presuming that the optimization satisfies a number of primary constraints. This paper is organized as follows: first the bundle adjustment, the mathematical model universally accepted by photogrammetrists, is reviewed in order to obtain a criterion useful to the optimization process. Then, a brief summary of the constraints on network design is presented. The problem of photogrammetric network design in terms of a stochastic global optimization is described together with implementation details about visibility and occlusion constraints related to the complexity of the search space. Finally, results are presented followed by a conclusion.
2
Photogrammetric Network Modeling
Brown originally developed the bundle method in a fully general form. Today, the bundle method is recognized as a critical factor in exploiting the mensuration potential of photogrammetry and is almost exclusively used in applications requiring high accuracy. The method accords simultaneous consideration to all sets (or bundles) of photogrammetric rays from all cameras. The bundle method is based on a mathematical camera model comprised of separate functional and stochastic models. The functional model describing the relationship between the desired and measured quantities consists of the well-known collinearity equations. The collinearity equations, derived from the perspective transformation, are based on the fundamental assumption that the perspective center, the ground point and its corresponding image point, all lie on a straight line. For each pair of image coordinates (xij , yij ) observed on each image, the following pair of equations is written: m11 (Xj − Xic ) + m12 (Yj − Yic ) + m13 (Zj − Zic ) xij = xp − f m31 (Xj − Xic ) + m32 (Yj − Yic ) + m33 (Zj − Zic ) yij = yp − f
m21 (Xj − Xic ) + m22 (Yj − Yic ) + m23 (Zj − Zic ) m31 (Xj − Xic ) + m32 (Yj − Yic ) + m33 (Zj − Zic )
(1) ,
where (xij , yij ) denote the coordinates of point j on photograph i, f and (xp , yp ) are the camera constant and image coordinates of the principal point of the sensor defining the sensor’s orientation, (Xj , Yj , Zj ) are the object space coordinates of the corresponding point feature, (Xic , Yic , Zic ) are the object space coordinates of the perspective center, and mkl are elements of an orthogonal matrix which defines the rotation between the image and object coordinate systems. This system of equations assumes that light rays travel in straight lines, that all rays entering a camera lens system pass through a single point and that the lens
Autonomous Photogrammetric Network Design
357
system is distortion-less or, as is usual in highly accurate measurement, that distortion has been cancelled out after having been estimated. Due to the nature of the measurement process, observations are accompanied by errors. Because of random errors, as evidenced by the small differences between observations of the same quantity, observations can be regarded as random variables and their effects described by means of a stochastic model. Equation 1 can be linearized through the first order development using the Taylor series. A functional model can be given as v = Ay − l C1 = σ02 P−1 where l, v and y are the vectors of observations, residuals and unknown parameters, respectively; A is the design or configuration matrix; C1 the covariance matrix of observations; P the weight matrix; and σ02 the variance factor. In situations where A is of full rank (i.e., where redundant or explicit minimal constraints are imposed), the parameter estimates y and the corresponding cofactor matrix Qy and covariance matrix Cy are obtained as
and
y = (AT PA)−1 AT PL = Qy AT PL ,
(2)
Cy = σ02 Qy .
(3)
The ultimate aim of any photogrammetric measurement is the determination of triangulated object point coordinates along with estimates for their precision. The bundle method is simplified by considering two groups of parameters in the vector y ˆ: y1 comprising exterior orientation (self-calibration parameters were not considered for simplicity), and y2 containing object coordinate corrections. Equation 2, then assumes the form −1 T T A1 PA1 AT A PL y1 1 PA2 , = T y2 AT AT PL 2 PA1 A2 PA2 and the cofactor matrix Qy can be written Q1 Q1,2 . Qy = Q2,1 Q2 The design optimization goal for precision is to achieve an optimal form of Q2 and therefore the covariance matrix of object point coordinates (Xj , Yj , Zj ), considering the applicable design constraints. The criterion used in the minimization process was the average variance along the covariance matrix σc2 σc2 =
σ02 (trace Q2 ) . 3n
Before dramatic improvements in computer processing power in recent years, a valid criticism of designing close range networks by simulation was the computation time required for a bundle adjustment after each design-iteration even for
358
G. Olague
relatively small networks. As shown in [7], the covariance matrix can be obtained through the equation −1 + K] , Q2 = σ02 [(AT 2 PA2 )
where and
K = MQ1 MT , −1 T A2 PA1 . M = (AT 2 PA2 )
In this way, the determination of Q2 using this approach represents a rigorous approach that is termed Total Error Propagation (TEP.) On the other hand, it has been demonstrated [8] that for a wide range of convergent photogrammetric networks, K = 0. This consideration is non-rigorous in that it implicitly assumes that exterior orientation parameters exhibit no dispersion and is called Limited Error Propagation (LEP). The perspective parameters are assumed to be error free and the variances in object point coordinates arise solely from the propagation of random errors in the image coordinate measurements. What is remarkable from a network design standpoint is that for strong networks (convergent networks) LEP is sufficiently accurate compared to TEP, causing considerable computation savings.
3
Constraints on Network Design
The problem of photogrammetric network design (PND) must deal with a series of constraints in order to propose an optimal camera distribution. The accuracy of the system is related to the imaging geometry (main objective in PND) as well as the convergence angle of each camera with respect to each object surface. In order to answer the most basic question of a favorable imaging geometry (FOD or the configuration problem) we must distinguish among the several constraints limiting the search space. Mason [9] has proposed a set of constraints and objectives that we separate into two parts: 3.1
Main Objective and Primary Constraints
Considering the constraints limiting the search space we identify the following main objective and three constraints due to the characteristics of the FOD problem: – Contribution to intersection angles or the imaging geometry. Within a camera placement system the main objective is to know the contribution of each camera with respect to the others. Two fundamental questions need to be answered: how many cameras will be needed and where should they be placed. However, before answering the first question we need to answer the second one. Once we know where to place a given number of cameras, it is a trivial matter to decide on the number.
Autonomous Photogrammetric Network Design
359
– Convergence angle. The reliability of image measurements from directions close to coplanar are difficult and even impossible to obtain. The minimum allowable incidence angle is dependent on the type of feature, its geometry and material. The accuracy of the measurement with respect to the convergence angle is a function of the viewing direction and the surface normal at the feature. In the case of circular targets the minimum convergence angle is about 20 to 30 degrees for the kind of retro-reflective targets that are normally used. – Working space constraint. The workspace in which the photogrammetric survey is conducted can impose restrictions on the selection of an ideal imaging geometry. This constraint includes the walls of the room, any obstructions in the working environment, and the workspace of the robot where the camera could be mounted. – Visibility. This constraint is related to the problem of obstructions in the environment. Viewpoints affected by occlusions caused by other objects in the workspace, or the object itself, should be avoided if possible. A ray tracing technique (POV-RAY, a free software package) was used in order to obtain visibility information of an object from different viewpoints. We created a database that was then used into our optimization process. 3.2
Secondary Constraints
Optical constraints such as field of view, depth of field, resolution, and image scale will not be taken into account when estimating a favorable imaging geometry. PND is mainly a function of the imaging geometry, as well as the convergence angle. Optical constraints lack significant importance once the camera observes the entire object. In this way, an optimal distance of the camera to the object can be defined a priori in order to measure the different object points. Thus, for the purpose here all object points appear within the field-of-view, in focus, at a given resolution and depth of field. In addition, in order to compute the exterior orientation parameters photogrammetrists affirm that the total number of points is irrelevant once a sufficient number of points are used during the simulation.
4
The Multi-cellular Genetic Algorithm
The multi-cellular genetic algorithm (MGA) then proceeds as follows: 1. An initial random population of N convergent networks that satisfy the environment constraints is chosen and is represented by (αn , βn ), coded into a binary string representation. 2. Next, we evaluate each network, and store the corresponding maximum value of the diagonal of ΛPn for each tree structure. This corresponds to the fitness value which says how good the network is, compared with other solutions in the population P (t).
360
G. Olague
eval( Λ P_N)
eval( Λ P_1)
Cam_1
α _1
β _1
Cam_M
Cam_2
α _2
β _2
α _M
Cam_1
β _M
α _1
β_1
Cam_M
Cam_2
α _2
β _2
α _M
β _M
9 bits 101001010 0 --> 511
Fig. 2. The multi-cellular genetic algorithm is represented by a tree structure composed of a main node where the evaluation process is stored and several leaves corresponding to each camera. All cameras are codified in two parameters (α, β), which correspond to the cells of an artificial being. As network evaluation uses only the cells that satisfy the visibility constraints a combinatorial problem is then involved.
3. Then, we select a population of “good” networks by tournament selection: two networks are selected from P (t) and are compared selecting the best individual according to its fitness, yielding the population P (t + 1). 4. From this population, we recombine the binary strings (αn , βn ) for each camera using the following operations: – Crossover, with a probability1 Pc = 0.7. This operation was implemented using one-cut-point2 . Let the two parents be: αx = [αx1 αx2 αx3 αx4 αx5 αx6 αx7 αx8 αx9 ] , αy = [αy1 αy2 αy3 αy4 αy5 αy6 αy7 αy8 αy9 ] . If they are crossed after the random kth position = 4, the resulting offspring are: αx0 = [αx1 αx2 αx3 αx4 αy5 αy6 αy7 αy8 αy9 ] , αy0 = [αy1 αy2 αy3 αy4 αx5 αx6 αx7 αx8 αx9 ] . – Mutation, with a probability Pm = 0.005. This operation alters one or more genes. Assume that the αy5 = 1 gene of the chromosome αx0 is selected for a mutation. Since the gene is 1, it would be flipped into 0. These operations yield a new population, which we copy into P (t). 5. Steps 2,3 and 4 are repeated until the optimization criterion stabilizes. 1 2
The threshold values associated to Pc and Pm were adopted from to the literature. Due to the classification of the MGA this operation works like a multiple-cut-point.
Autonomous Photogrammetric Network Design
361
Finally, this algorithm minimizes the maximum average variance along the covariance matrix σc2 : (4) f itness = min (max σc2 ) . i=1...N
Thereby, the camera placement Mi relative to the world coordinate frame is optimized. Geometrically, each ΛPi represents a hyper-ellipsoid, which changes its orientation and size as each sensor placement Mi does. Thus, an optimal placement solution is proposed, where the combined uncertainty of all points is minimal.
a) 6 cameras over a plane.
b) Similar to Mason [3].
c) Fraser’s [10] configuration.
d) Multi-robot system.
Configurations produced by our evolutionary system. Fig. 3. Configurations reported in the literature b) and c) were reproduced by EPOCA. Figure a) improves upon Fraser’s configuration due to SOD operation, which is automatically generated. Moreover, EPOCA can be used in the case of complex objects, as can be appreciated from Figure d).
362
5
G. Olague
Examples and Conclusion
We have run a series of experiments to test the validity of our approach. We present select results in Figure 3, which show four configurations designed by EPOCA. The cameras are looking at a set of targets represented by their error ellipsoids aligned in one or two planes, as well as over a complex object. These configurations are a product of our evolutionary system. In fact, within a stochastic optimization process we cannot make conclusions from just one trial. Each configuration presented is the product of about 50 independent runs. Figure 3c illustrates a solution with four cameras looking at a planar surface. This solution is not the standard one used by the expert photogrammetrists: a photogrammetrist usually puts the four cameras at four-corners of a cube whose center contains the targets to be measured. In fact, Fraser [10] has already discussed our configuration; he noticed that this configuration is not atypical. Our experiments confirm Fraser’s statement, hence the equivalence between both configurations [9]. Acknowledgements. This work was supported by CONACyT under project 35267-A. Prof. Roger Mohr contributed to the development of EPOCA with many useful ideas, criticisms and suggestions. I am also grateful to Dr. Scott Mason and Dr. Marc Schoenauer for his helpful comments and interest.
References 1. Grafarend, E.W., 1974, Optimization of Geodetic Networks, Bollettino di Geodesia e Scienze Affini, 33(4):351-406. 2. Fraser, C.S., 1996, Network Design, Close-Range Photogrammetry and Machine Vision, K.B. Atkinson, editor, Whittles Publishing, Chapter 9, pp. 256-281. 3. Mason, S.O. and A. Gr¨ un, 1995, Automatic Sensor Placement for Accurate Dimensional Inspection, Computer Vision and Image Understanding, 61(3):454-467. 4. Mason, S.O., 1997, Heuristic Reasoning Strategy for Automated Sensor Placement, Photogrammetric Engineering & Remote Sensing, 63(9):1093-1102, September. 5. Olague, G., 1998, Planification du placement de cam´eras pour des mesures 3D de pr´ecision, PhD Thesis, Institut National Polytechnique de Grenoble, France. ftp://ftp.imag.fr/pub/Mediatheque.IMAG/theses/98-Olague.Gustavo/notice-francais.html
6. Olague, G. and R. Mohr, 1998, Optimal Camera Placement to Obtain Accurate 3D Point Positions, In Proceedings of the 14th International Conference on Pattern Recognition, Vol. 1, pages 8-10. 7. Brown, D.C., 1980, Application of Close-Range Photogrammetry to Measurements of Structures in Orbit, Vol. 1 and 2, GSI Technical Report No. 80-012, Melbourne. 8. Fraser, C.S., 1987, Limiting Error Propagation in Network Design,Photogrammetric Engineering & Remote Sensing, 53(5):487-493, May. 9. Mason, S.O., 1994, Expert System-Based Design of Photogrammetric Networks, PhD Thesis, Institut f¨ ur Geod´ asie und Photogrammetrie, Zurich. 10. Fraser, C.S., 1982, Optimization of Precision in Close-Range Photogrammetry, Photogrammetric Engineering & Remote Sensing, 48(4):561-570, April.
Autonomous Photogrammetric Network Design
363
11. Olague, G., 2000, Design and Simulation of Photogrammetric Networks using Genetic Algorithms, In American Society for Photogrammetry and Remote Sensing 2000, Annual Conference Proceedings, 12 pages, Washington DC, USA, Copyright.
