Dynamic fire 3D modeling using a real-time ... - Semantic Scholar
Oct. 2009, Volume 6, No.10 (Serial No.59)
Journal of Communication and Computer, ISSN 1548-7709, USA
Dynamic fire 3D modeling using a real-time stereovision system Rossi Lucile1, Akhloufi Moulay2, Tison Yves1 (1. UMR CNRS SPE 6134, University of Corsica, 20250 Corte, France; 2. CRVI, 205, rue Mgr-Bourget, Lévis (Québec), G6V 6Z9, Canada) Abstract: This work presents a new framework for three-dimensional modeling of dynamic fires present in unstructured scenes. The proposed approach addresses the problem of segmenting fire regions using information from YUV and RGB color spaces. Clustering is also used to extract salient points from a pair of stereo images. These points are then used to reconstruct 3D positions in the scene. A matching strategy is proposed to deal with mismatches due to occlusions and missing data. The obtained data are fitted in a 3D ellipsoid in order to model the enclosing fire volume. This form is then used to compute dynamic fire characteristics like its position, dimension, orientation, heading direction, etc. These results are of great importance for fire behavior monitoring and prediction. Key words: segmentation; stereovision; fire; 3D modeling
1. Introduction Research in the field of fire needs tools that permit to track fire characteristics over time. With the obtained results, we can compare the theoretical models to the situation in the ground. Also, in many real situations information about fire, like its position, orientation, height and volume, are crucial for firefighting. Examples of these situations are: compartments fire[1], propagation fire[2], oil platform fire[3]. For more than two decades, visual and infrared cameras have been used as complementary metrological instruments in fire and flame experiments[4-8]. Vision systems are now capable to reconstruct a 3D turbulent flame and its front structure when the flame is the only density field in the images[9, Corresponding author: Rossi Lucile, Ph. D.; research fields: image processing, stereovision. Akhloufi Moulay, MscA.; research fields: image processing, stereovision. Tison Yves, MscA.; research field: computer science. 54
10]
. Image processing methods are also applied to follow forest fire properties like the rate of spread of flame fronts, fire base contour, flame orientation and maximum flame height[11-14]. These techniques extract information from a scene using multiple viewpoints (in general, frontal and lateral views) and synthesize the data in subsequent steps. This type of processing cannot be used in all experimental situations. In computer vision, many works has been done for the reconstruction of 3D object shape. However, the majority of these works deal with rigid objects[15-17]. Little work has been done for modeling 3D non rigid objects and many hypotheses are made in order to achieve an acceptable 3D reconstruction in this case[18-20]. The lack of a framework for 3D reconstruction of complex fire fronts is due to the dynamic and random nature of the fire which makes it difficult for vision systems to efficiently locate and match salient points. In this paper, we present a new framework for a three dimensional modeling of dynamic fires. This framework is robust to outdoor unstructured conditions. It permits the extraction of important 3D data and models the fire by means of a 3D ellipsoid. This later shape permits the extraction of important fire characteristics, like its dimensions, angles and volume.
2. Proposed approach The proposed framework uses stereovision for three-dimensional modeling of parts extracted from fire regions. In order to build 3D data from stereo, we need to extract corresponding points in the left and
Dynamic fire 3D modeling using a real-time stereovision system
right images of the stereo pair. The steps involved in the proposed approach are shown in Fig. 1 and are: (1) Segmentation of fire images in order to extract fire regions. (2) Features detection algorithm for extracting salient points from the segmented regions. (3) Best features selection using correlation based matching strategy. This step permits the refinement and selection of salient points and the construction of a set of corresponding points. (4) Computation of three-dimensional fire points using stereo correspondence. (5) 3D Ellipsoid fit for volume reconstruction and fire characteristics computation.
Fig. 1 Framework for 3D modeling of flame
3. Vision framework dimensional modeling of fire 3.1 Stereo vision
for
three
Since our objective is to model three-dimensional shapes of fires in operational scenarios, we need a system than can be deployed quickly and efficiently in unstructured scenes. A stereo camera pair is the best choice in this case. A stereo vision system relies on epipolar geometry constraints to derive the 3D structure form 2D correspondences[21-24]. Stereo triangulation permits to extract the 3D coordinates of a point form its 2D projections in two or more images. 3D position is depicted by stereo disparity, defined by the difference in the position of the projected point in two images. In order to compute the 3D position from 2D correspondences, the system must be calibrated. Calibration allows the calculation of cameras intrinsic and extrinsic parameters. Since calibration can be a difficult task in an outdoor scenario, we chose to use a pre-calibrated camera. With this type of camera, there is no need for calibration. A Point Grey XB3 Trinocular stereo system[25] was used in our experiments. In stereo processing, image rectification is an important step. Image rectification permits to align epipolar lines in the two stereo images. Corresponding point in the right image are then located within the line passing through the same “Y” coordinate as their left counterpart[21-23]. In order to derive the 3D data from the obtained images, we need to find corresponding points in the left and right images. Since fire is of dynamic and random nature, we have developed a new approach to extract salient corresponding points present in the two stereo images. The following sections give more details about the new technique. 3.2 Two levels segmentation The fire image is characterized by dominant colors in yellow, orange and red intervals. Also, color variations inside the fire flame give rise to homogeneous color regions (Fig. 2). These characteristics will be used in the proposed two-level segmentation technique. The proposed approach is 55
Dynamic fire 3D modeling using a real-time stereovision system
robust and can handle large variations present in unstructured scenes.
We used K-means clustering technique with k = 2 (1 cluster for the fire regions and 1 cluster for the background). Fig. 2 shows the results of this processing.
Fig. 2 Original image
3.2.1 First level segmentation: A combination of YUV and RGB information Work we have conducted using different color spaces for a better segmentation of fire regions in complex unstructured scenes showed that “V” channel of the YUV color system[25-26] is interesting for finding fire regions. However, in outdoor scenes other areas not corresponding to fire can appear close to fire regions (Fig. 3). K-means clustering technique is applied to “V” channel in order to extract the most interesting areas corresponding to fire. K-means clustering is used to find k partitions based on a set of attributes. It permits to find iteratively the centers of natural clusters in the data. It assumes that the object attributes form a vector space. The objective of the algorithm is to minimize total intra-cluster variance: k
V =∑
∑x
i =1 x j ∈S i
j
− mi
2
(1)
Where: k is the number of clusters; S i are the
clusters i ∈ {1, K , k } ; mi is the centroid of all the points x j ∈ S i .
56
Fig. 3 “V” channel of YUV fire image
Assuming that fire region corresponds to the bigger extracted zone in the clustered “V” channel, we used a learning strategy in the RGB color space to compute a reference model for color classification of fire pixels. We use a 3D Gaussian model to represent the pixels present within the fire zone. The 3D Gaussian model is defined by the following mean and standard deviation:
m = ( m R , mG , m B )
σ = max(σ R ,σ G ,σ B ) Where: mi is the mean for channel I;
(2)
σ i is the
standard deviation for channel I; i ∈ {R, G , B}
The pixels present in the white clustered “V” channel are then verified in the RGB image in order to see if their colors are close to the reference fire color. A pixel is classified based on the model learned from a reference area. A pixel is represented by a 3D vector defined by its color components: p = ( p R , p G , p B ) . It is classified using the following formulation:
Dynamic fire 3D modeling using a real-time stereovision system
⎧⎪ p − m ≤ k × σ ⎨ ⎪⎩ Otherwise
z ∈ Fire z ∉ Fire
(3)
Where: p − m = [( p R −m R ) 2 + ( p G −mG ) 2 + ( p B − m B ) 2 ]1 / 2
k : is a constant. The result of this first segmentation is a fire region as shown in Fig. 4.
to extract the interior fire regions. Fig. 5 shows the result of clustering the fire image in 4 clusters (3 clusters for the fire regions and 1 cluster for the background) 3.3 Detection of feature points 3.3.1 Contour extraction
(a) Global contour of the fire region; Fig. 4 Extracted fire region
3.2.2 Second level clustering and segmentation
(b) Interior fire regions contours Fig. 6 Contour extraction
Fig. 5 Clustering of interior fire regions
After the first level segmentation we obtain a fire region with different homogeneous color zones. A second level segmentation is then performed in the resulting image. K-means clustering technique is used
The obtained image in the first level segmentation is used to extract the global contour of the fire region. The extracted region is binarized. The obtained binary image highlights abrupt discontinuities present in the image. A postprocessing step based on mathematical morphology is then conducted in order to eliminate spurious pixels, such as residual burning embers, and to 57
Dynamic fire 3D modeling using a real-time stereovision system
correct imperfect segmentation results like holes that appear in the fire area due to the presence of smoke. The final contour is then obtained using Canny edge detection algorithm. The result is a list of points representing the bordering pixels along the global fire region (Fig. 6 (a)). The two-level segmentation permits the labeling of interior homogeneous color regions. Theses regions are then processed separately in order to extract their contours. A labeled region is extracted in another image and binarized. Canny edge detection algorithm is then applied in order to extract the labeled region contour (Fig. 6 (b)). 3.3.2 Features detection Features like edges and textures are not easily found in fire images. In our approach we use peaks and valleys of the fire contour as feature points. A peak detector is used in order to find local positive and negative inflections along the extracted contour denoted f (x) : ⎧ f ( xi ) < f ( x j ) ⇒ Minima ⎨ ⎩ f ( xi ) > f ( x j ) ⇒ Maxima
j ∈ {i − 1, i + 1} j ∈ {i − 1, i + 1}
(4)
Fig. 7 shows the result obtained with the peak detector: 818 points where extracted from the left image (Fig. 7 (a)) and 830 points from the right image (Fig. 7 (b)).
(a) Left image;
58
(b) Right image Fig. 7 Features detection
3.3.3 Matching and refinement of features selection The previous step permits the extraction of all feature points satisfying our extrema detection criteria. Not all of these points can be matched due to occlusion and local color variations. The matching procedure selects the best features and finds corresponding points in the left and right images. The algorithm uses the following constraints during the matching process: epipolar, order and uniqueness constraints. Also, we have added a disparity constraint that restricts the search in a small interval along the epipolar search line.
(a) (to be continued)
Dynamic fire 3D modeling using a real-time stereovision system
(b) Fig. 8 Refinement of selected features and matching
For each detected feature in the rectified left image, we start a 1D search for its corresponding feature point in a small area along the horizontal line in a rectified right image (epipolar and disparity constraints). The search algorithm uses a normalized cross-correlation pattern matching algorithm in a
33x33 window around each potential corresponding point. The similarity criterion is the correlation score between windows in the two images[20]. When two possible correspondences to the left point are present in the right image and have close matching scores, priority is given to the leftmost unmatched point (order and uniqueness constraints). Fig. 8 shows the remaining points after the matching procedure: 281 corresponding points are detected. 3.4 3D position Once the corresponding features are extracted, a triangulation technique is used to compute their 3D coordinates. A line l is passed through the point in the left image and the left optical center Cl and a line r is passed through the corresponding point in the right image and the right optical center Cr. A mid-point method finds the point which lies exactly at the middle of the shortest line segment which joins the two projection lines. This point represents the 3D coordinate of the corresponding pixels.
Fig. 9 3D position of corresponding points 59
Dynamic fire 3D modeling using a real-time stereovision system
Fig. 9 shows the three dimensional positions of the computed points of flame. 3.5 Ellipsoid modeling From the computed 3D points, we use the Khachiyan’s algorithm[27] to obtain the minimum-volume enclosing ellipsoid. The radii and orientation of the ellipsoid were obtained by decomposing the ellipsoid matrix with the Singular Value Decomposition (SVD). This information was used to approximate the position, dimensions, orientation and volume of the fire shape model. Fig. 10 shows the ellipsoid obtained corresponding to the fire presented in Fig. 10.
Trinocular stereo system). This camera gives perfectly registered images (synchronization time is around 125 μs). The images were captured at full resolution 1280x960. Brightness and white balance were adjusted before acquisition and integration time was set to 0.1ms to avoid successive image averaging. The stereo system was positioned at approximately 12m from the fire front. The height of the fire was approximately 2.5m. The obtained results of fire 3D model are in concordance with these data. Captured images were stocked in RAW format and later processed for segmentation and 3D reconstruction. Image acquisition code was developed in C++ and optimized for real time processing. Image processing was done offline using Matlab. Fig. 11 shows the experimental setup used for capturing fire fronts in outdoor unstructured scenes. The arrow points to the XB3 stereo system. The figure shows also near Infra Red cameras which are used in our experiments. These cameras are part of ongoing research in multi-spectral processing of fire images.
Fig. 10 Ellipsoid modeling of fire
4. Experimental results Experiments were conducted in an operational scenario: outdoor unstructured environment. Tests were conducted outdoors during the day in Corte region (Corsica Island, France). Since the fire has a dynamic and random nature, the acquisition of the two images must be synchronized. We chose to use a pre-calibrated stereo camera form Point Grey (XB3
60
Fig. 11 Experimental setup
5. Conclusion In this work we present a new framework for robust 3D modeling of dynamic fires. The proposed approach deals with various problems arising from the dynamic nature of fires like occlusions, changing colors, etc. A two-level segmentation technique was
Dynamic fire 3D modeling using a real-time stereovision system
introduced. The first level starts with a global segmentation of fire regions from the unstructured scene, and the second level segments the fire image in multiple interior regions of corresponding colors. The two levels use clustering techniques. A new matching strategy is also presented. It handles efficiently mismatches due to occlusions and missing data. In order to obtain fire shape, an ellipsoid modeling based on the obtained 3d data is proposed. From this model, information like position, orientation, dimensions and volume of fire are computed. These data can be used efficiently in operational scenarios for tracking and predicting fire fronts propagation. Future work includes the fusion of different strategies for enhanced detection of feature points and the use of multi-spectral images in order to add more information to the processing steps. Also, of importance is the final rendering of the 3D data. Finally, work will be conducted to optimize the image processing for real time. This step will permit the deployment of the proposed setup in operational scenarios during fire fighting in conjunction with the fire spread models. References: [1] C. Abecassis-Empis et al..Characterisation of dalmarnock fire test one. Experimental Thermal and Fluid Science, 2008. [2] J. H. Balbi., J. L. Rossi, T. Marcelli, P. A. Santoni. A 3D physical real-time model of surface fires across fuel beds. Combustion Science and Technology, 2007,179: 2511-2537. [3] J. Huseynov, S. Baliga, A. Widmer., Z. Boger. 2008 special issue: An adaptive method for industrial hydrocarbon flame detection. Neural Networks, 2008, 21(2-3): 398-405. [4] C. M. Britton, B. L. Karr, F. A. Sneva. Correlation of weather and fuel variables to mesquite damage by fire. Journ. of Range Manag., 1977, 30: 395-397. [5] H. B. Clements. Measuring fire behavior with photography. Photogram, Engineer, and Remote Sensing. 1983, 49(10): 1563-1575. [6] G. Lu, Y. Yan, Y Huang and A. Reed. An intelligent monitoring and control system of combustion flames. Meas. Control, 1999, 32(7): 164--168. [7] D. X. Viegas, M.G.Cruz, L.M. Silva, A.J. Ollero, et al.. Forest fire research and wildland fire safety. Proc. of IV Intern. Conf. on Forest Research and Wildland and Fire Safty Summit, 2002. [8] H. C. Bheemul, G. Lu and Y. Yan. Digital imaging based three-dimensional characterization of flame front
[9]
[10]
[11]
[12]
[13]
[14]
[15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27]
structures in a turbulent flame. IEEE Trans. on Instr. and Measur., 2005, 54: 1073-1078. G. Lu, G. Gilabert and Y. Yan. Three dimensional visualisation and reconstruction of the luminosity distribution of a flame using digital imaging techniques. Journ. of Physics: Conference Series, 2005, 15: 194-200. S. W. Hasinoff and K.N. Kutulakos. Photo-consistent reconstruction of semitransparent scenes by density-sheet decomposition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2007, 29(5): 870-885. J. R. Martinez-de Dios, J. C. André, J. C. Gonçalves, B. Ch. Arrue, A. Ollero and D.X. Viegas. Laboratory fire spread analysis using visual and infrared cameras. Inter. Journ. of Wildland Fire, 2006, 15: 175-186. E. Pastor, A. Àgueda, J. Andrade-Cetto, M. Muñoz, Y. Pérez, E. Planas. Computing the rate of spread of linear flame fronts by thermal image processing. Fire Safety Journal, 2006, 41(8): 569-579. L. Merino, F. Caballero and J. R. Martínez-de Dios, J. Ferruz, A. Ollero. A cooperative perception system for multiple UAVs: Application to automatic detection of forest fire. Journal of Field Robotics, John Wiley, 2006, 23(3): 165 - 184. J. R. Martinez-de Dios, B.C. Arrue, A. Ollero, L. Merino, F. Gomez-Rodriguez. Computer vision techniques for forest fire perception. Image and Vision Computing, 2008, 26(4): 550-562. E. Trucco, A. Verri. Introductory Techniques for 3-D Computer Vision. Prentice Hall, 1998. Faugeras. Three Dimensional Computer Vision. MIT Press, 1993. R. Hartley, A. Zissermann. Multiple View Geometry in Computer Vision. Cambridge University Press, 2nd edition, 2003. Del Bue and L. Agapito. Non-rigid stereo factorization. International Journal of Computer Vision (IJCV), 2006, 66 (2): 193-207. Del Bue, X. Llado and L. Agapito. Non-rigid face modelling using shape priors. Analysis and Modelling of Faces and Gestures (AMFG 2005), Springer LNCS, 3723. Del Bue, L. Agapito. Non-rigid 3D shape recovery using stereo factorization. Asian Conference of Computer Vision (ACCV2004), 2004, 1: 25-30. N Ayache, P.T. Sander. Artificial Vision for Mobile Robots: Stereo Vision and Multisensory Perception. MIT Press, 1991. R. Hartley. Stereo from uncalibrated cameras. Proc. of the Second Joint European–US Workshop on Applications of Invariance in Computer Vision, 1994, pp. 237-256. Z. Zhang. Determining the Epipolar geometry and its uncertainty: A review. International Journal of Computer Vision, 1998, 27(2): 161-198. Point Grey Research. http://www.ptgrey.com. S. Westland and A. Ripamonti, Computational Colour Science. John Wiley, 2004. Tremeau, C. Fernandez-Maloigne, and P. Bonton, Image Numérique Couleur, Dunod, 2004. M. J. Todd, and E. A. Yıldırım. On Khachiyan’s algorithm for the computation of minimum-volume enclosing ellipsoids. Discrete Applied Mathematics, .2007, 155(13): 1731-1744.
(Edited by Jane, Sang)
61
Journal of Communication and Computer, ISSN 1548-7709, USA
Dynamic fire 3D modeling using a real-time stereovision system Rossi Lucile1, Akhloufi Moulay2, Tison Yves1 (1. UMR CNRS SPE 6134, University of Corsica, 20250 Corte, France; 2. CRVI, 205, rue Mgr-Bourget, Lévis (Québec), G6V 6Z9, Canada) Abstract: This work presents a new framework for three-dimensional modeling of dynamic fires present in unstructured scenes. The proposed approach addresses the problem of segmenting fire regions using information from YUV and RGB color spaces. Clustering is also used to extract salient points from a pair of stereo images. These points are then used to reconstruct 3D positions in the scene. A matching strategy is proposed to deal with mismatches due to occlusions and missing data. The obtained data are fitted in a 3D ellipsoid in order to model the enclosing fire volume. This form is then used to compute dynamic fire characteristics like its position, dimension, orientation, heading direction, etc. These results are of great importance for fire behavior monitoring and prediction. Key words: segmentation; stereovision; fire; 3D modeling
1. Introduction Research in the field of fire needs tools that permit to track fire characteristics over time. With the obtained results, we can compare the theoretical models to the situation in the ground. Also, in many real situations information about fire, like its position, orientation, height and volume, are crucial for firefighting. Examples of these situations are: compartments fire[1], propagation fire[2], oil platform fire[3]. For more than two decades, visual and infrared cameras have been used as complementary metrological instruments in fire and flame experiments[4-8]. Vision systems are now capable to reconstruct a 3D turbulent flame and its front structure when the flame is the only density field in the images[9, Corresponding author: Rossi Lucile, Ph. D.; research fields: image processing, stereovision. Akhloufi Moulay, MscA.; research fields: image processing, stereovision. Tison Yves, MscA.; research field: computer science. 54
10]
. Image processing methods are also applied to follow forest fire properties like the rate of spread of flame fronts, fire base contour, flame orientation and maximum flame height[11-14]. These techniques extract information from a scene using multiple viewpoints (in general, frontal and lateral views) and synthesize the data in subsequent steps. This type of processing cannot be used in all experimental situations. In computer vision, many works has been done for the reconstruction of 3D object shape. However, the majority of these works deal with rigid objects[15-17]. Little work has been done for modeling 3D non rigid objects and many hypotheses are made in order to achieve an acceptable 3D reconstruction in this case[18-20]. The lack of a framework for 3D reconstruction of complex fire fronts is due to the dynamic and random nature of the fire which makes it difficult for vision systems to efficiently locate and match salient points. In this paper, we present a new framework for a three dimensional modeling of dynamic fires. This framework is robust to outdoor unstructured conditions. It permits the extraction of important 3D data and models the fire by means of a 3D ellipsoid. This later shape permits the extraction of important fire characteristics, like its dimensions, angles and volume.
2. Proposed approach The proposed framework uses stereovision for three-dimensional modeling of parts extracted from fire regions. In order to build 3D data from stereo, we need to extract corresponding points in the left and
Dynamic fire 3D modeling using a real-time stereovision system
right images of the stereo pair. The steps involved in the proposed approach are shown in Fig. 1 and are: (1) Segmentation of fire images in order to extract fire regions. (2) Features detection algorithm for extracting salient points from the segmented regions. (3) Best features selection using correlation based matching strategy. This step permits the refinement and selection of salient points and the construction of a set of corresponding points. (4) Computation of three-dimensional fire points using stereo correspondence. (5) 3D Ellipsoid fit for volume reconstruction and fire characteristics computation.
Fig. 1 Framework for 3D modeling of flame
3. Vision framework dimensional modeling of fire 3.1 Stereo vision
for
three
Since our objective is to model three-dimensional shapes of fires in operational scenarios, we need a system than can be deployed quickly and efficiently in unstructured scenes. A stereo camera pair is the best choice in this case. A stereo vision system relies on epipolar geometry constraints to derive the 3D structure form 2D correspondences[21-24]. Stereo triangulation permits to extract the 3D coordinates of a point form its 2D projections in two or more images. 3D position is depicted by stereo disparity, defined by the difference in the position of the projected point in two images. In order to compute the 3D position from 2D correspondences, the system must be calibrated. Calibration allows the calculation of cameras intrinsic and extrinsic parameters. Since calibration can be a difficult task in an outdoor scenario, we chose to use a pre-calibrated camera. With this type of camera, there is no need for calibration. A Point Grey XB3 Trinocular stereo system[25] was used in our experiments. In stereo processing, image rectification is an important step. Image rectification permits to align epipolar lines in the two stereo images. Corresponding point in the right image are then located within the line passing through the same “Y” coordinate as their left counterpart[21-23]. In order to derive the 3D data from the obtained images, we need to find corresponding points in the left and right images. Since fire is of dynamic and random nature, we have developed a new approach to extract salient corresponding points present in the two stereo images. The following sections give more details about the new technique. 3.2 Two levels segmentation The fire image is characterized by dominant colors in yellow, orange and red intervals. Also, color variations inside the fire flame give rise to homogeneous color regions (Fig. 2). These characteristics will be used in the proposed two-level segmentation technique. The proposed approach is 55
Dynamic fire 3D modeling using a real-time stereovision system
robust and can handle large variations present in unstructured scenes.
We used K-means clustering technique with k = 2 (1 cluster for the fire regions and 1 cluster for the background). Fig. 2 shows the results of this processing.
Fig. 2 Original image
3.2.1 First level segmentation: A combination of YUV and RGB information Work we have conducted using different color spaces for a better segmentation of fire regions in complex unstructured scenes showed that “V” channel of the YUV color system[25-26] is interesting for finding fire regions. However, in outdoor scenes other areas not corresponding to fire can appear close to fire regions (Fig. 3). K-means clustering technique is applied to “V” channel in order to extract the most interesting areas corresponding to fire. K-means clustering is used to find k partitions based on a set of attributes. It permits to find iteratively the centers of natural clusters in the data. It assumes that the object attributes form a vector space. The objective of the algorithm is to minimize total intra-cluster variance: k
V =∑
∑x
i =1 x j ∈S i
j
− mi
2
(1)
Where: k is the number of clusters; S i are the
clusters i ∈ {1, K , k } ; mi is the centroid of all the points x j ∈ S i .
56
Fig. 3 “V” channel of YUV fire image
Assuming that fire region corresponds to the bigger extracted zone in the clustered “V” channel, we used a learning strategy in the RGB color space to compute a reference model for color classification of fire pixels. We use a 3D Gaussian model to represent the pixels present within the fire zone. The 3D Gaussian model is defined by the following mean and standard deviation:
m = ( m R , mG , m B )
σ = max(σ R ,σ G ,σ B ) Where: mi is the mean for channel I;
(2)
σ i is the
standard deviation for channel I; i ∈ {R, G , B}
The pixels present in the white clustered “V” channel are then verified in the RGB image in order to see if their colors are close to the reference fire color. A pixel is classified based on the model learned from a reference area. A pixel is represented by a 3D vector defined by its color components: p = ( p R , p G , p B ) . It is classified using the following formulation:
Dynamic fire 3D modeling using a real-time stereovision system
⎧⎪ p − m ≤ k × σ ⎨ ⎪⎩ Otherwise
z ∈ Fire z ∉ Fire
(3)
Where: p − m = [( p R −m R ) 2 + ( p G −mG ) 2 + ( p B − m B ) 2 ]1 / 2
k : is a constant. The result of this first segmentation is a fire region as shown in Fig. 4.
to extract the interior fire regions. Fig. 5 shows the result of clustering the fire image in 4 clusters (3 clusters for the fire regions and 1 cluster for the background) 3.3 Detection of feature points 3.3.1 Contour extraction
(a) Global contour of the fire region; Fig. 4 Extracted fire region
3.2.2 Second level clustering and segmentation
(b) Interior fire regions contours Fig. 6 Contour extraction
Fig. 5 Clustering of interior fire regions
After the first level segmentation we obtain a fire region with different homogeneous color zones. A second level segmentation is then performed in the resulting image. K-means clustering technique is used
The obtained image in the first level segmentation is used to extract the global contour of the fire region. The extracted region is binarized. The obtained binary image highlights abrupt discontinuities present in the image. A postprocessing step based on mathematical morphology is then conducted in order to eliminate spurious pixels, such as residual burning embers, and to 57
Dynamic fire 3D modeling using a real-time stereovision system
correct imperfect segmentation results like holes that appear in the fire area due to the presence of smoke. The final contour is then obtained using Canny edge detection algorithm. The result is a list of points representing the bordering pixels along the global fire region (Fig. 6 (a)). The two-level segmentation permits the labeling of interior homogeneous color regions. Theses regions are then processed separately in order to extract their contours. A labeled region is extracted in another image and binarized. Canny edge detection algorithm is then applied in order to extract the labeled region contour (Fig. 6 (b)). 3.3.2 Features detection Features like edges and textures are not easily found in fire images. In our approach we use peaks and valleys of the fire contour as feature points. A peak detector is used in order to find local positive and negative inflections along the extracted contour denoted f (x) : ⎧ f ( xi ) < f ( x j ) ⇒ Minima ⎨ ⎩ f ( xi ) > f ( x j ) ⇒ Maxima
j ∈ {i − 1, i + 1} j ∈ {i − 1, i + 1}
(4)
Fig. 7 shows the result obtained with the peak detector: 818 points where extracted from the left image (Fig. 7 (a)) and 830 points from the right image (Fig. 7 (b)).
(a) Left image;
58
(b) Right image Fig. 7 Features detection
3.3.3 Matching and refinement of features selection The previous step permits the extraction of all feature points satisfying our extrema detection criteria. Not all of these points can be matched due to occlusion and local color variations. The matching procedure selects the best features and finds corresponding points in the left and right images. The algorithm uses the following constraints during the matching process: epipolar, order and uniqueness constraints. Also, we have added a disparity constraint that restricts the search in a small interval along the epipolar search line.
(a) (to be continued)
Dynamic fire 3D modeling using a real-time stereovision system
(b) Fig. 8 Refinement of selected features and matching
For each detected feature in the rectified left image, we start a 1D search for its corresponding feature point in a small area along the horizontal line in a rectified right image (epipolar and disparity constraints). The search algorithm uses a normalized cross-correlation pattern matching algorithm in a
33x33 window around each potential corresponding point. The similarity criterion is the correlation score between windows in the two images[20]. When two possible correspondences to the left point are present in the right image and have close matching scores, priority is given to the leftmost unmatched point (order and uniqueness constraints). Fig. 8 shows the remaining points after the matching procedure: 281 corresponding points are detected. 3.4 3D position Once the corresponding features are extracted, a triangulation technique is used to compute their 3D coordinates. A line l is passed through the point in the left image and the left optical center Cl and a line r is passed through the corresponding point in the right image and the right optical center Cr. A mid-point method finds the point which lies exactly at the middle of the shortest line segment which joins the two projection lines. This point represents the 3D coordinate of the corresponding pixels.
Fig. 9 3D position of corresponding points 59
Dynamic fire 3D modeling using a real-time stereovision system
Fig. 9 shows the three dimensional positions of the computed points of flame. 3.5 Ellipsoid modeling From the computed 3D points, we use the Khachiyan’s algorithm[27] to obtain the minimum-volume enclosing ellipsoid. The radii and orientation of the ellipsoid were obtained by decomposing the ellipsoid matrix with the Singular Value Decomposition (SVD). This information was used to approximate the position, dimensions, orientation and volume of the fire shape model. Fig. 10 shows the ellipsoid obtained corresponding to the fire presented in Fig. 10.
Trinocular stereo system). This camera gives perfectly registered images (synchronization time is around 125 μs). The images were captured at full resolution 1280x960. Brightness and white balance were adjusted before acquisition and integration time was set to 0.1ms to avoid successive image averaging. The stereo system was positioned at approximately 12m from the fire front. The height of the fire was approximately 2.5m. The obtained results of fire 3D model are in concordance with these data. Captured images were stocked in RAW format and later processed for segmentation and 3D reconstruction. Image acquisition code was developed in C++ and optimized for real time processing. Image processing was done offline using Matlab. Fig. 11 shows the experimental setup used for capturing fire fronts in outdoor unstructured scenes. The arrow points to the XB3 stereo system. The figure shows also near Infra Red cameras which are used in our experiments. These cameras are part of ongoing research in multi-spectral processing of fire images.
Fig. 10 Ellipsoid modeling of fire
4. Experimental results Experiments were conducted in an operational scenario: outdoor unstructured environment. Tests were conducted outdoors during the day in Corte region (Corsica Island, France). Since the fire has a dynamic and random nature, the acquisition of the two images must be synchronized. We chose to use a pre-calibrated stereo camera form Point Grey (XB3
60
Fig. 11 Experimental setup
5. Conclusion In this work we present a new framework for robust 3D modeling of dynamic fires. The proposed approach deals with various problems arising from the dynamic nature of fires like occlusions, changing colors, etc. A two-level segmentation technique was
Dynamic fire 3D modeling using a real-time stereovision system
introduced. The first level starts with a global segmentation of fire regions from the unstructured scene, and the second level segments the fire image in multiple interior regions of corresponding colors. The two levels use clustering techniques. A new matching strategy is also presented. It handles efficiently mismatches due to occlusions and missing data. In order to obtain fire shape, an ellipsoid modeling based on the obtained 3d data is proposed. From this model, information like position, orientation, dimensions and volume of fire are computed. These data can be used efficiently in operational scenarios for tracking and predicting fire fronts propagation. Future work includes the fusion of different strategies for enhanced detection of feature points and the use of multi-spectral images in order to add more information to the processing steps. Also, of importance is the final rendering of the 3D data. Finally, work will be conducted to optimize the image processing for real time. This step will permit the deployment of the proposed setup in operational scenarios during fire fighting in conjunction with the fire spread models. References: [1] C. Abecassis-Empis et al..Characterisation of dalmarnock fire test one. Experimental Thermal and Fluid Science, 2008. [2] J. H. Balbi., J. L. Rossi, T. Marcelli, P. A. Santoni. A 3D physical real-time model of surface fires across fuel beds. Combustion Science and Technology, 2007,179: 2511-2537. [3] J. Huseynov, S. Baliga, A. Widmer., Z. Boger. 2008 special issue: An adaptive method for industrial hydrocarbon flame detection. Neural Networks, 2008, 21(2-3): 398-405. [4] C. M. Britton, B. L. Karr, F. A. Sneva. Correlation of weather and fuel variables to mesquite damage by fire. Journ. of Range Manag., 1977, 30: 395-397. [5] H. B. Clements. Measuring fire behavior with photography. Photogram, Engineer, and Remote Sensing. 1983, 49(10): 1563-1575. [6] G. Lu, Y. Yan, Y Huang and A. Reed. An intelligent monitoring and control system of combustion flames. Meas. Control, 1999, 32(7): 164--168. [7] D. X. Viegas, M.G.Cruz, L.M. Silva, A.J. Ollero, et al.. Forest fire research and wildland fire safety. Proc. of IV Intern. Conf. on Forest Research and Wildland and Fire Safty Summit, 2002. [8] H. C. Bheemul, G. Lu and Y. Yan. Digital imaging based three-dimensional characterization of flame front
[9]
[10]
[11]
[12]
[13]
[14]
[15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27]
structures in a turbulent flame. IEEE Trans. on Instr. and Measur., 2005, 54: 1073-1078. G. Lu, G. Gilabert and Y. Yan. Three dimensional visualisation and reconstruction of the luminosity distribution of a flame using digital imaging techniques. Journ. of Physics: Conference Series, 2005, 15: 194-200. S. W. Hasinoff and K.N. Kutulakos. Photo-consistent reconstruction of semitransparent scenes by density-sheet decomposition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2007, 29(5): 870-885. J. R. Martinez-de Dios, J. C. André, J. C. Gonçalves, B. Ch. Arrue, A. Ollero and D.X. Viegas. Laboratory fire spread analysis using visual and infrared cameras. Inter. Journ. of Wildland Fire, 2006, 15: 175-186. E. Pastor, A. Àgueda, J. Andrade-Cetto, M. Muñoz, Y. Pérez, E. Planas. Computing the rate of spread of linear flame fronts by thermal image processing. Fire Safety Journal, 2006, 41(8): 569-579. L. Merino, F. Caballero and J. R. Martínez-de Dios, J. Ferruz, A. Ollero. A cooperative perception system for multiple UAVs: Application to automatic detection of forest fire. Journal of Field Robotics, John Wiley, 2006, 23(3): 165 - 184. J. R. Martinez-de Dios, B.C. Arrue, A. Ollero, L. Merino, F. Gomez-Rodriguez. Computer vision techniques for forest fire perception. Image and Vision Computing, 2008, 26(4): 550-562. E. Trucco, A. Verri. Introductory Techniques for 3-D Computer Vision. Prentice Hall, 1998. Faugeras. Three Dimensional Computer Vision. MIT Press, 1993. R. Hartley, A. Zissermann. Multiple View Geometry in Computer Vision. Cambridge University Press, 2nd edition, 2003. Del Bue and L. Agapito. Non-rigid stereo factorization. International Journal of Computer Vision (IJCV), 2006, 66 (2): 193-207. Del Bue, X. Llado and L. Agapito. Non-rigid face modelling using shape priors. Analysis and Modelling of Faces and Gestures (AMFG 2005), Springer LNCS, 3723. Del Bue, L. Agapito. Non-rigid 3D shape recovery using stereo factorization. Asian Conference of Computer Vision (ACCV2004), 2004, 1: 25-30. N Ayache, P.T. Sander. Artificial Vision for Mobile Robots: Stereo Vision and Multisensory Perception. MIT Press, 1991. R. Hartley. Stereo from uncalibrated cameras. Proc. of the Second Joint European–US Workshop on Applications of Invariance in Computer Vision, 1994, pp. 237-256. Z. Zhang. Determining the Epipolar geometry and its uncertainty: A review. International Journal of Computer Vision, 1998, 27(2): 161-198. Point Grey Research. http://www.ptgrey.com. S. Westland and A. Ripamonti, Computational Colour Science. John Wiley, 2004. Tremeau, C. Fernandez-Maloigne, and P. Bonton, Image Numérique Couleur, Dunod, 2004. M. J. Todd, and E. A. Yıldırım. On Khachiyan’s algorithm for the computation of minimum-volume enclosing ellipsoids. Discrete Applied Mathematics, .2007, 155(13): 1731-1744.
(Edited by Jane, Sang)
61
