Multi-Stage Sensor Fusion for Landmine Detection ... - Semantic Scholar
Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems October 9 - 15, 2006, Beijing, China
Multi-Stage Sensor Fusion for Landmine Detection Svetlana Larionova, Lino Marques and An´ıbal Trac¸a de Almeida Institute of Systems and Robotics University of Coimbra Coimbra, Portugal Email: {sveta,lino,adealmeida}@isr.uc.pt Abstract— This paper proposes a multi-stage approach for landmine detection. It is based on a feature-level data fusion which combines data from several nonspecific sensors. The sensor fusion process is analysed and divided in three stages in order to improve the final result. During the first stage all suspected objects are detected against a background. Then, the detected objects are classified to be a man-made or natural object. And, finally, the landmines are distinguished among the identified manmade objects. The last two stages are described in detail in this paper, demonstrating the advantages for their separation. Classification features, which enable the sensor fusion, are also presented in this work together with an approach for their integration. The proposed ideas are tested using real experimental data obtained from pulsed and continuous metal detectors, infra-red camera and ground penetrating radar.
I. I NTRODUCTION This work is a part of the effort for the automation of humanitarian demining. Manual demining is a time-consuming and a very dangerous task, being still the only reliable solution for the world landmines problem [1]. One of the obstacles for the automated demining is the development of a robust approach for landmine detection including improvement of landmine detection sensors and sensor fusion algorithms. Landmines usually contain some amount of metal (case or only fuse) and appear as a dielectric and thermal anomaly in the ground (metal, plastic or wooden case of a certain shape). The available sensors are nonspecific making the landmine detection to be a challenging pattern recognition task: metal detectors confuse landmines with any metal object, IR sensors detect any thermal anomaly on the ground, and ground penetrating radars sense any dielectric disturbance in the soil. The problem of combining data from such nonspecific sensors by means of sensor fusion is considered in this work. It is assumed that the area is scanned by a scanning device [2] (on a test field) or by a mobile scanning platform [3] (which could be used on a real field). First, the data from sensors are mapped spatially according to the position of the scanning platform, and then they can be processed by the sensor fusion algorithm. It was proposed by the authors in [4] to consider a feature-level sensor fusion for landmine detection when the features are estimated from a Region-Of-Interest (ROI). This allows to consider complex features related to the shape and nature of the object to be detected [5]. The introduction of the ROIs extraction step provides the division of the whole task into two stages:
1-4244-0259-X/06/$20.00 ©2006 IEEE
Sensor data
Objects
Landmines
Background
Other Objects
Fig. 1. Two-step landmine detection proposed in [4] (classes highlighted with gray background can be associated with landmines)
1) detection of all suspected “objects” against a background 2) selection of landmines among the detected “objects” This strategy is shown in Fig 1. The second stage, consisting in distinguishing of landmines from other objects, is analysed in this work in more detail. There are many previous works which consider using of sensor fusion for landmine detection [6], [7], [8], [9], [10], [11], [12], [13]. However, the idea of a multi-stage sensor fusion is not widely utilized. A two-step strategy for landmine detection using handheld detector can be found in [13]. Several works also consider a preprocessing step of ROIs extraction [11], [6], [14], [7]. In this work it is proposed to farther divide the landmine detection stage into two subtasks to improve the results of sensor fusion. The described ideas are tested on the real experimental data obtained from the Multi-Sensor Mine Signature (MsMs) database maintained by the EU Joint Research Center in Ispra, Italy [2]. The database contains multi-sensor data recorded on 21 test fields of 7 different soil types, populated with mine surrogates and other objects. In this work, the data from four sensors were used: Vallon ML 1620C pulsed metal detector (PMD), Foerster Minex 2FD 4.500 continuous metal detector (CMD), AGEMA 570 infrared camera (IR) and experimental ground penetrating radar C scans (GPR). The paper is organized as follows. This section briefly describes the first stage of the sensor fusion strategy which consists in ROIs extraction and association. Section II presents the classification features used for sensor fusion and an approach for their integration. Section III shows the analysis of the classification for landmine detection and the idea of its division. Finally, the experimental results and conclusions are presented in Sections IV and V.
2943
(a) Raw Map
(b) Object Area
(a)
(c) Segmented Map
Fig. 2. Example of ROI maps (high-metal landmine seen by pulsed metal detector): (a) Raw Map, (b) Object Area and (c) Segmented map.
(b)
(c)
(d)
Fig. 4. Sample representation of an object sensed by (a) pulsed metal detector, (b) continuous metal detector, (c) IR camera and (d) ground penetrating radar (only Raw Maps are shown).
never achieve good results if the selected features do not reflect the difference between classes. The feature is a single value which characterizes some property of the entire ROI map (Raw Map, Object Area or Segmented Map). All the ROI maps are used for the estimation of features. The multiplications Raw Map×Object Area and Segmented Map×Object Area are also considered as additional ROI maps (providing an information less affected by the background around the object). (a) Pulsed MD on B6 field [2]
A. Feature Estimation
(b) Continuous MD on A1 field [2]
Fig. 3. Examples of results obtained by the ROIs extraction algorithm applied to the data from MsMs database [2]
A. Object detection The first stage of the proposed sensor fusion process was developed by the authors in [4]. This algorithm allows to distinguish from the spatially mapped sensor data all suspected “objects” against a background. Each detected “object”, also called Region-Of-Interest (ROI), consists of three grid-based maps (Fig. 2): • Raw Map represents spatially mapped raw sensor data. • Object Area is a binary map which shows approximate shape of the “object”. • Segmented Map represents a simpler fingerprint of the “object” obtained as an intermediate result during the detection of the ROI. Having two additional ROI maps provides an increased amount of information about the ROI. Moreover, the multiplication of the Object Area with the other two maps allows to farther eliminate the background around the object, which is used for estimation of the classification features (Section II). Fig. 3 presents sample results of ROIs extraction. The algorithm performs automatically and the ROI is detected online, right after it is fully scanned by the sensors. The ROIs from different sensors are associated together if the distance between them is less than a certain threshold (e.g. 100 mm) implying that they belong to the same “object”. An example of the associated ROIs is shown in Fig. 4. II. C LASSIFICATION F EATURES The choice of the right features for classification is one of the most important steps of sensor fusion. Classification can
2944
In previous works the most used features are statistical measures ([11], [6], [14]), such as mean value inside the ROI, and measures related with simple shapes, like the object radius for IR images in [11], or the parameters of hyperbolas for GPR in [7]. More complex features like entropy, contrast and correlation are considered in [14] for processing of IR data, but the results are difficult to analyse due to the small number of samples in the available data sets. In [15] the possibility to use shape symmetry features for landmine detection is analysed. However, the presented results were obtained only for well controlled conditions. The ROIs extraction algorithm, used as a first stage for sensor fusion in this work, provides additional information about the object (Object Area and Segmented Map) allowing the extraction of complex features related to its shape and nature. As there is no landmine specific sensor, the features which can be used for classification are also not specific for landmine detection. Other applications of pattern recognition (mostly related to computer vision) were investigated in order to reveal classification features which can be adopted for this work. Additionally, several new features were developed. The proposed features fall into three categories: • Based on the absolute sensor value: – statistical measures: mean, standard deviation, skewness, kurtosis – contrast C = max(Ci,j ), where Ci,j is the local contrast Ci,j = |xi,j − (xi−1,j + xi+1,j + xi,j−1 + xi,j+1 )/4| • Related to shape: – size, aspect ratio – vertical skewness of object, horizontal skewness of object, occupied part [5]
– compactness
Sensor 1
Mcmp = where μ00 , μ20 , μ02 are central shape moments – eccentricity √ (μ20 +μ02 )2 +4μ211 , where μ20 , μ02 , μ11 are Mect = μ20 +μ02 central shape moments – circularity
•
The sensor fusion in this work is carried out by using the features, estimated for different sensors, together in one feature list for classification. This feature list can be formed as a simple collection of all features. However, in this case the
detected the object
features
average
average
average
Features list
Classifier
log(Nr ) F D = log(1/r) , where Nr is a number of copies of the object scaled down by ratio r. F D is estimated using a differential box-counting approach [16] – entropy (measure of disorder) H(X) = − x P (x) log P (x), where P (x) is the probability that X is in the state x – GR measure (measure of golden ratio) The Segmented Map is analysed in order to determine how different is the averaged ratio between two neighboring segments from the Golden Ratio as follows: N −1 GRM = i=1 grmi /N , min(Si ,Si−1 ) − φ1 , grmi = max(S i ,Si−1 )
B. Combining of the Features
Sensor 3
missed the object
Common
F = 4πS P 2 , where S is area and P is perimeter of the object Related to the nature of the object: – fractal dimension (measure of self-similarity)
where Si is the size of √ i, N is a number of segment segments and φ = 12 1 + 5 is the golden ratio Some features are calculated using only one ROI map (for example, most shape features use only the Object Area) while the others can be considered both for the Raw Map and the Segmented Map. In the second case one best ROI map was chosen using Mutual information and Hausdorff distance as measures for feature evaluation as proposed in [5]. It is logic to consider that the features based on the absolute sensor value are highly dependent on the current environmental conditions, whereas the features related to the shape and nature may be less affected by them, and thus are preferable to be used for classification. Besides the sensor-based features, there is a possibility to consider the features which reflect relations between sensors (multi-sensor features): distance between ROIs, height/width correlation, image correlation, shape correlation and contour correlation.These features are calculated for a pair of ROIs considering the corresponding maps (e.g. the Object Area). If the object contains more than two ROIs, the feature is averaged over all the existing pairs.
Sensor 2
detected the object
μ00 μ20 +μ02 ,
Fig. 5.
Mixing features
problem of missed features arises because some objects may be not detected by all sensors (for example, a low-metal landmine may be not detected by a metal detector), which can be solved by training several classifiers for all possible combinations of sensors. On the other hand, using of complex features, proposed in this work, allows to assume that the feature value weakly depends on the particular sensor. Therefore, its values obtained for one object from different sensors can be mixed, for example, by averaging (Fig. 5). In this case the confidence in the feature value should increase with the number of sensors detecting the object. Moreover, even if there is only one sensor detecting the object, the feature is present, solving the problem of missed features. Of course, this operation cannot be considered for very simple features like mean or size, but the overall performance of classification using more complex features (like entropy) is good in this case. The experiments show that an optimal strategy would be to combine both ideas and use together the mixed features (when appropriate) and the features calculated for each sensor with several classifiers. The problem of feature selection is out of the scope of this paper, thus, to simplify the analysis here the classification is based on using only the mixed features as shown in Fig. 5 with one Bayesian classifier. This strategy provides reasonable results which allow to compare the presented approaches. III. C LASSIFICATION It was pointed in the previous section that the performance of the classification strongly depends on the used features. However, if the classes are not in principle separable, the right features may not exist at all. In this case a better strategy might be a division of the whole task into several subtasks each of which is better separable in the feature space. The detection of landmines is an ambiguous task because a landmine is specific only thanks to the presence of explosives, which are not detected by the used sensors. All features introduced in the previous section were considered to analyse this issue by using principal component analysis (PCA) [17].
2945
Sensor data
Objects
Man-made Objects
Landmines
Background
Natural Objects
Other Objects
Fig. 7. Three-step landmine detection (classes highlighted with gray background can be associated with landmines)
(a)
Logically the class Man-made Objects is divided between the class Landmines and the part of the class Other Objects, making the classification Landmines/Other Objects challenging. The same result of PCA is shown in Fig. 6(b) now highlighting the classes Man-made Objects and Natural Objects (these principal components provide only 31% of the whole variance in the data). It can be seen that the classes Manmade Objects/Natural Objects are better separated than the previous pair. This suggests the division of the classification process as shown in Fig. 7: after all objects are separated from the background, they are classified to be a man-made or natural, and then the landmines are detected among the man-made objects. The experimental testing of the proposed strategy is provided in the next section. •
IV. E XPERIMENTAL R ESULTS (b) Fig. 6. Principal component analysis of the feature space: (a) Landmines o, Other Objects - x, (b) Man-made Objects - o, Natural Objects - x
Fig. 6(a) shows a feature space produced by the first two principal components. It can be seen that the variance in the data revealed by PCA does not reflect the difference between classes Landmines and Other Objects. Thus, the idea followed in this work is to analyse the possibility to determine a better classification methodology. It is proposed to consider for landmine detection two classification subtasks instead of a single one (Landmines/Other Objects) which was utilized before [4]. Particularly, it is proposed to consider one more classification stage with classes Man-made Objects and Natural Objects. The objective for this division arose from the following problems: •
It was noticed that during the classification Landmines/Other objects most false detections consist in other man-made objects: the classifier is not able to distinguish landmines from other man-made objects.
Data from 9 fields of the experimental database [2] (A1, A2, A3, C1, C2, C3, C4, C5, C7) were used for evaluation of the proposed ideas (data for ground penetrating radar are not available for other fields). To estimate the performance of the ROIs extraction algorithm, the experimental data from pulsed MD and continuous MD were labeled to create a sensor-specific ground-truth map, which contains all the objects seen by a human on the sensor data with adjustable scales. In this experiment the detection rate (number of ground-truth objects detected by at least one ROI divided by their total amount) of 90% and the false alarm rate (area occupied by false detections divided by the total area of background) of 4% were achieved. From the 90% of the detected objects, 47% were detected more than once. This large number of the repeated detections increases the processing time but, on the other hand, most of such objects have weak (or confusing) fingerprints. Therefore, in the case of a landmine several detections may improve the probability to detect it by at least one of the ROIs. Evaluation of the sensor fusion approaches was performed using data from all four sensors. For the training of the sensor fusion algorithms, the data from two fields, one with loamy soil
2946
Fig. 8. Comparison of different classifications (on the training set) in terms of detection rate and false alarm rate (a)
and the other with sandy soil, of the experimental database [2] were taken. The rest of the data were used for the evaluation. In order to confirm the idea of multi-stage classification, a comparison of Receiver-Operating Characteristics (ROCs) for different classifications is shown in Fig. 8. It can be seen that, as expected, the classification with classes Manmade Objects/Natural Objects has better performance than the classification with classes Landmines/Other Objects. Moreover, the classification Landmines/Other Objects performs better when used to distinguish among the Man-made Objects. This suggests that the two-step classification (last two stages in Fig. 7) should perform better. However, in practice, the performance of the two-step classification is degraded because of the following reasons: • A part of the detected Man-made Objects by the first classifier appears as false alarms for landmine detection. • The second classifier Landmines/Other Objects is trained using only Man-made Objects of the training set, but has to deal with false alarms of the first classifier in practice. Finally, the performance of the two-step classification in comparison with the one-step is shown in Fig. 9, where in general the two-step classification outperforms. The proposed multi-stage landmine detection can be also seen as a step-by-step reduction of the false alarm rate. After each stage all objects which fall into the corresponding class (shown with gray background in Fig. 7) can be considered Landmines. Then each subsequent stage improves the false alarm rate of the previous one. Table I shows the changing of the number of false alarms per m2 over the stages of sensor fusion.
(b) Fig. 9. Comparison of the relative performance of one-step and two-step classifications on training (a) and evaluation (b) sets
TABLE I C HANGING OF FALSE ALARMS PER m2 FOR DETECTION RATE 95% OVER THE STAGES OF THE SENSOR FUSION
Two-step classification ROIs extraction FA/m2
V. C ONCLUSION This work proposed a multi-stage sensor fusion strategy for landmine detection (Fig. 7). The paper is particularly focused on the last stage of the sensor fusion process classification - providing a framework for its division into two
2947
12.1
Man-made Objects/ Landmines/ Natural Objects Other Objects 5.4 4.5 One-step classification 7
Fig. 10. Experimental data obtained from an IR sensor using a mobile scanning platform [3] and the results of ROIs extraction. Fig. 11.
classification subtasks which are better separable in the feature space. It is demonstrated with experimental data that the two-step classification with classes Man-made Object/Natural Object and Landmines/Other Objects has better performance than the previously proposed one-step classification (Fig. 9 and Table I). The proposed strategy has an important advantage for the practical implementation of landmine detection. The sensor fusion process can be terminated at any stage according to the quality of the available data considering, first of all, the quality of the data mapping. For example, if the data has low spatial resolution as shown in Fig. 10, it is reasonable to consider only the first stage of the process (ROIs extraction) because there is not enough information for a more sophisticated classification. In this experiment the data were obtained by a mobile scanning platform [3] shown in Fig. 11 and the improvement of its positioning is a part of future work. Similarly, the data may contain enough information only for the classification Man-made Object/Natural Object and provide no further improvement in distinguishing landmines from other man-made objects. The level at which the sensor fusion is terminated determines the false alarm ratio which can be obtained (see Table I): a smaller number of false alarms requires higher quality of the available data. The experimental data used in this work were obtained in well controlled conditions using a precise scanning device [2]. Thus, all the stages of the sensor fusion process could be evaluated. The introduction of the classification Manmade Objects/Natural Objects in this work allows to consider a more general task beyond the landmine detection: detection of all dangerous objects located on a minefield. However, the available experimental data do not unfortunately include other dangerous objects besides the antipersonnel landmines. ACKNOWLEDGMENT This research was supported by the FCT (Portuguese Foundation for Science and Technology). The authors would like to thank Dr. Adam Lewis (JRC - Ispra) for the help in supplying and processing of the GPR data.
2948
Mobile scanning platform for landmine detection [3]
R EFERENCES [1] R. Siegel, “Land mine detection,” IEEE Instrumentation and Measurement Magazine, vol. 5, no. 4, pp. 22–28, 2002. [2] “Joint Multi-Sensor Mine Signature Database (Joint Research Centre Ispra),” http://demining.jrc.it/msms/. [3] M. Y. Rachkov, L. Marques, and A. T. de Almeida, “Multisensor demining robot,” Autonomous Robots, no. 18, pp. 275–291, 2005. [4] S. Larionova, L. Marques, and A. de Almeida, “Toward practical implementation of sensor fusion for a demining robot,” in Int. Conf. on Intelligent Robots and Systems, IROS, 2004. [5] ——, “Features selection for sensor fusion in a demining robot,” in Int. Conf. on Robotics and Automation, ICRA, 2005. [6] F. Cremer, J. Schavemaker, W. Jong, and K. Schutte, “Comparison of vehicle-mounted forward-looking polarimetric infrared and downwardlooking infrared sensors for landmine detection,” in Detection and Remediation Technologies for Mines and Minelike Targets - SPIE, 2003. [7] H. Frigui, P. Gader, and J. M. Keller, “Fuzzy clustering for land mine detection,” in NAFIPS, 1998. [8] A. Gunatilaka and B. Baertlein, “Comparison of pre-detection and postdetection fusion for mine detection,” in Detection and Remediation Technologies for Mines and Minelike Targets - SPIE, vol. 3710, 1999, pp. 1212–1223. [9] D. W. McMichael, “Data fusion for vehicle-borne mine detection,” in Int. Conf. on the Detection of Abandoned Land Mines, 1998, pp. 167–171. [10] W. Messelink, K. Schutte, A. Vossepoel, F. Cremer, J. Schavemaker, and E. Breejen, “Feature-based detection of landmines in infrared images,” in Detection and Remediation Technologies for Mines and Minelike Targets - SPIE, vol. 4742, 2002, pp. 108–119. [11] M. Roughan and D. W. McMichael, “A comparison of methods of data fusion for land-mine detection,” in Int. Workshop on Image Analysis and Information Fusion, 1997. [12] S. Perrin, E. Duflos, P. Vanheeghe, and A. Bibaut, “Multisensor fusion in the frame of evidence theory for landmines detection,” IEEE Trans. on Systems, Man and Cybernetics, Part C, 2004. [13] K. C. Ho, L. M. Collins, L. G. Huettel, and P. D. Gader, “Discrimination mode processing for EMI and GPR sensors for hand-held land mine detection,” IEEE Trans. on Geoscience and Remote Sensing, vol. 42, no. 1, pp. 249–263, 2004. [14] G. A. Clark, S. K. Sengupta, D. Aimonetti, F. Roeske, and J. G. Donetti, “Multispectral image feature selection for land mine detection,” IEEE Trans. on Geoscience and Remote Sensing, vol. 38, no. 1, pp. 304–311, 2000. [15] J. M. Stiles, A. V. Apte, and B. Beh, “A group-theoretic analysis of symmetric target scattering with application to landmine detection,” IEEE Trans. on Geoscience and Remote Sensing, vol. 40, no. 8, pp. 1802–1814, 2002. [16] N. Sarkar and B. B. Chaudhuri, “An efficient differential box-counting approach to compute fractal dimension of image,” IEEE Trans. on Systems, Man, and Cybernetics, vol. 24, no. 1, pp. 115–120, 1994. [17] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. WileyInterscience Publication, 2000.
Multi-Stage Sensor Fusion for Landmine Detection Svetlana Larionova, Lino Marques and An´ıbal Trac¸a de Almeida Institute of Systems and Robotics University of Coimbra Coimbra, Portugal Email: {sveta,lino,adealmeida}@isr.uc.pt Abstract— This paper proposes a multi-stage approach for landmine detection. It is based on a feature-level data fusion which combines data from several nonspecific sensors. The sensor fusion process is analysed and divided in three stages in order to improve the final result. During the first stage all suspected objects are detected against a background. Then, the detected objects are classified to be a man-made or natural object. And, finally, the landmines are distinguished among the identified manmade objects. The last two stages are described in detail in this paper, demonstrating the advantages for their separation. Classification features, which enable the sensor fusion, are also presented in this work together with an approach for their integration. The proposed ideas are tested using real experimental data obtained from pulsed and continuous metal detectors, infra-red camera and ground penetrating radar.
I. I NTRODUCTION This work is a part of the effort for the automation of humanitarian demining. Manual demining is a time-consuming and a very dangerous task, being still the only reliable solution for the world landmines problem [1]. One of the obstacles for the automated demining is the development of a robust approach for landmine detection including improvement of landmine detection sensors and sensor fusion algorithms. Landmines usually contain some amount of metal (case or only fuse) and appear as a dielectric and thermal anomaly in the ground (metal, plastic or wooden case of a certain shape). The available sensors are nonspecific making the landmine detection to be a challenging pattern recognition task: metal detectors confuse landmines with any metal object, IR sensors detect any thermal anomaly on the ground, and ground penetrating radars sense any dielectric disturbance in the soil. The problem of combining data from such nonspecific sensors by means of sensor fusion is considered in this work. It is assumed that the area is scanned by a scanning device [2] (on a test field) or by a mobile scanning platform [3] (which could be used on a real field). First, the data from sensors are mapped spatially according to the position of the scanning platform, and then they can be processed by the sensor fusion algorithm. It was proposed by the authors in [4] to consider a feature-level sensor fusion for landmine detection when the features are estimated from a Region-Of-Interest (ROI). This allows to consider complex features related to the shape and nature of the object to be detected [5]. The introduction of the ROIs extraction step provides the division of the whole task into two stages:
1-4244-0259-X/06/$20.00 ©2006 IEEE
Sensor data
Objects
Landmines
Background
Other Objects
Fig. 1. Two-step landmine detection proposed in [4] (classes highlighted with gray background can be associated with landmines)
1) detection of all suspected “objects” against a background 2) selection of landmines among the detected “objects” This strategy is shown in Fig 1. The second stage, consisting in distinguishing of landmines from other objects, is analysed in this work in more detail. There are many previous works which consider using of sensor fusion for landmine detection [6], [7], [8], [9], [10], [11], [12], [13]. However, the idea of a multi-stage sensor fusion is not widely utilized. A two-step strategy for landmine detection using handheld detector can be found in [13]. Several works also consider a preprocessing step of ROIs extraction [11], [6], [14], [7]. In this work it is proposed to farther divide the landmine detection stage into two subtasks to improve the results of sensor fusion. The described ideas are tested on the real experimental data obtained from the Multi-Sensor Mine Signature (MsMs) database maintained by the EU Joint Research Center in Ispra, Italy [2]. The database contains multi-sensor data recorded on 21 test fields of 7 different soil types, populated with mine surrogates and other objects. In this work, the data from four sensors were used: Vallon ML 1620C pulsed metal detector (PMD), Foerster Minex 2FD 4.500 continuous metal detector (CMD), AGEMA 570 infrared camera (IR) and experimental ground penetrating radar C scans (GPR). The paper is organized as follows. This section briefly describes the first stage of the sensor fusion strategy which consists in ROIs extraction and association. Section II presents the classification features used for sensor fusion and an approach for their integration. Section III shows the analysis of the classification for landmine detection and the idea of its division. Finally, the experimental results and conclusions are presented in Sections IV and V.
2943
(a) Raw Map
(b) Object Area
(a)
(c) Segmented Map
Fig. 2. Example of ROI maps (high-metal landmine seen by pulsed metal detector): (a) Raw Map, (b) Object Area and (c) Segmented map.
(b)
(c)
(d)
Fig. 4. Sample representation of an object sensed by (a) pulsed metal detector, (b) continuous metal detector, (c) IR camera and (d) ground penetrating radar (only Raw Maps are shown).
never achieve good results if the selected features do not reflect the difference between classes. The feature is a single value which characterizes some property of the entire ROI map (Raw Map, Object Area or Segmented Map). All the ROI maps are used for the estimation of features. The multiplications Raw Map×Object Area and Segmented Map×Object Area are also considered as additional ROI maps (providing an information less affected by the background around the object). (a) Pulsed MD on B6 field [2]
A. Feature Estimation
(b) Continuous MD on A1 field [2]
Fig. 3. Examples of results obtained by the ROIs extraction algorithm applied to the data from MsMs database [2]
A. Object detection The first stage of the proposed sensor fusion process was developed by the authors in [4]. This algorithm allows to distinguish from the spatially mapped sensor data all suspected “objects” against a background. Each detected “object”, also called Region-Of-Interest (ROI), consists of three grid-based maps (Fig. 2): • Raw Map represents spatially mapped raw sensor data. • Object Area is a binary map which shows approximate shape of the “object”. • Segmented Map represents a simpler fingerprint of the “object” obtained as an intermediate result during the detection of the ROI. Having two additional ROI maps provides an increased amount of information about the ROI. Moreover, the multiplication of the Object Area with the other two maps allows to farther eliminate the background around the object, which is used for estimation of the classification features (Section II). Fig. 3 presents sample results of ROIs extraction. The algorithm performs automatically and the ROI is detected online, right after it is fully scanned by the sensors. The ROIs from different sensors are associated together if the distance between them is less than a certain threshold (e.g. 100 mm) implying that they belong to the same “object”. An example of the associated ROIs is shown in Fig. 4. II. C LASSIFICATION F EATURES The choice of the right features for classification is one of the most important steps of sensor fusion. Classification can
2944
In previous works the most used features are statistical measures ([11], [6], [14]), such as mean value inside the ROI, and measures related with simple shapes, like the object radius for IR images in [11], or the parameters of hyperbolas for GPR in [7]. More complex features like entropy, contrast and correlation are considered in [14] for processing of IR data, but the results are difficult to analyse due to the small number of samples in the available data sets. In [15] the possibility to use shape symmetry features for landmine detection is analysed. However, the presented results were obtained only for well controlled conditions. The ROIs extraction algorithm, used as a first stage for sensor fusion in this work, provides additional information about the object (Object Area and Segmented Map) allowing the extraction of complex features related to its shape and nature. As there is no landmine specific sensor, the features which can be used for classification are also not specific for landmine detection. Other applications of pattern recognition (mostly related to computer vision) were investigated in order to reveal classification features which can be adopted for this work. Additionally, several new features were developed. The proposed features fall into three categories: • Based on the absolute sensor value: – statistical measures: mean, standard deviation, skewness, kurtosis – contrast C = max(Ci,j ), where Ci,j is the local contrast Ci,j = |xi,j − (xi−1,j + xi+1,j + xi,j−1 + xi,j+1 )/4| • Related to shape: – size, aspect ratio – vertical skewness of object, horizontal skewness of object, occupied part [5]
– compactness
Sensor 1
Mcmp = where μ00 , μ20 , μ02 are central shape moments – eccentricity √ (μ20 +μ02 )2 +4μ211 , where μ20 , μ02 , μ11 are Mect = μ20 +μ02 central shape moments – circularity
•
The sensor fusion in this work is carried out by using the features, estimated for different sensors, together in one feature list for classification. This feature list can be formed as a simple collection of all features. However, in this case the
detected the object
features
average
average
average
Features list
Classifier
log(Nr ) F D = log(1/r) , where Nr is a number of copies of the object scaled down by ratio r. F D is estimated using a differential box-counting approach [16] – entropy (measure of disorder) H(X) = − x P (x) log P (x), where P (x) is the probability that X is in the state x – GR measure (measure of golden ratio) The Segmented Map is analysed in order to determine how different is the averaged ratio between two neighboring segments from the Golden Ratio as follows: N −1 GRM = i=1 grmi /N , min(Si ,Si−1 ) − φ1 , grmi = max(S i ,Si−1 )
B. Combining of the Features
Sensor 3
missed the object
Common
F = 4πS P 2 , where S is area and P is perimeter of the object Related to the nature of the object: – fractal dimension (measure of self-similarity)
where Si is the size of √ i, N is a number of segment segments and φ = 12 1 + 5 is the golden ratio Some features are calculated using only one ROI map (for example, most shape features use only the Object Area) while the others can be considered both for the Raw Map and the Segmented Map. In the second case one best ROI map was chosen using Mutual information and Hausdorff distance as measures for feature evaluation as proposed in [5]. It is logic to consider that the features based on the absolute sensor value are highly dependent on the current environmental conditions, whereas the features related to the shape and nature may be less affected by them, and thus are preferable to be used for classification. Besides the sensor-based features, there is a possibility to consider the features which reflect relations between sensors (multi-sensor features): distance between ROIs, height/width correlation, image correlation, shape correlation and contour correlation.These features are calculated for a pair of ROIs considering the corresponding maps (e.g. the Object Area). If the object contains more than two ROIs, the feature is averaged over all the existing pairs.
Sensor 2
detected the object
μ00 μ20 +μ02 ,
Fig. 5.
Mixing features
problem of missed features arises because some objects may be not detected by all sensors (for example, a low-metal landmine may be not detected by a metal detector), which can be solved by training several classifiers for all possible combinations of sensors. On the other hand, using of complex features, proposed in this work, allows to assume that the feature value weakly depends on the particular sensor. Therefore, its values obtained for one object from different sensors can be mixed, for example, by averaging (Fig. 5). In this case the confidence in the feature value should increase with the number of sensors detecting the object. Moreover, even if there is only one sensor detecting the object, the feature is present, solving the problem of missed features. Of course, this operation cannot be considered for very simple features like mean or size, but the overall performance of classification using more complex features (like entropy) is good in this case. The experiments show that an optimal strategy would be to combine both ideas and use together the mixed features (when appropriate) and the features calculated for each sensor with several classifiers. The problem of feature selection is out of the scope of this paper, thus, to simplify the analysis here the classification is based on using only the mixed features as shown in Fig. 5 with one Bayesian classifier. This strategy provides reasonable results which allow to compare the presented approaches. III. C LASSIFICATION It was pointed in the previous section that the performance of the classification strongly depends on the used features. However, if the classes are not in principle separable, the right features may not exist at all. In this case a better strategy might be a division of the whole task into several subtasks each of which is better separable in the feature space. The detection of landmines is an ambiguous task because a landmine is specific only thanks to the presence of explosives, which are not detected by the used sensors. All features introduced in the previous section were considered to analyse this issue by using principal component analysis (PCA) [17].
2945
Sensor data
Objects
Man-made Objects
Landmines
Background
Natural Objects
Other Objects
Fig. 7. Three-step landmine detection (classes highlighted with gray background can be associated with landmines)
(a)
Logically the class Man-made Objects is divided between the class Landmines and the part of the class Other Objects, making the classification Landmines/Other Objects challenging. The same result of PCA is shown in Fig. 6(b) now highlighting the classes Man-made Objects and Natural Objects (these principal components provide only 31% of the whole variance in the data). It can be seen that the classes Manmade Objects/Natural Objects are better separated than the previous pair. This suggests the division of the classification process as shown in Fig. 7: after all objects are separated from the background, they are classified to be a man-made or natural, and then the landmines are detected among the man-made objects. The experimental testing of the proposed strategy is provided in the next section. •
IV. E XPERIMENTAL R ESULTS (b) Fig. 6. Principal component analysis of the feature space: (a) Landmines o, Other Objects - x, (b) Man-made Objects - o, Natural Objects - x
Fig. 6(a) shows a feature space produced by the first two principal components. It can be seen that the variance in the data revealed by PCA does not reflect the difference between classes Landmines and Other Objects. Thus, the idea followed in this work is to analyse the possibility to determine a better classification methodology. It is proposed to consider for landmine detection two classification subtasks instead of a single one (Landmines/Other Objects) which was utilized before [4]. Particularly, it is proposed to consider one more classification stage with classes Man-made Objects and Natural Objects. The objective for this division arose from the following problems: •
It was noticed that during the classification Landmines/Other objects most false detections consist in other man-made objects: the classifier is not able to distinguish landmines from other man-made objects.
Data from 9 fields of the experimental database [2] (A1, A2, A3, C1, C2, C3, C4, C5, C7) were used for evaluation of the proposed ideas (data for ground penetrating radar are not available for other fields). To estimate the performance of the ROIs extraction algorithm, the experimental data from pulsed MD and continuous MD were labeled to create a sensor-specific ground-truth map, which contains all the objects seen by a human on the sensor data with adjustable scales. In this experiment the detection rate (number of ground-truth objects detected by at least one ROI divided by their total amount) of 90% and the false alarm rate (area occupied by false detections divided by the total area of background) of 4% were achieved. From the 90% of the detected objects, 47% were detected more than once. This large number of the repeated detections increases the processing time but, on the other hand, most of such objects have weak (or confusing) fingerprints. Therefore, in the case of a landmine several detections may improve the probability to detect it by at least one of the ROIs. Evaluation of the sensor fusion approaches was performed using data from all four sensors. For the training of the sensor fusion algorithms, the data from two fields, one with loamy soil
2946
Fig. 8. Comparison of different classifications (on the training set) in terms of detection rate and false alarm rate (a)
and the other with sandy soil, of the experimental database [2] were taken. The rest of the data were used for the evaluation. In order to confirm the idea of multi-stage classification, a comparison of Receiver-Operating Characteristics (ROCs) for different classifications is shown in Fig. 8. It can be seen that, as expected, the classification with classes Manmade Objects/Natural Objects has better performance than the classification with classes Landmines/Other Objects. Moreover, the classification Landmines/Other Objects performs better when used to distinguish among the Man-made Objects. This suggests that the two-step classification (last two stages in Fig. 7) should perform better. However, in practice, the performance of the two-step classification is degraded because of the following reasons: • A part of the detected Man-made Objects by the first classifier appears as false alarms for landmine detection. • The second classifier Landmines/Other Objects is trained using only Man-made Objects of the training set, but has to deal with false alarms of the first classifier in practice. Finally, the performance of the two-step classification in comparison with the one-step is shown in Fig. 9, where in general the two-step classification outperforms. The proposed multi-stage landmine detection can be also seen as a step-by-step reduction of the false alarm rate. After each stage all objects which fall into the corresponding class (shown with gray background in Fig. 7) can be considered Landmines. Then each subsequent stage improves the false alarm rate of the previous one. Table I shows the changing of the number of false alarms per m2 over the stages of sensor fusion.
(b) Fig. 9. Comparison of the relative performance of one-step and two-step classifications on training (a) and evaluation (b) sets
TABLE I C HANGING OF FALSE ALARMS PER m2 FOR DETECTION RATE 95% OVER THE STAGES OF THE SENSOR FUSION
Two-step classification ROIs extraction FA/m2
V. C ONCLUSION This work proposed a multi-stage sensor fusion strategy for landmine detection (Fig. 7). The paper is particularly focused on the last stage of the sensor fusion process classification - providing a framework for its division into two
2947
12.1
Man-made Objects/ Landmines/ Natural Objects Other Objects 5.4 4.5 One-step classification 7
Fig. 10. Experimental data obtained from an IR sensor using a mobile scanning platform [3] and the results of ROIs extraction. Fig. 11.
classification subtasks which are better separable in the feature space. It is demonstrated with experimental data that the two-step classification with classes Man-made Object/Natural Object and Landmines/Other Objects has better performance than the previously proposed one-step classification (Fig. 9 and Table I). The proposed strategy has an important advantage for the practical implementation of landmine detection. The sensor fusion process can be terminated at any stage according to the quality of the available data considering, first of all, the quality of the data mapping. For example, if the data has low spatial resolution as shown in Fig. 10, it is reasonable to consider only the first stage of the process (ROIs extraction) because there is not enough information for a more sophisticated classification. In this experiment the data were obtained by a mobile scanning platform [3] shown in Fig. 11 and the improvement of its positioning is a part of future work. Similarly, the data may contain enough information only for the classification Man-made Object/Natural Object and provide no further improvement in distinguishing landmines from other man-made objects. The level at which the sensor fusion is terminated determines the false alarm ratio which can be obtained (see Table I): a smaller number of false alarms requires higher quality of the available data. The experimental data used in this work were obtained in well controlled conditions using a precise scanning device [2]. Thus, all the stages of the sensor fusion process could be evaluated. The introduction of the classification Manmade Objects/Natural Objects in this work allows to consider a more general task beyond the landmine detection: detection of all dangerous objects located on a minefield. However, the available experimental data do not unfortunately include other dangerous objects besides the antipersonnel landmines. ACKNOWLEDGMENT This research was supported by the FCT (Portuguese Foundation for Science and Technology). The authors would like to thank Dr. Adam Lewis (JRC - Ispra) for the help in supplying and processing of the GPR data.
2948
Mobile scanning platform for landmine detection [3]
R EFERENCES [1] R. Siegel, “Land mine detection,” IEEE Instrumentation and Measurement Magazine, vol. 5, no. 4, pp. 22–28, 2002. [2] “Joint Multi-Sensor Mine Signature Database (Joint Research Centre Ispra),” http://demining.jrc.it/msms/. [3] M. Y. Rachkov, L. Marques, and A. T. de Almeida, “Multisensor demining robot,” Autonomous Robots, no. 18, pp. 275–291, 2005. [4] S. Larionova, L. Marques, and A. de Almeida, “Toward practical implementation of sensor fusion for a demining robot,” in Int. Conf. on Intelligent Robots and Systems, IROS, 2004. [5] ——, “Features selection for sensor fusion in a demining robot,” in Int. Conf. on Robotics and Automation, ICRA, 2005. [6] F. Cremer, J. Schavemaker, W. Jong, and K. Schutte, “Comparison of vehicle-mounted forward-looking polarimetric infrared and downwardlooking infrared sensors for landmine detection,” in Detection and Remediation Technologies for Mines and Minelike Targets - SPIE, 2003. [7] H. Frigui, P. Gader, and J. M. Keller, “Fuzzy clustering for land mine detection,” in NAFIPS, 1998. [8] A. Gunatilaka and B. Baertlein, “Comparison of pre-detection and postdetection fusion for mine detection,” in Detection and Remediation Technologies for Mines and Minelike Targets - SPIE, vol. 3710, 1999, pp. 1212–1223. [9] D. W. McMichael, “Data fusion for vehicle-borne mine detection,” in Int. Conf. on the Detection of Abandoned Land Mines, 1998, pp. 167–171. [10] W. Messelink, K. Schutte, A. Vossepoel, F. Cremer, J. Schavemaker, and E. Breejen, “Feature-based detection of landmines in infrared images,” in Detection and Remediation Technologies for Mines and Minelike Targets - SPIE, vol. 4742, 2002, pp. 108–119. [11] M. Roughan and D. W. McMichael, “A comparison of methods of data fusion for land-mine detection,” in Int. Workshop on Image Analysis and Information Fusion, 1997. [12] S. Perrin, E. Duflos, P. Vanheeghe, and A. Bibaut, “Multisensor fusion in the frame of evidence theory for landmines detection,” IEEE Trans. on Systems, Man and Cybernetics, Part C, 2004. [13] K. C. Ho, L. M. Collins, L. G. Huettel, and P. D. Gader, “Discrimination mode processing for EMI and GPR sensors for hand-held land mine detection,” IEEE Trans. on Geoscience and Remote Sensing, vol. 42, no. 1, pp. 249–263, 2004. [14] G. A. Clark, S. K. Sengupta, D. Aimonetti, F. Roeske, and J. G. Donetti, “Multispectral image feature selection for land mine detection,” IEEE Trans. on Geoscience and Remote Sensing, vol. 38, no. 1, pp. 304–311, 2000. [15] J. M. Stiles, A. V. Apte, and B. Beh, “A group-theoretic analysis of symmetric target scattering with application to landmine detection,” IEEE Trans. on Geoscience and Remote Sensing, vol. 40, no. 8, pp. 1802–1814, 2002. [16] N. Sarkar and B. B. Chaudhuri, “An efficient differential box-counting approach to compute fractal dimension of image,” IEEE Trans. on Systems, Man, and Cybernetics, vol. 24, no. 1, pp. 115–120, 1994. [17] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. WileyInterscience Publication, 2000.
