Automated Morphological Image Composition for Mosaicing Large Image Data Sets Conrad Bielski, Jacopo Grazzini and Pierre Soille
Spatial Data Infrastructures Unit Institute for Environment and Sustainability DG Joint Research Centre, European Commission I-21020 Ispra, (Va), Italy Email: {Conrad.Bielski,Jacopo.Grazzini,Pierre.Soille}@jrc.it Abstract—Users of remotely sensed imagery are often faced with the need to stitch two or more overlapping scenes together. This is generally referred to as image composition and the outcome is a single image with no overlapping regions which is also called an image mosaic. The most difficult part of image composition however, is deciding where to place the cut line in overlapping regions. In the days of hardcopy remote sensing, technicians would cut pictures along salient image structures or features when creating mosaics in order to minimise the ability of visually detecting the cut lines. Recently, a morphological image compositing algorithm was proposed that is able to automatically delineate cut lines along salient image structures. This technique was developed specifically to deal with very large image data sets automatically and generates a visually appealing image mosaic. To minimise the visual detection of a cut line, the contrast should look natural and therefore cuts should not be made through image objects but rather along the object. This paper presents the algorithm behind the morphological image compositing technique and how it was applied to automatically generate a European wide image mosaic based on over 800 Landsat ETM+ scenes. A quantitative measure was developed in order to try and quantify the ’quality’ of an automatically delineated seam line. The results demonstrate how invisible the delineated seam lines are to the human eye based on the morphological image compositing method but at the same time, the experiments show the difficulty of quantitatively determining the ’best’ one.
I. I NTRODUCTION Image mosaicing is a frequent remote sensing processing task whereby a seam or cut line must be delineated in regions where images overlap. The act of delineating cut lines is often referred to as image compositing or mosaicing. In the days of hardcopy remote sensing, the technician had the often sticky job of cutting and gluing the overlapping photographs in such a way as to make the cut lines invisible to the interpreter. Ideally, such cuts were made in the overlapping region by following a series of points corresponding to the same set of features in each image [1], [2]. Today, digital remotely sensed images require virtual cut lines to be defined in much the same manner with the added difficulty of managing much large image data sets. While automatic seam line delineation algorithms are available to remote sensing practitioners many times the cut lines are still delineated by hand. Furthermore, the resulting automatically delineated seam line is generally assessed qualitatively which makes it difficult to compare algorithms quantitatively. A definition of the ’best’ seam line
was presented by Milgram [3] as a line that minimises the visual confusion created by the artificial edge along the seam. Therefore, an observer should distinguish the location of the cut line with difficulty because the visual system is not confused by the delineated cut line. Recently, Soille [4] proposed the morphological image compositing algorithm which has been further developed to handle large image data sets in [5], [6]. The main goals of the morphological image compositing procedure are to produce visually pleasing seam lines without changing the image radiometry. While applying feathering along image edges is common practice when producing image mosaics, the radiometry is changed significantly especially when a wide buffer is used. Furthermore, radiometrically altered image mosaics are no longer fit for research purposes which is an important criterion due to the fact that the mosaic is not always the final product. Surveys on image mosaicing can be found in [4], [7] and a constantly updated bibliography can be found in [8]. The focus of this paper is on the quantification of the quality of a delineated seam line. The next section provides a brief introduction to the morphological image compositing algorithm [4] and discusses the methodology used for quantifying a ’best’ seam line. Section 3 presents the results of our initial attempt at quantifying the quality of automatically delineated seam lines by filtering the input images prior to compositing [4]. Finally, we discuss the obtained results and present our conclusions. II. M ETHODOLOGY This section briefly describes the steps in the morphological image compositing algorithm [4] followed by the method used to quantify a ’best’ seam line within an image mosaic. A. Morphological Image Compositing The morphological image compositing method is adaptive because the driver behind delineating seam lines is the image content. The main part of this technique is a region growing procedure that is initiated by seeds found in regions where there is no image overlap. The region growing procedure, a morphological watershed algorithm [11], is constrained by a gradient image at high gradient values. By basing the seam line
Fig. 1. The test data set is made up of 838 Landsat 7 ETM+ images from the Global Land Cover Facility (GLCF, http://glcf.umiacs.umd.edu/) from the summer of 2000 covering most of Europe. It was used specifically to test the morphological image compositing method on large image datasets. All original images were re-projected from Universal Transverse Mercator to Lambert Azimuthal Equal Area based on the European Grid System [9], [10].
delineation on the gradient intensity, the algorithm attempts as best as possible to follow salient image features. A more formal but simplified description of the methodology already presented in [6] and originally proposed in [4] is summarised below. The input set of images must first be geometrically corrected and resampled to the same spatial resolution. •
• •
•
•
Input: arbitrary number n of input overlapping images denoted by f1 , . . . , fn . No input image is fully included inside of another image. The so-called marker image indicates the number of images available for each pixel (i.e. number of overlapping images). The so-called mask image fmask is used to constrain the growth of the markers (seeds) defined below. In its basic form, it is simply defined as the pointwise minimum between the V morphological gradients of all input images: fmask = i ρB (fi ). Since this mask image is at the root of the adaptive definition of the seam lines, it can be modified so as to maximise some function. The marker image seeds the region growing process. The process starts where there is no image overlap.
The method is ordered in the sense that the growth of the seeds operates in those regions where only two images overlap, then three, and so forth until the maximum number of overlaps is reached. It should be noted that all lower overlap regions
must be completed prior to advancing to the next overlap. Fig. 1 presents the results of applying the methodology to a dataset made up of 838 Landsat 7 ETM+ scenes acquired in the year 2000 over Europe. B. Quantifying Seam Line Quality The delineation of seam lines is still quite often done by a human operator and therefore the question of whether the chosen path of the seam is ’best’ is answered qualitatively by the operators themselves who make a judgement. Ideally, as Milgram [3] stated, the qualitative goal should be to minimise the visual effect of the seam line. While feathering or any other technique that blurs and/or alters the image radiometry in order to hide the location of the seam line produces results that are pleasing to the eye, it is but a cosmetic solution. In order to achieve this qualitative goal without any change to the original image radiometry the solution is a little more complicated. The morphological image compositing algorithm is able to produce visually pleasing seam lines because the algorithm was devised to cut along salient image features present in all overlapping images. Essentially, cut lines are delineated mainly along the boundary between adjacent image objects whose grey level difference is greatest. This reduces the visual confusion and produces visually pleasing seam lines without changing the image radiometry. This is true of course as long as the overall differences between images are not too marked
(i.e. seasonal differences or atmospheric differences). The results of our previous experiments are favourable [4], [5], [6] and we were interested in checking whether filtering would improve or worsen the resulting seam line delineation. The ’best’ criterion was formulated as the sum of morphological gradient of the image mosaic along the seam line divided by the length of the seam (in number of pixels). This criterion was based on the idea that the better seam line would have a higher mean contrast computed along the delineated seam.
Fig. 2. The above graph presents the computed score for each image pair based on the automatically delineated seam line. The scores were computed by summing the morphological gradient of the resulting image mosaic along the seam line and dividing it by the length of the seam line (in pixels). Score results ranged between [138,155]. Note that filtering the input images did not produce significant differences in the quality of the automatically delineated seam line.
III. E XPERIMENTS
AND
R ESULTS
The above defined quality criterion was tested on two overlapping areas of adjacent Landsat 7 ETM+ scenes of band 5 (NIR). The size of the test areas were 1025 × 1025 pixels and different seam lines were delineated by generating a series of images through morphological filtering. The images were filtered using an alternating sequential filter (ASF) with geodesic reconstruction [12]. Such a filter preserves edges while smoothing homogeneous areas by alternating the open/close morphological operators. The pair of sub-sampled Landsat ETM+ test scenes were first filtered using Structuring Elements (SE) of different sizes ranging from 1 to 10 pixels in steps of 1 pixel. Each set of overlapping input images were then used to automatically delineate seam lines using the morphological image compositing method. The score of the quality criterion was computed for the seam lines estimated over these different sets (fig. 2). Even though the actual delineated seam lines varied based on the input imagery no significant differences between the scores were observed and no trend could be inferred from the resulting graphed scores. Fig. 3 presents four of the eleven test input image pairs including the computed gradient image, delineated seam line and resulting mosaic.
IV. D ISCUSSION High scores across the board indicate that the resulting seam lines in all cases cut along high gradient regions which generally are image object edges. The greater the radiometric difference between two image objects, the higher the gradient along the edge of such objects. In fig. 3, it is possible to compare the outputs of the seam line delineation more closely. From this figure it is clear that the input images do have an influence on the delineation of the seam lines. From a purely visual perspective, the mosaiced image is quite pleasing in all cases. The results suggest that, using the current approach based on ASF, no pre-processing is necessary to delineate the ’best’ seam line. This highlights the fact that even though the overall variance of the imagery is reduced due to filtering, the algorithm itself is still able to follow salient features and produces quality cut lines. However, the initial experiment only looks at the differences in scores between a single scene. Based on the graph of fig. 2, it is not clear whether an optimal filter (at an optimal scale) exists that would ensure that the morphological compositing method would always delineate the ’best’ seam lines for all types of scenes. Consequently, it is difficult to generalise. These results can be interpreted in two ways: either the pre-processing method used for generating the input images for delineating seam lines is not appropriate or the quality criterion is not capable of accounting for the visual properties of the final mosaic. Improvements regarding these two issues are possible. For instance, pre-processing of the input images could be performed based on a multiscale approach [13] whereby local scales, depending on the spatial location, would be considered for filtering instead of a single fixed scale as in the case of the ASF method. The definition of the quality criterion should further be investigated [14] by introducing different quantities to better target the goal of producing a ’visually pleasing’ image mosaic. V. C ONCLUSION The results show the difficulty of distinguishing between the resulting mosaics because no clear winner was found with respect to the ’best’ seam line. At this point it is unclear based on our initial testing whether the criterion used is too simple, whether the target is too vaguely defined or whether the preprocessing approach is not appropriate. Consequently, larger image sets and more complicated examples (i.e. greater than two overlapping images) must be tested. At the same time it may be interesting to produce some type of benchmark such as a linear seam line along the edge of an image. Moreover, it is our intention to use imagery from different sensors and spectral bands (or apply a different type of filter) in order to understand the influence of spatial and spectral resolution on the automatic delineation of the seam lines based on the morphological image compositing method. In particular, future experiments will use both very fine spatial resolution imagery (e.g. Ikonos/Quickbird, airborne) and coarse spatial resolution imagery (e.g. MODIS/Meris), as well as digital elevation data (derived from ASTER).
Fig. 3. Different seam lines were automatically delineated using the morphological image compositing method by applying edge preserving morphological filters with SEs of different sizes to the input overlapping image pairs. The four rows (from the top) correspond to the filter size used (original data, 3 pixels, 7, and 10). The first two columns from the left present the input images which are sub-samples of Landsat ETM+ band 5 (NIR) with WRS-2 path/row of 197/027 and 198/027 respectively. The central column is the input morphological gradient image; for visualisation purposes the images were inverted and equalised so that high gradient values are darker. The penultimate column is the automatically delineated seam line and the last column is the resulting mosaic. In all cases, the automatically delineated seam lines are difficult to discern and produce visually pleasing mosaics.
R EFERENCES [1] D. McNeil, “The wet process of laying mosaics,” Photogrammetric Engineering, vol. 15, no. 2, p. 315, 1949. [2] P. Wolf, Elements of Photogrammetry (with Air Photo Interpretation and Remote Sensing). New York: McGraw-Hill, 1974. [3] D. Milgram, “Adaptive techniques for photomosaicking,” IEEE Transactions on Computers, vol. 26, pp. 1175–1180, Nov. 1977. [4] P. Soille, “Morphological image compositing,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 5, pp. 673–683, May 2006. [5] C. Bielski and P. Soille, “Order independent image compositing,” Lecture Notes in Computer Science, vol. 3617, pp. 1076–1083, Sept. 2005. [6] ——, “Adaptive mosaicing: principle and application to the mosaicing of large image data sets,” Lecture Notes in Computer Science, vol. 4432, no. 2, pp. 500–507, Apr. 2007. [7] R. Szeliski, “Image alignment and stitching: a tutorial,” Foundations and Trends in Computer Graphics and Vision, vol. 2, no. 1, 2006. [8] K. Price, “Computer vision bibliography,” 2006. [Online]. Available: http://iris.usc.edu/Vision-Notes/bibliography/contents.html
[9] A. Annoni, C. Luzet, E. Gubler, and J. Ihde, Eds., Map Projections for Europe. European Commission, Joint Research Centre, 2003, vol. EUR 20120 EN. [Online]. Available: http://www.ec-gis.org/sdi/publist/ pdfs/annoni-etal2003eur.pdf [10] A. Annoni, Ed., European Reference Grids. European Commission, Joint Research Centre, 2005, vol. EUR 21494 EN. [Online]. Available: http://www.ec-gis.org/sdi/publist/pdfs/annoni2005eurgrids.pdf [11] L. Vincent and P. Soille, “Watersheds in digital spaces: an efficient algorithm based on immersion simulations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 6, pp. 583–598, June 1991. [12] P. Soille, Morphological Image Analysis: Principles and Applications, 2nd ed. Berlin Heidelberg New York: Springer-Verlag, 2003. [13] J. Grazzini, A. Turiel, H. Yahia, and I. Herlin, “Edge-preserving smoothing of high-resolution images with a partial multifractal reconstruction scheme,” in Proc. of International Society for Photogrammetry and Remote Sensing, 2003, pp. 1125–1129. [14] M. Kerschner, “Twin snakes for determining seam lines in orthoimage mosaicking,” International Archives of Photogrammetry and Remote Sensing, vol. 33, no. B4, pp. 454–461, 2000.