Multiresolution Image Fusion Based on Wavelet ... - Semantic Scholar
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
91
Multiresolution Image Fusion Based on Wavelet Transform By Using a Novel Technique for Selection Coefficients Yong Yang School of Information Technology Jiangxi University of Finance and Economics, Nanchang 330013, China Email: [email protected]
Abstract—Today, image fusion as one kind of information integrated technology has played an important role in many fields. This paper presents a novel wavelet based technique for image fusion, which is developed by taking into account the physical meaning of the wavelet coefficients. The images to be fused are firstly decomposed into high frequency and low frequency bands by wavelet transform. Then, by considering the physical meaning of the coefficients, a new selection strategy that treats the coefficients in different ways is proposed. This strategy selects the coefficients of high frequency bands with a variance based scheme, while selects the coefficients of low frequency band with an edge based scheme. Finally, the fused image is constructed by an inverse wavelet transform performing on the combined coefficients from all frequency bands. The proposed method is validated and compared with some existing fusion methods. Experimental results can clearly indicate the superiority of the proposed method through visual inspection and objective fusion performance measurements. Index Terms—wavelet transform, image fusion, coefficients, variance
I. INTRODUCTION Nowadays, with the rapid improvement of numerous imaging sensors, image fusion has been widely used in many fields, such as remote sensing, medical imaging, machine vision, etc. Image fusion can be generally defined as the process of combing multiple input images or some of their features into a single image without the introduction of distortion or loss of information [1]. The aim of image fusion is to integrate complementary as well as redundant information from multiple images to create a fused image output. Therefore, the new image generated should contain a more accurate description of the scene than any of the individual source images and is more suitable for human visual and machine perception or further image processing and analysis tasks [2]. Over the past decades, many techniques for image fusion have been proposed and a thorough overview of Manuscript received July 3, 2010; revised September 3, 2010; accepted September 23, 2010. Corresponding author: E-mail: [email protected].
© 2011 ACADEMY PUBLISHER doi:10.4304/jmm.6.1.91-98
these methods can be viewed in reference [3]. According to the stage at which the combination mechanism takes place, the image fusion methods can be generally grouped into three categories, namely, pixel level, feature level, and decision level [4]. Since the pixel level fusion has the advantage that the images used contain the original measured quantities, and the algorithms are computationally efficient and easy to implement, the most image fusion applications employ pixel level based methods [5]. Therefore, in this paper, we are still concerned about pixel level fusion The simplest way of pixel level image fusion is to take the average of the two images pixel by pixel. However, this method usually leads to undesirable side effect such as reduced contrast [6]. More robust algorithm for pixel level fusion is the weighted average approach. In this method, the fused pixel is estimated as the weighted average of the corresponding input pixels. However, the weight estimation usually requires a user-specific threshold. Artificial neural network (ANN) has also been introduced to make image fusion, as seen in [7]. However, the performance of ANN depends on the sample images and this is not an appealing characteristic. Yang et al. used a statistical approach to fuse the images [8]; however, in his method the distortion is modeled as a mixture of Gaussian probability density functions (pdfs) which is a limiting assumption. Due to the multiresolution transform can contribute a good mathematical model of human visual system (HVS) and can provide information on the contrast changes, the multiresolution techniques have then attracted more and more interest in image fusion. The multiresolution techniques involve two kinds, one is pyramid transform another is wavelet transform. Examples of pyramid approach include the Laplacian pyramid, the gradient pyramid, and the morphological pyramid, etc [7]. However, for the reason of the pyramid method fails to introduce any spatial orientation selectivity in the decomposition process, the above mentioned methods often cause blocking effects in the fusion results [9]. Another family of the multiresolution fusion techniques is the wavelet based method, which usually used the discrete wavelet transform (DWT) in the fusion. Since the DWT of image signals produces a
92
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
nonredundant image representation, it can provide better spatial and spectral localization of image information as compared to other multiresolution representations. Therefore, the wavelet based method has been popular widely used for image fusion [10]-[11]. In the wavelet transform based image fusion, the input images are first transformed into their multiresolution representations. The fusion process then creates a new composite multiresolution representation from these inputs by a certain fusion rule. The fused image is finally reconstructed by performing an inverse wavelet transform. Therefore, we can find the key step for wavelet based image fusion is the definition of the fusion rule. The widely used fusion rule is the maximum selection (MS) scheme [9]. This simple scheme just selects the largest absolute wavelet coefficient at each location from the input images as the coefficient at the location in the fused image. However, as we know that the noise and artifacts usually has higher salient features in the image, therefore, this method is sensitive to noise and artifacts. Zhao et al. [12] proposed another fusion rule that selected the coefficients in the low frequency band by a weighted average scheme, and selected the coefficients in the high frequency bands by the MS scheme. However, for the shortcoming of weighted average scheme this method will blur and reduce the contrast of features appearing in the image. More recently, Chen et al. [13] proposed a weighted fusion rule for fusion of PET and CT images. The disadvantage of this method is that the weights are determined by the user himself but can not be automatically obtained. In this paper, a novel and fully automated waveletbased method for image fusion is presented. The main contribution of this work is that by considering the physical meaning of the wavelet coefficients, we proposed a new fusion strategy for image fusion, in which the coefficients of the high frequency portions are performed by a variance based scheme, while the coefficients of the low frequency portion are performed by an edge based scheme. Both qualitative and quantitative experiments can prove our technique is effective and promising. The remainder of the paper is organized as follows. Section II describes the general wavelet transform based technique for image fusion. The proposed method for image fusion is presented in Section III. Experimental results and comparisons are given in IV. Finally, some conclusions are drawn in Section V. II. GENERAL IMAGE FUSION BASED ON DWT The original concept and theory of wavelet-based multiresolution analysis came from Mallat [14]. The wavelet transform is a mathematical tool that can detect local features in a signal process. It also can be used to decompose two-dimensional (2-D) signals such as 2-D gray-scale image signals into different resolution levels for multiresolution analysis.
A. DWT Analysis There are two main groups of transforms, continuous and discrete. Of particular interest is the DWT, which is a spatial-frequency decomposition that provides a flexible multiresolution analysis of an image [15]. In one dimension (1-D) the basic idea of the DWT is to represent the signal as a superposition of wavelets. Suppose a discrete signal is represented by f (t ) , the wavelet decomposition is then defined as:
f (t ) = ∑ cm ,nψ m ,n (t ) ,
(1)
m ,n
where
ψ m ,n (t ) = 2 − m 2ψ [2 − m t − n] , m
and
n are
integers. There exist very special choices of ψ such that
ψ m,n (t )
constitutes an orthonormal basis, so that the
wavelet transform coefficients can be obtained by an inner calculation:
cm,n = f ,ψ m,n = ∫ψ m,n (t ) f (t )dt
(2)
In order to develop a multiresolution analysis, a scaling function φ is needed, together with the dilated and translated version of it,
φm ,n (t ) = 2− m 2 φ [2− m t − n ] .
According to the characteristics of the scale spaces spanned by φ and ψ , the signal f (t ) can be decomposed in its coarse part and details of various sizes by projecting it onto the corresponding spaces. Therefore, to find such decomposition explicitly, additional coefficients am ,n are required at each scale. At each scale
am ,n and am−1,n describe the approximations
of the function resolution
2 m −1
f at resolution 2 m and at the coarser respectively, while the coefficients cm ,n
describe the information loss when going from one approximation to another. In order to obtain the coefficients cm ,n and a m ,n at each scale and position, a scaling function is needed that is similarly defined to equation (2). The approximation coefficients and wavelet coefficients can be obtained:
a m ,n = ∑ h2 n − k a m −1,k ,
(3)
cm ,n = ∑ g 2 n −k a m −1,k ,
(4)
k
k
where hn is a low pass FIR filter and g n is related high pass FIR filter. To reconstruct the original signal the analysis filters can be selected from a biorthogonal set which have a related set of synthesis filters. These
~
~ can be used to perfectly synthesis filters h and g reconstruct the signal using the reconstruction formula
~ am−1,l ( f ) = ∑ [h2 n −l am ,n ( f ) + g~2 n−l cm ,n ( f )] (5) n
Equations (3) and (4) are implemented by filtering and downsampling. Conversely equation (5) is implemented by an initial upsampling and a subsequent filtering. © 2011 ACADEMY PUBLISHER
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
In a 2-D DWT, a 1-D DWT is first performed on the rows and then columns of the data by separately filtering and downsampling. This results in one set of approximation coefficients and three sets of detail coefficients, which represent the horizontal, vertical and dialog directions of the image I , respectively. In the language of filter theory, these four subimages correspond to the outputs of low–low (LL), low–high (LH), high–low (HL), and high–high (HH) bands. By recursively applying the same scheme to the LL subband a multiresolution decomposition with a desires level can then be achieved. Therefore, a DWT with K decomposition levels will have M = 3 ∗ K + 1 such frequency bands. Fig.1 shows the 2-D structures of the wavelet transform with two decomposition levels. It should be noted that for a transform with K levels of decomposition, there is always only one low frequency band ( LLK in Fig.1), the rest of bands are high frequency bands in a given decomposition level. LL2
LH2 LH1
HL2
HH2
HL1
HH1
Fig.1. 2-D DWT structure with labeled subbands in two-level decomposition.
B. DWT Based Image Fusion In this subsection, to better understand the concept and procedure of the wavelet based fusion technique, a schematic diagram is first given in Fig.2. In general, the basic idea of image fusion based on wavelet transform is to perform a multiresolution decomposition on each source image, the coefficients are then performed with a certain fusion rule as displayed in the middle block of Fig.2. After that, the fused image is obtained by performing the inverse DWT (IDWT) for the corresponding combined wavelet coefficients. Therefore, as shown in Fig.2, the detailed fusion steps based on wavelet transform can be summarized below.
Fig.2. Schematic diagram of the wavelet based image fusion.
© 2011 ACADEMY PUBLISHER
93
Step 1: The images to be fused must be registered to assure the corresponding pixels are aligned. Step 2: These images are decomposed into wavelet transformed images respectively, based on DWT. The transformed images with K -level decomposition will include one low frequency portion and 3K high frequency portions. Step 3: The transform coefficients of different portions or bands are performed with a certain fusion rule. Step 4: The fused image is constructed by performing the IDWT based on the combined transform coefficients from step 3. III. THE PROPOSED FUSION TECHNIQUE As shown in the above schematic diagram of Fig.2, it is easy to see that the core step in wavelet based image fusion is the definition of the fusion rule because it will decide how to merge the coefficients in an appropriate way so that a high quality fused image can be obtained. However, as we know that the most popular widely used fusion rule is the aforementioned MS scheme. Although, this method can select the salient features from the source images; it is sensitive to noise and artifacts. Therefore, with this method some noise and artifacts are easily introduced into the fused image, which will reduce the resultant image quality. Averaging fusion rule may be another alternative method and it can lead to a stabilization of the fusion result. However, this scheme tends to blur images and reduce the contrast of features appearing in only one image. More importantly, all these fusion rules (including Chen et al.[13]) usually treat the coefficients in the same way. However, as we know that when an image is decomposed by the wavelet transform the coefficients of low frequency band and high frequency bands must have their quite different physical meaning. So, the coefficients in low frequency and high frequency portions should be performed with different fusion schemes. Therefore, in this paper, we proposed a new strategy for image fusion, which treats the coefficients by using different schemes in different frequency bands. A. Procedure of the proposed method In this section, as the procedure of the general wavelet based fusion method (shown in Fig.2), the block diagram of the proposed approach is given in Fig.3. The basic idea of our method is to perform a multiresolution decomposition on each source image; the coefficients are then performed with a certain fusion rule as displayed in the middle block of Fig.3. After taking in to account the physical meaning of the coefficient, this paper presents a novel fusion scheme that treats the coefficients of the high frequency bands and low frequency bands separately: the former are performed by a maximal variance based scheme, while the latter is performed by a maximal edge based scheme. Finally, the fused image is obtained by performing the IDWT on the combined wavelet coefficients. It is important to note that the high frequency bands include the vertical, horizontal, and
94
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
Fig. 3. The block diagram of the proposed fusion technique.
diagonal high frequencies of the image, respectively. Therefore, the fusion process must be performed in all these domains. To simplify the description of the different alternatives available in forming a fusion rule, as reference [10] we also consider only two source images, X and Y , and the fused image Z . The method can of course be easily extended to more than two images. Generally, an image I has its multiscale decomposition (MSD) representation denoted as DI . Hence we will encounter DX , DY , and DZ . Let p = (m, n, k , l ) indicate the index corresponding to a particular MSD coefficient, where m and n indicate the spatial position in a given frequency band, k is the decomposition level, and l is the frequency band of the MSD representation. Therefore, DI ( p ) denote the MSD value of the corresponding coefficient.
B. Selection Scheme for High Frequency Bands The high frequency bands contain the detail coefficients of an image, which usually have large absolute values correspond to sharp intensity changes and preserve salient information in the image. In addition, based on the requirements of image fusion [10], we know that the purpose of image fusion requires that the fused image must not discard any useful information contained in the source images and effectively preserve the details. Therefore, it is important to find appropriate scheme to merge the details of the input images. On the other hand, according to characteristic of HVS it is easy to see that for the high resolution region the human visual interest is concentrated on the detection of changes in contrast between regions on the edges separate these regions. Therefore, a good method for the high frequency bands should produce large coefficients on those edges. Based on the above analysis, we propose a scheme by computing the variance in a neighborhood to select the high frequency coefficients. The procedure can be formulated as the following:
1 σ I ( p) = S ×T
⎛ DI (m + s, n + t , k , l )⎞ ⎟⎟ (6) ⎜⎜ ∑ ∑ s =−S 2 t =−T 2 ⎝ − meanI ( p ) ⎠ S 2
T 2
© 2011 ACADEMY PUBLISHER
2
meanI ( p ) =
1 S ×T
S 2
T 2
∑ ∑ D (m + s, n + t, k , l )
s =−S 2 t =−T 2
I
(7)
where S × T is the neighboring size, meanI ( p ) ,
σ I ( p ) denote the mean value and variance value of the
coefficients centered at
(m, n )
in the window of
S × T respectively. Then, the fusion scheme used for the high frequency bands can be illustrated as the following:
⎧D ( p ) σ X ( p ) ≥ σ Y ( p ) DZ ( p ) = ⎨ X ⎩ DY ( p ) σ X ( p ) < σ Y ( p )
(8)
C. Selection Scheme for Low Frequency Band However, differing from the high frequency bands the low frequency band is the original image at the coarser resolution level, which can be considered as a smoothed and subsampled version of the original image. Therefore, most information of their source images is kept in the low frequency band. So, if the scheme mentioned above is adopted here, the fused results will be blocked. In this paper, in order to better improve the quality of the fused result, an edge-based scheme for selection the coefficients in low frequency band is performed consequently. This scheme firstly calculated the edges in the horizontal, vertical and diagonal directions as the following [16]:
Edge( p ) = (F1 ∗ D ) ( p ) + (F2 ∗ D ) ( p ) 2
2
(9)
+ (F3 ∗ D ) ( p ) where the symbol ∗ denote the convolution, and F1 = {{− 1, − 1, − 1}, {2, 2, 2}, {− 1, − 1, − 1}}, F2 = {{− 1, 2, − 1}, {− 1, 2, − 1}, {− 1, 2, − 1}}, F3 = {{− 1, 0, − 1}, {0, 4, 0}, {− 1, 0, − 1}}. 2
After obtaining the edges of the source images and in order to retain much more information from those images, the coefficients of the low frequency band are then chosen by the following formula: DZ ( p ) = wX ( p )DX ( p ) + wY ( p )DY ( p ) (10)
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
⎧1, Edge X ( p ) > EdgeY ( p ) , otherwise ⎩0, ⎧1, Edge X ( p ) ≤ EdgeY ( p ) . wY ( p ) = ⎨ otherwise ⎩0,
where, w X ( p ) = ⎨
Once all the coefficients are achieved from the above two procedures, an inverse wavelet transform is then performed on them, the fused image is thus constructed. IV. EXPERIMENTAL RESULTS AND ANALYSIS In this section, the application results of the proposed DWT based method for image fusion are presented. The performance of the proposed method is compared with those of pixel averaging method [9], the DWT method proposed by Zhao et al. [12]. To better describe the experiments, as in reference [10], in all test cases we assume the source images to be in perfect registration. We use the Daubechies’ db8, with a decomposition level of 3, as the wavelet basis for DWT method and the proposed method. A 3 × 3 window size for calculating the variance is considered in this paper.
95
The first experiment is performed on a popular widely used standard image Lena of size 256× 256 as shown in Fig. 4 (a), which served as the ideal reference image here. Then two other images are generated by filtering the reference image with a Gaussian blurring process as reference [2]. Fig. 4 (b) is the image blurred on the left, while Fig. 4 (c) is the image blurred on the right. Figs.4 (d)-(f) are the fused results obtained by fusing Fig. 4 (b) and Fig. 4 (c) with the three methods mentioned above, respectively. It is easy to see that from the fused images by visual inspection it is difficult to find the difference of the three methods except that Fig.4 (d) has a lower contrast. Therefore, a frequently used metric, root mean square error (RMSE) [9] is employed here to objectively evaluate the performance of the three methods. This metric can indicate how much error the fused image conveys about the reference image. Hence, the lower the RMSE, the better the fused result. The RMSE is defined as: 12
⎛ 1 N M (xR (n, m) − xF (n, m))2 ⎞⎟ RMSE = ⎜ ∑∑ ⎠ ⎝ MN n=1 m=1
(11)
xR and xF denote the ideal reference image and fused image, respectively, and M and N are the where
dimensions of the image. The RMSE values of the three methods are calculated and showed in Table I. It can be seen from Table I that the RMSE value of the proposed method is the smallest in the three methods, and the RMSE value of the pixel averaging method is the largest. The results presented here can demonstrate that the proposed technique can fuse the image with conveying less error than other two methods. (a)
(c)
(b)
(d)
(e) (f) Fig. 4. Image fusion with the simulated pair from Lena image. (a) The original image (reference image or ground truth); (b) image blurred on the left; (c) image blurred on the right; (d) fused image by pixel averaging; (e) fused image by DWT; (f) fused image by the proposed method.
© 2011 ACADEMY PUBLISHER
TABLE I.
RMSE OF THE THREE FUSION METHODS IN FIG.4.
Methods
Pixel averaging
RMSE
8.762
DWT 5.382
Proposed method 4.165
The second examples are two medical images, one is a T1-weighted MR image, and another is a MRA image as show in Fig.5 (a) and (b), respectively. From these two images, it can be seen that in the T1-weighted MR image, the soft tissue is clear and easy to recognize, but the some important medical information as shown in the marked ellipse area of Fig.5 (b) has been lost. On the contrary, although the MRA image contains the medical information which can not be seen in the Fig.5 (a), the soft tissues are very difficult to distinguish due to its lower spatial resolution. Therefore, in order to support entire and accurate medical information, the fusion of these two images is required. The three methods mentioned above are then used to fuse the two images, and their corresponding results are displayed in Figs.5 (c)-(e), respectively. As can be seen, with all the methods the fused images now preserve the whole regions of interest (ROI) presented in the two images. However, by visual inspection it can be seen that the fused result of the proposed method is more clearly
96
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
and has a higher contrast than other methods, but it is hard to discriminate the fused images of the pixel averaging, and the DWT methods in this case. Hence, in order to better evaluate the above fusion methods, quantitative assessments of the performance of the three methods are needed. However, as we know that in fact for image fusion it is often hard to get the ideal or reference composite image, so the above RMSE metric cannot be used here. As a result, three other evaluation criteria including the average gradient, the information entropy, and the overall cross entropy are then introduced and employed in this paper [17]-[18].
The above three evaluation criteria are then used to evaluate the performance of the methods, and their corresponding results are listed in Table II. From Table II, we can find that the proposed method is the best in the three methods because it not only has the highest average gradient and information entropy values, but also has the lowest value of the overall cross entropy, which means our method can get higher contrast fused image with more information and less difference to the source images.
(i) Average gradient The average gradient of an image with size of M × N is defined as:
Avg =
1 (M − 1)× (N − 1) (12)
⎡⎛ ∂f (m, n ) ⎞ 2 ⎛ ∂f (m, n ) ⎞ 2 ⎤ ⎟ ⎥ 2 ⎟ +⎜ ⎢⎜ ∑∑ m=1 n =1 ⎢⎣⎝ ∂m ⎠ ⎝ ∂n ⎠ ⎥⎦ where f (m, n ) is the pixel value of the fused image at the position (m, n ) . The average gradient reflects the M −1 N −1
(a)
(b)
clarity of the fused image. It is used to measure the spatial resolution of the fused image, i.e., larger average gradient means a higher resolution. (ii) Information entropy The formulation of the classical information entropy of an image is defined as:
(c)
(d)
L −1
H = −∑ Pl log 2 Pl ,
(13)
l =0
where
L is the number of gray level, Pl equals the ratio
between the number of pixels whose gray value is l (0 ≤ l ≤ L − 1) and the total pixel number contained in the image. The information entropy measures the richness of information in an image. Thus, the higher the entropy, the better the performance. (iii) Overall Cross entropy (OCE) The cross entropy is used to measure the difference between the source images and the fused image. Small value corresponds to good fusion result obtained: L −1
P (14) CE = ∑ Pl log 2 l . Ql l =0 where Pl and Ql denote the gray level histogram of the source image and fused image, respectively. Therefore, the overall cross entropy can be defined as:
CE ( A, F ) + CE (B, F ) (15) 2 where A and B are the source images, and F is the OCE =
fused image.
© 2011 ACADEMY PUBLISHER
(e) Fig.5. Fusion results of the T1-weighted MR and MRA images with different methods. (a) Original T1-weighted MR image; (b) original MRA image; (c) fused image by pixel averaging; (d) fused image by DWT; (e) fused image by the proposed method.
TABLE II.
PERFORMANCE COMPARISON OF THE THREE FUSION METHODS IN FIG.5.
Methods Average gradient Information entropy Overall cross entropy
Pixel averaging
DWT
Proposed method
6.478
10.343
10.570
5.959
6.233
6.577
2.603
2.509
2.194
The last examples are two remote sensing images [19], which were captured by Daedalus scanner. Fig.6 (a) and (b) are the two panchromatic source images. The two
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
images are then fused by the above three methods and their resultant images are presented in Figs.6 (c)-(e), respectively. From the fusion results, we can easily observe that the result of pixel averaging method has a lower contrast than those of the DWT method and the proposed method, but it is difficult to discriminate the difference of the results of the latter two methods. Therefore, as the second examples we also carried out the quantitative assessments for the three methods with the criteria (12)-(15), and their evaluation results are given in Table III. From Table III, we can find that although the proposed method has a little bigger value of the overall cross entropy than that of the pixel averaging, it has the best values of the average gradient and the information entropy. Thus, we can conclude that our fusion method performs best in the three methods.
(a)
(b)
97
Therefore, based on the all the experimental results presented here, we can see that all the quantitative evaluations are basically corresponding to the visual effects, and the proposed DWT based fusion method outperforms both the pixel averaging method and the DWT method. V. CONCLUSION Image fusion is to integrate complementary information from multiple images of the same scene into an image, so that the resultant image is more suitable for the purpose of human visual perception and computer processing tasks. In this paper, a novel image fusion technique based on wavelet transform is presented, which is developed by taking into account the physical meaning of the wavelet coefficients. In our method, after the images to be fused are decomposed by the wavelet transform, the coefficients of the high frequency bands and low frequency band are performed with new fusion schemes: the former is selected by a variance based scheme and the latter is selected by an edge based scheme. The performance of the proposed method is quantitatively compared with those of pixel averaging method and wavelet based methods. Experimental results clearly demonstrate the feasibility and effectiveness of the proposed method. ACKNOWLEDGMENT This work is supported by the National Natural Science Foundation of China under the grant No.60963012, and by the China Postdoctoral Special Science Foundation funded project under the grant No. 200902614.
(c)
(d)
REFERENCES
(e) Fig.6. Fusion results of different algorithms on two remote sensing images.(a) Original source image 1; (b) original source image 2; (c) fused image by pixel averaging; (d) fused image by DWT; (e) fused image by the proposed method. TABLE III.
PERFORMANCE COMPARISON OF THE THREE FUSION METHODS IN FIG.6.
Methods Average gradient Information entropy Overall cross entropy
Pixel averaging
DWT
Proposed method
5.811
10.701
10.743
6.412
6.651
7.146
0.931
1.137
1.057
© 2011 ACADEMY PUBLISHER
[1] V. S. Petrovic, C. S. Xydeas, “Gradient-Based Multiresolution Image Fusion,” IEEE Transactions on Image Processing, vol. 13, no.2, pp. 228–237, 2004. [2] Z. Zhang and R. S. Blum, “A categorization of multiscaledecomposition-based image fusion schemes with a performance study for a digital camera application,” Poceedings of the IEEE, vol. 87, no. 8, pp. 1315–1326, 1999. [3] Y. Wang, B. Lohmann, “Multisensor image fusion: concept, method and applications,” Tech. Report, Institute of Automatic Technology, Univ. Bremen, 2000. [4] S. Zheng, W. Z. Shi, J. Liu, G. X. Zhu, and J. W. Tian, “Multisource Image Fusion Method Using Support Value Transform,” IEEE Transactions on Image Processing, vol. 16, no. 7, pp. 1831–1839, 2007. [5] R. Redondo, F. Sroubek, S. Fischer, G. Cristobal, “Multifocus image fusion using the log-Gabor transform and a Multisize Windows technique,” Information Fusion, vol. 10, no. 2, pp. 163–171, 2009. [6] S. T. Li, B. Yang, “Multifocus image fusion using region segmentation and spatial frequency,” Image and Vision Computing, vol. 26, no. 7, pp. 971–979, 2008. [7] S. T. Li, J. T. Kwok, Y. N. Wang, “Multifocus image fusion using artificial neural networks,” Pattern Recognition Letters, vol. 23, no. 8, pp. 985–997, 2002.
98
[8] J. Yang and R. S. Blum, “A statistical signal processing approach to image fusion for concealed weapon detection,” in Proc. IEEE International Conference on Image Processing, vol. 1, pp. 513–516, 2002. [9] H. Li, B. S. Manjunath, and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. [10] G. Pajares, J. M. D. L. Cruz, “A wavelet-based image fusion tutorial, Pattern Recognition,” vol. 37, no. 9, pp. 1855–1872, 2004. [11] K. Amolins, Y. Zhang, P. Dare, “Wavelet based image fusion techniques—An introduction, review and comparison,” ISPRS Journal of Photogrammetry & Remote Sensing, vol. 62, no. 4, pp. 249–263, 2007. [12] R. Z. Zhao, L. Xu, G. X. Song, “Multiscale Image Data Fusion with Wavelet Transform,” Journal of ComputerAided Design & Computer Graphics, vol. 14, no. 4, pp. 361–364, 2002. [13] S. L. Cheng, J. M. He, Z. W. Lv, “Medical Image of PET/CT Weighted Fusion Based on Wavelet Transform,” in IEEE Proc. the 2nd International Conference on Bioinformatics and Biomedical Engineering, pp. 2523– 2525, 2008. [14] S. G. Mallat, “A theory for multiresolution signal decomposition: the wavelet representation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 7, pp. 674–693, 1989. [15] S. G. Nikolov, P. Hill, D. R. Bull, and C. N. Canagarajah, “Wavelets for image fusion,” in Wavelets in Signal and Image Analysis, Computational Imaging and Vision Series, Kluwer Academic Publishers, pp. 213–244, 2001.
© 2011 ACADEMY PUBLISHER
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
[16] R. Chao, K. Zhang, Y. J. Yan, “An Image Fusion Algorithm Using Wavelet Transform,” Acta Electronica Sinica, vol. 32, no. 5, pp. 750–753, 2004. [17] Z. Xue, R. S. Blum, and Y. Li, “Fusion of Visual and IR Images for Concealed Weapon Detection,” Proceedings of the Fifth International Conference on Information Fusion, vol. 2, pp. 1198–1205, 2002. [18] Y .Q. Zhao, Q. Pan, H. C. Zhang, “New polarization imaging method based on spatially adaptive wavelet image fusion,” Optical Engineering, vol. 45, no. 12, Article Number: 123202, 2006. [19] http://www.imagefusion.org/.
Yong Yang received his Ph.D. degree in Biomedical Engineering from Xi’an Jiaotong University, China in 2005. He has been an associate professor in the School of Information Technology, Jiangxi University of Finance and Economics since 2007. From June 2009 to February 2010, he was a postdoctoral research fellow at the Chonbuk National University, Korea. His main research interests include image processing, pattern recognition and machine vision. He has published more than 50 research articles in international journals and conferences.
91
Multiresolution Image Fusion Based on Wavelet Transform By Using a Novel Technique for Selection Coefficients Yong Yang School of Information Technology Jiangxi University of Finance and Economics, Nanchang 330013, China Email: [email protected]
Abstract—Today, image fusion as one kind of information integrated technology has played an important role in many fields. This paper presents a novel wavelet based technique for image fusion, which is developed by taking into account the physical meaning of the wavelet coefficients. The images to be fused are firstly decomposed into high frequency and low frequency bands by wavelet transform. Then, by considering the physical meaning of the coefficients, a new selection strategy that treats the coefficients in different ways is proposed. This strategy selects the coefficients of high frequency bands with a variance based scheme, while selects the coefficients of low frequency band with an edge based scheme. Finally, the fused image is constructed by an inverse wavelet transform performing on the combined coefficients from all frequency bands. The proposed method is validated and compared with some existing fusion methods. Experimental results can clearly indicate the superiority of the proposed method through visual inspection and objective fusion performance measurements. Index Terms—wavelet transform, image fusion, coefficients, variance
I. INTRODUCTION Nowadays, with the rapid improvement of numerous imaging sensors, image fusion has been widely used in many fields, such as remote sensing, medical imaging, machine vision, etc. Image fusion can be generally defined as the process of combing multiple input images or some of their features into a single image without the introduction of distortion or loss of information [1]. The aim of image fusion is to integrate complementary as well as redundant information from multiple images to create a fused image output. Therefore, the new image generated should contain a more accurate description of the scene than any of the individual source images and is more suitable for human visual and machine perception or further image processing and analysis tasks [2]. Over the past decades, many techniques for image fusion have been proposed and a thorough overview of Manuscript received July 3, 2010; revised September 3, 2010; accepted September 23, 2010. Corresponding author: E-mail: [email protected].
© 2011 ACADEMY PUBLISHER doi:10.4304/jmm.6.1.91-98
these methods can be viewed in reference [3]. According to the stage at which the combination mechanism takes place, the image fusion methods can be generally grouped into three categories, namely, pixel level, feature level, and decision level [4]. Since the pixel level fusion has the advantage that the images used contain the original measured quantities, and the algorithms are computationally efficient and easy to implement, the most image fusion applications employ pixel level based methods [5]. Therefore, in this paper, we are still concerned about pixel level fusion The simplest way of pixel level image fusion is to take the average of the two images pixel by pixel. However, this method usually leads to undesirable side effect such as reduced contrast [6]. More robust algorithm for pixel level fusion is the weighted average approach. In this method, the fused pixel is estimated as the weighted average of the corresponding input pixels. However, the weight estimation usually requires a user-specific threshold. Artificial neural network (ANN) has also been introduced to make image fusion, as seen in [7]. However, the performance of ANN depends on the sample images and this is not an appealing characteristic. Yang et al. used a statistical approach to fuse the images [8]; however, in his method the distortion is modeled as a mixture of Gaussian probability density functions (pdfs) which is a limiting assumption. Due to the multiresolution transform can contribute a good mathematical model of human visual system (HVS) and can provide information on the contrast changes, the multiresolution techniques have then attracted more and more interest in image fusion. The multiresolution techniques involve two kinds, one is pyramid transform another is wavelet transform. Examples of pyramid approach include the Laplacian pyramid, the gradient pyramid, and the morphological pyramid, etc [7]. However, for the reason of the pyramid method fails to introduce any spatial orientation selectivity in the decomposition process, the above mentioned methods often cause blocking effects in the fusion results [9]. Another family of the multiresolution fusion techniques is the wavelet based method, which usually used the discrete wavelet transform (DWT) in the fusion. Since the DWT of image signals produces a
92
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
nonredundant image representation, it can provide better spatial and spectral localization of image information as compared to other multiresolution representations. Therefore, the wavelet based method has been popular widely used for image fusion [10]-[11]. In the wavelet transform based image fusion, the input images are first transformed into their multiresolution representations. The fusion process then creates a new composite multiresolution representation from these inputs by a certain fusion rule. The fused image is finally reconstructed by performing an inverse wavelet transform. Therefore, we can find the key step for wavelet based image fusion is the definition of the fusion rule. The widely used fusion rule is the maximum selection (MS) scheme [9]. This simple scheme just selects the largest absolute wavelet coefficient at each location from the input images as the coefficient at the location in the fused image. However, as we know that the noise and artifacts usually has higher salient features in the image, therefore, this method is sensitive to noise and artifacts. Zhao et al. [12] proposed another fusion rule that selected the coefficients in the low frequency band by a weighted average scheme, and selected the coefficients in the high frequency bands by the MS scheme. However, for the shortcoming of weighted average scheme this method will blur and reduce the contrast of features appearing in the image. More recently, Chen et al. [13] proposed a weighted fusion rule for fusion of PET and CT images. The disadvantage of this method is that the weights are determined by the user himself but can not be automatically obtained. In this paper, a novel and fully automated waveletbased method for image fusion is presented. The main contribution of this work is that by considering the physical meaning of the wavelet coefficients, we proposed a new fusion strategy for image fusion, in which the coefficients of the high frequency portions are performed by a variance based scheme, while the coefficients of the low frequency portion are performed by an edge based scheme. Both qualitative and quantitative experiments can prove our technique is effective and promising. The remainder of the paper is organized as follows. Section II describes the general wavelet transform based technique for image fusion. The proposed method for image fusion is presented in Section III. Experimental results and comparisons are given in IV. Finally, some conclusions are drawn in Section V. II. GENERAL IMAGE FUSION BASED ON DWT The original concept and theory of wavelet-based multiresolution analysis came from Mallat [14]. The wavelet transform is a mathematical tool that can detect local features in a signal process. It also can be used to decompose two-dimensional (2-D) signals such as 2-D gray-scale image signals into different resolution levels for multiresolution analysis.
A. DWT Analysis There are two main groups of transforms, continuous and discrete. Of particular interest is the DWT, which is a spatial-frequency decomposition that provides a flexible multiresolution analysis of an image [15]. In one dimension (1-D) the basic idea of the DWT is to represent the signal as a superposition of wavelets. Suppose a discrete signal is represented by f (t ) , the wavelet decomposition is then defined as:
f (t ) = ∑ cm ,nψ m ,n (t ) ,
(1)
m ,n
where
ψ m ,n (t ) = 2 − m 2ψ [2 − m t − n] , m
and
n are
integers. There exist very special choices of ψ such that
ψ m,n (t )
constitutes an orthonormal basis, so that the
wavelet transform coefficients can be obtained by an inner calculation:
cm,n = f ,ψ m,n = ∫ψ m,n (t ) f (t )dt
(2)
In order to develop a multiresolution analysis, a scaling function φ is needed, together with the dilated and translated version of it,
φm ,n (t ) = 2− m 2 φ [2− m t − n ] .
According to the characteristics of the scale spaces spanned by φ and ψ , the signal f (t ) can be decomposed in its coarse part and details of various sizes by projecting it onto the corresponding spaces. Therefore, to find such decomposition explicitly, additional coefficients am ,n are required at each scale. At each scale
am ,n and am−1,n describe the approximations
of the function resolution
2 m −1
f at resolution 2 m and at the coarser respectively, while the coefficients cm ,n
describe the information loss when going from one approximation to another. In order to obtain the coefficients cm ,n and a m ,n at each scale and position, a scaling function is needed that is similarly defined to equation (2). The approximation coefficients and wavelet coefficients can be obtained:
a m ,n = ∑ h2 n − k a m −1,k ,
(3)
cm ,n = ∑ g 2 n −k a m −1,k ,
(4)
k
k
where hn is a low pass FIR filter and g n is related high pass FIR filter. To reconstruct the original signal the analysis filters can be selected from a biorthogonal set which have a related set of synthesis filters. These
~
~ can be used to perfectly synthesis filters h and g reconstruct the signal using the reconstruction formula
~ am−1,l ( f ) = ∑ [h2 n −l am ,n ( f ) + g~2 n−l cm ,n ( f )] (5) n
Equations (3) and (4) are implemented by filtering and downsampling. Conversely equation (5) is implemented by an initial upsampling and a subsequent filtering. © 2011 ACADEMY PUBLISHER
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
In a 2-D DWT, a 1-D DWT is first performed on the rows and then columns of the data by separately filtering and downsampling. This results in one set of approximation coefficients and three sets of detail coefficients, which represent the horizontal, vertical and dialog directions of the image I , respectively. In the language of filter theory, these four subimages correspond to the outputs of low–low (LL), low–high (LH), high–low (HL), and high–high (HH) bands. By recursively applying the same scheme to the LL subband a multiresolution decomposition with a desires level can then be achieved. Therefore, a DWT with K decomposition levels will have M = 3 ∗ K + 1 such frequency bands. Fig.1 shows the 2-D structures of the wavelet transform with two decomposition levels. It should be noted that for a transform with K levels of decomposition, there is always only one low frequency band ( LLK in Fig.1), the rest of bands are high frequency bands in a given decomposition level. LL2
LH2 LH1
HL2
HH2
HL1
HH1
Fig.1. 2-D DWT structure with labeled subbands in two-level decomposition.
B. DWT Based Image Fusion In this subsection, to better understand the concept and procedure of the wavelet based fusion technique, a schematic diagram is first given in Fig.2. In general, the basic idea of image fusion based on wavelet transform is to perform a multiresolution decomposition on each source image, the coefficients are then performed with a certain fusion rule as displayed in the middle block of Fig.2. After that, the fused image is obtained by performing the inverse DWT (IDWT) for the corresponding combined wavelet coefficients. Therefore, as shown in Fig.2, the detailed fusion steps based on wavelet transform can be summarized below.
Fig.2. Schematic diagram of the wavelet based image fusion.
© 2011 ACADEMY PUBLISHER
93
Step 1: The images to be fused must be registered to assure the corresponding pixels are aligned. Step 2: These images are decomposed into wavelet transformed images respectively, based on DWT. The transformed images with K -level decomposition will include one low frequency portion and 3K high frequency portions. Step 3: The transform coefficients of different portions or bands are performed with a certain fusion rule. Step 4: The fused image is constructed by performing the IDWT based on the combined transform coefficients from step 3. III. THE PROPOSED FUSION TECHNIQUE As shown in the above schematic diagram of Fig.2, it is easy to see that the core step in wavelet based image fusion is the definition of the fusion rule because it will decide how to merge the coefficients in an appropriate way so that a high quality fused image can be obtained. However, as we know that the most popular widely used fusion rule is the aforementioned MS scheme. Although, this method can select the salient features from the source images; it is sensitive to noise and artifacts. Therefore, with this method some noise and artifacts are easily introduced into the fused image, which will reduce the resultant image quality. Averaging fusion rule may be another alternative method and it can lead to a stabilization of the fusion result. However, this scheme tends to blur images and reduce the contrast of features appearing in only one image. More importantly, all these fusion rules (including Chen et al.[13]) usually treat the coefficients in the same way. However, as we know that when an image is decomposed by the wavelet transform the coefficients of low frequency band and high frequency bands must have their quite different physical meaning. So, the coefficients in low frequency and high frequency portions should be performed with different fusion schemes. Therefore, in this paper, we proposed a new strategy for image fusion, which treats the coefficients by using different schemes in different frequency bands. A. Procedure of the proposed method In this section, as the procedure of the general wavelet based fusion method (shown in Fig.2), the block diagram of the proposed approach is given in Fig.3. The basic idea of our method is to perform a multiresolution decomposition on each source image; the coefficients are then performed with a certain fusion rule as displayed in the middle block of Fig.3. After taking in to account the physical meaning of the coefficient, this paper presents a novel fusion scheme that treats the coefficients of the high frequency bands and low frequency bands separately: the former are performed by a maximal variance based scheme, while the latter is performed by a maximal edge based scheme. Finally, the fused image is obtained by performing the IDWT on the combined wavelet coefficients. It is important to note that the high frequency bands include the vertical, horizontal, and
94
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
Fig. 3. The block diagram of the proposed fusion technique.
diagonal high frequencies of the image, respectively. Therefore, the fusion process must be performed in all these domains. To simplify the description of the different alternatives available in forming a fusion rule, as reference [10] we also consider only two source images, X and Y , and the fused image Z . The method can of course be easily extended to more than two images. Generally, an image I has its multiscale decomposition (MSD) representation denoted as DI . Hence we will encounter DX , DY , and DZ . Let p = (m, n, k , l ) indicate the index corresponding to a particular MSD coefficient, where m and n indicate the spatial position in a given frequency band, k is the decomposition level, and l is the frequency band of the MSD representation. Therefore, DI ( p ) denote the MSD value of the corresponding coefficient.
B. Selection Scheme for High Frequency Bands The high frequency bands contain the detail coefficients of an image, which usually have large absolute values correspond to sharp intensity changes and preserve salient information in the image. In addition, based on the requirements of image fusion [10], we know that the purpose of image fusion requires that the fused image must not discard any useful information contained in the source images and effectively preserve the details. Therefore, it is important to find appropriate scheme to merge the details of the input images. On the other hand, according to characteristic of HVS it is easy to see that for the high resolution region the human visual interest is concentrated on the detection of changes in contrast between regions on the edges separate these regions. Therefore, a good method for the high frequency bands should produce large coefficients on those edges. Based on the above analysis, we propose a scheme by computing the variance in a neighborhood to select the high frequency coefficients. The procedure can be formulated as the following:
1 σ I ( p) = S ×T
⎛ DI (m + s, n + t , k , l )⎞ ⎟⎟ (6) ⎜⎜ ∑ ∑ s =−S 2 t =−T 2 ⎝ − meanI ( p ) ⎠ S 2
T 2
© 2011 ACADEMY PUBLISHER
2
meanI ( p ) =
1 S ×T
S 2
T 2
∑ ∑ D (m + s, n + t, k , l )
s =−S 2 t =−T 2
I
(7)
where S × T is the neighboring size, meanI ( p ) ,
σ I ( p ) denote the mean value and variance value of the
coefficients centered at
(m, n )
in the window of
S × T respectively. Then, the fusion scheme used for the high frequency bands can be illustrated as the following:
⎧D ( p ) σ X ( p ) ≥ σ Y ( p ) DZ ( p ) = ⎨ X ⎩ DY ( p ) σ X ( p ) < σ Y ( p )
(8)
C. Selection Scheme for Low Frequency Band However, differing from the high frequency bands the low frequency band is the original image at the coarser resolution level, which can be considered as a smoothed and subsampled version of the original image. Therefore, most information of their source images is kept in the low frequency band. So, if the scheme mentioned above is adopted here, the fused results will be blocked. In this paper, in order to better improve the quality of the fused result, an edge-based scheme for selection the coefficients in low frequency band is performed consequently. This scheme firstly calculated the edges in the horizontal, vertical and diagonal directions as the following [16]:
Edge( p ) = (F1 ∗ D ) ( p ) + (F2 ∗ D ) ( p ) 2
2
(9)
+ (F3 ∗ D ) ( p ) where the symbol ∗ denote the convolution, and F1 = {{− 1, − 1, − 1}, {2, 2, 2}, {− 1, − 1, − 1}}, F2 = {{− 1, 2, − 1}, {− 1, 2, − 1}, {− 1, 2, − 1}}, F3 = {{− 1, 0, − 1}, {0, 4, 0}, {− 1, 0, − 1}}. 2
After obtaining the edges of the source images and in order to retain much more information from those images, the coefficients of the low frequency band are then chosen by the following formula: DZ ( p ) = wX ( p )DX ( p ) + wY ( p )DY ( p ) (10)
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
⎧1, Edge X ( p ) > EdgeY ( p ) , otherwise ⎩0, ⎧1, Edge X ( p ) ≤ EdgeY ( p ) . wY ( p ) = ⎨ otherwise ⎩0,
where, w X ( p ) = ⎨
Once all the coefficients are achieved from the above two procedures, an inverse wavelet transform is then performed on them, the fused image is thus constructed. IV. EXPERIMENTAL RESULTS AND ANALYSIS In this section, the application results of the proposed DWT based method for image fusion are presented. The performance of the proposed method is compared with those of pixel averaging method [9], the DWT method proposed by Zhao et al. [12]. To better describe the experiments, as in reference [10], in all test cases we assume the source images to be in perfect registration. We use the Daubechies’ db8, with a decomposition level of 3, as the wavelet basis for DWT method and the proposed method. A 3 × 3 window size for calculating the variance is considered in this paper.
95
The first experiment is performed on a popular widely used standard image Lena of size 256× 256 as shown in Fig. 4 (a), which served as the ideal reference image here. Then two other images are generated by filtering the reference image with a Gaussian blurring process as reference [2]. Fig. 4 (b) is the image blurred on the left, while Fig. 4 (c) is the image blurred on the right. Figs.4 (d)-(f) are the fused results obtained by fusing Fig. 4 (b) and Fig. 4 (c) with the three methods mentioned above, respectively. It is easy to see that from the fused images by visual inspection it is difficult to find the difference of the three methods except that Fig.4 (d) has a lower contrast. Therefore, a frequently used metric, root mean square error (RMSE) [9] is employed here to objectively evaluate the performance of the three methods. This metric can indicate how much error the fused image conveys about the reference image. Hence, the lower the RMSE, the better the fused result. The RMSE is defined as: 12
⎛ 1 N M (xR (n, m) − xF (n, m))2 ⎞⎟ RMSE = ⎜ ∑∑ ⎠ ⎝ MN n=1 m=1
(11)
xR and xF denote the ideal reference image and fused image, respectively, and M and N are the where
dimensions of the image. The RMSE values of the three methods are calculated and showed in Table I. It can be seen from Table I that the RMSE value of the proposed method is the smallest in the three methods, and the RMSE value of the pixel averaging method is the largest. The results presented here can demonstrate that the proposed technique can fuse the image with conveying less error than other two methods. (a)
(c)
(b)
(d)
(e) (f) Fig. 4. Image fusion with the simulated pair from Lena image. (a) The original image (reference image or ground truth); (b) image blurred on the left; (c) image blurred on the right; (d) fused image by pixel averaging; (e) fused image by DWT; (f) fused image by the proposed method.
© 2011 ACADEMY PUBLISHER
TABLE I.
RMSE OF THE THREE FUSION METHODS IN FIG.4.
Methods
Pixel averaging
RMSE
8.762
DWT 5.382
Proposed method 4.165
The second examples are two medical images, one is a T1-weighted MR image, and another is a MRA image as show in Fig.5 (a) and (b), respectively. From these two images, it can be seen that in the T1-weighted MR image, the soft tissue is clear and easy to recognize, but the some important medical information as shown in the marked ellipse area of Fig.5 (b) has been lost. On the contrary, although the MRA image contains the medical information which can not be seen in the Fig.5 (a), the soft tissues are very difficult to distinguish due to its lower spatial resolution. Therefore, in order to support entire and accurate medical information, the fusion of these two images is required. The three methods mentioned above are then used to fuse the two images, and their corresponding results are displayed in Figs.5 (c)-(e), respectively. As can be seen, with all the methods the fused images now preserve the whole regions of interest (ROI) presented in the two images. However, by visual inspection it can be seen that the fused result of the proposed method is more clearly
96
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
and has a higher contrast than other methods, but it is hard to discriminate the fused images of the pixel averaging, and the DWT methods in this case. Hence, in order to better evaluate the above fusion methods, quantitative assessments of the performance of the three methods are needed. However, as we know that in fact for image fusion it is often hard to get the ideal or reference composite image, so the above RMSE metric cannot be used here. As a result, three other evaluation criteria including the average gradient, the information entropy, and the overall cross entropy are then introduced and employed in this paper [17]-[18].
The above three evaluation criteria are then used to evaluate the performance of the methods, and their corresponding results are listed in Table II. From Table II, we can find that the proposed method is the best in the three methods because it not only has the highest average gradient and information entropy values, but also has the lowest value of the overall cross entropy, which means our method can get higher contrast fused image with more information and less difference to the source images.
(i) Average gradient The average gradient of an image with size of M × N is defined as:
Avg =
1 (M − 1)× (N − 1) (12)
⎡⎛ ∂f (m, n ) ⎞ 2 ⎛ ∂f (m, n ) ⎞ 2 ⎤ ⎟ ⎥ 2 ⎟ +⎜ ⎢⎜ ∑∑ m=1 n =1 ⎢⎣⎝ ∂m ⎠ ⎝ ∂n ⎠ ⎥⎦ where f (m, n ) is the pixel value of the fused image at the position (m, n ) . The average gradient reflects the M −1 N −1
(a)
(b)
clarity of the fused image. It is used to measure the spatial resolution of the fused image, i.e., larger average gradient means a higher resolution. (ii) Information entropy The formulation of the classical information entropy of an image is defined as:
(c)
(d)
L −1
H = −∑ Pl log 2 Pl ,
(13)
l =0
where
L is the number of gray level, Pl equals the ratio
between the number of pixels whose gray value is l (0 ≤ l ≤ L − 1) and the total pixel number contained in the image. The information entropy measures the richness of information in an image. Thus, the higher the entropy, the better the performance. (iii) Overall Cross entropy (OCE) The cross entropy is used to measure the difference between the source images and the fused image. Small value corresponds to good fusion result obtained: L −1
P (14) CE = ∑ Pl log 2 l . Ql l =0 where Pl and Ql denote the gray level histogram of the source image and fused image, respectively. Therefore, the overall cross entropy can be defined as:
CE ( A, F ) + CE (B, F ) (15) 2 where A and B are the source images, and F is the OCE =
fused image.
© 2011 ACADEMY PUBLISHER
(e) Fig.5. Fusion results of the T1-weighted MR and MRA images with different methods. (a) Original T1-weighted MR image; (b) original MRA image; (c) fused image by pixel averaging; (d) fused image by DWT; (e) fused image by the proposed method.
TABLE II.
PERFORMANCE COMPARISON OF THE THREE FUSION METHODS IN FIG.5.
Methods Average gradient Information entropy Overall cross entropy
Pixel averaging
DWT
Proposed method
6.478
10.343
10.570
5.959
6.233
6.577
2.603
2.509
2.194
The last examples are two remote sensing images [19], which were captured by Daedalus scanner. Fig.6 (a) and (b) are the two panchromatic source images. The two
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
images are then fused by the above three methods and their resultant images are presented in Figs.6 (c)-(e), respectively. From the fusion results, we can easily observe that the result of pixel averaging method has a lower contrast than those of the DWT method and the proposed method, but it is difficult to discriminate the difference of the results of the latter two methods. Therefore, as the second examples we also carried out the quantitative assessments for the three methods with the criteria (12)-(15), and their evaluation results are given in Table III. From Table III, we can find that although the proposed method has a little bigger value of the overall cross entropy than that of the pixel averaging, it has the best values of the average gradient and the information entropy. Thus, we can conclude that our fusion method performs best in the three methods.
(a)
(b)
97
Therefore, based on the all the experimental results presented here, we can see that all the quantitative evaluations are basically corresponding to the visual effects, and the proposed DWT based fusion method outperforms both the pixel averaging method and the DWT method. V. CONCLUSION Image fusion is to integrate complementary information from multiple images of the same scene into an image, so that the resultant image is more suitable for the purpose of human visual perception and computer processing tasks. In this paper, a novel image fusion technique based on wavelet transform is presented, which is developed by taking into account the physical meaning of the wavelet coefficients. In our method, after the images to be fused are decomposed by the wavelet transform, the coefficients of the high frequency bands and low frequency band are performed with new fusion schemes: the former is selected by a variance based scheme and the latter is selected by an edge based scheme. The performance of the proposed method is quantitatively compared with those of pixel averaging method and wavelet based methods. Experimental results clearly demonstrate the feasibility and effectiveness of the proposed method. ACKNOWLEDGMENT This work is supported by the National Natural Science Foundation of China under the grant No.60963012, and by the China Postdoctoral Special Science Foundation funded project under the grant No. 200902614.
(c)
(d)
REFERENCES
(e) Fig.6. Fusion results of different algorithms on two remote sensing images.(a) Original source image 1; (b) original source image 2; (c) fused image by pixel averaging; (d) fused image by DWT; (e) fused image by the proposed method. TABLE III.
PERFORMANCE COMPARISON OF THE THREE FUSION METHODS IN FIG.6.
Methods Average gradient Information entropy Overall cross entropy
Pixel averaging
DWT
Proposed method
5.811
10.701
10.743
6.412
6.651
7.146
0.931
1.137
1.057
© 2011 ACADEMY PUBLISHER
[1] V. S. Petrovic, C. S. Xydeas, “Gradient-Based Multiresolution Image Fusion,” IEEE Transactions on Image Processing, vol. 13, no.2, pp. 228–237, 2004. [2] Z. Zhang and R. S. Blum, “A categorization of multiscaledecomposition-based image fusion schemes with a performance study for a digital camera application,” Poceedings of the IEEE, vol. 87, no. 8, pp. 1315–1326, 1999. [3] Y. Wang, B. Lohmann, “Multisensor image fusion: concept, method and applications,” Tech. Report, Institute of Automatic Technology, Univ. Bremen, 2000. [4] S. Zheng, W. Z. Shi, J. Liu, G. X. Zhu, and J. W. Tian, “Multisource Image Fusion Method Using Support Value Transform,” IEEE Transactions on Image Processing, vol. 16, no. 7, pp. 1831–1839, 2007. [5] R. Redondo, F. Sroubek, S. Fischer, G. Cristobal, “Multifocus image fusion using the log-Gabor transform and a Multisize Windows technique,” Information Fusion, vol. 10, no. 2, pp. 163–171, 2009. [6] S. T. Li, B. Yang, “Multifocus image fusion using region segmentation and spatial frequency,” Image and Vision Computing, vol. 26, no. 7, pp. 971–979, 2008. [7] S. T. Li, J. T. Kwok, Y. N. Wang, “Multifocus image fusion using artificial neural networks,” Pattern Recognition Letters, vol. 23, no. 8, pp. 985–997, 2002.
98
[8] J. Yang and R. S. Blum, “A statistical signal processing approach to image fusion for concealed weapon detection,” in Proc. IEEE International Conference on Image Processing, vol. 1, pp. 513–516, 2002. [9] H. Li, B. S. Manjunath, and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” Graphical Models and Image Processing, vol. 57, no. 3, pp. 235–245, 1995. [10] G. Pajares, J. M. D. L. Cruz, “A wavelet-based image fusion tutorial, Pattern Recognition,” vol. 37, no. 9, pp. 1855–1872, 2004. [11] K. Amolins, Y. Zhang, P. Dare, “Wavelet based image fusion techniques—An introduction, review and comparison,” ISPRS Journal of Photogrammetry & Remote Sensing, vol. 62, no. 4, pp. 249–263, 2007. [12] R. Z. Zhao, L. Xu, G. X. Song, “Multiscale Image Data Fusion with Wavelet Transform,” Journal of ComputerAided Design & Computer Graphics, vol. 14, no. 4, pp. 361–364, 2002. [13] S. L. Cheng, J. M. He, Z. W. Lv, “Medical Image of PET/CT Weighted Fusion Based on Wavelet Transform,” in IEEE Proc. the 2nd International Conference on Bioinformatics and Biomedical Engineering, pp. 2523– 2525, 2008. [14] S. G. Mallat, “A theory for multiresolution signal decomposition: the wavelet representation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 7, pp. 674–693, 1989. [15] S. G. Nikolov, P. Hill, D. R. Bull, and C. N. Canagarajah, “Wavelets for image fusion,” in Wavelets in Signal and Image Analysis, Computational Imaging and Vision Series, Kluwer Academic Publishers, pp. 213–244, 2001.
© 2011 ACADEMY PUBLISHER
JOURNAL OF MULTIMEDIA, VOL. 6, NO. 1, FEBRUARY 2011
[16] R. Chao, K. Zhang, Y. J. Yan, “An Image Fusion Algorithm Using Wavelet Transform,” Acta Electronica Sinica, vol. 32, no. 5, pp. 750–753, 2004. [17] Z. Xue, R. S. Blum, and Y. Li, “Fusion of Visual and IR Images for Concealed Weapon Detection,” Proceedings of the Fifth International Conference on Information Fusion, vol. 2, pp. 1198–1205, 2002. [18] Y .Q. Zhao, Q. Pan, H. C. Zhang, “New polarization imaging method based on spatially adaptive wavelet image fusion,” Optical Engineering, vol. 45, no. 12, Article Number: 123202, 2006. [19] http://www.imagefusion.org/.
Yong Yang received his Ph.D. degree in Biomedical Engineering from Xi’an Jiaotong University, China in 2005. He has been an associate professor in the School of Information Technology, Jiangxi University of Finance and Economics since 2007. From June 2009 to February 2010, he was a postdoctoral research fellow at the Chonbuk National University, Korea. His main research interests include image processing, pattern recognition and machine vision. He has published more than 50 research articles in international journals and conferences.
