Evaluation of ICA Based Fusion of Hyperspectral ... - Semantic Scholar
Evaluation of ICA Based Fusion of Hyperspectral Images for Color Display Yingxuan Zhu Department of Electrical Engineering and Computer Science Syracuse University Syracuse, NY, U.S.A. [email protected]
Pramod K. Varshney Department of Electrical Engineering and Computer Science Syracuse University Syracuse, NY, U.S.A. [email protected]
Abstract - Hyperspectral imaging is becoming increasingly important in a variety of applications. These images contain a large number of contiguous bands to provide information at a fine spectral resolution and, therefore, cannot be displayed directly using an RGB color display. There has been some recent work on the problem of fusing hyperspectral images to three-band images for color display purposes. In this paper, we evaluate the performance of our recently proposed approach based on independent component analysis, correlation coefficient and mutual information (ICACCMI) to fuse the information from a large number of bands to three images suitable for color display. Depending on whether the reference images are available or not, several image quality metrics such as entropy and edge correlation have been proposed and employed to evaluate the fusion performance via three widely used hyperspectral image datasets. Keywords: Hyperspectral images, image quality metrics, independent component analysis, information fusion.
1
Introduction
Hyperspectral images, taken from satellites, or airplanes, consist of images taken in dozens or hundreds of narrow, adjacent spectral bands providing near-continuous spectral information [1]. Because hyperspectral images are usually collected over more than 100 bands [2], the visualization of hyperspectral images is different from that of normal images which are taken by cameras. There are several color models that can be employed for the display of hyperspectral images, e.g., CMY, RGB, CIELAB. In this paper, the RGB color model is employed for display because of its simplicity. One of the primary problems in the visualization of hyperspectral images is how to mathematically and conceptually map the N-band images into RGB images while keeping as much information as possible. In 1988, Robertson and O’Callaghan presented a spatial and spectral computational framework for the display of remotely sensed imagery in [3]. The principal component transformation (PCT) method was proposed to reduce the number of bands and display the hyperspectral remotely
Hao Chen Department of Electrical Engineering and Computer Science Syracuse University Syracuse, NY, U.S.A. [email protected]
sensed images in [4-5]. The principal component based display strategy is also used for spectral imagery in [6], where the first three principal components that contain over 99.3% of the total eigenvalues are mapped to the H, S, V color space. Another important problem in the display of spectral images is how to evaluate the visualization quality. A visual information fidelity measure for image quality assessment is proposed in [7], and validated with an extensive subjective study. Jacobson and Gupta discussed the design goals for hyperspectral image display systems and provided a display method using fixed linear spectral weighting envelopes in [8]. They also introduced a metric based on the relationship between Spectral Angle and CIELAB distance for pairs of pixels to evaluate the visualization quality of spectrally weighted envelopes. Wang, et al. assumed that human visual perception is highly adapted for extracting structural information and used structural similarity (SSIM) for quality assessment in [9]. Despite the progress made in the literature on the display methods for hyperspectral images, little has been done for the image quality assessment of the fused RGB image, especially for the case when no reference information is available. In this paper, to evaluate the display performance, several objective statistical metrics are proposed and employed to evaluate the display performance. The paper is organized as follows. In Section 2, the application of ICA for the visualization of hyperspectral images is briefly introduced and the ICA based dimensionality reduction method, ICA-CCMI, is described. Section 3 introduces the application of some statistical methods for the evaluation of visualization results. Section 4 provides experimental results based on three different hyperspectral image datasets, and compares the results in terms of metrics mentioned in Section 3. Finally, Section 5 includes some concluding remarks.
2 2.1
Using ICA in color representation of hyperspectral data Independent component analysis
As a blind source separation method, ICA is employed to separate sounds, images or telecommunication channel data into individual information components [10].
∑ ∑ (H
The classical ICA model is given by [11-12]: x = As
C H i Aj =
(1)
m
2.2
− H i )( Amnj − A j )
(3)
( ∑ ∑ ( H mni − H i ) )( ∑ ∑ ( Amnj − A j ) ) 2
m
where x is the vector of observed signals, A is the mixing matrix, s is the vector of original signals. ICA attempts to find the matrix W=A-1, which estimates the signal vector s by optimizing a statistical independence criterion [13]. Based on the Central Limit Theorem, the sum of two independent random variables has a closer Gaussian distribution than the two individual variables, so the general idea behind ICA is to try to estimate the matrix W to maximize the non-Gaussianity of Wx.
mni
n
2
n
m
n
where H i and A j indicate the means of the ith and jth images in H and A, respectively. We will disregard the sign of C H A and will employ its absolute value. i
j
3) Calculate mutual information I H i A j of the ith image in H and jth component in A for all i and j;
pH A (x, y) IHi Aj = ∑∑pHi Aj (x, y)log i j = HHi + HAj − HHi Aj pHi (x)pAj (y) y x
FastICA
Depending on different criteria of non-Gaussianity, there are several algorithms to implement ICA; e.g., kurtosis and gradient-based algorithms [10][14]. Considering the enormous computation involved when applying ICA to hyperspectral images, this paper applies the FastICA algorithm [11-12] to realize ICA. The image data is preprocessed by centering and whitening before applying ICA. The FastICA algorithm uses negentropy J as a measure of the non-Gaussianity [11-12].
(4)
where H H and H A are entropies of Hi and Aj, respectively. i j
H Hi Aj is the joint entropy of Hi and Aj. 4) Calculate the average absolute correlation of the jth component in A with hyperspectral image H for all j as:
C A j = ∑ CH i A j / L
(5)
i
J ( y ) = H ( y gauss ) − H ( y )
(2)
where H represents the entropy of a discrete random variable, y gauss is a Gaussian random variable with the same covariance matrix as y. Detailed discussion on ICA can be found in [11-12][14]. The MatlabTM implementation of the FastICA algorithm is available at [15].
The larger the average correlation coefficient value of an independent component, the closer the relationship of this component with the original hyperspectral image dataset. 5) Similar to step 4), compute the average mutual information for the jth components of A for all j as: I A j = ∑ I H i Aj / L
(6)
i
2.3
Dimensionality reduction
Unlike principal component analysis (PCA), which ranks the components by the magnitude of eigenvalues, the order of the independent components after each application of ICA is generally not the same. In our experiments, the components which have the most information with respect to the original hyperspectral images are selected. The proposed idea is realized by using both the correlation coefficient and mutual information. We name it ICACCMI. Specifically, we assume that the intensity of hyperspectral image data is given as an M*N*L matrix H, where L is the number of spectral bands. The FastICA algorithm is applied to H, and the resulting output matrix A consisting of the independent components is also an M*N*L matrix. The three RGB components need to be selected from these L independent components. The main idea of ICA-CCMI is summarized in the following steps. 1) Apply FastICA to the original dataset to obtain A; 2) Calculate the correlation coefficient CHi Aj between every pair of hyperspectral bands (ith band) and independent components (jth component);
Again larger the value of I A j , more the information regarding the original dataset the jth component has. 6) Find the maximum value of C AJ , called max C A . j
Normalize each C AJ using max C A as: j
CA j
N
= CA j / max CAj
(7)
7) Normalize each I A using max I A j as: j
I Aj
N
= I Aj / max I Aj
8) Find the product 1 of CAj
N
and I A j
(8) N
, and then the
selection criterion for independent components is:
1
In this paper, we employ the product to combine the two individual selection criteria. Other combination methods are also possible and are currently being investigated.
CCMI j = CAj
N
/ I Aj
N
(9)
Component(s) with the largest value(s) of CCMIj are to be selected for display.
2.4
Segmentation approaches
One may employ ICA-CCMI to the entire hyperspectral dataset and select the three bands with largest values of CCMIj for display purposes. However, there are two obvious drawbacks when ICA-CCMI is applied to the entire dataset. One is the huge computational burden, and the other is that the use of all the bands may not reveal the information in certain bands. To alleviate this situation we consider segmentation or partitioning of the bands prior to the application of ICA. These segmentation methods are introduced below. 1) Segmented ICA based on equal subgroups: This segmentation method is simple and direct. It divides the total number of bands into three equal size subgroups. Theoretically, the equal subgroup partitioning approach can significantly reduce the computational load, but there is no special information included in each subgroup for further image analysis. After the application of ICA-CCMI to each subgroup, the most suitable component from each subgroup is selected to construct the final RGB image. 2) Segmented ICA based on correlation coefficient: The significant degree of correlation between bands results in a high degree of spectral redundancy [9]. Figure 1 shows the correlation coefficients between every pair of spectral bands for the HYMAP dataset, where the white points represent high degree of correlation and dark points represent less correlation between two bands. According to the correlation values in this figure, the 126 bands can be separated into three subgroups where bands in each subgroup exhibit high correlation amongst each other. These three subgroups can be handled as described earlier. 3) Segmented ICA based on RGB spectrum: According to the human visual system and the region of the electromagnetic spectrum, three subgroups can be selected from all the bands to represent the blue, green and red regions of the visible part of the electromagnetic spectrum [4]. In particular, instead of using all of the bands, this partitioning method just deals with the bands in the spectrum regions of blue, green and red, and ICA-CCMI based methodology is applied to these subgroups.
Figure 1. Correlation coefficient matrix of bands
3
Performance evaluation metrics
To comparatively evaluate the performance of our recently proposed ICA-CCMI algorithm, one possible approach is to examine the images visually in a qualitative manner. However, in order to compare the performance of different approaches, it is desirable to employ quantitative measures. Several visualization design goals are provided in [8] for displaying hyperspectral imagery, including consistent rendering, edge preservation, and smallest effective differences. Based on these goals, we employ several image quality metrics [17] to evaluate the results. We separate the quantitative measures used in this paper into two groups, some with reference images and others without reference images.
3.1
Evaluation without reference images
1) Entropy The entropy of the RGB components represents the information content of each component. Thus, higher the entropy, richer the information content, resulting in a more meaningful representation [4]. H x = −∑ p X ( x) ln p X ( x)
(10)
x
where pX(x)is the probability of pixel value x. 2) Correlation coefficients between the RGB components In statistics, the correlation coefficient denotes the accuracy of a least square fitting to the original data. It is a normalized measure of the strength of the linear relationship between two variables. Correlation is employed in many types of applications, such as image processing where it is used to measure and to quantitatively compare the similarity between images. The two-dimensional normalized correlation function for image processing is shown below: C X ,Y =
∑ ∑ ( X − X )(Y ∑ ∑ (X − X ) ∑ ∑ m
n
mn
mn
2
m
n
mn
m
n
−Y) (Ymn − Y )
(11) 2
where X, Y are two image vectors; and X , Y are the means of X and Y, respectively. C X ,Y is a real number between -1 and 1. The natural color images have a high degree of correlation between the RGB components, so the correlation coefficients between the RGB components in the visualization results will show the similarity of color display images to the true color images.
3.2
Evaluation with reference images
In this part, we provide some evaluation metrics to compare the display results and the available true color images that serve as reference images. 1) Root mean square error Visualization of hyperspectral image data as a color image results in a loss of real visual information [18]. The goal of visualization is to use the hyperspectral bands to create an image as similar as possible to its corresponding true color image, or to satisfy the specific requirement. For this reason, the Root Mean Square Error (RMSE) between the original data and the display image is very direct and useful way to evaluate the quality of the visualization. Let Y[m, n] represent the pixel values in the visualized image, and let X[m, n] denote those of the reference image. The RMSE between Y and X is defined by:
RMSE =
1 MN
N −1 M −1
∑∑ {Y [m, n] − X [m, n]}
2
(12)
n =0 m =0
where the size of the images is M*N. The RMSE is applied to each RGB component in our experiment. Lower the RMSE, more similar the fused image and the reference image. 2) Edge preservation performance Entropy, correlation, and root mean square error mentioned above denote the overall quality of an image, but how to judge the clarity of specific features and relevant information in a scene? Edge detection is the most common approach for characterizing object boundaries and detecting these meaningful discontinuities in images. Because edges represent the jumps in intensity, the goal of edge detection is to mark the points in an image at which the luminous intensity changes sharply. These changes include the discontinuities in depth, the discontinuities in surface orientation, the differences in material properties and the variations in scene illumination. Edge detection reduces significant amount of data, filters huge number of unnecessary details and preserves the important structural properties of an image. In our experiments, the Canny method [19] is employed for edge detection. Canny method provides an almost ideal edge detection scheme. It is less likely to be "fooled" by noise, and more likely to detect true weak edges. This method looks for local maxima of the gradient of an object.
In this method, the gradient is calculated by using the derivative of a Gaussian filter and two thresholds are used to detect strong and weak edges. To evaluate the visualization results, we first apply the Canny method to visualized color images, then compare the binary edge images with those of the true color images. 3) Correlation with true color images As mentioned above, the correlation coefficient measures the degree and direction of correlation between two variables; it also denotes the proportion of covariation between these variables. As a metric here, under certain statistical assumptions, the correlation coefficient gives a linear indication of the similarity between images, and provides useful information about these images. As defined in formula (12), the correlation coefficient between the true color image and the visualized image is used as a useful and potentially powerful tool to quantitatively measure how likely it is that the visualized image is the true color image.
4
Experimental results
Three different hyperspectral image datasets are used here to evaluate the performance of different visualization methods. These three data sets are: (i) Airborne Visible InfraRed Imaging Spectrometer, AVIRIS, 224 bands; (ii) Hyperspectral Digital Imagery Collection Experiment, HYDICE, 210 bands; (iii) HyMap, 126 bands. The PCA based and ICA-CCMI visualization methods are applied to these data sets respectively and the resulting RGB images are evaluated by the statistical metrics. According to the information given regarding the HYMAP dataset, the true color images, used here as a reference to obtain the experimental results, are the 14th, 8th, and 2nd bands, for R, G, B, respectively. The performance comparisons between the ICA based visualization and the PCA based visualization [4] are shown below.
4.1
Visual comparison images
of
color
display
Figures 2-4 show the color display of the ICA based and the PCA based visualizations from AVIRIS, HYDICE and HyMap datasets, respectively. The displays are all based on the equal subgroup segmentation method.
(a) ICA Visualization (b) PCA Visualization Figure 2 ICA and PCA based Visualization of AVIRIS dataset
Entropy AVIRIS HYDICE HyMap
4.3 (a) ICA Visualization (b)PCA Visualization Figure 3 ICA and PCA based Visualization of HYDICE dataset
(a) ICA Visualization (b) PCA Visualization Figure 4 ICA and PCA based Visualization of HyMap dataset From Figures 2-4, it is obvious that the ICA based visualization is much closer to the natural color images than the PCA based visualization. Without saturated regions, the ICA based visualization has a more consistent display, which helps the viewer to have a real impression about the original scene. In contrast, more regions in the PCA based visualization results “pop-out” from the images and attract viewer’s unnecessary attention [9][16]. For instance, in Figure 2, the entire PCA based visualization image is mainly separated into two colors, and the edges are difficult to distinguish, especially in the middle of the image.
4.2
Entropy
Entropy reveals the information content of each component and indicates the significance of representation [4]. Table 1 shows the entropies of each of the RGB components of the ICA and PCA based visualization. The RGB components of the PCA based visualization in each data set are selected to be the first principal components of each equal subgroup, which have most of the energy of that subgroup. Obviously, the entropies of the RGB components from the ICA based visualization method are significantly larger than that of the PCA based visualization method. For instance, the mean value of the entropies of the AVIRIS data set is 6.272 for the ICA based visualization, while it is 2.183 for the PCA based visualization.
Table 1 Entropies of Components Component 1 Component 2 Component 3 ICA PCA ICA PCA ICA PCA 5.945 6.541 5.684
2.270 2.095 1.839
6.274 6.613 7.011
2.537 4.14 3.572
6.598 6.836 5.928
1.737 5.148 1.599
Correlation coefficient between the RGB components
Correlation coefficients between RGB components are presented in Table 2. The degree of correlation between the RGB components of the ICA based visualization method is similar to the correlation found in natural color image [4]. Table 2 Correlation Coefficient between the RGB Components of Different Visualization Methods HyMap R G B R 1 0.979 0.980 True Color G 0.979 1 0.981 B 0.980 0.981 1 R 1 0.831 0.837 ICA(RGB) G 0.831 1 0.805 B 0.837 0.805 1 R 1 0.994 -0.045 G 0.994 1 4.25 × 10 −15 PCA(RGB) 1 B -0.045 4.25 × 10−15
4.4
Root mean square error
The RMSE between the true color image and the ICA and PCA based visualization images, both segmented by RGB spectrum method, are calculated below. The ‘ICA(RGB)’ and ‘PCA(RGB)’ in Table 3 stand for the ICA and PCA results by RGB spectrum partitioning. All the results are for the HYMAP dataset. Promising results for the ICA based visualization are shown in Table 3. The PCA(RGB) visualization has RMSE that is more than twice that of the ICA(RGB) visualization. Table 3 RMSE for Different Visualization Methods RMSE ICARGB PCARGB R 4.16 109.95 G 45.05 119.61 B 38.16 81.63
4.5
Edge preservation in RGB images
The edges and lines in an image usually represent the most important information to the viewer, and the sharp changes in image properties normally reflect significant events and changes of the scene. Hence, edge detection is a frequently used technique in image analysis and evaluation. In this part, a classical edge detection scheme, the Canny method, is applied to all visualization results, and the comparisons are shown in Figure 5. Because the performance of the Canny method depends on the threshold values, the edge correlation coefficient between true color images and visualization images will change according to different threshold values. Figure 5
shows the edge images of the true color image, the ICA based visualization image, and the PCA based visualization image. The threshold values while obtaining these results are all automatically chosen by the toolbox provided in Matlab [19], which are [0.0313, 0.0781] for the ICA visualization and [0.0625 0.1563] for the PCA visualization. The comparisons with different threshold values are shown in the next section. The results in Figure 5 show that though both edge images of the ICA and PCA based visualization have noise, the edge image of the ICA based visualization method is much closer to the edge image of the true color image than the edge image of the PCA based visualization. To quantitatively evaluate the relationship between the true color image and the visualization images, we compare their correlation coefficient next.
(a) Edges of true color image
4.6
Correlation coefficients of edges
Figure 6 shows the edge correlation coefficients between the true color image and the ICA based visualization image with different threshold values used in the Canny edge detection method. In this graph, the maximum correlation coefficient value for the ICA based display is around 0.25, while the maximum value for the PCA based display is less then 0.1, which is much less than the ICA based display. A higher edge correlation coefficient value indicates that the display image preserves more edge information of the scene and provides a more correct display of the original images.
(a) Edge correlation between true color image and ICA based visualization image
(b) Edges of the ICA based visualization image (b) Edge correlation between the true color image and PCA based visualization image Figure 6 Edge correlation between true color image and visualization images.
4.7
(c) Edges of the PCA based visualization image Figure 5 Edges of the true color image, the ICA and PCA based visualization images
Correlation coefficient with the true color image
Correlations between the true color image and the visualization images are presented and compared in Table 4. As earlier, ICA (RGB) and PCA(RGB) mean the ICA and PCA based visualization method with the RGB spectrum partitioning approach. The correlation values in Table 4 show that the ICA based visualization has a considerable correlation with true color images.
Table 4 Correlation With The True Color Images Correlation Coefficient with True Color Image ICA(RGB) PCA(RGB) R 0.999 0.9986 HyMap G 0.8405 0.9921 B 0.8461 -0.0134
Transactions on Geoscience and Remote Sensing, Vol. 41, No. 3, pp. 708-718, March 2003.
5
[8] Nathaniel P. Jacobson and Maya R. Gupta, Design Goals and Solutions for Display of Hyperspectral Images, IEEE Transactions on Geoscience and Remote Sensing, Vol. 43, No. 11, pp. 2684-2692, Nov. 2005.,
Conclusion
Fusion of large-dimensional hyperspectral dataset to enable their visualization as color images is an important problem. This paper described a recently developed novel ICA based approach for this purpose. A comparative performance evaluation of a prior PCA based approach and the ICA based approach was carried out in terms of several image quality metrics. Each metric evaluates an aspect of the results, for example, the entropy evaluates the information contents of the images, the RMSE directly shows the differences between color display images and true color images, etc. It was shown that the ICA based approach performs quite well. Because the hyperspectral images represent more information than the true color images, a good performance when compared with the true color image may not necessarily mean that more information is preserved. Some new quality metrics should be developed to alleviate this situation, for instance, for performance evaluation without reference images. Improved fusion methods for color display also need to be developed.
References [1] TNTmips, Introduction to Hyperspectral imaging, www.microimages.com/getstart/pdf_new/hyprspec.pdf [2] Gerrit Polder and Gerie W.A. M van der Heijden, Visualization of Spectral Images, Proceedings of SPIE Vol. 4553, pp. 132-137, 2001. [3] Philip K. Robertson and John F. O’Callaghan, The Application of Perceptual Color Spaces to the Display of Remotely Sensed Imagery, IEEE Transactions on Geoscience and Remote Sensing, Vol. 26, No. 1, pp. 49-59, January 1988. [4] Vassilis Tsagaris, Vassilis Anastassopoulos and George A. Lampropoulos, Fusion of Hyperspectral Data Using Segmented PCT for Color Representation and Classification, IEEE Transactions on Geoscience and Remote Sensing, Vol.26, No.10, pp. 2365-2375, Jan. 2005. [5] Xiuping Jia and John A.Richard, Segmented Principal Components Transformation for Efficient Hyperspectral Remote-Sensing Image Display and Classification, IEEE Transactions on Geoscience and Remote Sensing, Vol. 37, No. 1, pp. 538-542, January 1999. [6] J. Scott Tyo, Athanasios Konsolakis, David I. Diersen, and Richard Christopher Olsen, Principal-ComponentsBased Display Strategy for Spectral Imagery, IEEE
[7] Hamid Rahim Sheikh and Alan C. Bovik, Image information and visual quality, IEEE Transactions on Image Processing, Vol. 15, pp. 430–444, February. 2006.
[9] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli, Image Quality Assessment: From Error Visibility to structural Similarity, IEEE Transactions on Image Processing, Vol. 13, No. 4, pp. 600612, Apr. 2004. [10] James V. Stone, Independent Component Analysis: a Tutorial Introduction, MIT, MA, 2003 [11] Stephen Roberts and Richard Everson, Independent Component Analysis: Principles and Practice, Cambridge University, United Kingdom, 2001 [12] A. Hyvärinen and E. Oja, Independent component analysis: algorithms and applications, Neural Networks, vol. 13, Issue 4, pp. 411--430, 2000. [13] M. Lennon, G. Mercier, M. C. Mouchot and L. Hubert-Moy, Independent Component Analysis as a tool for the dimensionality reduction and the representation of hyperspectral images, IGARSS, IEEE, 2001 [14] A. Hyvärinen, Survey on Independent Component Analysis, Neural Computing Surveys, 2, pp. 94-128, 1999. [15] [Online]: http://www.cis.hut.fi/projects/ica/fastica/. [16] Christopher G. Healey, Kellogg S. Booth and James T. Enns, Visualizing Real-time Multivariate Data Using Preattentive Processing, ACM Transactions on Modeling an Computer Simulation, Vol. 5, pp. 190-221, July 1995. [17] Ronaldo de Freitas Zampolo and Rui Seara, “A Comparison of Image Quality Metric Performances under Practical Condition”, IEEE International Conference on Image Processing, Vol. 3, pp.1192-1195, September 2005. [18] Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, Prentice Hall, 2002. [19] Canny, John. "A Computational Approach to Edge Detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986. Vol. PAMI-8, pp. 679-698.
Pramod K. Varshney Department of Electrical Engineering and Computer Science Syracuse University Syracuse, NY, U.S.A. [email protected]
Abstract - Hyperspectral imaging is becoming increasingly important in a variety of applications. These images contain a large number of contiguous bands to provide information at a fine spectral resolution and, therefore, cannot be displayed directly using an RGB color display. There has been some recent work on the problem of fusing hyperspectral images to three-band images for color display purposes. In this paper, we evaluate the performance of our recently proposed approach based on independent component analysis, correlation coefficient and mutual information (ICACCMI) to fuse the information from a large number of bands to three images suitable for color display. Depending on whether the reference images are available or not, several image quality metrics such as entropy and edge correlation have been proposed and employed to evaluate the fusion performance via three widely used hyperspectral image datasets. Keywords: Hyperspectral images, image quality metrics, independent component analysis, information fusion.
1
Introduction
Hyperspectral images, taken from satellites, or airplanes, consist of images taken in dozens or hundreds of narrow, adjacent spectral bands providing near-continuous spectral information [1]. Because hyperspectral images are usually collected over more than 100 bands [2], the visualization of hyperspectral images is different from that of normal images which are taken by cameras. There are several color models that can be employed for the display of hyperspectral images, e.g., CMY, RGB, CIELAB. In this paper, the RGB color model is employed for display because of its simplicity. One of the primary problems in the visualization of hyperspectral images is how to mathematically and conceptually map the N-band images into RGB images while keeping as much information as possible. In 1988, Robertson and O’Callaghan presented a spatial and spectral computational framework for the display of remotely sensed imagery in [3]. The principal component transformation (PCT) method was proposed to reduce the number of bands and display the hyperspectral remotely
Hao Chen Department of Electrical Engineering and Computer Science Syracuse University Syracuse, NY, U.S.A. [email protected]
sensed images in [4-5]. The principal component based display strategy is also used for spectral imagery in [6], where the first three principal components that contain over 99.3% of the total eigenvalues are mapped to the H, S, V color space. Another important problem in the display of spectral images is how to evaluate the visualization quality. A visual information fidelity measure for image quality assessment is proposed in [7], and validated with an extensive subjective study. Jacobson and Gupta discussed the design goals for hyperspectral image display systems and provided a display method using fixed linear spectral weighting envelopes in [8]. They also introduced a metric based on the relationship between Spectral Angle and CIELAB distance for pairs of pixels to evaluate the visualization quality of spectrally weighted envelopes. Wang, et al. assumed that human visual perception is highly adapted for extracting structural information and used structural similarity (SSIM) for quality assessment in [9]. Despite the progress made in the literature on the display methods for hyperspectral images, little has been done for the image quality assessment of the fused RGB image, especially for the case when no reference information is available. In this paper, to evaluate the display performance, several objective statistical metrics are proposed and employed to evaluate the display performance. The paper is organized as follows. In Section 2, the application of ICA for the visualization of hyperspectral images is briefly introduced and the ICA based dimensionality reduction method, ICA-CCMI, is described. Section 3 introduces the application of some statistical methods for the evaluation of visualization results. Section 4 provides experimental results based on three different hyperspectral image datasets, and compares the results in terms of metrics mentioned in Section 3. Finally, Section 5 includes some concluding remarks.
2 2.1
Using ICA in color representation of hyperspectral data Independent component analysis
As a blind source separation method, ICA is employed to separate sounds, images or telecommunication channel data into individual information components [10].
∑ ∑ (H
The classical ICA model is given by [11-12]: x = As
C H i Aj =
(1)
m
2.2
− H i )( Amnj − A j )
(3)
( ∑ ∑ ( H mni − H i ) )( ∑ ∑ ( Amnj − A j ) ) 2
m
where x is the vector of observed signals, A is the mixing matrix, s is the vector of original signals. ICA attempts to find the matrix W=A-1, which estimates the signal vector s by optimizing a statistical independence criterion [13]. Based on the Central Limit Theorem, the sum of two independent random variables has a closer Gaussian distribution than the two individual variables, so the general idea behind ICA is to try to estimate the matrix W to maximize the non-Gaussianity of Wx.
mni
n
2
n
m
n
where H i and A j indicate the means of the ith and jth images in H and A, respectively. We will disregard the sign of C H A and will employ its absolute value. i
j
3) Calculate mutual information I H i A j of the ith image in H and jth component in A for all i and j;
pH A (x, y) IHi Aj = ∑∑pHi Aj (x, y)log i j = HHi + HAj − HHi Aj pHi (x)pAj (y) y x
FastICA
Depending on different criteria of non-Gaussianity, there are several algorithms to implement ICA; e.g., kurtosis and gradient-based algorithms [10][14]. Considering the enormous computation involved when applying ICA to hyperspectral images, this paper applies the FastICA algorithm [11-12] to realize ICA. The image data is preprocessed by centering and whitening before applying ICA. The FastICA algorithm uses negentropy J as a measure of the non-Gaussianity [11-12].
(4)
where H H and H A are entropies of Hi and Aj, respectively. i j
H Hi Aj is the joint entropy of Hi and Aj. 4) Calculate the average absolute correlation of the jth component in A with hyperspectral image H for all j as:
C A j = ∑ CH i A j / L
(5)
i
J ( y ) = H ( y gauss ) − H ( y )
(2)
where H represents the entropy of a discrete random variable, y gauss is a Gaussian random variable with the same covariance matrix as y. Detailed discussion on ICA can be found in [11-12][14]. The MatlabTM implementation of the FastICA algorithm is available at [15].
The larger the average correlation coefficient value of an independent component, the closer the relationship of this component with the original hyperspectral image dataset. 5) Similar to step 4), compute the average mutual information for the jth components of A for all j as: I A j = ∑ I H i Aj / L
(6)
i
2.3
Dimensionality reduction
Unlike principal component analysis (PCA), which ranks the components by the magnitude of eigenvalues, the order of the independent components after each application of ICA is generally not the same. In our experiments, the components which have the most information with respect to the original hyperspectral images are selected. The proposed idea is realized by using both the correlation coefficient and mutual information. We name it ICACCMI. Specifically, we assume that the intensity of hyperspectral image data is given as an M*N*L matrix H, where L is the number of spectral bands. The FastICA algorithm is applied to H, and the resulting output matrix A consisting of the independent components is also an M*N*L matrix. The three RGB components need to be selected from these L independent components. The main idea of ICA-CCMI is summarized in the following steps. 1) Apply FastICA to the original dataset to obtain A; 2) Calculate the correlation coefficient CHi Aj between every pair of hyperspectral bands (ith band) and independent components (jth component);
Again larger the value of I A j , more the information regarding the original dataset the jth component has. 6) Find the maximum value of C AJ , called max C A . j
Normalize each C AJ using max C A as: j
CA j
N
= CA j / max CAj
(7)
7) Normalize each I A using max I A j as: j
I Aj
N
= I Aj / max I Aj
8) Find the product 1 of CAj
N
and I A j
(8) N
, and then the
selection criterion for independent components is:
1
In this paper, we employ the product to combine the two individual selection criteria. Other combination methods are also possible and are currently being investigated.
CCMI j = CAj
N
/ I Aj
N
(9)
Component(s) with the largest value(s) of CCMIj are to be selected for display.
2.4
Segmentation approaches
One may employ ICA-CCMI to the entire hyperspectral dataset and select the three bands with largest values of CCMIj for display purposes. However, there are two obvious drawbacks when ICA-CCMI is applied to the entire dataset. One is the huge computational burden, and the other is that the use of all the bands may not reveal the information in certain bands. To alleviate this situation we consider segmentation or partitioning of the bands prior to the application of ICA. These segmentation methods are introduced below. 1) Segmented ICA based on equal subgroups: This segmentation method is simple and direct. It divides the total number of bands into three equal size subgroups. Theoretically, the equal subgroup partitioning approach can significantly reduce the computational load, but there is no special information included in each subgroup for further image analysis. After the application of ICA-CCMI to each subgroup, the most suitable component from each subgroup is selected to construct the final RGB image. 2) Segmented ICA based on correlation coefficient: The significant degree of correlation between bands results in a high degree of spectral redundancy [9]. Figure 1 shows the correlation coefficients between every pair of spectral bands for the HYMAP dataset, where the white points represent high degree of correlation and dark points represent less correlation between two bands. According to the correlation values in this figure, the 126 bands can be separated into three subgroups where bands in each subgroup exhibit high correlation amongst each other. These three subgroups can be handled as described earlier. 3) Segmented ICA based on RGB spectrum: According to the human visual system and the region of the electromagnetic spectrum, three subgroups can be selected from all the bands to represent the blue, green and red regions of the visible part of the electromagnetic spectrum [4]. In particular, instead of using all of the bands, this partitioning method just deals with the bands in the spectrum regions of blue, green and red, and ICA-CCMI based methodology is applied to these subgroups.
Figure 1. Correlation coefficient matrix of bands
3
Performance evaluation metrics
To comparatively evaluate the performance of our recently proposed ICA-CCMI algorithm, one possible approach is to examine the images visually in a qualitative manner. However, in order to compare the performance of different approaches, it is desirable to employ quantitative measures. Several visualization design goals are provided in [8] for displaying hyperspectral imagery, including consistent rendering, edge preservation, and smallest effective differences. Based on these goals, we employ several image quality metrics [17] to evaluate the results. We separate the quantitative measures used in this paper into two groups, some with reference images and others without reference images.
3.1
Evaluation without reference images
1) Entropy The entropy of the RGB components represents the information content of each component. Thus, higher the entropy, richer the information content, resulting in a more meaningful representation [4]. H x = −∑ p X ( x) ln p X ( x)
(10)
x
where pX(x)is the probability of pixel value x. 2) Correlation coefficients between the RGB components In statistics, the correlation coefficient denotes the accuracy of a least square fitting to the original data. It is a normalized measure of the strength of the linear relationship between two variables. Correlation is employed in many types of applications, such as image processing where it is used to measure and to quantitatively compare the similarity between images. The two-dimensional normalized correlation function for image processing is shown below: C X ,Y =
∑ ∑ ( X − X )(Y ∑ ∑ (X − X ) ∑ ∑ m
n
mn
mn
2
m
n
mn
m
n
−Y) (Ymn − Y )
(11) 2
where X, Y are two image vectors; and X , Y are the means of X and Y, respectively. C X ,Y is a real number between -1 and 1. The natural color images have a high degree of correlation between the RGB components, so the correlation coefficients between the RGB components in the visualization results will show the similarity of color display images to the true color images.
3.2
Evaluation with reference images
In this part, we provide some evaluation metrics to compare the display results and the available true color images that serve as reference images. 1) Root mean square error Visualization of hyperspectral image data as a color image results in a loss of real visual information [18]. The goal of visualization is to use the hyperspectral bands to create an image as similar as possible to its corresponding true color image, or to satisfy the specific requirement. For this reason, the Root Mean Square Error (RMSE) between the original data and the display image is very direct and useful way to evaluate the quality of the visualization. Let Y[m, n] represent the pixel values in the visualized image, and let X[m, n] denote those of the reference image. The RMSE between Y and X is defined by:
RMSE =
1 MN
N −1 M −1
∑∑ {Y [m, n] − X [m, n]}
2
(12)
n =0 m =0
where the size of the images is M*N. The RMSE is applied to each RGB component in our experiment. Lower the RMSE, more similar the fused image and the reference image. 2) Edge preservation performance Entropy, correlation, and root mean square error mentioned above denote the overall quality of an image, but how to judge the clarity of specific features and relevant information in a scene? Edge detection is the most common approach for characterizing object boundaries and detecting these meaningful discontinuities in images. Because edges represent the jumps in intensity, the goal of edge detection is to mark the points in an image at which the luminous intensity changes sharply. These changes include the discontinuities in depth, the discontinuities in surface orientation, the differences in material properties and the variations in scene illumination. Edge detection reduces significant amount of data, filters huge number of unnecessary details and preserves the important structural properties of an image. In our experiments, the Canny method [19] is employed for edge detection. Canny method provides an almost ideal edge detection scheme. It is less likely to be "fooled" by noise, and more likely to detect true weak edges. This method looks for local maxima of the gradient of an object.
In this method, the gradient is calculated by using the derivative of a Gaussian filter and two thresholds are used to detect strong and weak edges. To evaluate the visualization results, we first apply the Canny method to visualized color images, then compare the binary edge images with those of the true color images. 3) Correlation with true color images As mentioned above, the correlation coefficient measures the degree and direction of correlation between two variables; it also denotes the proportion of covariation between these variables. As a metric here, under certain statistical assumptions, the correlation coefficient gives a linear indication of the similarity between images, and provides useful information about these images. As defined in formula (12), the correlation coefficient between the true color image and the visualized image is used as a useful and potentially powerful tool to quantitatively measure how likely it is that the visualized image is the true color image.
4
Experimental results
Three different hyperspectral image datasets are used here to evaluate the performance of different visualization methods. These three data sets are: (i) Airborne Visible InfraRed Imaging Spectrometer, AVIRIS, 224 bands; (ii) Hyperspectral Digital Imagery Collection Experiment, HYDICE, 210 bands; (iii) HyMap, 126 bands. The PCA based and ICA-CCMI visualization methods are applied to these data sets respectively and the resulting RGB images are evaluated by the statistical metrics. According to the information given regarding the HYMAP dataset, the true color images, used here as a reference to obtain the experimental results, are the 14th, 8th, and 2nd bands, for R, G, B, respectively. The performance comparisons between the ICA based visualization and the PCA based visualization [4] are shown below.
4.1
Visual comparison images
of
color
display
Figures 2-4 show the color display of the ICA based and the PCA based visualizations from AVIRIS, HYDICE and HyMap datasets, respectively. The displays are all based on the equal subgroup segmentation method.
(a) ICA Visualization (b) PCA Visualization Figure 2 ICA and PCA based Visualization of AVIRIS dataset
Entropy AVIRIS HYDICE HyMap
4.3 (a) ICA Visualization (b)PCA Visualization Figure 3 ICA and PCA based Visualization of HYDICE dataset
(a) ICA Visualization (b) PCA Visualization Figure 4 ICA and PCA based Visualization of HyMap dataset From Figures 2-4, it is obvious that the ICA based visualization is much closer to the natural color images than the PCA based visualization. Without saturated regions, the ICA based visualization has a more consistent display, which helps the viewer to have a real impression about the original scene. In contrast, more regions in the PCA based visualization results “pop-out” from the images and attract viewer’s unnecessary attention [9][16]. For instance, in Figure 2, the entire PCA based visualization image is mainly separated into two colors, and the edges are difficult to distinguish, especially in the middle of the image.
4.2
Entropy
Entropy reveals the information content of each component and indicates the significance of representation [4]. Table 1 shows the entropies of each of the RGB components of the ICA and PCA based visualization. The RGB components of the PCA based visualization in each data set are selected to be the first principal components of each equal subgroup, which have most of the energy of that subgroup. Obviously, the entropies of the RGB components from the ICA based visualization method are significantly larger than that of the PCA based visualization method. For instance, the mean value of the entropies of the AVIRIS data set is 6.272 for the ICA based visualization, while it is 2.183 for the PCA based visualization.
Table 1 Entropies of Components Component 1 Component 2 Component 3 ICA PCA ICA PCA ICA PCA 5.945 6.541 5.684
2.270 2.095 1.839
6.274 6.613 7.011
2.537 4.14 3.572
6.598 6.836 5.928
1.737 5.148 1.599
Correlation coefficient between the RGB components
Correlation coefficients between RGB components are presented in Table 2. The degree of correlation between the RGB components of the ICA based visualization method is similar to the correlation found in natural color image [4]. Table 2 Correlation Coefficient between the RGB Components of Different Visualization Methods HyMap R G B R 1 0.979 0.980 True Color G 0.979 1 0.981 B 0.980 0.981 1 R 1 0.831 0.837 ICA(RGB) G 0.831 1 0.805 B 0.837 0.805 1 R 1 0.994 -0.045 G 0.994 1 4.25 × 10 −15 PCA(RGB) 1 B -0.045 4.25 × 10−15
4.4
Root mean square error
The RMSE between the true color image and the ICA and PCA based visualization images, both segmented by RGB spectrum method, are calculated below. The ‘ICA(RGB)’ and ‘PCA(RGB)’ in Table 3 stand for the ICA and PCA results by RGB spectrum partitioning. All the results are for the HYMAP dataset. Promising results for the ICA based visualization are shown in Table 3. The PCA(RGB) visualization has RMSE that is more than twice that of the ICA(RGB) visualization. Table 3 RMSE for Different Visualization Methods RMSE ICARGB PCARGB R 4.16 109.95 G 45.05 119.61 B 38.16 81.63
4.5
Edge preservation in RGB images
The edges and lines in an image usually represent the most important information to the viewer, and the sharp changes in image properties normally reflect significant events and changes of the scene. Hence, edge detection is a frequently used technique in image analysis and evaluation. In this part, a classical edge detection scheme, the Canny method, is applied to all visualization results, and the comparisons are shown in Figure 5. Because the performance of the Canny method depends on the threshold values, the edge correlation coefficient between true color images and visualization images will change according to different threshold values. Figure 5
shows the edge images of the true color image, the ICA based visualization image, and the PCA based visualization image. The threshold values while obtaining these results are all automatically chosen by the toolbox provided in Matlab [19], which are [0.0313, 0.0781] for the ICA visualization and [0.0625 0.1563] for the PCA visualization. The comparisons with different threshold values are shown in the next section. The results in Figure 5 show that though both edge images of the ICA and PCA based visualization have noise, the edge image of the ICA based visualization method is much closer to the edge image of the true color image than the edge image of the PCA based visualization. To quantitatively evaluate the relationship between the true color image and the visualization images, we compare their correlation coefficient next.
(a) Edges of true color image
4.6
Correlation coefficients of edges
Figure 6 shows the edge correlation coefficients between the true color image and the ICA based visualization image with different threshold values used in the Canny edge detection method. In this graph, the maximum correlation coefficient value for the ICA based display is around 0.25, while the maximum value for the PCA based display is less then 0.1, which is much less than the ICA based display. A higher edge correlation coefficient value indicates that the display image preserves more edge information of the scene and provides a more correct display of the original images.
(a) Edge correlation between true color image and ICA based visualization image
(b) Edges of the ICA based visualization image (b) Edge correlation between the true color image and PCA based visualization image Figure 6 Edge correlation between true color image and visualization images.
4.7
(c) Edges of the PCA based visualization image Figure 5 Edges of the true color image, the ICA and PCA based visualization images
Correlation coefficient with the true color image
Correlations between the true color image and the visualization images are presented and compared in Table 4. As earlier, ICA (RGB) and PCA(RGB) mean the ICA and PCA based visualization method with the RGB spectrum partitioning approach. The correlation values in Table 4 show that the ICA based visualization has a considerable correlation with true color images.
Table 4 Correlation With The True Color Images Correlation Coefficient with True Color Image ICA(RGB) PCA(RGB) R 0.999 0.9986 HyMap G 0.8405 0.9921 B 0.8461 -0.0134
Transactions on Geoscience and Remote Sensing, Vol. 41, No. 3, pp. 708-718, March 2003.
5
[8] Nathaniel P. Jacobson and Maya R. Gupta, Design Goals and Solutions for Display of Hyperspectral Images, IEEE Transactions on Geoscience and Remote Sensing, Vol. 43, No. 11, pp. 2684-2692, Nov. 2005.,
Conclusion
Fusion of large-dimensional hyperspectral dataset to enable their visualization as color images is an important problem. This paper described a recently developed novel ICA based approach for this purpose. A comparative performance evaluation of a prior PCA based approach and the ICA based approach was carried out in terms of several image quality metrics. Each metric evaluates an aspect of the results, for example, the entropy evaluates the information contents of the images, the RMSE directly shows the differences between color display images and true color images, etc. It was shown that the ICA based approach performs quite well. Because the hyperspectral images represent more information than the true color images, a good performance when compared with the true color image may not necessarily mean that more information is preserved. Some new quality metrics should be developed to alleviate this situation, for instance, for performance evaluation without reference images. Improved fusion methods for color display also need to be developed.
References [1] TNTmips, Introduction to Hyperspectral imaging, www.microimages.com/getstart/pdf_new/hyprspec.pdf [2] Gerrit Polder and Gerie W.A. M van der Heijden, Visualization of Spectral Images, Proceedings of SPIE Vol. 4553, pp. 132-137, 2001. [3] Philip K. Robertson and John F. O’Callaghan, The Application of Perceptual Color Spaces to the Display of Remotely Sensed Imagery, IEEE Transactions on Geoscience and Remote Sensing, Vol. 26, No. 1, pp. 49-59, January 1988. [4] Vassilis Tsagaris, Vassilis Anastassopoulos and George A. Lampropoulos, Fusion of Hyperspectral Data Using Segmented PCT for Color Representation and Classification, IEEE Transactions on Geoscience and Remote Sensing, Vol.26, No.10, pp. 2365-2375, Jan. 2005. [5] Xiuping Jia and John A.Richard, Segmented Principal Components Transformation for Efficient Hyperspectral Remote-Sensing Image Display and Classification, IEEE Transactions on Geoscience and Remote Sensing, Vol. 37, No. 1, pp. 538-542, January 1999. [6] J. Scott Tyo, Athanasios Konsolakis, David I. Diersen, and Richard Christopher Olsen, Principal-ComponentsBased Display Strategy for Spectral Imagery, IEEE
[7] Hamid Rahim Sheikh and Alan C. Bovik, Image information and visual quality, IEEE Transactions on Image Processing, Vol. 15, pp. 430–444, February. 2006.
[9] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli, Image Quality Assessment: From Error Visibility to structural Similarity, IEEE Transactions on Image Processing, Vol. 13, No. 4, pp. 600612, Apr. 2004. [10] James V. Stone, Independent Component Analysis: a Tutorial Introduction, MIT, MA, 2003 [11] Stephen Roberts and Richard Everson, Independent Component Analysis: Principles and Practice, Cambridge University, United Kingdom, 2001 [12] A. Hyvärinen and E. Oja, Independent component analysis: algorithms and applications, Neural Networks, vol. 13, Issue 4, pp. 411--430, 2000. [13] M. Lennon, G. Mercier, M. C. Mouchot and L. Hubert-Moy, Independent Component Analysis as a tool for the dimensionality reduction and the representation of hyperspectral images, IGARSS, IEEE, 2001 [14] A. Hyvärinen, Survey on Independent Component Analysis, Neural Computing Surveys, 2, pp. 94-128, 1999. [15] [Online]: http://www.cis.hut.fi/projects/ica/fastica/. [16] Christopher G. Healey, Kellogg S. Booth and James T. Enns, Visualizing Real-time Multivariate Data Using Preattentive Processing, ACM Transactions on Modeling an Computer Simulation, Vol. 5, pp. 190-221, July 1995. [17] Ronaldo de Freitas Zampolo and Rui Seara, “A Comparison of Image Quality Metric Performances under Practical Condition”, IEEE International Conference on Image Processing, Vol. 3, pp.1192-1195, September 2005. [18] Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, Prentice Hall, 2002. [19] Canny, John. "A Computational Approach to Edge Detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986. Vol. PAMI-8, pp. 679-698.
