Image fusion based on topographic mappings ... - Semantic Scholar
Information Visualization (2005), 1–10 &
2005 Palgrave Macmillan Ltd. All rights reserved 1473-8716 $30.00
www.palgrave-journals.com/ivs
Image fusion based on topographic mappings using the hyperbolic space Axel Saalbach1 Jo ¨ rg Ontrup2 Helge Ritter2 Tim W. Nattkemper1 1
Applied Neuroinformatics Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany; 2Neuroinformatics Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany Correspondence: Axel Saalbach, Neuroinformatics Group, Faculty of Technology, Bielefeld University, PO Box 10 01 31, 33501 Bielefeld, Germany. Tel: þ 49 (0)521 106 6054; Fax: þ 49 (0)521 106 6011; E-mail: [email protected]
Abstract The analysis of multivariate image data is a field of research that is becoming increasingly important in a broad range of applications from remote sensing to medical imaging. While traditional scientific visualization techniques are often not suitable for the analysis of this kind of data, methods of image fusion have evolved as a promising approach for synergistic data integration. In this paper, a new approach for the analysis of multivariate image data by means of image fusion is presented, which employs topographic mapping techniques based on non-Euclidean geometry. The hyperbolic self-organizing map (HSOM) facilitates the exploration of high-dimensional data and provides an interface in the tradition of distortion-oriented presentation techniques. For the analysis of hidden patterns and spatial relationships, the HSOM gives rise to an intuitive and efficient framework for the dynamic visualization of multivariate image data by means of color. In an application, the hyperbolic data explorer (HyDE) is employed for the visualization of image data from dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). Using 12 image sequences from breast cancer research, the method is introduced by different visual representations of the data and is also quantitatively analyzed. The HSOM is compared to different standard classifiers and evaluated with respect to topology preservation. Information Visualization advance online publication, 13 October 2005; doi:10.1057/palgrave.ivs.9500106 Keywords: Image fusion; topographic mapping; hyperbolic self-organizing map; pseudocoloring
Introduction
Received: 20 September 2004 Revised: 11 February 2005 Accepted: 16 February 2005
Nowadays, multivariate image data occur in a growing number of different domains such as remote sensing or medical imaging. Multispectral and hyperspectral imaging techniques based on air- and satellite-borne sensor systems are employed in remote sensing for data acquisition at different wavelengths and for the monitoring of temporal changes.1 Combinations of multiple imaging techniques such as PET, CT and MRI are employed for diagnostic purposes in medicine.2 In studies of brain functions and breast cancer research, functional and DCE-MRI is used for the generation of image sequences over time.3 While many modern imaging systems provide detailed measurements at high spatial and temporal resolution, fully automatic analysis of large sets of high-dimensional data is still a major problem in image processing. As pointed out by Landgrebe, many of the most efficient approaches for the analysis of multispectral and hyperspectral data are based on a combination of the capabilities of humans and computers and could be considered as human-assisted machine or machine-assisted human schemes.4
Image Fusion
Axel Saalbach et al
2
Although certain information visualization techniques such as glyph-based methods could be employed in an integrated representation for the visualization of the characteristics of spatial data, they are often not suitable in the context of large data sets.5 With respect to this issue, image fusion techniques offer powerful options for studies of complex data distributions and their underlying structure in multivariate imaging. While the integration of complex visual data from different sources is a challenging task, high-dimensional data can often be summarized in a low-dimensional representation. As discussed by Villmann et al.6 in the context of remote sensing, multivariate image data tend to be highly correlated and the data might lie on a low-dimensional, nonlinear manifold. In a specific application, they showed that hyperspectral image data with 194 bands could be represented successfully in a three-dimensional projection. The application of image fusion techniques provides an informative basis for the detection of hidden structures in the data such as correlations between different variables as well as spatial relationships. The combination of image fusion and visualization techniques provides a framework to facilitate human understanding of the data. In the context of multivariate imaging it can be employed to take advantage of the exceptional human capabilities with respect to pattern recognition, generalization and to utilize expert knowledge.
Image fusion The term image fusion comprises a large variety of techniques and has a long tradition in remote sensing where data from different sensors and dates have to be processed as well as data with different resolution and polarization. In a review article about concepts, methods and applications of multisensor fusion, Pohl and Genderen7 defined image fusion, as the combination of two or more different images to form a new image by using a certain algorithm. Following their definition,two major categories of image fusion techniques can be distinguished. The first category consists of color-related techniques that can be used for the direct visualization of up to three different data dimensions using color models such as RGB, HSV or CIELuv. The second category comprises statistical and numerical methods that include arithmetic combinations of the image data (e.g. difference images) as well as more sophisticated techniques for the integration of data from a larger number of sources. In a survey, Simone et al.8 discussed typical applications of data fusion in remote sensing. In three studies they employed synthetic aperture radar (SAR) interferometry for the creation of elevation maps and fusion techniques for classification purposes using multitemporal and multisensor data. Finally, they considered fusion techniques for multifrequency, multipolarization and multiresolution images using wavelet transform and a multiscale Kalman filter. Apart from these rather special methods, image fusion at pixel level is often achieved by means of techniques from statistics, machine learning and artificial neural net-
Information Visualization
works. This includes among others principal component analysis (PCA),1 methods based on specific projection indices9 and self-organizing maps.10 In this context, each pixel is represented by a vector containing components from different sources. Using image fusion, even for very high-dimensional data compact descriptions can be generated according to different objective functions. Moreover, these techniques can be employed in a broad range of applications in order to facilitate visual analysis of spatial data and are especially suitable for the exploration of data with unknown characteristics. In image fusion, PCA is a standard technique that is commonly employed to generate compact data representations and to make hidden structures evident. This includes, for example, the application of PCA for the generation of color composite images by means of multiple principal components1,11 and the detection of land cover changes based on selected components in remote sensing.12 Further, PCA has been employed for the display of brain lesions in medical imaging.13 Since the capabilities of linear projection techniques such as PCA are limited, nonlinear methods based on artificial neural networks such as the SOM have received much attention. The notion of the SOM has been introduced by Kohonen more than 20 years ago.10 The majority of uses still follows its original motivation: to use a deformable template to translate data similarities from a highdimensional feature space X into spatial relations on a low-dimensional template L of formal neurons usually positioned on a regular lattice in 2D Euclidean space. In this way dimension-reduced topographic feature maps are created most prominently for the purpose of data visualization. The SOM exhibits many favorable properties, which makes it interesting for visualization tasks including the exploratory analysis of multivariate image data. The SOM can represent nonlinearities, is robust in handling outliers and noise, and is inherently parallel. A SOM can further adapt to changes in the data and can be applied even to very large data sets.6 Using SOMs, multivariate image data can be represented at the pixel level by means of a small number of variables. In combination with, for example, color-based visualization techniques, an integrated representation of even high-dimensional data can be generated. SOMs have been used for instance by Gro and Seibert14 for the visualization of burning oil fields in Kuwait. For visualization purposes, a color was assigned to each pixel in the image based on the best matching prototype vector of the SOM. To this end, a mapping of the position of the corresponding node in the threedimensional SOM lattice into the RGB color model was employed. Recently, Villmann et al.6 gave an extensive overview of the application of artificial neural networks in remote sensing with respect to visualization, classification and cluster analysis. However, the SOM can not only be employed for the visualization of image data in a classical sense, but has also been used by Koua and
Image Fusion
Axel Saalbach et al
3
Kraak15 for the exploration of health and demographic survey data in the context of geovisualization. For this purpose, Koua and Kraak proposed a SOM-based framework to support visual data mining and knowledge discovery. Among different standard visualization techniques they used the output of the SOM analysis to display the characteristics of different countries according to their multivariate attributes using a projection onto a geographic map.
A new approach for the visualization of multivariate image data In previous work, Ritter16 has introduced a generalization of the SOM to non-Euclidean curved spaces by means of spherical and hyperbolic geometry. In this context, the favorable properties of the hyperbolic self-organizing map (HSOM) with respect to text categorization and semantic browsing have been demonstrated by Ontrup and Ritter.17 For the first time, the HSOM is utilized in this contribution for the exploration and visualization of multivariate image data in a framework called HyDE. By means of the focus and context like user interface of the HSOM, a framework for the visualization of multivariate image data is developed, which can be adjusted dynamically by the user with respect to his information needs. Using a color mapping that is controlled by the HSOM interface, the user becomes able to customize the visualization due to the selection of a focus region in the HSOM. As more space is provided on the display for the focus region, more variation in color is utilized to visualize image regions with corresponding characteristics. In the next section, basic properties of are described as well as the extension of the SOM to the HSOM. Subsequently important aspects for the visualization of multivariate image data by means of color are discussed. In an experimental section, the proposed framework is applied to the visualization of DCE-MRI image sequences from breast cancer research. To evaluate the capabilities of the HSOM for the visualization and analysis of multivariate image data, several experiments are conducted. The results are analyzed by means of detection accuracy using receiver operating characteristics (ROC) and by a measure of topology preservation.
offered by the hyperbolic space: It is characterized by uniform negative curvature and provides a geometry where the neighborhood around each point increases exponentially with the distance.18 As a result, it offers much ‘more’ space in which complex data can be embedded as compared to the Euclidean case. This exponential behavior was firstly exploited (and patented) by the ‘hyperbolic tree browser’ from Lamping and Rao,19 followed by a web content viewer by Munzner.20 Naturally, it becomes apparent to combine the SOM algorithm with the favorable properties of hyperbolic space.16 The core idea of the HSOM is to employ a grid of nodes in the hyperbolic plane , which is then projected onto 2D Euclidean space for inspection on a ‘flat’ computer screen.
Hyperbolic lattice In order to construct a regular structure of formal neurons in the hyperbolic plane , we employ a tessellation of with congruent polygons.21,22 For the case considered here of a uniform mesh of equilateral triangles, we begin with a ‘starting triangle’ in the center of the map as indicated by the bold outline in Figure 1. By repeatedly applying a set of hyperbolic rotations and translations around the already generated triangle vertices, we iteratively add ‘ring’ for ‘ring’ to the tessellation; for more details cf.16 The HSOM is then formed in the usual way. To each node a 2 L, a reference vector wa is attached, projecting into the input data space X . During a learning phase, the distribution of reference vectors wa is iteratively adapted by a sequence of training vectors xt 2 X : After finding the ‘best-match’ neuron a ¼ argmin8a2L jjwa xjj, that is, the node that has its prototype vector wa closest to the given input x, all reference vectors are updated by the adaptation rule Dwa ¼ e h(a, a*)(xwa), with 2 2 hða; a Þ ¼ exp½ðda;a þ =2s Þ . Here h(a, a*) is a bell-shaped Gaussian centered at the ‘winner’ a* and decaying with increasing distance da, a* ¼ |lala*| in the neuron lattice L. Thus, each formal neuron in the neighborhood of the ‘winner’ a* participates in the current learning step. During the course of learning, the width s of the neighborhood bell function and the learning step size
Methods In the following sections, two methods are described, which are the main ingredients of HyDE: The HSOM and the dynamic coloring technique based on the mapping of the HSV color model to the Poincare´ disk.
Hyberbolic self-organizing map When using the SOM as a data display, a rather limiting factor is the two-dimensional Euclidean space in which the formal neurons are embedded: The neighborhood that ‘fits’ around each neuron is restricted by the square of the distance to that node. An interesting loophole is
Figure 1 Navigation snapshots showing the isometric Mo ¨ bius transformation acting on a regular tessellation grid. The three images were acquired while moving the focus from the center of the map to the highlighted region near the outer perimeter. Note the ‘fish-eye’ effect: All triangles are congruent, but appear smaller as they are further away from the focus.
Information Visualization
Image Fusion
Axel Saalbach et al
4
parameter e are continuously decreased in order to allow more and more specialization and fine tuning of the (then increasingly) weakly coupled neurons. The structure of this neighborhood is essentially governed by the shape of h(a, a*) ¼ h(da, a*), and is therefore also called the neighborhood function.
Projections of There are a number of models of 2D hyperbolic geometry that map the infinite space of into Euclidean space. Naturally, none of them achieves a perfect embedding, since a projection of the negatively curved into our flat Euclidean world introduces distortions in either length, area or angles, that is, shape. The most common models are the finite Poincare´ disk, the Klein Beltrami model, the infinite upper half plane and the Minkowski model.23,24 In the following, we will concentrate on Poincare´’s model of because of its favorable properties for a visualization framework:
It maps the infinitely large area of entirely into the Euclidean unit disc.
The projection is conformal, that is, preserving shapes and angles, but at the cost of distorting distances.
The nonisometric projection exhibits a strong ‘fisheye’ effect: The origin of – corresponding to the ‘fish-eye’ fovea – is mapped almost faithfully, while distant regions become exponentially ‘squeezed’ (cf. Figure 1).
The model allows one to translate the original in an elegant way, where the fovea can be moved to any other part of the infinite hyperbolic plane. This enables the user to selectively focus on interesting portions of a map painted on while still keeping a coarser view of its surrounding context. As stated in the ‘Introduction’, an intuitive navigation and interaction method is a crucial element for a wellfitted visualization framework. For adjusting the fish-eye fovea, we need a translation operation that can be bound to mouse click and drag events. In the Poincare´ disk model the Mo¨bius transformation T(z) is an appropriate solution. By describing the Poincare´ disk PD as the unit circle in the complex plane, the isometric transformations for a point zAPD can be written as z0 ¼ T(z; c, y) ¼ (yz þ c)/(c¯yz þ 1), |y| ¼ 1, |c|o1. Here, the complex number y describes a pure rotation of PD around the origin 0. The following translation by c maps the origin to c and c becomes the new center (if y ¼ 1). For further details, see.19,25 An example for the application of ¨ bius transformation is depicted in Figure 1. It the Mo shows a navigation sequence where the focus was moved towards a highlighted region of interest. Note that from the left to the right, details in the target area get increasingly magnified, as the colored region occupies more and more display space. In contrast to standard zoom operations, the current surrounding context is not clipped, but remains visible, gradually compressed at the periphery of the field of view. Since all operations are continuous, the focus can be positioned in a smooth and
Information Visualization
natural way. This allows for a visualization scheme that enables the user to interactively zoom into portions of a data set that can then be rendered with dynamically increasing resolution.
Dynamic coloring In previous work, SOM-based visualization of multivariate image data has been achieved by using the space spanned by the template of formal neurons L for a color coding of the data.6,14 In doing so, it is not only possible to translate data similarities into spatial relationships, but also to visualize data similarities by means of color. By using the color coded position of the corresponding lattice node a* for each pixel in the image, a compact visual representation of the data can be generated. Typically, SOMs with a one-dimensional lattice are used in combination with a gray scale and the position of a node in a three-dimensional lattice is directly associated with an RGB color model. To this end, the faithful preservation of the data topology by means of the SOM is essential in order to prevent false interpretations of the results. However, the color coding is another important aspect since it affects the user’s perception of the data. While in many applications arbitrarily selected color scales are employed for data visualization, sophisticated techniques have been discussed in detail in computer graphics,26 medicine,27 statistics28 and cartography.29 Since the visualization of the data has to convey a variety of different aspects of the data such as value and form information, the color coding has to be done carefully. The goals of the visualization, the data and the abilities and limitations of the observer have to be considered. Especially for the visualization of univariate maps, a variety of color-based techniques have been proposed. Apart from simple gray and rainbow scales, approaches based on perceptually uniform color spaces have been developed. In order to design an optimal color scale, Levkowitz and Hermann proposed the principles of order, uniformity and representative distance and boundaries.30 Additionally, several tools have been developed to assist the selection of appropriate color-coding techniques. Examples are the rule-based architecture PRAVDA31 and the online tool ColorBrewer.29 The presentation of bi- and trivariate information by means of color is even more challenging since data from different sources have to be displayed simultaneously. The results are usually difficult to interpret because the decomposition of a displayed color into its components is often not intuitive. Especially a coding based on the components of the RGB color model is problematic since it maps the data not to perceptual channels, although it corresponds to the capacities of the display device.32 Some basic principles for the generation of bivariate color scales have been presented by Trumbo.28 Other commonly employed color scales are based on the components of different color models and complementary display parameters.26
Image Fusion
Axel Saalbach et al
5
With respect to this issue, Ware32 argued convincingly that the most efficient way to generate an effective color scale might result from a color editing tool in combination with someone who understands the data and the display requirements. In a study of color mappings of bivariate qualitative data, Rheingans33 showed further that the dynamic control of the parameters of the mapping process can increase the effectiveness of the visualization compared to a static representation. Here, dynamic techniques are defined differently from interactive techniques in the sense that dynamic techniques can be used to manipulate the visualization directly while interactive approaches are used to optimize a static representation. For the HSOM, a simple and direct way to visualize the outcome of the image fusion process by means of color is to employ a mapping of the HSOM lattice L onto perceptual color dimensions using the HSV color model.34 The HSV model can be obtained by simple transformations from the RGB color model and is widely used in computer graphics. It describes color more intuitively in terms of hue (H), saturation (S) and value (V). The hue and saturation components are commonly represented as a disk-like structure where the hue value, that is, the color angle, is mapped to an angular component and the saturation value to a radial component. In this context, bivariate color scales based on a mapping to hue and saturation or hue and lightness are commonly employed for data visualization.26,32 By identifying the coordinates on the Poincare´ disk to which L is projected (cf. Figure 2) a convenient color mapping based on the HSOM can be achieved. While the HSV color model is not perceptual uniform, it might not be suitable for the visualization of interval or ratio data. Since the HSOM performs a mapping from a highdimensional data space on the nodes of a lattice, it is important that an ordering in the mapping can be perceived, making the described procedure a rather favorable approach. Thus, the focus and context-like projection can not only be seen as an interface for the
HSOM-based data exploration, but also as a color editing tool that allows the operator to emphasize critical image regions dynamically depending on the projection of L. However, the approach is obviously not restricted to the visualization of multivariate image data by means of color using value and saturation but could also be employed in combination with other color models or different visual variables such as texture.
Application In order to demonstrate the application of the HSOMbased approach to the visualization of multivariate image data, a set of DCE-MRI image sequences from breast cancer research was employed. Breast cancer is one of the most common malignancies among women35 and the detection and classification of lesions is very important. With respect to this issue DCE-MRI has evolved in the last years as a valuable addition to classical X-ray mammography. In DCE-MRI, a sequence of images is acquired before and after the application of a paramagnetic contrast agent. The agent accumulates over time in suspicious tissue with rich vascularity and becomes visible as an enhancement in the image sequence. Moreover, because of differences in, for example, the vascularity, certain lesion types can be identified based on their signal intensity time course.36 For diagnostic purposes, signal intensities for every image in the sequence have to be considered and visualizations based on difference images are frequently utilized (cf. Figure 2). The image data employed in this experiment consist of sequences of dynamic T1 weighted images from 12 clinical cases with malignant and benign lesions and were acquired using a 1.5 T MRI system (Magnetom Vision, Siemens, Germany). The first sequence image was created before the injection of a contrast agent (Magnevist, Schering, Germany), the following five in steps of 110 s. A single sequence consists of six 3D data sets with 256 256 34 voxels with a resolution of 1.37, 1.37 and 4 mm. More precisely the data could be considered as
Figure 2 Integration of the HSOM setup into a visualization tool for volume data. On the left side slices from one image in the MRI sequence and the corresponding difference images are presented. On the right side the representation of the hyperbolic lattice on the Poincare´ disk is given together with a corresponding 3D visualization of the sequence data.
Information Visualization
Image Fusion
Axel Saalbach et al
6
multidimensional,37 although the signal intensity time course of a voxel is used in the following for its characterization. For further details of the imaging protocol, cf.3 To estimate the ability to differentiate between handlabeled tissue classes based on the HSOM representation of the data, the system was applied for classification. Therefore, ROC analysis was employed and the results were compared to those of standard classification techniques. Since for a faithful representation of the data the preservation of the topology is essential, additionally the degree of topology preservation was computed using Spearman’s rank correlation coefficient. For all experiments in this study, a HSOM topology based on a tessellation of the hyperbolic plane with R ¼ 3 rings and n ¼ 7 neighbors was selected, which results in a lattice with 85 nodes. In an iterative process, HSOMs were trained with identical learning parameters after a random initialization of their prototype vectors. Since a lesion in the data set typically contains only a small number of voxels, for each sequence an equal number of voxels from normal tissue of the same sequence was
randomly selected to create balanced training data sets, G 0c ¼ fðxi ; yi Þ 2 X Y jj i ¼ 1; . . . ; Nc g with X ¼ 2, Y ¼ f1; 1g and c ¼ 1, y, 12, where the signal intensity time course of a voxel is represented by the vector xi. For the estimation of the correspondence between HSOM prototype vectors and different tissue types, an associated class label yi was used to denote the tissue class. In order to assess the generalization capabilities of the approach for new data, a leave-one-case-out strategy was employed. Thus, 11 data sets were combined into a data set Gc, while the HSOM was applied to the entire data from the remaining sequence. To average over the random effects from tissue sampling and HSOM initialization, all experiments were repeated 10 times.
HSOM visualization and exploration A HyDE application to DCE-MRI data is shown in Figure 3 and discussed below. The figure shows an axial slice through one of the MRI images with a malignant lesion, whose characteristics are exposed in more detail by zooming into the corresponding HSOM region.
Figure 3 Different representations of a slice from a MRI data set. The upper part of the figure shows a visualization based on a HSOM coloring with the focus in the center of the HSOM. In the lower part, a region of the HSOM was moved in the focus which contains characteristic patterns for lesions.
Information Visualization
Image Fusion
Axel Saalbach et al
7
In the initial configuration of the interface (Figure 3A), the analysis of the nodes from the center of the HSOM lattice is facilitated. Nodes from the outer rings are presented in lower detail at the rim of the Poincare´ disk. Starting from the center of the lattice, groups of nodes with different signal characteristics can be identified. In this example, the smooth transition between enhancement patterns associated with neighboring nodes is directly visible by means of bar charts that depict the signal time course. While a lesion could hardly be identified based on signal intensities alone in a single MRI image, the corresponding voxels become visible by different shades of blue and magenta in the HSOM-based visualization (Figure 3B). By changing the focus region, different parts of the HSOM itself can be explored in detail since more space is provided in the representation. Nodes from different parts of the lattice can be moved dynamically into the center of the Poincare´ disk, affecting the visualization of the image data simultaneously due to the projection of the lattice to the HSV color model. Since neighboring nodes outside of the focus region are assigned to similar colors, data regions with only subtle differences are visually grouped together. The selection of different focus regions can be used to facilitate the visualization of interesting regions such as the lesion. In Figure 3C the focus was moved towards a region of the HSOM lattice whose prototype vectors reflect a signal time course characteristic for lesions. In this configuration, the contrast within the lesion is enhanced at the expense of the normal tissue, which reveals important differences within the lesion (Figure 3D). The upper section of the lesion is dominated by voxels of blue color and is characterized by a fast and early signal enhancement and a decline at the end of the sequence. This is a typical pattern for malignant lesions. In contrast, the lower section consists mainly of voxels of red color and shows aslow and continuous enhancement that is more likely for a benign lesion. Although this representation of the data is very appealing, it depends on the capability of the HSOM to reflect the characteristics of the data distribution. In order to access this ability for a larger data set quantitatively, we evaluate the correspondence of the prototype vectors to certain tissue classes by means of detection accuracy and the preservation of the data topology based on Spearman’s rank correlation coefficient.
Detection accuracy While the lesion and its characteristics could be clearly identified in the visualization, the correspondence between the HSOM prototype vectors and the tissue classes could be estimated quantitatively by means of classification accuracy for all 12 cases. In order for the HSOM to be able to generate useful visualizations, or to be employed as part of an automated preprocessing step for a computer-aided diagnosis system, it must allow the reliable detection of suspicious tissue. Using hand-labeled
data of suspicious tissue, generated by a medical expert, the HSOM was employed for the classification of normal and suspicious tissue. The classification accuracy of the HSOM was computed and compared to those of different classifiers. In the following experiments, the HSOM was compared with two of the most commonly applied and basic schemes for classification, the linear discriminant analysis (LDA) and the k-nearest-neighbor classifier (k-NN) with k ¼ 1,3,5. For the evaluation, the responses of the three different approaches were calculated for every voxel of the 12 image sequences. The outputs of the k-NN and the LDA classifier were therefore defined as follows: 1 X CðxÞk-NN ¼ y ; CðxÞLDA ¼ wT x; ð1Þ k x 2N ðxÞ i i
k
where Nk(x) is a set of k samples from the training data set with the smallest Euclidean distance from x. For LDA the vector w is used to project the data into a onedimensional subspace with the best class separation in terms of a maximum ratio of the between-class scatter to the within-class scatter. For a two-class problem the solution can be computed as w ¼ S1 xt¯ xl), where Sw is w (¯ the matrix of the within-class scatter, while x ¯t and x ¯l are the average signal patterns of the different tissue classes, respectively. Similar to,17 the outcome C(x)HSOM of the HSOM was calculated based on the prototype vectors with the smallest Euclidian distance to the sample vectors. Therefore, confidence values for the prototype vectors were estimated based on all training samples within the Voronoi set of a particular prototype. To evaluate the performance of the HSOM with respect to the other classifiers, ROC graphs were employed, which is a standard method in medical decision-making and has recently received increased interest in machine learning and pattern recognition.38 ROC analysis is based on the description of the classifier performance in terms of correct (true positives and true negatives,) and wrong classifications (false positives and false negatives), which results from the application of different thresholds to the continuous classifier outcome. In order to compare the different approaches based on a single scalar value, the area under the ROC curve (AUC) was calculated for all classifiers. The ROC curve depicts the true-positive and false-positive rate of a classifier at different thresholds. These quantities are the ratios of correctly classified positives to the total number of positives and the incorrectly classified negatives to the total number of negatives, respectively. An advantage of this approach is the fact that it allows a rating of classifiers according to the ranking of its outcome and not on the absolute values. As can be seen in Figure 4, all classifiers achieved very similar results and high AUC values in this experiment. Even though the HSOM is typically employed as an unsupervised algorithm, the performance of the HSOM in this experiment was on average slightly higher and more stable than those of the LDA and the most simple
Information Visualization
Image Fusion
Axel Saalbach et al
8
1.00
0.65
Rank correlation coefficient
Area under the Curve
0.98
0.96
0.94
0.63
0.61
0.59
0.92
0.57 1
2
3
4
5
0.90 LDA
1-NN
3-NN
5-NN
HSOM
Figure 4 Results for the classification experiments using LDA, three different k-NN classifiers and the HSOM for tissue classification . The box plot shows the area under the ROC curve for 10 repetitions of the experiments.
nearest-neighbor classifier. However, each classifier has situations where it works best. LDA makes strong assumptions about the structure of the data but it is usually stable to fit; a k-NN is less biased but can have, especially for small k, a high variance. While modern machine learning techniques might achieve an even higher classification accuracy, the HSOM can provide additional information about, for example, cluster structure and regularities in the data that are otherwise not available.
Preservation of topology While the correspondence of HSOM prototype vectors to certain classes can be quantified by means of classification results, the ability of the HSOM to reflect the underlying structure of the data might be even more important. To prevent false interpretations of the data, the preservation of the data topology is essential. A common intuition of a topographic mapping is that it preserves neighborhood relations in high-dimensional data distributions in low-dimensional representations. To quantify this property a large variety of different measures have been proposed. Two well-known approaches are the topographic product proposed by Bauer and Pawelzik39 and the C-measure of Goodhill and Sejnowsky which can be found together with an extensive overview of this topic in.40 In this application, the approach proposed by Bezdek and Pal41 based on Spearman’s rank correlation coefficient was employed to measure the degree of topology
Information Visualization
6 7 8 Case ID
9
10 11 12
Figure 5 Evaluation of the topology preservation of the HSOM using Spearman’s r.
preservation. It is based on the preservation of the rank order of all pairwise distances, P i SÞ ðRi RÞðS i ffi; r ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð2Þ P 2 2 ðSi SÞ ðRi RÞ i
where the vectors R and S denote the ranks in the highdimensional space and in the low-dimensional representation. Unlike many other topographic measures, r has a fixed range of values, 1rrr1. A mapping with r ¼ 1 is called metric topology preserving transformation.41 Figure 5 shows the obtained results for the HSOM using Spearman’s r. The results of the different HSOMs were grouped according to the leave-one-case-out strategy, which was employed for the generation of the data sets. It can be seen that the HSOM achieved very stable results and an average correlation level between the ranking of the pairwise distances in and in the HSOM lattice. Since the ability of the HSOM to preserve the data topology is affected by the intrinsic data dimensionality and the output dimensionality of the HSOM lattice, Grassberger–Procaccia analysis, which is commonly applied in the studies of strange attractors, as well as PCA was carried out. An individual analysis of the data from different tissue types using both techniques indicated a rather heterogeneous structure of the data set. While, for data from normal tissue, a dimension DGPE1.0 could be estimated, data from lesion tissue showed a locally varying scaling behavior with a much higher intrinsic dimensionality. Although the visual inspection showed promising results, a perfect preservation of the topology in the sense of Spearman’s r is rather difficult to achieve for this kind of data. For the HSOM, techniques for
Image Fusion
Axel Saalbach et al
9
structure adaption can be employed to further increase the accuracy of the representation.42 However, it should be noted that for a discrimination of different tissue types a perfect preservation of the topology might not be necessary, and that the Euclidean distance accounts only partially for the subtle characteristic of the signal intensity time course.
Discussion In this contribution, a new approach for the visualization and analysis of multivariate image data was presented by means of topographic mapping techniques using concepts of non-Euclidian geometry. The presented approach gives students and engineers new options for the visual exploration of multivariate image data in a broad range of applications from remote sensing to medical imaging. Based on the HSOM it was demonstrated how transformations of data similarities into spatial similarities in can be applied for data visualization in general and for image fusion in particular. In previous applications it was demonstrated that the HSOM offers remarkable options for the visualization of hierarchical and high-dimensional data.16,17 The representation of the HSOM lattice based on a projection of onto the Poincare´ disk facilitates the exploration of even
very large data structures in the tradition of distortion¨ bius transoriented presentation techniques. Using Mo formations, a convenient interface for the HSOM-based data analysis can be generated, which allows the selection of specific lattice regions that are represented in detail while the context remains visible. The integration of this navigation scheme with a color-coding technique using the HSV color model results in a framework for the fusion and visualization of multivariate image data, which can be employed to dynamically emphasize image regions with specific characteristics. In an application, the framework was employed to the analysis of image data from breast cancer research. Using HSOM-based data representations, it was possible to detect suspicious tissue and to resolve important characteristics of lesions visually. By means of ROC analysis it was demonstrated that the capability of the HSOM to discriminate between signal time curves of normal and suspicious tissue was superior to those of basic classification schemes like a LDA or a simple nearest-neighbor classifier. Even for the heterogeneous data employed in this experiment a reasonable degree of topology preservation was estimated using a measure based on Spearman’s rank correlation coefficient.
References 1 Richards JA. Remote Sensing Digital Image Analysis: An Introduction, 2nd edn. Springer: Berlin, Heidelberg, New York, 1993. 2 Bankman IN (Ed). Handbook of Medical Imaging: Processing and Analysis. Academic Press: San Diego, San Francisco, New York, Boston, London, Sydney, Tokyo, 2000. 3 Wismu¨ller A, Lange O, Dersch DR, Leinsinger GL, Hahn K, Pu¨tz B, Auer D. Cluster analysis of biomedical image time-series. International Journal of Computer Vision 2002; 46: 103–128. 4 Landgrebe D. Information extraction principles and methods. In: Chen CH (Ed). Information Processing for Remote Sensing. World Scientific Publishing Co: River Edge, New Jersey, 2000; 3–38. 5 Schumann H, Mu¨ller W. Visualisierung - Grundlagen und allgemeine Methoden. Springer: Berlin, Heiderberg, New York, 2000. 6 Villmann T, Mere´nyi E, Hammer B. Neural maps in remote sensing image analysis. Neural Networks 2003; 16: 389–403. 7 Pohl C, van Genderen JL. Multisensor image fusion in remote sensing: concepts, methods and applications. International Journal of Remote Sensing 1998; 19: 823–854. 8 Simone G, Farina A, Morabito FC, Serpico SB, Bruzzone L. Image fusion techniques for remote sensing applications. Information Fusion 2002; 3: 3–15. 9 Harikumar G, Bresler Y. Feature extraction techniques for exploratory visualization of vector-valued imagery. IEEE Transactions on Image Processing 1996; 5: 1324–1334. 10 Kohonen T. Self-Organizing Maps. Springer Series in Information Science, 3rd edn. Springer: Berlin, Heidelberg, New York, 2001. 11 Geladi P, Grahn H. Multivariate Image Analysis. Wiley: Chichester, 1996. 12 Fung T, LeDrew E. Application of principal component analysis to change detection. Photogrammetric Engineering & Remote Sensing 1987; 53: 1649–1659. 13 Schmiedl U, Orthendahl DA, Mark AS, Berry I, Kaufman L. The utility of principal component analysis for the image display of brain lesions: a preliminary, comparative study. Magnetic Resonance in Medicine 1987; 4: 471–486. 14 Groß MH, Seibert F. Visualization of multidimensional image data sets using a neural network. The Visual Computer 1993; 1: 145–159.
15 Koua E, Kraak MJ. Geovisualization to support the exploration of large health and demographic survey data. International Journal of Health Geographics 2004; 3: 12. 16 Ritter H. Self-organizing Maps in non-euclidean Spaces. In: Oja E, Kaski S (Eds). Kohonen Maps. Elsevier: Amsterdam, 1999; 97–108. 17 Ontrup J, Ritter H. Text categorization and semantic browsing with self-organizing maps on non-euclidean spaces. In: Proceedings of PKDD-2001. Springer: Berlin, 2001; 338–349. 18 Morgan F. Riemannian Geometry: A Beginner’s Guide. Jones and Bartlett Publishers: Boston, London, 1993. 19 Lamping J, Rao R, Pirolli P. A focus+content technique based on hyperbolic geometry for viewing large hierarchies. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, Denver. ACM: NewYork, 1995; 401–408. 20 Munzner T. Exploring large graphs in 3D hyperbolic space. IEEE Computer Graphics and Applications 1998; 18: 18–23. 21 Fricke R, Klein F. Vorlesungen u¨ber die Theorie der automorphen Funktionen, Vol. 1. Teubner: Leipzig 1897 (Reprinted by Johnson Reprint: New York, 1965). 22 Magnus W. Noneuclidean Tesselations and Their Groups. Academic Press: New York, 1974. 23 Cannon JW, Floyd WJ, Kenyon R, Parry WR. Hyperbolic geometry. In: Levy S (Ed). Flavors of Geometry. Cambridge University Press: Cambridge and New York, 1997. 24 Coxeter HSM. Non Euclidean Geometry. University of Toronto Press: Toronto, 1957. 25 Walter J. H-MDS: a new approach for interactive visualization with multidimensional scaling in the hyperbolic space. Information Systems 2004; 29: 273–292. 26 Rheingans P. Task-based color scale design. In: Proceedings of Applied Image and Pattern Recognition. SPIE Press: Bellingham, Washington, 1999; 35–43. 27 Li H, Burgess EA. Evaluation of signal detection performance with pseudocolour display and lumpy backgrounds. In: Proceedings of Medical Imaging. SPIE Press: Bellingham, Washington, 1997; 143–149.
Information Visualization
Image Fusion
Axel Saalbach et al
10
28 Trumbo B. Theory for coloring bivariate statistical maps. The American Statistician 1981; 35: 220–226. 29 Harrower MA, Brewer CA. ColorBrewer.org: an online tool for selecting color schemes for maps. The Cartographic Journal 2003; 40: 27–37. 30 Levkowitz H, Herman GT. Color scales for image data. IEEE Computer Graphics & Applications 1992; 12: 72–80. 31 Rogowitz BE, Treinish LA. An Architecture for Perceptual Rule-Based Visualization. In: Proceedings of the IEEE Visualization 1993 Conference. IEE Publications, 1993; 236–243. 32 Ware C. Information Visualization, 2nd edn. Morgan Kaufmann: San Francisco, California, 2004. 33 Rheingans P. Dynamic color mapping of bivariate qualitative data. In: Proceedings of IEEE Visualization’97. IEEE Computer Society Press: Los Alamitos, California, 1997; 159–166. 34 Smith AR. Color gamut transform pairs. In: Proceedings of SIGGRAPH ’78, ACM Press: New York, 1978; 12–19. 35 Hellwig G, Brix G, Griebel J, Lucht R, Delorme S, Siebert M, Englmeier KH. Dynamic MR mammography: three-dimensional real-time visualization of contrast enhancement in virtual reality. Academic Radiology 2002; 9: 1255–1263.
Information Visualization
36 Kuhl CK, Mielcareck P, Klaschik S, Leutner C, Wardelmann E, Gieseke J, Schild HH. Dynamic breast MR imaging: are signal intensity time course data useful for differential diagnosis of enhancing lesions? Radiology 1999; 1: 101–110. 37 Wong P, Bergeron R. 30 Years of multidimensional multivariate visualization. In: Nielson GM, Hagen H, Mueller H (Eds). Scientific Visualization - Overviews, Methodologies and Techniques. IEEE Computer Society Press: Los Alamitos, California, 1997. 38 Fawcett T. ROC Graphs: notes and practical considerations for researchers. Technical report, HP Labs Tech Report HPL-2003-4, 2003. 39 Bauer HU, Pawelzik K. Quantifying the neighbourhood preservation of self-organizing feature maps. IEEE Transactions on Neural Networks 1992; 8: 226–233. 40 Goodhill GJ, Sejnowsky TJ. A unifying objective function for topographic mappings. Neural Computation 1997; 9: 1291–1303. 41 Bezdek JC, Pal NR. An index of topological preservation for feature extraction. Pattern Recognition 1995; 28: 381–391. 42 Bauer HU, Villmann T. Growing a hypercubical output space in selforganizing feature maps. IEEE Transactions on Neural Networks 1997; 8: 218–226.
2005 Palgrave Macmillan Ltd. All rights reserved 1473-8716 $30.00
www.palgrave-journals.com/ivs
Image fusion based on topographic mappings using the hyperbolic space Axel Saalbach1 Jo ¨ rg Ontrup2 Helge Ritter2 Tim W. Nattkemper1 1
Applied Neuroinformatics Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany; 2Neuroinformatics Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany Correspondence: Axel Saalbach, Neuroinformatics Group, Faculty of Technology, Bielefeld University, PO Box 10 01 31, 33501 Bielefeld, Germany. Tel: þ 49 (0)521 106 6054; Fax: þ 49 (0)521 106 6011; E-mail: [email protected]
Abstract The analysis of multivariate image data is a field of research that is becoming increasingly important in a broad range of applications from remote sensing to medical imaging. While traditional scientific visualization techniques are often not suitable for the analysis of this kind of data, methods of image fusion have evolved as a promising approach for synergistic data integration. In this paper, a new approach for the analysis of multivariate image data by means of image fusion is presented, which employs topographic mapping techniques based on non-Euclidean geometry. The hyperbolic self-organizing map (HSOM) facilitates the exploration of high-dimensional data and provides an interface in the tradition of distortion-oriented presentation techniques. For the analysis of hidden patterns and spatial relationships, the HSOM gives rise to an intuitive and efficient framework for the dynamic visualization of multivariate image data by means of color. In an application, the hyperbolic data explorer (HyDE) is employed for the visualization of image data from dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). Using 12 image sequences from breast cancer research, the method is introduced by different visual representations of the data and is also quantitatively analyzed. The HSOM is compared to different standard classifiers and evaluated with respect to topology preservation. Information Visualization advance online publication, 13 October 2005; doi:10.1057/palgrave.ivs.9500106 Keywords: Image fusion; topographic mapping; hyperbolic self-organizing map; pseudocoloring
Introduction
Received: 20 September 2004 Revised: 11 February 2005 Accepted: 16 February 2005
Nowadays, multivariate image data occur in a growing number of different domains such as remote sensing or medical imaging. Multispectral and hyperspectral imaging techniques based on air- and satellite-borne sensor systems are employed in remote sensing for data acquisition at different wavelengths and for the monitoring of temporal changes.1 Combinations of multiple imaging techniques such as PET, CT and MRI are employed for diagnostic purposes in medicine.2 In studies of brain functions and breast cancer research, functional and DCE-MRI is used for the generation of image sequences over time.3 While many modern imaging systems provide detailed measurements at high spatial and temporal resolution, fully automatic analysis of large sets of high-dimensional data is still a major problem in image processing. As pointed out by Landgrebe, many of the most efficient approaches for the analysis of multispectral and hyperspectral data are based on a combination of the capabilities of humans and computers and could be considered as human-assisted machine or machine-assisted human schemes.4
Image Fusion
Axel Saalbach et al
2
Although certain information visualization techniques such as glyph-based methods could be employed in an integrated representation for the visualization of the characteristics of spatial data, they are often not suitable in the context of large data sets.5 With respect to this issue, image fusion techniques offer powerful options for studies of complex data distributions and their underlying structure in multivariate imaging. While the integration of complex visual data from different sources is a challenging task, high-dimensional data can often be summarized in a low-dimensional representation. As discussed by Villmann et al.6 in the context of remote sensing, multivariate image data tend to be highly correlated and the data might lie on a low-dimensional, nonlinear manifold. In a specific application, they showed that hyperspectral image data with 194 bands could be represented successfully in a three-dimensional projection. The application of image fusion techniques provides an informative basis for the detection of hidden structures in the data such as correlations between different variables as well as spatial relationships. The combination of image fusion and visualization techniques provides a framework to facilitate human understanding of the data. In the context of multivariate imaging it can be employed to take advantage of the exceptional human capabilities with respect to pattern recognition, generalization and to utilize expert knowledge.
Image fusion The term image fusion comprises a large variety of techniques and has a long tradition in remote sensing where data from different sensors and dates have to be processed as well as data with different resolution and polarization. In a review article about concepts, methods and applications of multisensor fusion, Pohl and Genderen7 defined image fusion, as the combination of two or more different images to form a new image by using a certain algorithm. Following their definition,two major categories of image fusion techniques can be distinguished. The first category consists of color-related techniques that can be used for the direct visualization of up to three different data dimensions using color models such as RGB, HSV or CIELuv. The second category comprises statistical and numerical methods that include arithmetic combinations of the image data (e.g. difference images) as well as more sophisticated techniques for the integration of data from a larger number of sources. In a survey, Simone et al.8 discussed typical applications of data fusion in remote sensing. In three studies they employed synthetic aperture radar (SAR) interferometry for the creation of elevation maps and fusion techniques for classification purposes using multitemporal and multisensor data. Finally, they considered fusion techniques for multifrequency, multipolarization and multiresolution images using wavelet transform and a multiscale Kalman filter. Apart from these rather special methods, image fusion at pixel level is often achieved by means of techniques from statistics, machine learning and artificial neural net-
Information Visualization
works. This includes among others principal component analysis (PCA),1 methods based on specific projection indices9 and self-organizing maps.10 In this context, each pixel is represented by a vector containing components from different sources. Using image fusion, even for very high-dimensional data compact descriptions can be generated according to different objective functions. Moreover, these techniques can be employed in a broad range of applications in order to facilitate visual analysis of spatial data and are especially suitable for the exploration of data with unknown characteristics. In image fusion, PCA is a standard technique that is commonly employed to generate compact data representations and to make hidden structures evident. This includes, for example, the application of PCA for the generation of color composite images by means of multiple principal components1,11 and the detection of land cover changes based on selected components in remote sensing.12 Further, PCA has been employed for the display of brain lesions in medical imaging.13 Since the capabilities of linear projection techniques such as PCA are limited, nonlinear methods based on artificial neural networks such as the SOM have received much attention. The notion of the SOM has been introduced by Kohonen more than 20 years ago.10 The majority of uses still follows its original motivation: to use a deformable template to translate data similarities from a highdimensional feature space X into spatial relations on a low-dimensional template L of formal neurons usually positioned on a regular lattice in 2D Euclidean space. In this way dimension-reduced topographic feature maps are created most prominently for the purpose of data visualization. The SOM exhibits many favorable properties, which makes it interesting for visualization tasks including the exploratory analysis of multivariate image data. The SOM can represent nonlinearities, is robust in handling outliers and noise, and is inherently parallel. A SOM can further adapt to changes in the data and can be applied even to very large data sets.6 Using SOMs, multivariate image data can be represented at the pixel level by means of a small number of variables. In combination with, for example, color-based visualization techniques, an integrated representation of even high-dimensional data can be generated. SOMs have been used for instance by Gro and Seibert14 for the visualization of burning oil fields in Kuwait. For visualization purposes, a color was assigned to each pixel in the image based on the best matching prototype vector of the SOM. To this end, a mapping of the position of the corresponding node in the threedimensional SOM lattice into the RGB color model was employed. Recently, Villmann et al.6 gave an extensive overview of the application of artificial neural networks in remote sensing with respect to visualization, classification and cluster analysis. However, the SOM can not only be employed for the visualization of image data in a classical sense, but has also been used by Koua and
Image Fusion
Axel Saalbach et al
3
Kraak15 for the exploration of health and demographic survey data in the context of geovisualization. For this purpose, Koua and Kraak proposed a SOM-based framework to support visual data mining and knowledge discovery. Among different standard visualization techniques they used the output of the SOM analysis to display the characteristics of different countries according to their multivariate attributes using a projection onto a geographic map.
A new approach for the visualization of multivariate image data In previous work, Ritter16 has introduced a generalization of the SOM to non-Euclidean curved spaces by means of spherical and hyperbolic geometry. In this context, the favorable properties of the hyperbolic self-organizing map (HSOM) with respect to text categorization and semantic browsing have been demonstrated by Ontrup and Ritter.17 For the first time, the HSOM is utilized in this contribution for the exploration and visualization of multivariate image data in a framework called HyDE. By means of the focus and context like user interface of the HSOM, a framework for the visualization of multivariate image data is developed, which can be adjusted dynamically by the user with respect to his information needs. Using a color mapping that is controlled by the HSOM interface, the user becomes able to customize the visualization due to the selection of a focus region in the HSOM. As more space is provided on the display for the focus region, more variation in color is utilized to visualize image regions with corresponding characteristics. In the next section, basic properties of are described as well as the extension of the SOM to the HSOM. Subsequently important aspects for the visualization of multivariate image data by means of color are discussed. In an experimental section, the proposed framework is applied to the visualization of DCE-MRI image sequences from breast cancer research. To evaluate the capabilities of the HSOM for the visualization and analysis of multivariate image data, several experiments are conducted. The results are analyzed by means of detection accuracy using receiver operating characteristics (ROC) and by a measure of topology preservation.
offered by the hyperbolic space: It is characterized by uniform negative curvature and provides a geometry where the neighborhood around each point increases exponentially with the distance.18 As a result, it offers much ‘more’ space in which complex data can be embedded as compared to the Euclidean case. This exponential behavior was firstly exploited (and patented) by the ‘hyperbolic tree browser’ from Lamping and Rao,19 followed by a web content viewer by Munzner.20 Naturally, it becomes apparent to combine the SOM algorithm with the favorable properties of hyperbolic space.16 The core idea of the HSOM is to employ a grid of nodes in the hyperbolic plane , which is then projected onto 2D Euclidean space for inspection on a ‘flat’ computer screen.
Hyperbolic lattice In order to construct a regular structure of formal neurons in the hyperbolic plane , we employ a tessellation of with congruent polygons.21,22 For the case considered here of a uniform mesh of equilateral triangles, we begin with a ‘starting triangle’ in the center of the map as indicated by the bold outline in Figure 1. By repeatedly applying a set of hyperbolic rotations and translations around the already generated triangle vertices, we iteratively add ‘ring’ for ‘ring’ to the tessellation; for more details cf.16 The HSOM is then formed in the usual way. To each node a 2 L, a reference vector wa is attached, projecting into the input data space X . During a learning phase, the distribution of reference vectors wa is iteratively adapted by a sequence of training vectors xt 2 X : After finding the ‘best-match’ neuron a ¼ argmin8a2L jjwa xjj, that is, the node that has its prototype vector wa closest to the given input x, all reference vectors are updated by the adaptation rule Dwa ¼ e h(a, a*)(xwa), with 2 2 hða; a Þ ¼ exp½ðda;a þ =2s Þ . Here h(a, a*) is a bell-shaped Gaussian centered at the ‘winner’ a* and decaying with increasing distance da, a* ¼ |lala*| in the neuron lattice L. Thus, each formal neuron in the neighborhood of the ‘winner’ a* participates in the current learning step. During the course of learning, the width s of the neighborhood bell function and the learning step size
Methods In the following sections, two methods are described, which are the main ingredients of HyDE: The HSOM and the dynamic coloring technique based on the mapping of the HSV color model to the Poincare´ disk.
Hyberbolic self-organizing map When using the SOM as a data display, a rather limiting factor is the two-dimensional Euclidean space in which the formal neurons are embedded: The neighborhood that ‘fits’ around each neuron is restricted by the square of the distance to that node. An interesting loophole is
Figure 1 Navigation snapshots showing the isometric Mo ¨ bius transformation acting on a regular tessellation grid. The three images were acquired while moving the focus from the center of the map to the highlighted region near the outer perimeter. Note the ‘fish-eye’ effect: All triangles are congruent, but appear smaller as they are further away from the focus.
Information Visualization
Image Fusion
Axel Saalbach et al
4
parameter e are continuously decreased in order to allow more and more specialization and fine tuning of the (then increasingly) weakly coupled neurons. The structure of this neighborhood is essentially governed by the shape of h(a, a*) ¼ h(da, a*), and is therefore also called the neighborhood function.
Projections of There are a number of models of 2D hyperbolic geometry that map the infinite space of into Euclidean space. Naturally, none of them achieves a perfect embedding, since a projection of the negatively curved into our flat Euclidean world introduces distortions in either length, area or angles, that is, shape. The most common models are the finite Poincare´ disk, the Klein Beltrami model, the infinite upper half plane and the Minkowski model.23,24 In the following, we will concentrate on Poincare´’s model of because of its favorable properties for a visualization framework:
It maps the infinitely large area of entirely into the Euclidean unit disc.
The projection is conformal, that is, preserving shapes and angles, but at the cost of distorting distances.
The nonisometric projection exhibits a strong ‘fisheye’ effect: The origin of – corresponding to the ‘fish-eye’ fovea – is mapped almost faithfully, while distant regions become exponentially ‘squeezed’ (cf. Figure 1).
The model allows one to translate the original in an elegant way, where the fovea can be moved to any other part of the infinite hyperbolic plane. This enables the user to selectively focus on interesting portions of a map painted on while still keeping a coarser view of its surrounding context. As stated in the ‘Introduction’, an intuitive navigation and interaction method is a crucial element for a wellfitted visualization framework. For adjusting the fish-eye fovea, we need a translation operation that can be bound to mouse click and drag events. In the Poincare´ disk model the Mo¨bius transformation T(z) is an appropriate solution. By describing the Poincare´ disk PD as the unit circle in the complex plane, the isometric transformations for a point zAPD can be written as z0 ¼ T(z; c, y) ¼ (yz þ c)/(c¯yz þ 1), |y| ¼ 1, |c|o1. Here, the complex number y describes a pure rotation of PD around the origin 0. The following translation by c maps the origin to c and c becomes the new center (if y ¼ 1). For further details, see.19,25 An example for the application of ¨ bius transformation is depicted in Figure 1. It the Mo shows a navigation sequence where the focus was moved towards a highlighted region of interest. Note that from the left to the right, details in the target area get increasingly magnified, as the colored region occupies more and more display space. In contrast to standard zoom operations, the current surrounding context is not clipped, but remains visible, gradually compressed at the periphery of the field of view. Since all operations are continuous, the focus can be positioned in a smooth and
Information Visualization
natural way. This allows for a visualization scheme that enables the user to interactively zoom into portions of a data set that can then be rendered with dynamically increasing resolution.
Dynamic coloring In previous work, SOM-based visualization of multivariate image data has been achieved by using the space spanned by the template of formal neurons L for a color coding of the data.6,14 In doing so, it is not only possible to translate data similarities into spatial relationships, but also to visualize data similarities by means of color. By using the color coded position of the corresponding lattice node a* for each pixel in the image, a compact visual representation of the data can be generated. Typically, SOMs with a one-dimensional lattice are used in combination with a gray scale and the position of a node in a three-dimensional lattice is directly associated with an RGB color model. To this end, the faithful preservation of the data topology by means of the SOM is essential in order to prevent false interpretations of the results. However, the color coding is another important aspect since it affects the user’s perception of the data. While in many applications arbitrarily selected color scales are employed for data visualization, sophisticated techniques have been discussed in detail in computer graphics,26 medicine,27 statistics28 and cartography.29 Since the visualization of the data has to convey a variety of different aspects of the data such as value and form information, the color coding has to be done carefully. The goals of the visualization, the data and the abilities and limitations of the observer have to be considered. Especially for the visualization of univariate maps, a variety of color-based techniques have been proposed. Apart from simple gray and rainbow scales, approaches based on perceptually uniform color spaces have been developed. In order to design an optimal color scale, Levkowitz and Hermann proposed the principles of order, uniformity and representative distance and boundaries.30 Additionally, several tools have been developed to assist the selection of appropriate color-coding techniques. Examples are the rule-based architecture PRAVDA31 and the online tool ColorBrewer.29 The presentation of bi- and trivariate information by means of color is even more challenging since data from different sources have to be displayed simultaneously. The results are usually difficult to interpret because the decomposition of a displayed color into its components is often not intuitive. Especially a coding based on the components of the RGB color model is problematic since it maps the data not to perceptual channels, although it corresponds to the capacities of the display device.32 Some basic principles for the generation of bivariate color scales have been presented by Trumbo.28 Other commonly employed color scales are based on the components of different color models and complementary display parameters.26
Image Fusion
Axel Saalbach et al
5
With respect to this issue, Ware32 argued convincingly that the most efficient way to generate an effective color scale might result from a color editing tool in combination with someone who understands the data and the display requirements. In a study of color mappings of bivariate qualitative data, Rheingans33 showed further that the dynamic control of the parameters of the mapping process can increase the effectiveness of the visualization compared to a static representation. Here, dynamic techniques are defined differently from interactive techniques in the sense that dynamic techniques can be used to manipulate the visualization directly while interactive approaches are used to optimize a static representation. For the HSOM, a simple and direct way to visualize the outcome of the image fusion process by means of color is to employ a mapping of the HSOM lattice L onto perceptual color dimensions using the HSV color model.34 The HSV model can be obtained by simple transformations from the RGB color model and is widely used in computer graphics. It describes color more intuitively in terms of hue (H), saturation (S) and value (V). The hue and saturation components are commonly represented as a disk-like structure where the hue value, that is, the color angle, is mapped to an angular component and the saturation value to a radial component. In this context, bivariate color scales based on a mapping to hue and saturation or hue and lightness are commonly employed for data visualization.26,32 By identifying the coordinates on the Poincare´ disk to which L is projected (cf. Figure 2) a convenient color mapping based on the HSOM can be achieved. While the HSV color model is not perceptual uniform, it might not be suitable for the visualization of interval or ratio data. Since the HSOM performs a mapping from a highdimensional data space on the nodes of a lattice, it is important that an ordering in the mapping can be perceived, making the described procedure a rather favorable approach. Thus, the focus and context-like projection can not only be seen as an interface for the
HSOM-based data exploration, but also as a color editing tool that allows the operator to emphasize critical image regions dynamically depending on the projection of L. However, the approach is obviously not restricted to the visualization of multivariate image data by means of color using value and saturation but could also be employed in combination with other color models or different visual variables such as texture.
Application In order to demonstrate the application of the HSOMbased approach to the visualization of multivariate image data, a set of DCE-MRI image sequences from breast cancer research was employed. Breast cancer is one of the most common malignancies among women35 and the detection and classification of lesions is very important. With respect to this issue DCE-MRI has evolved in the last years as a valuable addition to classical X-ray mammography. In DCE-MRI, a sequence of images is acquired before and after the application of a paramagnetic contrast agent. The agent accumulates over time in suspicious tissue with rich vascularity and becomes visible as an enhancement in the image sequence. Moreover, because of differences in, for example, the vascularity, certain lesion types can be identified based on their signal intensity time course.36 For diagnostic purposes, signal intensities for every image in the sequence have to be considered and visualizations based on difference images are frequently utilized (cf. Figure 2). The image data employed in this experiment consist of sequences of dynamic T1 weighted images from 12 clinical cases with malignant and benign lesions and were acquired using a 1.5 T MRI system (Magnetom Vision, Siemens, Germany). The first sequence image was created before the injection of a contrast agent (Magnevist, Schering, Germany), the following five in steps of 110 s. A single sequence consists of six 3D data sets with 256 256 34 voxels with a resolution of 1.37, 1.37 and 4 mm. More precisely the data could be considered as
Figure 2 Integration of the HSOM setup into a visualization tool for volume data. On the left side slices from one image in the MRI sequence and the corresponding difference images are presented. On the right side the representation of the hyperbolic lattice on the Poincare´ disk is given together with a corresponding 3D visualization of the sequence data.
Information Visualization
Image Fusion
Axel Saalbach et al
6
multidimensional,37 although the signal intensity time course of a voxel is used in the following for its characterization. For further details of the imaging protocol, cf.3 To estimate the ability to differentiate between handlabeled tissue classes based on the HSOM representation of the data, the system was applied for classification. Therefore, ROC analysis was employed and the results were compared to those of standard classification techniques. Since for a faithful representation of the data the preservation of the topology is essential, additionally the degree of topology preservation was computed using Spearman’s rank correlation coefficient. For all experiments in this study, a HSOM topology based on a tessellation of the hyperbolic plane with R ¼ 3 rings and n ¼ 7 neighbors was selected, which results in a lattice with 85 nodes. In an iterative process, HSOMs were trained with identical learning parameters after a random initialization of their prototype vectors. Since a lesion in the data set typically contains only a small number of voxels, for each sequence an equal number of voxels from normal tissue of the same sequence was
randomly selected to create balanced training data sets, G 0c ¼ fðxi ; yi Þ 2 X Y jj i ¼ 1; . . . ; Nc g with X ¼ 2, Y ¼ f1; 1g and c ¼ 1, y, 12, where the signal intensity time course of a voxel is represented by the vector xi. For the estimation of the correspondence between HSOM prototype vectors and different tissue types, an associated class label yi was used to denote the tissue class. In order to assess the generalization capabilities of the approach for new data, a leave-one-case-out strategy was employed. Thus, 11 data sets were combined into a data set Gc, while the HSOM was applied to the entire data from the remaining sequence. To average over the random effects from tissue sampling and HSOM initialization, all experiments were repeated 10 times.
HSOM visualization and exploration A HyDE application to DCE-MRI data is shown in Figure 3 and discussed below. The figure shows an axial slice through one of the MRI images with a malignant lesion, whose characteristics are exposed in more detail by zooming into the corresponding HSOM region.
Figure 3 Different representations of a slice from a MRI data set. The upper part of the figure shows a visualization based on a HSOM coloring with the focus in the center of the HSOM. In the lower part, a region of the HSOM was moved in the focus which contains characteristic patterns for lesions.
Information Visualization
Image Fusion
Axel Saalbach et al
7
In the initial configuration of the interface (Figure 3A), the analysis of the nodes from the center of the HSOM lattice is facilitated. Nodes from the outer rings are presented in lower detail at the rim of the Poincare´ disk. Starting from the center of the lattice, groups of nodes with different signal characteristics can be identified. In this example, the smooth transition between enhancement patterns associated with neighboring nodes is directly visible by means of bar charts that depict the signal time course. While a lesion could hardly be identified based on signal intensities alone in a single MRI image, the corresponding voxels become visible by different shades of blue and magenta in the HSOM-based visualization (Figure 3B). By changing the focus region, different parts of the HSOM itself can be explored in detail since more space is provided in the representation. Nodes from different parts of the lattice can be moved dynamically into the center of the Poincare´ disk, affecting the visualization of the image data simultaneously due to the projection of the lattice to the HSV color model. Since neighboring nodes outside of the focus region are assigned to similar colors, data regions with only subtle differences are visually grouped together. The selection of different focus regions can be used to facilitate the visualization of interesting regions such as the lesion. In Figure 3C the focus was moved towards a region of the HSOM lattice whose prototype vectors reflect a signal time course characteristic for lesions. In this configuration, the contrast within the lesion is enhanced at the expense of the normal tissue, which reveals important differences within the lesion (Figure 3D). The upper section of the lesion is dominated by voxels of blue color and is characterized by a fast and early signal enhancement and a decline at the end of the sequence. This is a typical pattern for malignant lesions. In contrast, the lower section consists mainly of voxels of red color and shows aslow and continuous enhancement that is more likely for a benign lesion. Although this representation of the data is very appealing, it depends on the capability of the HSOM to reflect the characteristics of the data distribution. In order to access this ability for a larger data set quantitatively, we evaluate the correspondence of the prototype vectors to certain tissue classes by means of detection accuracy and the preservation of the data topology based on Spearman’s rank correlation coefficient.
Detection accuracy While the lesion and its characteristics could be clearly identified in the visualization, the correspondence between the HSOM prototype vectors and the tissue classes could be estimated quantitatively by means of classification accuracy for all 12 cases. In order for the HSOM to be able to generate useful visualizations, or to be employed as part of an automated preprocessing step for a computer-aided diagnosis system, it must allow the reliable detection of suspicious tissue. Using hand-labeled
data of suspicious tissue, generated by a medical expert, the HSOM was employed for the classification of normal and suspicious tissue. The classification accuracy of the HSOM was computed and compared to those of different classifiers. In the following experiments, the HSOM was compared with two of the most commonly applied and basic schemes for classification, the linear discriminant analysis (LDA) and the k-nearest-neighbor classifier (k-NN) with k ¼ 1,3,5. For the evaluation, the responses of the three different approaches were calculated for every voxel of the 12 image sequences. The outputs of the k-NN and the LDA classifier were therefore defined as follows: 1 X CðxÞk-NN ¼ y ; CðxÞLDA ¼ wT x; ð1Þ k x 2N ðxÞ i i
k
where Nk(x) is a set of k samples from the training data set with the smallest Euclidean distance from x. For LDA the vector w is used to project the data into a onedimensional subspace with the best class separation in terms of a maximum ratio of the between-class scatter to the within-class scatter. For a two-class problem the solution can be computed as w ¼ S1 xt¯ xl), where Sw is w (¯ the matrix of the within-class scatter, while x ¯t and x ¯l are the average signal patterns of the different tissue classes, respectively. Similar to,17 the outcome C(x)HSOM of the HSOM was calculated based on the prototype vectors with the smallest Euclidian distance to the sample vectors. Therefore, confidence values for the prototype vectors were estimated based on all training samples within the Voronoi set of a particular prototype. To evaluate the performance of the HSOM with respect to the other classifiers, ROC graphs were employed, which is a standard method in medical decision-making and has recently received increased interest in machine learning and pattern recognition.38 ROC analysis is based on the description of the classifier performance in terms of correct (true positives and true negatives,) and wrong classifications (false positives and false negatives), which results from the application of different thresholds to the continuous classifier outcome. In order to compare the different approaches based on a single scalar value, the area under the ROC curve (AUC) was calculated for all classifiers. The ROC curve depicts the true-positive and false-positive rate of a classifier at different thresholds. These quantities are the ratios of correctly classified positives to the total number of positives and the incorrectly classified negatives to the total number of negatives, respectively. An advantage of this approach is the fact that it allows a rating of classifiers according to the ranking of its outcome and not on the absolute values. As can be seen in Figure 4, all classifiers achieved very similar results and high AUC values in this experiment. Even though the HSOM is typically employed as an unsupervised algorithm, the performance of the HSOM in this experiment was on average slightly higher and more stable than those of the LDA and the most simple
Information Visualization
Image Fusion
Axel Saalbach et al
8
1.00
0.65
Rank correlation coefficient
Area under the Curve
0.98
0.96
0.94
0.63
0.61
0.59
0.92
0.57 1
2
3
4
5
0.90 LDA
1-NN
3-NN
5-NN
HSOM
Figure 4 Results for the classification experiments using LDA, three different k-NN classifiers and the HSOM for tissue classification . The box plot shows the area under the ROC curve for 10 repetitions of the experiments.
nearest-neighbor classifier. However, each classifier has situations where it works best. LDA makes strong assumptions about the structure of the data but it is usually stable to fit; a k-NN is less biased but can have, especially for small k, a high variance. While modern machine learning techniques might achieve an even higher classification accuracy, the HSOM can provide additional information about, for example, cluster structure and regularities in the data that are otherwise not available.
Preservation of topology While the correspondence of HSOM prototype vectors to certain classes can be quantified by means of classification results, the ability of the HSOM to reflect the underlying structure of the data might be even more important. To prevent false interpretations of the data, the preservation of the data topology is essential. A common intuition of a topographic mapping is that it preserves neighborhood relations in high-dimensional data distributions in low-dimensional representations. To quantify this property a large variety of different measures have been proposed. Two well-known approaches are the topographic product proposed by Bauer and Pawelzik39 and the C-measure of Goodhill and Sejnowsky which can be found together with an extensive overview of this topic in.40 In this application, the approach proposed by Bezdek and Pal41 based on Spearman’s rank correlation coefficient was employed to measure the degree of topology
Information Visualization
6 7 8 Case ID
9
10 11 12
Figure 5 Evaluation of the topology preservation of the HSOM using Spearman’s r.
preservation. It is based on the preservation of the rank order of all pairwise distances, P i SÞ ðRi RÞðS i ffi; r ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð2Þ P 2 2 ðSi SÞ ðRi RÞ i
where the vectors R and S denote the ranks in the highdimensional space and in the low-dimensional representation. Unlike many other topographic measures, r has a fixed range of values, 1rrr1. A mapping with r ¼ 1 is called metric topology preserving transformation.41 Figure 5 shows the obtained results for the HSOM using Spearman’s r. The results of the different HSOMs were grouped according to the leave-one-case-out strategy, which was employed for the generation of the data sets. It can be seen that the HSOM achieved very stable results and an average correlation level between the ranking of the pairwise distances in and in the HSOM lattice. Since the ability of the HSOM to preserve the data topology is affected by the intrinsic data dimensionality and the output dimensionality of the HSOM lattice, Grassberger–Procaccia analysis, which is commonly applied in the studies of strange attractors, as well as PCA was carried out. An individual analysis of the data from different tissue types using both techniques indicated a rather heterogeneous structure of the data set. While, for data from normal tissue, a dimension DGPE1.0 could be estimated, data from lesion tissue showed a locally varying scaling behavior with a much higher intrinsic dimensionality. Although the visual inspection showed promising results, a perfect preservation of the topology in the sense of Spearman’s r is rather difficult to achieve for this kind of data. For the HSOM, techniques for
Image Fusion
Axel Saalbach et al
9
structure adaption can be employed to further increase the accuracy of the representation.42 However, it should be noted that for a discrimination of different tissue types a perfect preservation of the topology might not be necessary, and that the Euclidean distance accounts only partially for the subtle characteristic of the signal intensity time course.
Discussion In this contribution, a new approach for the visualization and analysis of multivariate image data was presented by means of topographic mapping techniques using concepts of non-Euclidian geometry. The presented approach gives students and engineers new options for the visual exploration of multivariate image data in a broad range of applications from remote sensing to medical imaging. Based on the HSOM it was demonstrated how transformations of data similarities into spatial similarities in can be applied for data visualization in general and for image fusion in particular. In previous applications it was demonstrated that the HSOM offers remarkable options for the visualization of hierarchical and high-dimensional data.16,17 The representation of the HSOM lattice based on a projection of onto the Poincare´ disk facilitates the exploration of even
very large data structures in the tradition of distortion¨ bius transoriented presentation techniques. Using Mo formations, a convenient interface for the HSOM-based data analysis can be generated, which allows the selection of specific lattice regions that are represented in detail while the context remains visible. The integration of this navigation scheme with a color-coding technique using the HSV color model results in a framework for the fusion and visualization of multivariate image data, which can be employed to dynamically emphasize image regions with specific characteristics. In an application, the framework was employed to the analysis of image data from breast cancer research. Using HSOM-based data representations, it was possible to detect suspicious tissue and to resolve important characteristics of lesions visually. By means of ROC analysis it was demonstrated that the capability of the HSOM to discriminate between signal time curves of normal and suspicious tissue was superior to those of basic classification schemes like a LDA or a simple nearest-neighbor classifier. Even for the heterogeneous data employed in this experiment a reasonable degree of topology preservation was estimated using a measure based on Spearman’s rank correlation coefficient.
References 1 Richards JA. Remote Sensing Digital Image Analysis: An Introduction, 2nd edn. Springer: Berlin, Heidelberg, New York, 1993. 2 Bankman IN (Ed). Handbook of Medical Imaging: Processing and Analysis. Academic Press: San Diego, San Francisco, New York, Boston, London, Sydney, Tokyo, 2000. 3 Wismu¨ller A, Lange O, Dersch DR, Leinsinger GL, Hahn K, Pu¨tz B, Auer D. Cluster analysis of biomedical image time-series. International Journal of Computer Vision 2002; 46: 103–128. 4 Landgrebe D. Information extraction principles and methods. In: Chen CH (Ed). Information Processing for Remote Sensing. World Scientific Publishing Co: River Edge, New Jersey, 2000; 3–38. 5 Schumann H, Mu¨ller W. Visualisierung - Grundlagen und allgemeine Methoden. Springer: Berlin, Heiderberg, New York, 2000. 6 Villmann T, Mere´nyi E, Hammer B. Neural maps in remote sensing image analysis. Neural Networks 2003; 16: 389–403. 7 Pohl C, van Genderen JL. Multisensor image fusion in remote sensing: concepts, methods and applications. International Journal of Remote Sensing 1998; 19: 823–854. 8 Simone G, Farina A, Morabito FC, Serpico SB, Bruzzone L. Image fusion techniques for remote sensing applications. Information Fusion 2002; 3: 3–15. 9 Harikumar G, Bresler Y. Feature extraction techniques for exploratory visualization of vector-valued imagery. IEEE Transactions on Image Processing 1996; 5: 1324–1334. 10 Kohonen T. Self-Organizing Maps. Springer Series in Information Science, 3rd edn. Springer: Berlin, Heidelberg, New York, 2001. 11 Geladi P, Grahn H. Multivariate Image Analysis. Wiley: Chichester, 1996. 12 Fung T, LeDrew E. Application of principal component analysis to change detection. Photogrammetric Engineering & Remote Sensing 1987; 53: 1649–1659. 13 Schmiedl U, Orthendahl DA, Mark AS, Berry I, Kaufman L. The utility of principal component analysis for the image display of brain lesions: a preliminary, comparative study. Magnetic Resonance in Medicine 1987; 4: 471–486. 14 Groß MH, Seibert F. Visualization of multidimensional image data sets using a neural network. The Visual Computer 1993; 1: 145–159.
15 Koua E, Kraak MJ. Geovisualization to support the exploration of large health and demographic survey data. International Journal of Health Geographics 2004; 3: 12. 16 Ritter H. Self-organizing Maps in non-euclidean Spaces. In: Oja E, Kaski S (Eds). Kohonen Maps. Elsevier: Amsterdam, 1999; 97–108. 17 Ontrup J, Ritter H. Text categorization and semantic browsing with self-organizing maps on non-euclidean spaces. In: Proceedings of PKDD-2001. Springer: Berlin, 2001; 338–349. 18 Morgan F. Riemannian Geometry: A Beginner’s Guide. Jones and Bartlett Publishers: Boston, London, 1993. 19 Lamping J, Rao R, Pirolli P. A focus+content technique based on hyperbolic geometry for viewing large hierarchies. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, Denver. ACM: NewYork, 1995; 401–408. 20 Munzner T. Exploring large graphs in 3D hyperbolic space. IEEE Computer Graphics and Applications 1998; 18: 18–23. 21 Fricke R, Klein F. Vorlesungen u¨ber die Theorie der automorphen Funktionen, Vol. 1. Teubner: Leipzig 1897 (Reprinted by Johnson Reprint: New York, 1965). 22 Magnus W. Noneuclidean Tesselations and Their Groups. Academic Press: New York, 1974. 23 Cannon JW, Floyd WJ, Kenyon R, Parry WR. Hyperbolic geometry. In: Levy S (Ed). Flavors of Geometry. Cambridge University Press: Cambridge and New York, 1997. 24 Coxeter HSM. Non Euclidean Geometry. University of Toronto Press: Toronto, 1957. 25 Walter J. H-MDS: a new approach for interactive visualization with multidimensional scaling in the hyperbolic space. Information Systems 2004; 29: 273–292. 26 Rheingans P. Task-based color scale design. In: Proceedings of Applied Image and Pattern Recognition. SPIE Press: Bellingham, Washington, 1999; 35–43. 27 Li H, Burgess EA. Evaluation of signal detection performance with pseudocolour display and lumpy backgrounds. In: Proceedings of Medical Imaging. SPIE Press: Bellingham, Washington, 1997; 143–149.
Information Visualization
Image Fusion
Axel Saalbach et al
10
28 Trumbo B. Theory for coloring bivariate statistical maps. The American Statistician 1981; 35: 220–226. 29 Harrower MA, Brewer CA. ColorBrewer.org: an online tool for selecting color schemes for maps. The Cartographic Journal 2003; 40: 27–37. 30 Levkowitz H, Herman GT. Color scales for image data. IEEE Computer Graphics & Applications 1992; 12: 72–80. 31 Rogowitz BE, Treinish LA. An Architecture for Perceptual Rule-Based Visualization. In: Proceedings of the IEEE Visualization 1993 Conference. IEE Publications, 1993; 236–243. 32 Ware C. Information Visualization, 2nd edn. Morgan Kaufmann: San Francisco, California, 2004. 33 Rheingans P. Dynamic color mapping of bivariate qualitative data. In: Proceedings of IEEE Visualization’97. IEEE Computer Society Press: Los Alamitos, California, 1997; 159–166. 34 Smith AR. Color gamut transform pairs. In: Proceedings of SIGGRAPH ’78, ACM Press: New York, 1978; 12–19. 35 Hellwig G, Brix G, Griebel J, Lucht R, Delorme S, Siebert M, Englmeier KH. Dynamic MR mammography: three-dimensional real-time visualization of contrast enhancement in virtual reality. Academic Radiology 2002; 9: 1255–1263.
Information Visualization
36 Kuhl CK, Mielcareck P, Klaschik S, Leutner C, Wardelmann E, Gieseke J, Schild HH. Dynamic breast MR imaging: are signal intensity time course data useful for differential diagnosis of enhancing lesions? Radiology 1999; 1: 101–110. 37 Wong P, Bergeron R. 30 Years of multidimensional multivariate visualization. In: Nielson GM, Hagen H, Mueller H (Eds). Scientific Visualization - Overviews, Methodologies and Techniques. IEEE Computer Society Press: Los Alamitos, California, 1997. 38 Fawcett T. ROC Graphs: notes and practical considerations for researchers. Technical report, HP Labs Tech Report HPL-2003-4, 2003. 39 Bauer HU, Pawelzik K. Quantifying the neighbourhood preservation of self-organizing feature maps. IEEE Transactions on Neural Networks 1992; 8: 226–233. 40 Goodhill GJ, Sejnowsky TJ. A unifying objective function for topographic mappings. Neural Computation 1997; 9: 1291–1303. 41 Bezdek JC, Pal NR. An index of topological preservation for feature extraction. Pattern Recognition 1995; 28: 381–391. 42 Bauer HU, Villmann T. Growing a hypercubical output space in selforganizing feature maps. IEEE Transactions on Neural Networks 1997; 8: 218–226.
