Affine registration of multimodality images by ... - IEEE Xplore
Affine registration of multimodality images by optimization of mutual information using a stochastic gradient approximation technique Qi Li, Isao Sato and Yutaka Murakami Institute of Geology and Geoinformation National Institute of Advanced Industrial Science and Technology (AIST) Tsukuba Central 7, 1-1-1 Higashi, Tsukuba 305-8567, Japan [email protected] Abstract—This paper focuses mainly on development of an efficient affine registration scheme for multimodality images. The stochastic gradient optimization of mutual information is accomplished using a simultaneous perturbation stochastic approximation (SPSA) technique to do the tuning of the affine parameters. The experimental results show that the SPSA based stochastic gradient optimization is strong for the multimodality image registration. The main contribution of this paper is that we extended the SPSA technique to simultaneously register the affine transformation for more than two images, and successfully accomplished the SPSA based stochastic gradient algorithm to optimize the information theory based similarity measure such as mutual information.
general, the stochastic gradient optimization provides significant improvements on the optimal solution over the conventional methods such as hill climbing and simplex optimizations in terms of accuracy and robustness. In the current work, the SPSA based stochastic gradient algorithm is further tailored to accomplish an affine registration more precisely. The six mapping parameters of the affine transformation are simultaneously registered by optimization of the mutual information. Our registration experiments are associated with several pairs of multimodality images, which are supposed to be misaligned. The experimental results show that the proposed approach is effective and robust.
Keywords-image registration; affine transformation; stochastic gradient; mutual information; SPSA; optimization
The main contribution of this paper is the accomplishment of an efficient SPSA based stochastic gradient optimization on the affine registration of remote sensed images using a mutual information criterion.
I.
INTRODUCTION
With the earnest need for the information integration of multi-source image data, such as satellite image, airborne photograph, geological map, and various thematic maps, etc., the image fusion analysis are becoming urgent topics. During this process, the image registration is the key step. Although image registration has been studied over many years, and many registration algorithms have been proposed in the literatures, it is still a challenging topic in the field of digital image processing [1].
II.
A. Registration Optimization Image registration is to define a transform T that will map one image onto another image of the same scene acquired by different sensors, or taken by the same sensor at different times, such that a certain distance measure ( S ) is optimal. To register a float image ( I F ) to a reference image ( I R ) can be mathematically expressed as:
In our past research [2, 3], an intensity based registration prototype has been proposed and successfully applied to the image registration by optimization of the mutual information criterion. This paper focuses mainly on an efficient automatic affine registration method for multimodality images using the popular similarity measure, i.e., mutual information. The stochastic gradient algorithm for optimizing the mutual information is accomplished, based on the simultaneous perturbation stochastic approximation (SPSA) technique, which is firstly proposed by J.C. Spall [4]. In [5] and our previous work [6], the SPSA based stochastic gradient optimization scheme has been successfully used in the registration study of the rigid (translation and/or rotation) transformation by maximization of the standard mutual information. Our previous work [7, 8] also shows that, in
1-4244-1212-9/07/$25.00 ©2007 IEEE.
REGISTRATION METHOD
I R ( x, y ) = ζ ( I F (Tα ( x, y ))) ,
(1)
where Tα is a transformation function, which maps two spatial
coordinates x and y , to the new spatial coordinates x ′ and
y ′ by the set of mapping parameters α : ( x′, y ′) = Tα ( x, y ) ,
397
(2)
ζ
In the process of the intensity based automatic image registration, the similarity measure such as the mutual information is the cost function, let us call L, to be optimized during the SPSA search. At iteration k, a steepest ascent update law for the transform parameters is written as:
is the intensity or radiometric calibration function. The intensity based image registration can be mapped as a typical optimization problem [1]. This can be expressed as:
α * = Arg optima( STα ( I F , I R )) .
(3)
ψ k +1 = ψ k + a k g k ,
Tα
Although the image registration can be regarded as an optimization problem, the interpolation artifact [9] complicates the cost (or registration) function of (3), and also with the increase of the transformation parameters, the registration becomes into a complicated non-convex optimization problem. During the process of the automatic image registration, the optimization module is the key element for the whole registration process.
where g k is the gradient vector for the parameter space
(tx, ty , ta11 , ta12 , ta 21 , ta 22 ) in our study, and it can be written as:
g k = [( g k )1 ( g k ) 2
B. Affine Transformation In general, mathematical transformation for digital image registration is consisted of some elementary operations such as translation, rotation, scaling, stretching, and shearing. The traditional registration transformations are rigid, affine, projective, and curved. In remote sensing applications, the twodimensional (2D) affine transformation is usually sufficient for image registration. The general 2D affine transformation can be expressed as:
⎛ x ′ ⎞ ⎛ tx ⎞ ⎛ ta11 ta12 ⎞⎛ x ⎞ ⎟⎟⎜⎜ ⎟⎟ , ⎜⎜ ⎟⎟ = ⎜⎜ ⎟⎟ + ⎜⎜ ⎝ y ′ ⎠ ⎝ ty ⎠ ⎝ ta 21 ta 22 ⎠⎝ y ⎠
( g k )i =
L(ψ k + ck Δ k ) − L(ψ k − ck Δ k ) , 2 ck ( Δ k ) i
(7)
updated at each iteration. The vector Δ k is generated by a standard Bernoulli distribution. This vector represents simultaneous perturbations applied to all search space components in approximating the gradient. The gain sequences of ak and ck are positive.
(4)
Spall presents sufficient conditions for convergence of the SPSA iteration algorithm in the stochastic almost sure. The choice of control parameters is addressed in [4, 7] in detail.
ta12 ⎞ ⎟ can be interpreted as rotation, ta 22 ⎟⎠
III.
EXPERIMENTAL RESULTS
The goal of image registration is to find the optimal mapping parameters, discussed in Section II, which define the relative position and orientation of the two sensed images in an efficient and robust way. In this section, two demonstration experiments are chosen to be discussed. Experiment one is designed by standard image set with knowing the ground truth to verify our proposed approach. Experiment two is to register three images without knowing the ground truth. Throughout the experiments, the partial volume interpolation was used for the estimation of joint histogram during the computation of mutual information.
C. SPSA based Parameter Search The SPSA technique recently becomes very popular for solving some challenging optimization problems. The prominent merit of the SPSA based stochastic gradient optimization is that it does not require an explicit knowledge of the gradient of the cost function, or measurements of this gradient. At each of iterations, it only needs an approximation to the gradient via simultaneous perturbations. The gradient approximation is based on only two function measurements regardless of the dimensions of the search parameter space. The SPSA based stochastic gradient is very strong [7, 8], which can get through some local optima of the cost function to successfully find the global optima because of the stochastic nature of the gradient approximation. In this paper, the implementation of the extended SPSA based stochastic gradient algorithm is introduced to the optimization of nonconvex registration function.
1-4244-1212-9/07/$25.00 ©2007 IEEE.
(6)
where i=1, 2, …6. Six transform parameters of ψ are to be
⎛ tx ⎞ ⎟⎟ denotes the two directional translations, ⎝ ty ⎠
⎛ ta11
( g k )3 ...( g k ) 6 ] .
The gradient vector is determined by a two-sided perturbation of the cost function (i.e. the mutual information) L. It is written as:
where the matrix ⎜⎜
and the matrix ⎜⎜ ⎝ ta 21 scaling, or shearing.
(5)
A. Experiment One In this experiment, two QuickBird images, shown in Fig. 1, were registered. The first one is a 2.4-meter QuickBird near infrared (NIR) multispectral image which accentuates vegetated areas in red, with the burn region clearly distinguished in green. The second one is a natural color multispectral image of the same area. The registered results are shown in Fig. 2. The registered natural color image with the
398
estimated transform parameters as follow: (Shift X, Shift Y, Scaling, Shear X, Shear Y, Rotation) = (2, 8, 0.9, 0.2, 0.1, 3). The disparity is successfully calibrated by the six-parameter affine transform. The information theory based similarity measure such as the mutual information is robust to the scenes with such heavy smoke contamination. B. Experiment Two In the second experiment, three geology-related multimodality images shown in Fig. 3 were registered in a lump other than two or three co-registrations by optimization of a mutual information criterion using a SPSA based stochastic gradient algorithm. Since we do not know the ground truth of this image set, a pseudo-color visualization technique is used to intuitionally check the registration results. The results shown in Fig. 4 are encouraging.
(i) Checkerboard before registration
(i) NIR image
(ii) Natural color image (ii) Checkerboard after registration
Figure 1. QuickBird multispectral image set (Size: 712x650 pixels, Location: Southern Arizona, Date: 06/21/2003).
Figure 2. Checkerboard visualization of QuickBird image set.
1-4244-1212-9/07/$25.00 ©2007 IEEE.
399
IV.
CONCLUSIONS
The stochastic gradient algorithm for optimizing the similarity measure is accomplished according to a SPSA technique. The experimental results show that the SPSA based stochastic gradient optimization is strong for the multimodality image registration. The main contribution of this paper is that we extended this technique to simultaneously register the affine transform, and successfully accomplished the SPSA based stochastic gradient algorithm to optimize a similarity measure such as the mutual information for the geology-related multimodality image registration. (i) ASTER VNIR image
REFERENCES
(ii) DSM image [1]
[2]
[3]
(iii) Digital geological map
[4]
Figure 3. Image set of geological images (size: 256x256 pixels). [5]
[6]
[7]
[8] Figure 4. Pseudo-color visualization of images: RGB = (ASTER, Map, DSM) Left: visualization of unregistered images Right: visualization of registered images.
1-4244-1212-9/07/$25.00 ©2007 IEEE.
[9]
400
Q. Li, "Challenging registration of geologic image data: initialization, estimation, and decision," Seminar Notes of Mathematical Sciences, Ibaraki University, Japan, vol. 10, pp. 1-10, April 2007. Q. Li, I. Sato, and Y. Murakami, "Toward the establishment of robust automatic system for multimodal image registration," in The 40th (2006 Spring) Annual Meeting of the Remote Sensing Society of Japan, Keyaki Kaikan, Chiba University, Chiba, Japan, 2006, pp. 101-102. (in Japanese) I. Sato, Q. Li, L. Nomura, J. Bandibus, K. Nishida, T. Masuda, and Y. Kita, "Research and development of resource fusion technology (Part of Advanced Research of Utilization Technology of Satellite Images)," National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, AIST Technical Research Report March 2006. (in Japanese) J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation and Control. New Jersey: John Wiley & Sons, Inc., 2003. A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, "Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient," IEEE Transactions on Image Processing, vol. 12, pp. 1495-1511, December 2003. Q. Li, I. Sato, and Y. Murakami, "Simultaneous perturbation stochastic approximation algorithm for automated image registration optimization," in 2006 IEEE International Geoscience and Remote Sensing Symposium & 27th Canadian Symposium on Remote Sensing (IGARSS 2006). Remote Sensing: A Natural Global Partnership, Denver, Colorado, USA, 2006, pp. 184-187. Q. Li, I. Sato, and Y. Murakami, "Automated image registration using stochastic optimization strategy of mutual information," Dynamics of Continuous Discrete and Impulsive Systems, Series B: Application and Algorithms, Supplement 1 Vol.7, pp. 2872-2877, 2006. Q. Li, I. Sato, and Y. Murakami, "Efficient stochastic gradient search for automatic image registration," International Journal of Simulation Modelling (IJSIMM), vol. 6, 2007. (in press) Q. Li, I. Sato, and Y. Murakami, "Interpolation effects on accuracy of mutual information based image registration," in 2006 IEEE International Geoscience and Remote Sensing Symposium & 27th Canadian Symposium on Remote Sensing (IGARSS 2006). Remote Sensing: A Natural Global Partnership, Denver, Colorado, USA, 2006, pp. 180-183.
general, the stochastic gradient optimization provides significant improvements on the optimal solution over the conventional methods such as hill climbing and simplex optimizations in terms of accuracy and robustness. In the current work, the SPSA based stochastic gradient algorithm is further tailored to accomplish an affine registration more precisely. The six mapping parameters of the affine transformation are simultaneously registered by optimization of the mutual information. Our registration experiments are associated with several pairs of multimodality images, which are supposed to be misaligned. The experimental results show that the proposed approach is effective and robust.
Keywords-image registration; affine transformation; stochastic gradient; mutual information; SPSA; optimization
The main contribution of this paper is the accomplishment of an efficient SPSA based stochastic gradient optimization on the affine registration of remote sensed images using a mutual information criterion.
I.
INTRODUCTION
With the earnest need for the information integration of multi-source image data, such as satellite image, airborne photograph, geological map, and various thematic maps, etc., the image fusion analysis are becoming urgent topics. During this process, the image registration is the key step. Although image registration has been studied over many years, and many registration algorithms have been proposed in the literatures, it is still a challenging topic in the field of digital image processing [1].
II.
A. Registration Optimization Image registration is to define a transform T that will map one image onto another image of the same scene acquired by different sensors, or taken by the same sensor at different times, such that a certain distance measure ( S ) is optimal. To register a float image ( I F ) to a reference image ( I R ) can be mathematically expressed as:
In our past research [2, 3], an intensity based registration prototype has been proposed and successfully applied to the image registration by optimization of the mutual information criterion. This paper focuses mainly on an efficient automatic affine registration method for multimodality images using the popular similarity measure, i.e., mutual information. The stochastic gradient algorithm for optimizing the mutual information is accomplished, based on the simultaneous perturbation stochastic approximation (SPSA) technique, which is firstly proposed by J.C. Spall [4]. In [5] and our previous work [6], the SPSA based stochastic gradient optimization scheme has been successfully used in the registration study of the rigid (translation and/or rotation) transformation by maximization of the standard mutual information. Our previous work [7, 8] also shows that, in
1-4244-1212-9/07/$25.00 ©2007 IEEE.
REGISTRATION METHOD
I R ( x, y ) = ζ ( I F (Tα ( x, y ))) ,
(1)
where Tα is a transformation function, which maps two spatial
coordinates x and y , to the new spatial coordinates x ′ and
y ′ by the set of mapping parameters α : ( x′, y ′) = Tα ( x, y ) ,
397
(2)
ζ
In the process of the intensity based automatic image registration, the similarity measure such as the mutual information is the cost function, let us call L, to be optimized during the SPSA search. At iteration k, a steepest ascent update law for the transform parameters is written as:
is the intensity or radiometric calibration function. The intensity based image registration can be mapped as a typical optimization problem [1]. This can be expressed as:
α * = Arg optima( STα ( I F , I R )) .
(3)
ψ k +1 = ψ k + a k g k ,
Tα
Although the image registration can be regarded as an optimization problem, the interpolation artifact [9] complicates the cost (or registration) function of (3), and also with the increase of the transformation parameters, the registration becomes into a complicated non-convex optimization problem. During the process of the automatic image registration, the optimization module is the key element for the whole registration process.
where g k is the gradient vector for the parameter space
(tx, ty , ta11 , ta12 , ta 21 , ta 22 ) in our study, and it can be written as:
g k = [( g k )1 ( g k ) 2
B. Affine Transformation In general, mathematical transformation for digital image registration is consisted of some elementary operations such as translation, rotation, scaling, stretching, and shearing. The traditional registration transformations are rigid, affine, projective, and curved. In remote sensing applications, the twodimensional (2D) affine transformation is usually sufficient for image registration. The general 2D affine transformation can be expressed as:
⎛ x ′ ⎞ ⎛ tx ⎞ ⎛ ta11 ta12 ⎞⎛ x ⎞ ⎟⎟⎜⎜ ⎟⎟ , ⎜⎜ ⎟⎟ = ⎜⎜ ⎟⎟ + ⎜⎜ ⎝ y ′ ⎠ ⎝ ty ⎠ ⎝ ta 21 ta 22 ⎠⎝ y ⎠
( g k )i =
L(ψ k + ck Δ k ) − L(ψ k − ck Δ k ) , 2 ck ( Δ k ) i
(7)
updated at each iteration. The vector Δ k is generated by a standard Bernoulli distribution. This vector represents simultaneous perturbations applied to all search space components in approximating the gradient. The gain sequences of ak and ck are positive.
(4)
Spall presents sufficient conditions for convergence of the SPSA iteration algorithm in the stochastic almost sure. The choice of control parameters is addressed in [4, 7] in detail.
ta12 ⎞ ⎟ can be interpreted as rotation, ta 22 ⎟⎠
III.
EXPERIMENTAL RESULTS
The goal of image registration is to find the optimal mapping parameters, discussed in Section II, which define the relative position and orientation of the two sensed images in an efficient and robust way. In this section, two demonstration experiments are chosen to be discussed. Experiment one is designed by standard image set with knowing the ground truth to verify our proposed approach. Experiment two is to register three images without knowing the ground truth. Throughout the experiments, the partial volume interpolation was used for the estimation of joint histogram during the computation of mutual information.
C. SPSA based Parameter Search The SPSA technique recently becomes very popular for solving some challenging optimization problems. The prominent merit of the SPSA based stochastic gradient optimization is that it does not require an explicit knowledge of the gradient of the cost function, or measurements of this gradient. At each of iterations, it only needs an approximation to the gradient via simultaneous perturbations. The gradient approximation is based on only two function measurements regardless of the dimensions of the search parameter space. The SPSA based stochastic gradient is very strong [7, 8], which can get through some local optima of the cost function to successfully find the global optima because of the stochastic nature of the gradient approximation. In this paper, the implementation of the extended SPSA based stochastic gradient algorithm is introduced to the optimization of nonconvex registration function.
1-4244-1212-9/07/$25.00 ©2007 IEEE.
(6)
where i=1, 2, …6. Six transform parameters of ψ are to be
⎛ tx ⎞ ⎟⎟ denotes the two directional translations, ⎝ ty ⎠
⎛ ta11
( g k )3 ...( g k ) 6 ] .
The gradient vector is determined by a two-sided perturbation of the cost function (i.e. the mutual information) L. It is written as:
where the matrix ⎜⎜
and the matrix ⎜⎜ ⎝ ta 21 scaling, or shearing.
(5)
A. Experiment One In this experiment, two QuickBird images, shown in Fig. 1, were registered. The first one is a 2.4-meter QuickBird near infrared (NIR) multispectral image which accentuates vegetated areas in red, with the burn region clearly distinguished in green. The second one is a natural color multispectral image of the same area. The registered results are shown in Fig. 2. The registered natural color image with the
398
estimated transform parameters as follow: (Shift X, Shift Y, Scaling, Shear X, Shear Y, Rotation) = (2, 8, 0.9, 0.2, 0.1, 3). The disparity is successfully calibrated by the six-parameter affine transform. The information theory based similarity measure such as the mutual information is robust to the scenes with such heavy smoke contamination. B. Experiment Two In the second experiment, three geology-related multimodality images shown in Fig. 3 were registered in a lump other than two or three co-registrations by optimization of a mutual information criterion using a SPSA based stochastic gradient algorithm. Since we do not know the ground truth of this image set, a pseudo-color visualization technique is used to intuitionally check the registration results. The results shown in Fig. 4 are encouraging.
(i) Checkerboard before registration
(i) NIR image
(ii) Natural color image (ii) Checkerboard after registration
Figure 1. QuickBird multispectral image set (Size: 712x650 pixels, Location: Southern Arizona, Date: 06/21/2003).
Figure 2. Checkerboard visualization of QuickBird image set.
1-4244-1212-9/07/$25.00 ©2007 IEEE.
399
IV.
CONCLUSIONS
The stochastic gradient algorithm for optimizing the similarity measure is accomplished according to a SPSA technique. The experimental results show that the SPSA based stochastic gradient optimization is strong for the multimodality image registration. The main contribution of this paper is that we extended this technique to simultaneously register the affine transform, and successfully accomplished the SPSA based stochastic gradient algorithm to optimize a similarity measure such as the mutual information for the geology-related multimodality image registration. (i) ASTER VNIR image
REFERENCES
(ii) DSM image [1]
[2]
[3]
(iii) Digital geological map
[4]
Figure 3. Image set of geological images (size: 256x256 pixels). [5]
[6]
[7]
[8] Figure 4. Pseudo-color visualization of images: RGB = (ASTER, Map, DSM) Left: visualization of unregistered images Right: visualization of registered images.
1-4244-1212-9/07/$25.00 ©2007 IEEE.
[9]
400
Q. Li, "Challenging registration of geologic image data: initialization, estimation, and decision," Seminar Notes of Mathematical Sciences, Ibaraki University, Japan, vol. 10, pp. 1-10, April 2007. Q. Li, I. Sato, and Y. Murakami, "Toward the establishment of robust automatic system for multimodal image registration," in The 40th (2006 Spring) Annual Meeting of the Remote Sensing Society of Japan, Keyaki Kaikan, Chiba University, Chiba, Japan, 2006, pp. 101-102. (in Japanese) I. Sato, Q. Li, L. Nomura, J. Bandibus, K. Nishida, T. Masuda, and Y. Kita, "Research and development of resource fusion technology (Part of Advanced Research of Utilization Technology of Satellite Images)," National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, AIST Technical Research Report March 2006. (in Japanese) J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation and Control. New Jersey: John Wiley & Sons, Inc., 2003. A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, "Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient," IEEE Transactions on Image Processing, vol. 12, pp. 1495-1511, December 2003. Q. Li, I. Sato, and Y. Murakami, "Simultaneous perturbation stochastic approximation algorithm for automated image registration optimization," in 2006 IEEE International Geoscience and Remote Sensing Symposium & 27th Canadian Symposium on Remote Sensing (IGARSS 2006). Remote Sensing: A Natural Global Partnership, Denver, Colorado, USA, 2006, pp. 184-187. Q. Li, I. Sato, and Y. Murakami, "Automated image registration using stochastic optimization strategy of mutual information," Dynamics of Continuous Discrete and Impulsive Systems, Series B: Application and Algorithms, Supplement 1 Vol.7, pp. 2872-2877, 2006. Q. Li, I. Sato, and Y. Murakami, "Efficient stochastic gradient search for automatic image registration," International Journal of Simulation Modelling (IJSIMM), vol. 6, 2007. (in press) Q. Li, I. Sato, and Y. Murakami, "Interpolation effects on accuracy of mutual information based image registration," in 2006 IEEE International Geoscience and Remote Sensing Symposium & 27th Canadian Symposium on Remote Sensing (IGARSS 2006). Remote Sensing: A Natural Global Partnership, Denver, Colorado, USA, 2006, pp. 180-183.
