Efficient registration of optical and infrared images ... - Semantic Scholar
Computers and Electrical Engineering 38 (2012) 1213–1221
Contents lists available at SciVerse ScienceDirect
Computers and Electrical Engineering journal homepage: www.elsevier.com/locate/compeleceng
Efficient registration of optical and infrared images via modified Sobel edging for plant canopy temperature estimation q Weiping Yang a,⇑, Xuezhi Wang b, Bill Moran b, Ashley Wheaton c, Nicola Cooley c a
School of Electronic Science & Engineering, National University of Defense Technology, Hunan 410073, China Melbourne Systems Laboratory, School of Engineering, University of Melbourne, Vic 3010, Australia c Melbourne School of Land and Environment, University of Melbourne, Vic 3010, Australia b
a r t i c l e
i n f o
Article history: Available online 20 June 2012
a b s t r a c t Automatic optical and infrared (IR) image registration is a crucial requirement in the estimation of plant canopy temperature. The latter is used to compute the plant water stress status which can potentially be applied to an automated control system for optimal crop irrigation management. Due to the nature of image sources and plant scene, it is difficult to find enough common features between the pair of images to register them using a simple method. In this paper an improved registration algorithm is described, where a modified Sobel edge detector is adopted and a variable resolution scheme is applied. The algorithm can improve the registration efficiency considerably compared to our previous developed registration method. Experiment results show that the modified algorithm can achieve the same registration accuracy and a better success rate compared to that from the original algorithm with a substantial reduction of computational complexity. Ó 2012 Elsevier Ltd. All rights reserved.
1. Introduction Water is the key resource for the agriculture sector, more than 70% freshwater is used for agriculture irrigation alone in many countries. How to use the limited freshwater efficiently and effectively is one of the most active and important research topics in the world. Advanced plant sensing techniques, which can determine plant water stress in a non-contact and non-destructive manner, enable the use of automated and smart irrigation systems which are able to irrigate the exact amount of water needed by a plant [1–5]. As descried in [1,2], the crop water stress index (CWSI) can be used as an effective indicator for plant water stress level. The perennial horticulture industry is deficient in plant based sensing techniques that can aid in irrigation decisions, ‘that is when, how often and how much to water’. A ‘Smart’ plant sensing technique would be required to provide automated realtime plant water status information and thus allow dynamic water management [1–5]. One such plant based sensor that can describe plant water status is based on plant temperature measurement; the crop water stress index (CWSI). It has the potential to adhere to many of the ideal ‘Smart’ sensor specifications, being an effective indicator for plant water stress with data that can be collected in real time, in ‘on farm’ robust situations. Canopy temperature may be estimated via the infrared images (IR) and an algorithm proposed [6]. The precision of the computed CWSI depends extensively on the accuracy of the canopy temperature estimation (IR data) and how this data is interoperated biologically. Leaf area and colour information are obtained via the optical image. The colour information allows the detection of leaf function in a ‘farm environment’.
q
Reviews processed and recommended for publication to Editor-in-Chief by Deputy Editor Dr. Ferat Sahin.
⇑ Corresponding author. Tel.: +86 133 0849 1470. E-mail address: [email protected] (W. Yang).
0045-7906/$ - see front matter Ó 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.compeleceng.2012.05.014
1214
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
Therefore, the image registration process is required to align the optical image with its associated IR image; so that temperature values corresponding to functioning plant leaves can be obtained. The development of an automated plant canopy temperature estimation algorithm was developed [6]. It is observed that the quality of the image pair alignment is crucial to the canopy temperature estimation accuracy and the computational complexity of image registration dominates the overall performance of the algorithm. The registration and matching of images have been well documented in the literature and many effective techniques are available, for example, cross correlation [7], mutual information [8], correlation ratio [9] and Scale-Invariant Feature Transform (SIFT) based methods [10]. A collection of fast algorithms have been cited, such as automatic registration approach based on point features [11], automatic image registration algorithm with sub-pixel accuracy [12], image registration using importance sampling [13] and image registration using shape matching [14]. While SIFT based methods are the best choice for many applications, they may only apply to the registration of rigid objects using the image pairs of the same source. When the pair of images are taken from different sources, possibly at different time, there are some practical challenges for the alignment of the optical and IR images, which have been discussed in our initial work in [15]. One of the major issues is the main objects in the images are time-varying plant leaves. In addition, images from different sources possess different imaging characteristics. Fig. 1 shows an example of registration failure using a SIFT based method because the number of matching key points found are insufficient. Wavelet transform was used in [16] to extract a number of feature points as the basis for image registration, but the method is only effective for the images with rigid objects taken by the same type of sensors, and the variations of the intensity fluctuations between the two images are assumed to be small. The registration algorithm based on the linear features between the two images was proposed in [17], where the images considered are in the man-made environments and are rich with linear features. Like SIFT, it is also not suitable for the underlying application which contains non-rigid objects from quite different image sources. Therefore it may be concluded that currently cited algorithms are not pertinent for the development of an algorithm that allows for non-rigid objects while being biological relevant. The work presented in this paper is the follow up work presented in [10] to improve the algorithm performance in terms of registration accuracy and efficiency. As analyzed in [10], apart from the overall contour structures and colours, the optical image is generally quite different from the associated IR image. Furthermore, each pair of images cannot be taken simultaneously. Therefore, an edge detector is used to extract edges from the pair of images. The image registration based on normalized cross correlation (NCC) is then performed on these edge images. In our experiment, it was observed that (1) the choice of edge detectors can heavily influence the registration accuracy; (2) the implemented NCC is time consuming. In this paper, we describe an improved registration procedure, where a modified Sobel edge detector is proposed for the edge detection and a variable resolution approach to the system implementation is considered to improve the registration algorithm efficiency. The experiment results presented in this paper show that the modified algorithm can achieve a similar registration accuracy and success rate compared to the existing approach but algorithm computational overhead has been substantially reduced. The modified Sobel edge detector and the variable resolution implementation ensure that we can obtain efficient image registration while maintaining biological significance. The paper is organized as follows: In Section 2 the algorithm is described and the flow diagram is given. In Section 3, the modified Sobel operators and the NCC algorithm implementation in variable resolutions are discussed. The experiment results are presented in Section 4, which is followed by the discussion and conclusion.
(a) 5906 key-points found in optical image
(b) 447 key-points found in IR image
(d) 605 key-points found in IR image
(c) 5666 key-points found in optical image Fig. 1. Results of SIFT based registration algorithm (no matching key-points) (optical image size is 2848 2136 pixels, IR image size is 320 240 pixels, the same below).
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
1215
2. Registration procedure The initial stage in registering an IR image to the accompanying optical image is to obtain location and rotation angle. For the problem at hand, the IR image is wholly overlapped with the reference optical image. Moreover, we assume that the space resolution of the two images is known and in our case is the same. The flowchart of the variable resolution algorithm based on normalized cross correlation is shown in Fig. 2. Since pixel by pixel based computation is required, implementation in full the normalized cross correlation (NCC) algorithm is time consuming. The algorithm efficiency issue can be addressed by several methods such as Sequential Similarity Detection Algorithm (SSDA) [18], Bit Plane Matching Algorithm (BPMA) [19], and Variable Resolution Correlation Algorithm (VRCA). Based on the merits of these techniques, the variable resolution algorithm is adopted for this application to improve the efficiency of the algorithm. The main idea of the variable resolution approach is that we initially perform a normalized cross correlation between the images using a reduced resolution and this may roughly identify the overlapping area of the two images. A high resolution correlation is then considered near the vicinity of each possible registration point and the correct registration point can be found by such a refined cross correlation procedure. Let ImO and ImIR (image size is M N) stand for the optical and IR image respectively, f be the image zoom factor; ImOl and ImIRl (image size is Mh Nh) stand for the low resolution images. Let ImOe, ImIRe, ImOle, ImIRle denote the edged images of the corresponding images, then the generalize cross correlation can be expressed as
ql ðu; v Þ ¼
PMh PNh
j¼0 ðImOleu;v ði; jÞ
i¼0
ð1Þ
rOlu;v rIRl
PM PN
qðk; lÞ ¼
ImIRle ði; jÞÞ
i¼0
j¼0 ðImOek;l ði; jÞ
ImIRe ði; jÞÞ
ð2Þ
rOk;l rIR
where ql ðu; v Þ is the cross correlation coefficient calculated from the low resolution image pair ImOle and ImIRle. (u,v) is the coordinate index of the optical image ImOle. ImOleu;v is an image located in the (u,v)th of image ImOle and its size is the same as the image ImIRle. rOlu,v and rIRl are the standard deviations of the corresponding images respectively. qðk; lÞ is the cross correlation coefficient calculated from the low resolution image pair ImOe and ImIRe. (k,l) is the coordinate index of the optical image ImOe. ImOek;l is an image located in the (k,l)th pixel of the image ImOe and its size is the same as image ImIRe. rOk,l and rIR are the standard deviations of the corresponding images respectively.
Optical image
IR image
Edge extraction
Edge extraction
Resolution Reducing
Resolution Reducing
NCC
Roughly registration
NCC at certain positions
Selecting optimal position
Rotation angle estimation
Image registration parameters Fig. 2. Flow chart of the NCC algorithm via the variable resolution approach.
1216
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
3. Modified Sobel operator To capture the common structural features between optical and IR images, the edge detectors play a key role. There are several methods for edge detection and extraction, such as Sobel, Roberts operators and Canny algorithm, and so on. In the underlying application the optical and IR images cannot be taken at the same time. The physical nature of leaves in the natural environment must also be considered, and often leaf edges in the two images are different. In the underlying application the optical and IR images cannot be taken at the same time. The physical nature of leaves in the natural environment must also be considered, and often leaf edges in the two images are different. It is observed that the standard Sobel operators and other available edge algorithms are too ‘‘sensitive’’ for these small leaf edges and their edge detection results have caused weak correlation between the optical and IR images to be registered. Therefore registration failures have been observed in our experiments. To preserve more global (larger) edges and ignore those locally fluctuated (smaller) leaf edges we propose the modified Sobel operators, which include four operators rather than the standard two operators. The four operators are
2
1 0 1
3
2
6 7 S1 ¼ 4 2 0 2 5; 1 0 1 2
2
1
6 S3 ¼ 4 1
0
0
0
3
7 1 5;
1 2
1
2
6 S2 ¼ 4 0
1
3
7 0 5 1 2 1
2
0
0
6 S4 ¼ 4 1
1 0
2
ð3aÞ
3
7 15
ð3bÞ
2 1 0
Using these four operators to convolute with the raw images, we can get
Ek ði; jÞ ¼
2 X 2 X Imði þ m 1; j þ n 1Þ Sk ðm; nÞ k ¼ 1; 2; 3; 4
ð4Þ
m¼0 n¼0
Eði; jÞ ¼ max Ek ði; jÞ k
ð5Þ
where Eði; jÞ is the edge of the point (i,j) of image Im. The differences of edge images between the standard and modified Sobel operators are illustrated in Fig. 3. It is observed that using the proposed modified Sobel operators the smaller leaf edges, most of which contain locally time-varying disturbance, have lower gray levels (Fig. 3c) while those larger edges, most of which represent the contours of the plant and are common to both the optical and IR images, have higher gray levels compared to the result in Fig. 3b processed using the standard Sobel operators. Therefore, a higher correlation between the edge filtered optical and IR images can be obtained compared to that between the edge images produced using the standard Sobel or other edge detectors. So, in the underlying application, a robust registration is obtained via the modified Sobel operators. Following edge detection, the registration process is initially performed via the NCC algorithm using the edged images ImOe, ImIRe, ImOle, ImIRle in reduced resolution as described in the flowchart. In order to take into account both the system and image uncertainties, a set of points fðuk ; v k Þg of the highest values of the correlation coefficient are labeled as the registration solution candidates. As indicated in Fig. 2, a fine registration step is carried out nearby each of the candidate points fðuk ; v k Þg using the images of full resolution. A weighted coefficient 0 < c < 1 is used to ensure that the final point with the maximum correlation coefficient approaches optimal.
qk ¼ cql ðuk ; v k Þ þ ð1 cÞqðu0k ; v 0k Þ
ð6Þ
Fig. 3. Edge detection results comparison for an IR image of a Grapevines. (a) Original IR image; (b) edge image using standard Sobel operators; (c) edge image using modified Sobel operators.
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
(a) Image pair 1(Optical(left), IR(right))
1217
(b) registration result of image pair 1 (287,299,-0.2)
(c) Image pair 1(Optical(left), IR(right))
(d) registration result of image pair 2 (343,166,0)
Fig. 4. Raw image pairs and their registration results.
Table 1 Comparison of the NCC registration performance. NCC implementation
Number of image pairs
Success number
Run-time per pair (average) (s)
Registration error (max, min, average)
Full-resolution Variable-resolution
20 20
20 20
610 15
(6, 0, 2.6) (6, 0, 2.6)
(a) Image pair 1(Optical(left), IR(right))
(b) Registration result 1 of Variable Res Algo (299,120,-0.8)
(c) Image pair 2(Optical(left), IR(right)) (d) Registration result 2 of Variable Res Algo (364,153,1.5) Fig. 5. Raw image pairs and their registration results using different algorithms.
qk ¼ max qk k
ð7Þ
where ðu0k ; v 0k Þ is the point on the image ImOe of full resolution, qðu0k ; v 0k Þ is the correlation coefficient of the fine image registration, ql ðuk ; v k Þ signifies the correlation coefficient corresponding to the image registration at reduced resolution. Following the steps (6) and (7), we can get the registration position ðu0k ; v 0k Þ. The final step is the estimation of the rotation angle between the optical and IR images. In the developed software, the range of rotation angle h is from 10° to 10°. The angle range is divided into 200 grids. The correlation coefficient is calculated at each of these grids and the rotation angle h associated with the maximum correlation value is then chosen. 4. Experiment results The refined NCC algorithm for image registration discussed in this paper was tested using grapevine image pairs, where the registration error, success rate and computational load are examined statistically. All images were collected using a FLIR
1218
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
Table 2 Registration performance comparison: Modified Sobel vs. Sobel and other edge method.
a
Edge detector
Number of image pairs
Success rate (%)a
Registration error (max, min, average)
Modified Sobel Standard Sobel Canny edge Robert edge
68 68 68 68
100 82.4 88.2 79.4
(7, (8, (6, (9,
0, 0, 0, 0,
2.6) 2.8) 2.6) 3.2)
A registration is counted as successful if the position error is less than 10 pixels.
(a) Optical raw image(left) and infrared raw image(right)
(b) Results of edge extraction using Canny algorithm(left) and registration result(235,148,-3.0)
(c) results of edge extraction using Roberts operators(left) and registration result(227,128,0)
(d) results of edge extraction using Standard Sobel operators(left) and registration result(223,98,-0.2)
(e) results of edge extraction using Modified Sobel operators(left) and registration result(224,98, 0) Fig. 6. Performance illustration of variable resolution algorithm based on different edge extraction methods.
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
1219
(f) optical raw image(left) and infrared raw image(right)
(g) results of edge extraction using Canny algorithm(left) and registration result(192,132,0)
(h) results of edge extraction using Roberts operators(left) and registration result(187,139,0.6)
(i) results of edge extraction using Standard Sobel operators(left) and registration result(187,138,0.7)
(j) results of edge extraction using Modified Sobel operators(left) and registration result(186,138,0.7) Fig. 6 (continued)
PM570 320X240 thermal imager and a standard Fujitsu S5600 digital camera from the Dookie College Estate vineyard, Victoria, Australia, where a research group from the University of Melbourne has conducted plant water stress monitoring experimental research using non-destructive and automated sensing technologies since 2007 [20]. The above algorithm was tested using many image pairs. All of the results are successful with errors in a tolerable range as shown in Fig. 4. The computational overhead of the proposed variable resolution method is compared to that of standard NCC under the same environment and conditions and the results are given in Table 1, which demonstrate both algorithms can achieve the same registration accuracy but the efficiency of the variable resolution algorithm is remarkably better, with only about one fortieth of the standard NCC running time.
1220
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
All of the tests are run in the condition of f = 2 and using Modified Sobel operator to get the image edges. Two registration examples based on our presented algorithm are illustrated in Fig. 5, which demonstrate registration effectiveness. The registration performance illustration is given in Table 2 and Fig 6. From the results we can see that the performance of the algorithm using Modified Sobel edge is the best. It demonstrates the effectiveness of our proposed algorithm. 5. Discussion From the experiment results we can see: 1. The underlying application requires an accurate registration between optical and IR images for a plant scene. This enables the plant leaf area from the IR image to be identified and thus the temperature data of the plant canopy can be obtained. The variable resolution algorithm presented is shown to be very efficient over the standard NCC method without compromising registration accuracy. 2. The use of the modified Sobel edge detector has resulted in the improvement in the processing abilities while maintaining biological significance in the following sense: Smaller edges – represent the plant leaves, most of which are time-varying and should be ignored; Larger edges – represent the contours of the plant, most of which are common to both the optical and IR images and should be considered as the common features. These special requirements have caused the standard edge detectors fail to obtain desired common features from the source images. 6. Conclusion The paper presented a fast image registration algorithm for the application of plant water stress monitoring using optical and infrared images. This is a refinement of an existing approach presented in [10]. By utilizing a variable resolution scheme and a modified Sobel edge detector, the proposed implementation has substantially improved the registration efficiency compared to a previous implementation. From our experimental results, the proposed algorithm runs about forty times faster than a standard implementation which has previously coded. In addition, as demonstrated by the statistics of field experiment results, an enhanced registration performance in terms of success rate is observed. Moreover, the improvement of this special registration algorithm makes it possible to establish a robust, real-time data sampling method for automated plant canopy temperature data collection. One would be able to determine how much, when and how often the plant should be watered by processing the data. References [1] Jones HG. Use of thermography for quantitative studies of spatial and temporal variation of stomatal conductance over leaf surfaces. J Plant Cell Environ 1999;22:1043–55. [2] Jones HG. Use of infrared thermometry for estimation of stomatal conductance as a possible aid to irrigation scheduling. J Agric For Meteorol 1999;95:139–49. [3] Guilioni L, Jones HG, Leinonen I, Lhomme JP. On the relationships between stomatal resistance and leaf temperatures in thermography. J Agric For Meteorol 2008;148:1908–12. [4] Grant O, Tronina1 L, Jones HG, Chaves MM. Exploring thermal imaging variables for the detection of stress responses in grapevine under different irrigation regimes. J Exp Botany 2007;58(4):815–25. [5] Wheaton AD, Cooley N, Dunn G, Goodwin I, Needs S. Evaluation of infrared thermography to determine the crop water status of Cabernet Sauvignon grapevines. In: Poster paper, 13th Australian wine industry technical conference, Adelaide; 28 July–2 August, 2007. [6] Wang XZ, Yang WP, Wheaton A, Cooley N, Moran B. Automated canopy temperature estimation via infrared thermography: a first step towards automated plant water stress monitoring. J Comput Electron Agric 2010;73(1):74–83. [7] Tsai Du-Ming, Lin Chien-Ta. Fast normalized cross correlation for defect detection. J Pattern Recog Lett 2003;24:2625–31. [8] Roche A, Malandain G, Pennec X, Ayache N. The correlation ratio as a new similarity measure for multimodal image registration. Medical image computing and computer assisted intervention – MICCAI’98, vol. 1496; 1998. p. 1115–24. [9] Klein S. Evaluation of optimization methods for nonrigid medical image registration using mutual information and B-splines. IEEE Trans Image Process 2007;16(12):2879–90. [10] Chen Jian, Tian Jie. Real-time multi-modal rigid registration based on a novel symmetric-SIFT descriptor. Prog Nat Sci 2009;19(5):643–51. [11] Le Yu, Zhang Dengrong, Holden Eun-Jung. A fast and fully automatic registration approach based on point features for multi-source remote-sensing images. Comput Geosci 2008;34(7):838–48. [12] Althof RJ, Wind MGJ, Dobbins JT. A rapid and automatic image registration algorithm with subpixel accuracy. IEEE Trans Med Imag 1997;16(3):308–16. [13] Bhagalia R, Fessler JA, Kim B. Accelerated non-rigid intensity-based image registration using importance sampling. IEEE Trans Med Imag 2009;28(8):1208–16. [14] Lima HS, Cheraghia SH. An optimization approach to shape matching and recognition. Comput Electr Eng 1998;24(3):183–200. [15] Yang Weiping, Wang Xuezhi, Wheaton A, Cooley N, Moran B. Automatic optical and IR image fusion for plant water stress analysis. In: Proc of 12th international conference on information fusion, Seatle; July, 2009. p. 1053–9. [16] Hsieh JW, Mark Liao HY, Fan KC, Ko MT, Hung YP. Image registration using a new edge-based approach. Comput Vis Image Understand 1997;67(2):112–30. [17] Rami Issa AR. Data fusion of multi-source imagery based on linear features registration. Int J Remote Sens 2010;31(19):5011–21. [18] Barnea DI, Silverman HF. A Class of algorithm for fast digital image registration. IEEE Trans Comput 1972;21(2):179–86. [19] Ko Sung-Jea, Lee Sung-Hee, Jeon Seung-Won, Kang Eui-Sung. Fast digital image stabilizer based on Gray-coded bit-plane matching. IEEE Trans Consumer Electron 1999;45(3):598–603.
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
1221
[20] Wheaton A, Cooley NC, Dunn GM, Goodwin I, Wang XZ, Moran B, et al. Use of thermal imagery to detect water stress during berry ripening in Vitis vinifera L. ‘Cabernet Sauvignon’. In: Proc of the VI international symposium on irrigation of horticultural crops, Chile. Acta Horticulturae 889; November 2009. p. 123–30. Weiping Yang received the B.S. degree in System Engineering from College of Communication Engineering, Xian, China in 1991 and received Master Degree and PhD degree in Information and Communication Engineering from the Department of Electronic Engineering, National University of Defence Technology in 1994 and 1998, respectively. He is now an association professor in School of Electronic Science and Engineering, National University of Defence Technology, Changsha, China. His research interests are signal and image processing, information fusion, and pattern recognition. Xuezhi Wang received the B. S. degree in avionics from Northwest Polytechnical University, Xian, China in 1982 and received PhD degree in Signal and Systems in 2001 in the Department of Electrical and Electronic Engineering, University of Melbourne, Australia. He is currently working as a research fellow in the Melbourne Systems Laboratory, department of Electrical and Electronic Engineering, University of Melbourne. His research interests are in stochastic signal processing, information theory, Bayesian estimation and plant sensing technologies. Bill Moran received the B. S. Hons. degree in Mathematics from University of Birmingham in 1965 and the PhD degree in Mathematics from University of Sheffield, London in 1968. He is a professor of electrical engineering at the University of Melbourne, Australia, where he serves as the Research Director of Defence Science Institute and Technical Director of Melbourne Systems Laboratory. Ashley Wheaton is completing a PhD titled ‘‘The use of remotely sensed thermal imagery to determine the spatial and temporal variation in grapevine water status’’. He was Lecturer in Viticulture at The University of Melbourne, Dookie Campus, from 2003 to 2009. The 5 years prior, Ashley worked jointly at The University and The Victorian Department of Primary Industries, Tatura where he was Viticultural Research Scientist from 1995. Ashley holds a BAgrSc (Hon) and MAgrSc from La Trobe University. Nicola Cooley receives her B. S. Hons. degree in Biological Sciences from University Plymouth, UK in 1995 and the PhD degree from Dept. of Environment Sciences, University of East London in 2001. She is working with the Melbourne School of Land and Environment, University of Melbourne as a research fellow. Her expertise falls in irrigated agriculture, particularly, in the area of plant water status, automated real time ‘‘smart’’ irrigation and canopy temperature data interpretation.
Contents lists available at SciVerse ScienceDirect
Computers and Electrical Engineering journal homepage: www.elsevier.com/locate/compeleceng
Efficient registration of optical and infrared images via modified Sobel edging for plant canopy temperature estimation q Weiping Yang a,⇑, Xuezhi Wang b, Bill Moran b, Ashley Wheaton c, Nicola Cooley c a
School of Electronic Science & Engineering, National University of Defense Technology, Hunan 410073, China Melbourne Systems Laboratory, School of Engineering, University of Melbourne, Vic 3010, Australia c Melbourne School of Land and Environment, University of Melbourne, Vic 3010, Australia b
a r t i c l e
i n f o
Article history: Available online 20 June 2012
a b s t r a c t Automatic optical and infrared (IR) image registration is a crucial requirement in the estimation of plant canopy temperature. The latter is used to compute the plant water stress status which can potentially be applied to an automated control system for optimal crop irrigation management. Due to the nature of image sources and plant scene, it is difficult to find enough common features between the pair of images to register them using a simple method. In this paper an improved registration algorithm is described, where a modified Sobel edge detector is adopted and a variable resolution scheme is applied. The algorithm can improve the registration efficiency considerably compared to our previous developed registration method. Experiment results show that the modified algorithm can achieve the same registration accuracy and a better success rate compared to that from the original algorithm with a substantial reduction of computational complexity. Ó 2012 Elsevier Ltd. All rights reserved.
1. Introduction Water is the key resource for the agriculture sector, more than 70% freshwater is used for agriculture irrigation alone in many countries. How to use the limited freshwater efficiently and effectively is one of the most active and important research topics in the world. Advanced plant sensing techniques, which can determine plant water stress in a non-contact and non-destructive manner, enable the use of automated and smart irrigation systems which are able to irrigate the exact amount of water needed by a plant [1–5]. As descried in [1,2], the crop water stress index (CWSI) can be used as an effective indicator for plant water stress level. The perennial horticulture industry is deficient in plant based sensing techniques that can aid in irrigation decisions, ‘that is when, how often and how much to water’. A ‘Smart’ plant sensing technique would be required to provide automated realtime plant water status information and thus allow dynamic water management [1–5]. One such plant based sensor that can describe plant water status is based on plant temperature measurement; the crop water stress index (CWSI). It has the potential to adhere to many of the ideal ‘Smart’ sensor specifications, being an effective indicator for plant water stress with data that can be collected in real time, in ‘on farm’ robust situations. Canopy temperature may be estimated via the infrared images (IR) and an algorithm proposed [6]. The precision of the computed CWSI depends extensively on the accuracy of the canopy temperature estimation (IR data) and how this data is interoperated biologically. Leaf area and colour information are obtained via the optical image. The colour information allows the detection of leaf function in a ‘farm environment’.
q
Reviews processed and recommended for publication to Editor-in-Chief by Deputy Editor Dr. Ferat Sahin.
⇑ Corresponding author. Tel.: +86 133 0849 1470. E-mail address: [email protected] (W. Yang).
0045-7906/$ - see front matter Ó 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.compeleceng.2012.05.014
1214
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
Therefore, the image registration process is required to align the optical image with its associated IR image; so that temperature values corresponding to functioning plant leaves can be obtained. The development of an automated plant canopy temperature estimation algorithm was developed [6]. It is observed that the quality of the image pair alignment is crucial to the canopy temperature estimation accuracy and the computational complexity of image registration dominates the overall performance of the algorithm. The registration and matching of images have been well documented in the literature and many effective techniques are available, for example, cross correlation [7], mutual information [8], correlation ratio [9] and Scale-Invariant Feature Transform (SIFT) based methods [10]. A collection of fast algorithms have been cited, such as automatic registration approach based on point features [11], automatic image registration algorithm with sub-pixel accuracy [12], image registration using importance sampling [13] and image registration using shape matching [14]. While SIFT based methods are the best choice for many applications, they may only apply to the registration of rigid objects using the image pairs of the same source. When the pair of images are taken from different sources, possibly at different time, there are some practical challenges for the alignment of the optical and IR images, which have been discussed in our initial work in [15]. One of the major issues is the main objects in the images are time-varying plant leaves. In addition, images from different sources possess different imaging characteristics. Fig. 1 shows an example of registration failure using a SIFT based method because the number of matching key points found are insufficient. Wavelet transform was used in [16] to extract a number of feature points as the basis for image registration, but the method is only effective for the images with rigid objects taken by the same type of sensors, and the variations of the intensity fluctuations between the two images are assumed to be small. The registration algorithm based on the linear features between the two images was proposed in [17], where the images considered are in the man-made environments and are rich with linear features. Like SIFT, it is also not suitable for the underlying application which contains non-rigid objects from quite different image sources. Therefore it may be concluded that currently cited algorithms are not pertinent for the development of an algorithm that allows for non-rigid objects while being biological relevant. The work presented in this paper is the follow up work presented in [10] to improve the algorithm performance in terms of registration accuracy and efficiency. As analyzed in [10], apart from the overall contour structures and colours, the optical image is generally quite different from the associated IR image. Furthermore, each pair of images cannot be taken simultaneously. Therefore, an edge detector is used to extract edges from the pair of images. The image registration based on normalized cross correlation (NCC) is then performed on these edge images. In our experiment, it was observed that (1) the choice of edge detectors can heavily influence the registration accuracy; (2) the implemented NCC is time consuming. In this paper, we describe an improved registration procedure, where a modified Sobel edge detector is proposed for the edge detection and a variable resolution approach to the system implementation is considered to improve the registration algorithm efficiency. The experiment results presented in this paper show that the modified algorithm can achieve a similar registration accuracy and success rate compared to the existing approach but algorithm computational overhead has been substantially reduced. The modified Sobel edge detector and the variable resolution implementation ensure that we can obtain efficient image registration while maintaining biological significance. The paper is organized as follows: In Section 2 the algorithm is described and the flow diagram is given. In Section 3, the modified Sobel operators and the NCC algorithm implementation in variable resolutions are discussed. The experiment results are presented in Section 4, which is followed by the discussion and conclusion.
(a) 5906 key-points found in optical image
(b) 447 key-points found in IR image
(d) 605 key-points found in IR image
(c) 5666 key-points found in optical image Fig. 1. Results of SIFT based registration algorithm (no matching key-points) (optical image size is 2848 2136 pixels, IR image size is 320 240 pixels, the same below).
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
1215
2. Registration procedure The initial stage in registering an IR image to the accompanying optical image is to obtain location and rotation angle. For the problem at hand, the IR image is wholly overlapped with the reference optical image. Moreover, we assume that the space resolution of the two images is known and in our case is the same. The flowchart of the variable resolution algorithm based on normalized cross correlation is shown in Fig. 2. Since pixel by pixel based computation is required, implementation in full the normalized cross correlation (NCC) algorithm is time consuming. The algorithm efficiency issue can be addressed by several methods such as Sequential Similarity Detection Algorithm (SSDA) [18], Bit Plane Matching Algorithm (BPMA) [19], and Variable Resolution Correlation Algorithm (VRCA). Based on the merits of these techniques, the variable resolution algorithm is adopted for this application to improve the efficiency of the algorithm. The main idea of the variable resolution approach is that we initially perform a normalized cross correlation between the images using a reduced resolution and this may roughly identify the overlapping area of the two images. A high resolution correlation is then considered near the vicinity of each possible registration point and the correct registration point can be found by such a refined cross correlation procedure. Let ImO and ImIR (image size is M N) stand for the optical and IR image respectively, f be the image zoom factor; ImOl and ImIRl (image size is Mh Nh) stand for the low resolution images. Let ImOe, ImIRe, ImOle, ImIRle denote the edged images of the corresponding images, then the generalize cross correlation can be expressed as
ql ðu; v Þ ¼
PMh PNh
j¼0 ðImOleu;v ði; jÞ
i¼0
ð1Þ
rOlu;v rIRl
PM PN
qðk; lÞ ¼
ImIRle ði; jÞÞ
i¼0
j¼0 ðImOek;l ði; jÞ
ImIRe ði; jÞÞ
ð2Þ
rOk;l rIR
where ql ðu; v Þ is the cross correlation coefficient calculated from the low resolution image pair ImOle and ImIRle. (u,v) is the coordinate index of the optical image ImOle. ImOleu;v is an image located in the (u,v)th of image ImOle and its size is the same as the image ImIRle. rOlu,v and rIRl are the standard deviations of the corresponding images respectively. qðk; lÞ is the cross correlation coefficient calculated from the low resolution image pair ImOe and ImIRe. (k,l) is the coordinate index of the optical image ImOe. ImOek;l is an image located in the (k,l)th pixel of the image ImOe and its size is the same as image ImIRe. rOk,l and rIR are the standard deviations of the corresponding images respectively.
Optical image
IR image
Edge extraction
Edge extraction
Resolution Reducing
Resolution Reducing
NCC
Roughly registration
NCC at certain positions
Selecting optimal position
Rotation angle estimation
Image registration parameters Fig. 2. Flow chart of the NCC algorithm via the variable resolution approach.
1216
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
3. Modified Sobel operator To capture the common structural features between optical and IR images, the edge detectors play a key role. There are several methods for edge detection and extraction, such as Sobel, Roberts operators and Canny algorithm, and so on. In the underlying application the optical and IR images cannot be taken at the same time. The physical nature of leaves in the natural environment must also be considered, and often leaf edges in the two images are different. In the underlying application the optical and IR images cannot be taken at the same time. The physical nature of leaves in the natural environment must also be considered, and often leaf edges in the two images are different. It is observed that the standard Sobel operators and other available edge algorithms are too ‘‘sensitive’’ for these small leaf edges and their edge detection results have caused weak correlation between the optical and IR images to be registered. Therefore registration failures have been observed in our experiments. To preserve more global (larger) edges and ignore those locally fluctuated (smaller) leaf edges we propose the modified Sobel operators, which include four operators rather than the standard two operators. The four operators are
2
1 0 1
3
2
6 7 S1 ¼ 4 2 0 2 5; 1 0 1 2
2
1
6 S3 ¼ 4 1
0
0
0
3
7 1 5;
1 2
1
2
6 S2 ¼ 4 0
1
3
7 0 5 1 2 1
2
0
0
6 S4 ¼ 4 1
1 0
2
ð3aÞ
3
7 15
ð3bÞ
2 1 0
Using these four operators to convolute with the raw images, we can get
Ek ði; jÞ ¼
2 X 2 X Imði þ m 1; j þ n 1Þ Sk ðm; nÞ k ¼ 1; 2; 3; 4
ð4Þ
m¼0 n¼0
Eði; jÞ ¼ max Ek ði; jÞ k
ð5Þ
where Eði; jÞ is the edge of the point (i,j) of image Im. The differences of edge images between the standard and modified Sobel operators are illustrated in Fig. 3. It is observed that using the proposed modified Sobel operators the smaller leaf edges, most of which contain locally time-varying disturbance, have lower gray levels (Fig. 3c) while those larger edges, most of which represent the contours of the plant and are common to both the optical and IR images, have higher gray levels compared to the result in Fig. 3b processed using the standard Sobel operators. Therefore, a higher correlation between the edge filtered optical and IR images can be obtained compared to that between the edge images produced using the standard Sobel or other edge detectors. So, in the underlying application, a robust registration is obtained via the modified Sobel operators. Following edge detection, the registration process is initially performed via the NCC algorithm using the edged images ImOe, ImIRe, ImOle, ImIRle in reduced resolution as described in the flowchart. In order to take into account both the system and image uncertainties, a set of points fðuk ; v k Þg of the highest values of the correlation coefficient are labeled as the registration solution candidates. As indicated in Fig. 2, a fine registration step is carried out nearby each of the candidate points fðuk ; v k Þg using the images of full resolution. A weighted coefficient 0 < c < 1 is used to ensure that the final point with the maximum correlation coefficient approaches optimal.
qk ¼ cql ðuk ; v k Þ þ ð1 cÞqðu0k ; v 0k Þ
ð6Þ
Fig. 3. Edge detection results comparison for an IR image of a Grapevines. (a) Original IR image; (b) edge image using standard Sobel operators; (c) edge image using modified Sobel operators.
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
(a) Image pair 1(Optical(left), IR(right))
1217
(b) registration result of image pair 1 (287,299,-0.2)
(c) Image pair 1(Optical(left), IR(right))
(d) registration result of image pair 2 (343,166,0)
Fig. 4. Raw image pairs and their registration results.
Table 1 Comparison of the NCC registration performance. NCC implementation
Number of image pairs
Success number
Run-time per pair (average) (s)
Registration error (max, min, average)
Full-resolution Variable-resolution
20 20
20 20
610 15
(6, 0, 2.6) (6, 0, 2.6)
(a) Image pair 1(Optical(left), IR(right))
(b) Registration result 1 of Variable Res Algo (299,120,-0.8)
(c) Image pair 2(Optical(left), IR(right)) (d) Registration result 2 of Variable Res Algo (364,153,1.5) Fig. 5. Raw image pairs and their registration results using different algorithms.
qk ¼ max qk k
ð7Þ
where ðu0k ; v 0k Þ is the point on the image ImOe of full resolution, qðu0k ; v 0k Þ is the correlation coefficient of the fine image registration, ql ðuk ; v k Þ signifies the correlation coefficient corresponding to the image registration at reduced resolution. Following the steps (6) and (7), we can get the registration position ðu0k ; v 0k Þ. The final step is the estimation of the rotation angle between the optical and IR images. In the developed software, the range of rotation angle h is from 10° to 10°. The angle range is divided into 200 grids. The correlation coefficient is calculated at each of these grids and the rotation angle h associated with the maximum correlation value is then chosen. 4. Experiment results The refined NCC algorithm for image registration discussed in this paper was tested using grapevine image pairs, where the registration error, success rate and computational load are examined statistically. All images were collected using a FLIR
1218
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
Table 2 Registration performance comparison: Modified Sobel vs. Sobel and other edge method.
a
Edge detector
Number of image pairs
Success rate (%)a
Registration error (max, min, average)
Modified Sobel Standard Sobel Canny edge Robert edge
68 68 68 68
100 82.4 88.2 79.4
(7, (8, (6, (9,
0, 0, 0, 0,
2.6) 2.8) 2.6) 3.2)
A registration is counted as successful if the position error is less than 10 pixels.
(a) Optical raw image(left) and infrared raw image(right)
(b) Results of edge extraction using Canny algorithm(left) and registration result(235,148,-3.0)
(c) results of edge extraction using Roberts operators(left) and registration result(227,128,0)
(d) results of edge extraction using Standard Sobel operators(left) and registration result(223,98,-0.2)
(e) results of edge extraction using Modified Sobel operators(left) and registration result(224,98, 0) Fig. 6. Performance illustration of variable resolution algorithm based on different edge extraction methods.
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
1219
(f) optical raw image(left) and infrared raw image(right)
(g) results of edge extraction using Canny algorithm(left) and registration result(192,132,0)
(h) results of edge extraction using Roberts operators(left) and registration result(187,139,0.6)
(i) results of edge extraction using Standard Sobel operators(left) and registration result(187,138,0.7)
(j) results of edge extraction using Modified Sobel operators(left) and registration result(186,138,0.7) Fig. 6 (continued)
PM570 320X240 thermal imager and a standard Fujitsu S5600 digital camera from the Dookie College Estate vineyard, Victoria, Australia, where a research group from the University of Melbourne has conducted plant water stress monitoring experimental research using non-destructive and automated sensing technologies since 2007 [20]. The above algorithm was tested using many image pairs. All of the results are successful with errors in a tolerable range as shown in Fig. 4. The computational overhead of the proposed variable resolution method is compared to that of standard NCC under the same environment and conditions and the results are given in Table 1, which demonstrate both algorithms can achieve the same registration accuracy but the efficiency of the variable resolution algorithm is remarkably better, with only about one fortieth of the standard NCC running time.
1220
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
All of the tests are run in the condition of f = 2 and using Modified Sobel operator to get the image edges. Two registration examples based on our presented algorithm are illustrated in Fig. 5, which demonstrate registration effectiveness. The registration performance illustration is given in Table 2 and Fig 6. From the results we can see that the performance of the algorithm using Modified Sobel edge is the best. It demonstrates the effectiveness of our proposed algorithm. 5. Discussion From the experiment results we can see: 1. The underlying application requires an accurate registration between optical and IR images for a plant scene. This enables the plant leaf area from the IR image to be identified and thus the temperature data of the plant canopy can be obtained. The variable resolution algorithm presented is shown to be very efficient over the standard NCC method without compromising registration accuracy. 2. The use of the modified Sobel edge detector has resulted in the improvement in the processing abilities while maintaining biological significance in the following sense: Smaller edges – represent the plant leaves, most of which are time-varying and should be ignored; Larger edges – represent the contours of the plant, most of which are common to both the optical and IR images and should be considered as the common features. These special requirements have caused the standard edge detectors fail to obtain desired common features from the source images. 6. Conclusion The paper presented a fast image registration algorithm for the application of plant water stress monitoring using optical and infrared images. This is a refinement of an existing approach presented in [10]. By utilizing a variable resolution scheme and a modified Sobel edge detector, the proposed implementation has substantially improved the registration efficiency compared to a previous implementation. From our experimental results, the proposed algorithm runs about forty times faster than a standard implementation which has previously coded. In addition, as demonstrated by the statistics of field experiment results, an enhanced registration performance in terms of success rate is observed. Moreover, the improvement of this special registration algorithm makes it possible to establish a robust, real-time data sampling method for automated plant canopy temperature data collection. One would be able to determine how much, when and how often the plant should be watered by processing the data. References [1] Jones HG. Use of thermography for quantitative studies of spatial and temporal variation of stomatal conductance over leaf surfaces. J Plant Cell Environ 1999;22:1043–55. [2] Jones HG. Use of infrared thermometry for estimation of stomatal conductance as a possible aid to irrigation scheduling. J Agric For Meteorol 1999;95:139–49. [3] Guilioni L, Jones HG, Leinonen I, Lhomme JP. On the relationships between stomatal resistance and leaf temperatures in thermography. J Agric For Meteorol 2008;148:1908–12. [4] Grant O, Tronina1 L, Jones HG, Chaves MM. Exploring thermal imaging variables for the detection of stress responses in grapevine under different irrigation regimes. J Exp Botany 2007;58(4):815–25. [5] Wheaton AD, Cooley N, Dunn G, Goodwin I, Needs S. Evaluation of infrared thermography to determine the crop water status of Cabernet Sauvignon grapevines. In: Poster paper, 13th Australian wine industry technical conference, Adelaide; 28 July–2 August, 2007. [6] Wang XZ, Yang WP, Wheaton A, Cooley N, Moran B. Automated canopy temperature estimation via infrared thermography: a first step towards automated plant water stress monitoring. J Comput Electron Agric 2010;73(1):74–83. [7] Tsai Du-Ming, Lin Chien-Ta. Fast normalized cross correlation for defect detection. J Pattern Recog Lett 2003;24:2625–31. [8] Roche A, Malandain G, Pennec X, Ayache N. The correlation ratio as a new similarity measure for multimodal image registration. Medical image computing and computer assisted intervention – MICCAI’98, vol. 1496; 1998. p. 1115–24. [9] Klein S. Evaluation of optimization methods for nonrigid medical image registration using mutual information and B-splines. IEEE Trans Image Process 2007;16(12):2879–90. [10] Chen Jian, Tian Jie. Real-time multi-modal rigid registration based on a novel symmetric-SIFT descriptor. Prog Nat Sci 2009;19(5):643–51. [11] Le Yu, Zhang Dengrong, Holden Eun-Jung. A fast and fully automatic registration approach based on point features for multi-source remote-sensing images. Comput Geosci 2008;34(7):838–48. [12] Althof RJ, Wind MGJ, Dobbins JT. A rapid and automatic image registration algorithm with subpixel accuracy. IEEE Trans Med Imag 1997;16(3):308–16. [13] Bhagalia R, Fessler JA, Kim B. Accelerated non-rigid intensity-based image registration using importance sampling. IEEE Trans Med Imag 2009;28(8):1208–16. [14] Lima HS, Cheraghia SH. An optimization approach to shape matching and recognition. Comput Electr Eng 1998;24(3):183–200. [15] Yang Weiping, Wang Xuezhi, Wheaton A, Cooley N, Moran B. Automatic optical and IR image fusion for plant water stress analysis. In: Proc of 12th international conference on information fusion, Seatle; July, 2009. p. 1053–9. [16] Hsieh JW, Mark Liao HY, Fan KC, Ko MT, Hung YP. Image registration using a new edge-based approach. Comput Vis Image Understand 1997;67(2):112–30. [17] Rami Issa AR. Data fusion of multi-source imagery based on linear features registration. Int J Remote Sens 2010;31(19):5011–21. [18] Barnea DI, Silverman HF. A Class of algorithm for fast digital image registration. IEEE Trans Comput 1972;21(2):179–86. [19] Ko Sung-Jea, Lee Sung-Hee, Jeon Seung-Won, Kang Eui-Sung. Fast digital image stabilizer based on Gray-coded bit-plane matching. IEEE Trans Consumer Electron 1999;45(3):598–603.
W. Yang et al. / Computers and Electrical Engineering 38 (2012) 1213–1221
1221
[20] Wheaton A, Cooley NC, Dunn GM, Goodwin I, Wang XZ, Moran B, et al. Use of thermal imagery to detect water stress during berry ripening in Vitis vinifera L. ‘Cabernet Sauvignon’. In: Proc of the VI international symposium on irrigation of horticultural crops, Chile. Acta Horticulturae 889; November 2009. p. 123–30. Weiping Yang received the B.S. degree in System Engineering from College of Communication Engineering, Xian, China in 1991 and received Master Degree and PhD degree in Information and Communication Engineering from the Department of Electronic Engineering, National University of Defence Technology in 1994 and 1998, respectively. He is now an association professor in School of Electronic Science and Engineering, National University of Defence Technology, Changsha, China. His research interests are signal and image processing, information fusion, and pattern recognition. Xuezhi Wang received the B. S. degree in avionics from Northwest Polytechnical University, Xian, China in 1982 and received PhD degree in Signal and Systems in 2001 in the Department of Electrical and Electronic Engineering, University of Melbourne, Australia. He is currently working as a research fellow in the Melbourne Systems Laboratory, department of Electrical and Electronic Engineering, University of Melbourne. His research interests are in stochastic signal processing, information theory, Bayesian estimation and plant sensing technologies. Bill Moran received the B. S. Hons. degree in Mathematics from University of Birmingham in 1965 and the PhD degree in Mathematics from University of Sheffield, London in 1968. He is a professor of electrical engineering at the University of Melbourne, Australia, where he serves as the Research Director of Defence Science Institute and Technical Director of Melbourne Systems Laboratory. Ashley Wheaton is completing a PhD titled ‘‘The use of remotely sensed thermal imagery to determine the spatial and temporal variation in grapevine water status’’. He was Lecturer in Viticulture at The University of Melbourne, Dookie Campus, from 2003 to 2009. The 5 years prior, Ashley worked jointly at The University and The Victorian Department of Primary Industries, Tatura where he was Viticultural Research Scientist from 1995. Ashley holds a BAgrSc (Hon) and MAgrSc from La Trobe University. Nicola Cooley receives her B. S. Hons. degree in Biological Sciences from University Plymouth, UK in 1995 and the PhD degree from Dept. of Environment Sciences, University of East London in 2001. She is working with the Melbourne School of Land and Environment, University of Melbourne as a research fellow. Her expertise falls in irrigated agriculture, particularly, in the area of plant water status, automated real time ‘‘smart’’ irrigation and canopy temperature data interpretation.
