adaptive shape prior in graph cut segmentation - Semantic Scholar
Proceedings of 2010 IEEE 17th International Conference on Image Processing
September 26-29, 2010, Hong Kong
ADAPTIVE SHAPE PRIOR IN GRAPH CUT SEGMENTATION Hui Wang and Hong Zhang Department of Computing Science University of Alberta, Edmonton, Alberta, CANADA {wanghui,zhang}@cs.ualberta.ca ABSTRACT In this paper, we propose a novel method to adaptively apply shape prior in graph cut segmentation. By incorporating shape priors in an adaptive way, we introduce a robust way to harness shape prior in graph cut segmentation. Since traditional graph cut approaches with shape prior may fail in cases where parameters for shape prior term are not set appropriately, incorporation of shape priors adaptively within this framework mitigates these problems. To address this issue, we propose to adaptively apply shape prior based on a shape probability map, defined to reflect the need of shape prior at each location of an image. We show that the proposed method can be easily applied to existing algorithms of graph cut segmentation with shape prior, such as level set based shape prior method, and star shape prior graph cut. We validate our method in various types of images corrupted by significant noise and intensity inhomogeneities. Convincing results are obtained. Index Terms— Adaptive, Shape Prior, Graph Cut, Interactive Image Segmentation 1. INTRODUCTION Image segmentation has always been an important and challenging task in computer vision. Since Boykov and Jolly [1] introduced the application of graph cut algorithm into image segmentation, graph cut based segmentation has become very popular in image segmentation in the last decade because it not only allows one to incorporate user interaction easily, but it is also efficient and globally optimal. More recently, in order to effectively handle noisy images or images with object occlusion, new graph cut based methods have been developed that are able to exploit the knowledge of shape prior. Freedman and Zhang [2] proposed to incorporate shape priors within the graph cut framework by matching the segmented curve with a level-set based shape template. Veksler [3] showed how to implement a star shape prior into graph cut segmentation to deal with a certain class of shapes called star shape. Das et al. presented a similar idea to incorporate shape prior to graph cut for shapes defined as compact shape [4]. Various related issues to the use of shape
978-1-4244-7993-1/10/$26.00 ©2010 IEEE
3029
prior have been investigated. Some focus on one or two partucular types of shapes [5, 6], some focus on incorporating multiple shape priors into one image [7], and some focus on shape representation and more general shape constraints [8, 9]. One of the outstanding issues of the graph cut framework is parameter selection. Parameters are usually fixed beforehand by the developer of the algorithm to achieve the best results for certain types of images [10]. Peng and Veksler [10] designed a parameter selection method by measuring segmentation quality based on different features of segmentation. Then they run the graph cut segmentation for different parameter values and choose the segmentation of the highest quality. However, their method only targets at issues on selecting the parameters between the data term and the boundary term, not on the newly added shape prior term. For images corrupted by significant noise and intensity inhomogeneities, the needs of shape priors at different pixels of an image are in general different. Therefore, setting a constant parameter on shape prior term in the energy function for all pixels may not be appropriate. Columns (b) to (d) in Figure 2 and 3 show that different parameter settings for shape prior can lead to very different segmentation results. To solve the above issue, we propose to impose shape constraint judiciously, by applying shape prior adaptively in graph cut segmentation. To determine the need of a pixel for shape prior, we derive a probability map based on image intensity information. The intuition behind this is that pixels with low probability values need a stronger shape term than pixels with high probability values. As will be seen, this probability can be easily calculated in a preprocessing step by making use of a classifier trained from labeled images. Following the similar idea, Song et al. [11] proposed a framework for automatic brain MRI tissue segmentation within an iterative scheme by incorporating the adaptive probabilistic atlas priors and intensity inhomogeneities correction for image segmentation into the graph cut energy function. However, the performance of their method relies on the accuracy of the probabilistic atlas and several other parameter settings. In another study, Bar-Yosef et al. [12] proposed a variational method for model based segmentation with adaptive shape prior. They propose to adaptively incorporate shape prior with the help of a shape confidence map. Their shape
ICIP 2010
confidence map is defined to reflect how correct each shape template candidate fits the current image and how much the template could help with current object segmentation. In contrast, our proposed method tackles adaptive shape prior from a differen point of view. Our shape probability map is defined to be based on the available image information, and it should reflect how much each pixel of the current image needs shape prior to help with segmentation, with the assumption that the provided shape prior is correct. The rest of the paper is organized as follows. In Section 2 we present the background for graph cut based segmentation with shape prior and the issue of parameter selection. In Section 3 we present our proposed adaptive shape prior idea and examples of applying it on two existing graph cut based methods using shape prior. In Section 4 we provide the experimental results to demonstrate the superiority of our approach. In Section 5 we state conclusions and future work. 2. BACKGROUND 2.1. Graph Cut Image Segmentation With Shape Prior
(1)
where 𝐸𝑖 is the original image energy, 𝐸𝑠 is an energy term based on the shape prior, and 𝜆 is a constant to balance between 𝐸𝑖 and 𝐸𝑠 . 𝐸𝑖 is usually defined as: ∑ ∑ 𝐸𝑖 = 𝜇 𝐷𝑝 (𝑓𝑝 ) + 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ) (2) 𝑝∈𝑃
In this section, we first present our proposed method for adaptively applying shape prior. Then, we apply our method to two existing graph cut based methods with shape priors. 3.1. Adaptive Shape Prior We propose to modify the energy function of graph cut segmentation with shape prior in the following way: 𝐸(𝑓 ) =
𝜇
∑
∑
𝑝∈𝑃
𝐷𝑝 (𝑓𝑝 ) +
∑ (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
1 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑒(𝛼𝑝 −𝛼𝑞 )2
𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+
𝐸𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )
(3) where 𝛼𝑝 represents the probability value from the probability map 𝛼 at pixel 𝑝, to be discussed in the next section. 𝐸𝑝 𝑞(𝑓𝑝 , 𝑓𝑞 ) represents a pairwise shape constraint term at pixel 𝑝 and 𝑞. 3.2. Shape Probability Map
A number of existing graph cut based segmentation methods incorporate shape prior by modifying the original graph cut energy function [1] by adding a shape prior energy term in the following way: 𝐸 = 𝐸𝑖 + 𝜆𝐸𝑠
3. PROPOSED METHOD
(𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
where 𝑁 is the set of neighboring pixels, 𝜇 is a balance factor between region term 𝐷𝑝 (𝑓𝑝 ) and boundary term 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ), and 𝑓𝑝 represents assigning label 𝑓𝑝 to pixel 𝑝. The particular forms for 𝐷𝑝 (𝑓𝑝 ) and 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ) are discussed in [1]. The shape energy term 𝐸𝑠 is defined in various ways depending on the type of shape prior.
𝛼 can be defined in various ways which reflect the need for shape prior at each pixel of an image. We refer to 𝛼 as shape prior probability map. This probability map should be generated from available image level information. In our experiment, 𝛼 is obtained from supervised learning [13]. The training is a very straight forward method. As an alternative, 𝛼 can also be defined as a matting map [14], as long as 𝛼 reflects the confidence of each pixel belonging to background or foreground. Hence, the above modified energy function (3) reflects that for pixel pair 𝑝 and 𝑞 which are neighboring pixels, if the probability value difference at these two locations is low, there should be a higher weight for shape term in the energy function. Two examples of the original images and their corresponding probability map 𝛼 are shown in Figure 1.
(a)
2.2. Parameter Selection for Shape Prior It is difficult to select a proper constant 𝜆 for Equation (1) to obtain proper segmentation result. Several shape prior graph cut papers have mentioned that users need to adjust 𝜆 depending on the types of images. Especially for images corrupted by significant noise and intensity inhomogeneities, the need of shape priors at different pixels of an image might vary significantly. In other words, setting a constant value 𝜆 for all pixels on the whole image is not appropriate. Again, we refer to Figure 2 and 3 for the sensitivity of the segmentation to the choice of 𝜆.
3030
(b)
Fig. 1. Examples of shape probablity map. For both (a) and (b), the left one shows the original image while the right one shows the probability map learned from supervised learning [13]. 3.3. Applying Adaptive Shape Prior 3.3.1. Level Set Based Shape Prior Method Freedman et al. [2] introduced to incorporate shape prior template in the form of level-set. In order to incorporate a shape
prior term, they modified the original graph cut energy function by adding a shape term in a similar way to Equation (1): ′
′
𝐸 = (1 − 𝜆 )𝐸𝑖 + 𝜆 𝐸𝑠
(4)
′
where 𝜆 is a constant. Using the idea of a level-set template, they defined shape energy term 𝐸𝑠 in Equation (4) in the following way: ) ( ∑ 𝑝+𝑞 ¯ (5) 𝐸𝑠 = 𝜙 2 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
where 𝜙¯ is a regular, unsigned distance function whose zero level set corresponds to the shape template curve 𝑐¯. To apply adaptive shape prior idea to Freedman et al.’s method, energy function (4) is modified to be: ′ ∑ 𝐸(𝑓 ) = (1 − 𝜆 ) 𝑝∈𝑃 𝐷𝑝 (𝑓𝑝 )+ ′
𝜆[ ∑
∑ (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+
1 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑒(𝛼𝑝 −𝛼𝑞 )2
𝜙¯
( 𝑝+𝑞 ) 2
(6) ]
where 𝛼𝑝 is from the probability map. 3.3.2. Star Shape Prior Another interesting method called star shape prior [3] applied more general shape prior to graph cut segmentation. However, it still shares the common limitations with parameter selection for the shape term. The star shape prior energy function is defined as: ∑ ∑ 𝐸(𝑓 ) = 𝜇 𝑝∈𝑃 𝐷𝑝 (𝑓𝑝 ) + (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+ ∑ (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
𝑆𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )
(7) where 𝑆𝑝𝑞 is a shape energy term defined as star shape [3]. Energy function (7) appears to have only one parameter 𝜇 to tune, but there is actually a hidden parameter inside shape term 𝑆𝑝𝑞 , which is defined as edge value 𝛽. 𝛽 needs to be chosen appropriately by the user in order to obtain good segmentation result [3]. On the other hand, Peng and Veksler [10] have already designed a parameter selection method to find the right 𝜇 between region and boundary term, by measuring segmentation quality based on different features of segmentation. On top of that, our proposed method solves the remaining issue with shape prior term by adaptively applying the shape prior 𝑆𝑝𝑞 . The adaptive star shape prior makes the process of incorporating shape prior flexible. After applying adaptive shape prior, Equation (7) becomes ∑ ∑ 𝐸(𝑓 ) = 𝑝∈𝑃 𝐷𝑝 (𝑓𝑝 ) + (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+ ∑
1 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑒(𝛼𝑝 −𝛼𝑞 )2
𝑆𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ) (8)
3031
4. EXPERIMENTAL RESULTS To validate the proposed adaptive shape prior method, we have run experiments on two exisiting graph cut based methods with shape prior detailed in Section 3.3. We use a MATLAB wrapper to interface with the C++ maxflow code by Boykov and Kolmogorov [1]. We have performed our experiments on various types of images including medical images, oil sand images and other star shape images which satisfy the assumption of star shape prior method. To compare our method with the original methods in Section 3.3, we perform experiments based on the same user initializations required from the original methods. Shape prior templates are also provided in the same way for both our method and the original method from [2], so shape alignment step is not considered in the experiments. All 𝛼 probability maps are obtained from a supervised learning algorithm [13] for all images. Visual results are shown in Figure 2 and 3. The original images are shown in column (a), our results are shown in column (e). Results obtained by original methods are shown in columns (b) to (d). The results look convincing. It is clear that our method obtains much better segmentation results without the need to optimize with regard to 𝜆, while the two original graph cut mehtods both need to tune parameter 𝜆. In all cases, our results are better than or similar to the optimal results in the previous studies. 5. CONCLUSIONS AND FUTURE WORK We have proposed an adaptive method for incorporating shape prior into graph cut based segmentation framework to eliminate failure cases in previous approaches in which parameters had to be tuned to fit the image. Adaptive shape prior works by adding the shape term in the energy function based on a probability map. We have shown that the proposed method can be easily applied to various types of graph cut based image segmentation method with shape prior, such as Freedman and Zhang’s original graph cut method with shape prior, and star shape prior method with graph cut. We have validated our method in various types of images and obtained very positive results. Although we need a preprocessing step to obtain the probability map 𝛼, the preprocessing step is straight forward and does not add much additional cost to graph cut segmentation. One major direction of future work is to see if this adaptive shape prior idea could be extended to segmentation algorithms other than graph cut based algorithms. Another major direction for future research is to examine whether more types of shapes can be easily incorporated into the framework. While we have seen that the current method is relative robust compared to other exsiting shape prior methods, it will be interesting to see whether this robustness holds in the case of greater variations in shape and images, and, if not, how the
algorithm may be modified to account for these changes. In addition, applying shape prior locally instead of globally can be another direction to pursue.
[2] D. Freedman and T. Zhang, “Interactive graph cut based segmentation with shape priors,” in IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 755–762. [3] O. Veskler, “Star shape prior for graph-cut image segmentation,” in IEEE European Conference on Computer Vision, 2008, pp. 454–467. [4] P. Das, O. Veskler, V. Zavadsky, and Y. Boykov, “Semiautomatic segmentation with compact shape prior,” in Image and Vision Computing, 2009, vol. 27, pp. 206– 219. [5] G. Slabaugh and G. Unal, “Graph cuts segmentation using an elliptical shape prior,” in International Conference on Image Processing, 2005, vol. 2. [6] M. Kumar, P. Torr, and A. Zisserman, “Obj cut,” in IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 18–25.
Fig. 2. Results from level-set shape prior method [2]. Column (a) shows the original images. Columns (b)-(d) show segmentation results from level-set shape prior method with 𝜆 = 0.1, 0.5 and 0.9. Column (e) shows segmentation results from our adaptive shape prior applied to level-set shape prior method .
[7] N. Vu and B. S. Manjunath, “Shape prior segmentation of multiple objects with graph cuts,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8. [8] J. Malcolm, Y. Rathi, and A. Tannenbaum, “Graph cut segmentation with nonlinear shape priors,” in Proceedings of the 14th IEEE International Conference on Image Processing, 2007, vol. 4, pp. 365–368. [9] V. Lempitsky, P. Kohli, C. Rother, and T. Sharp, “Image segmentation with a bounding box prior,” in IEEE International Conference on Computer Vision, 2009. [10] B. Peng and O. Veskler, “Parameter selection for graph cut based image segmentation.,” in British Machine Vision Conference, 2008. [11] Z. Song, N. Tustison, B. Avants, and J. Gee, “Adaptive graph cuts with tissue priors for brain mri segmentation,” in IEEE International Symposium on Biomedical Imaging, 2006.
Fig. 3. Results from star shape prior method [3]. Column (a) shows original images. Columns (b)-(d) show segmentation results from star shape prior method with 𝜆 = 0.1, 0.5 and 0.8. Column (e) shows segmentation results from our adaptive shape prior applied to star shape prior method.
[12] I. Bar-Yosef, A. Mokeichev, K. Kedem, I. Dinstein, and U. Ehrlich, “Adaptive shape prior for recognition and variational segmentation of degraded historical characters,” in Pattern Recognition, 2009, vol. 42, pp. 3348– 3354. [13] I. Levner, “Data driven object segmentation,” PhD Thesis, Department of Computing Science, University of Alberta, 2008.
6. REFERENCES [1] Y. Boykov and M.-P. Jolly, “Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images,” in IEEE Signal Processing Letters, 2001, vol. 1, pp. 105–112.
3032
[14] A. Levin, D. Lischinski, and Y. Weiss, “A closed-form solution to natural image matting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, pp. 228–242, 2008.
September 26-29, 2010, Hong Kong
ADAPTIVE SHAPE PRIOR IN GRAPH CUT SEGMENTATION Hui Wang and Hong Zhang Department of Computing Science University of Alberta, Edmonton, Alberta, CANADA {wanghui,zhang}@cs.ualberta.ca ABSTRACT In this paper, we propose a novel method to adaptively apply shape prior in graph cut segmentation. By incorporating shape priors in an adaptive way, we introduce a robust way to harness shape prior in graph cut segmentation. Since traditional graph cut approaches with shape prior may fail in cases where parameters for shape prior term are not set appropriately, incorporation of shape priors adaptively within this framework mitigates these problems. To address this issue, we propose to adaptively apply shape prior based on a shape probability map, defined to reflect the need of shape prior at each location of an image. We show that the proposed method can be easily applied to existing algorithms of graph cut segmentation with shape prior, such as level set based shape prior method, and star shape prior graph cut. We validate our method in various types of images corrupted by significant noise and intensity inhomogeneities. Convincing results are obtained. Index Terms— Adaptive, Shape Prior, Graph Cut, Interactive Image Segmentation 1. INTRODUCTION Image segmentation has always been an important and challenging task in computer vision. Since Boykov and Jolly [1] introduced the application of graph cut algorithm into image segmentation, graph cut based segmentation has become very popular in image segmentation in the last decade because it not only allows one to incorporate user interaction easily, but it is also efficient and globally optimal. More recently, in order to effectively handle noisy images or images with object occlusion, new graph cut based methods have been developed that are able to exploit the knowledge of shape prior. Freedman and Zhang [2] proposed to incorporate shape priors within the graph cut framework by matching the segmented curve with a level-set based shape template. Veksler [3] showed how to implement a star shape prior into graph cut segmentation to deal with a certain class of shapes called star shape. Das et al. presented a similar idea to incorporate shape prior to graph cut for shapes defined as compact shape [4]. Various related issues to the use of shape
978-1-4244-7993-1/10/$26.00 ©2010 IEEE
3029
prior have been investigated. Some focus on one or two partucular types of shapes [5, 6], some focus on incorporating multiple shape priors into one image [7], and some focus on shape representation and more general shape constraints [8, 9]. One of the outstanding issues of the graph cut framework is parameter selection. Parameters are usually fixed beforehand by the developer of the algorithm to achieve the best results for certain types of images [10]. Peng and Veksler [10] designed a parameter selection method by measuring segmentation quality based on different features of segmentation. Then they run the graph cut segmentation for different parameter values and choose the segmentation of the highest quality. However, their method only targets at issues on selecting the parameters between the data term and the boundary term, not on the newly added shape prior term. For images corrupted by significant noise and intensity inhomogeneities, the needs of shape priors at different pixels of an image are in general different. Therefore, setting a constant parameter on shape prior term in the energy function for all pixels may not be appropriate. Columns (b) to (d) in Figure 2 and 3 show that different parameter settings for shape prior can lead to very different segmentation results. To solve the above issue, we propose to impose shape constraint judiciously, by applying shape prior adaptively in graph cut segmentation. To determine the need of a pixel for shape prior, we derive a probability map based on image intensity information. The intuition behind this is that pixels with low probability values need a stronger shape term than pixels with high probability values. As will be seen, this probability can be easily calculated in a preprocessing step by making use of a classifier trained from labeled images. Following the similar idea, Song et al. [11] proposed a framework for automatic brain MRI tissue segmentation within an iterative scheme by incorporating the adaptive probabilistic atlas priors and intensity inhomogeneities correction for image segmentation into the graph cut energy function. However, the performance of their method relies on the accuracy of the probabilistic atlas and several other parameter settings. In another study, Bar-Yosef et al. [12] proposed a variational method for model based segmentation with adaptive shape prior. They propose to adaptively incorporate shape prior with the help of a shape confidence map. Their shape
ICIP 2010
confidence map is defined to reflect how correct each shape template candidate fits the current image and how much the template could help with current object segmentation. In contrast, our proposed method tackles adaptive shape prior from a differen point of view. Our shape probability map is defined to be based on the available image information, and it should reflect how much each pixel of the current image needs shape prior to help with segmentation, with the assumption that the provided shape prior is correct. The rest of the paper is organized as follows. In Section 2 we present the background for graph cut based segmentation with shape prior and the issue of parameter selection. In Section 3 we present our proposed adaptive shape prior idea and examples of applying it on two existing graph cut based methods using shape prior. In Section 4 we provide the experimental results to demonstrate the superiority of our approach. In Section 5 we state conclusions and future work. 2. BACKGROUND 2.1. Graph Cut Image Segmentation With Shape Prior
(1)
where 𝐸𝑖 is the original image energy, 𝐸𝑠 is an energy term based on the shape prior, and 𝜆 is a constant to balance between 𝐸𝑖 and 𝐸𝑠 . 𝐸𝑖 is usually defined as: ∑ ∑ 𝐸𝑖 = 𝜇 𝐷𝑝 (𝑓𝑝 ) + 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ) (2) 𝑝∈𝑃
In this section, we first present our proposed method for adaptively applying shape prior. Then, we apply our method to two existing graph cut based methods with shape priors. 3.1. Adaptive Shape Prior We propose to modify the energy function of graph cut segmentation with shape prior in the following way: 𝐸(𝑓 ) =
𝜇
∑
∑
𝑝∈𝑃
𝐷𝑝 (𝑓𝑝 ) +
∑ (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
1 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑒(𝛼𝑝 −𝛼𝑞 )2
𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+
𝐸𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )
(3) where 𝛼𝑝 represents the probability value from the probability map 𝛼 at pixel 𝑝, to be discussed in the next section. 𝐸𝑝 𝑞(𝑓𝑝 , 𝑓𝑞 ) represents a pairwise shape constraint term at pixel 𝑝 and 𝑞. 3.2. Shape Probability Map
A number of existing graph cut based segmentation methods incorporate shape prior by modifying the original graph cut energy function [1] by adding a shape prior energy term in the following way: 𝐸 = 𝐸𝑖 + 𝜆𝐸𝑠
3. PROPOSED METHOD
(𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
where 𝑁 is the set of neighboring pixels, 𝜇 is a balance factor between region term 𝐷𝑝 (𝑓𝑝 ) and boundary term 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ), and 𝑓𝑝 represents assigning label 𝑓𝑝 to pixel 𝑝. The particular forms for 𝐷𝑝 (𝑓𝑝 ) and 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ) are discussed in [1]. The shape energy term 𝐸𝑠 is defined in various ways depending on the type of shape prior.
𝛼 can be defined in various ways which reflect the need for shape prior at each pixel of an image. We refer to 𝛼 as shape prior probability map. This probability map should be generated from available image level information. In our experiment, 𝛼 is obtained from supervised learning [13]. The training is a very straight forward method. As an alternative, 𝛼 can also be defined as a matting map [14], as long as 𝛼 reflects the confidence of each pixel belonging to background or foreground. Hence, the above modified energy function (3) reflects that for pixel pair 𝑝 and 𝑞 which are neighboring pixels, if the probability value difference at these two locations is low, there should be a higher weight for shape term in the energy function. Two examples of the original images and their corresponding probability map 𝛼 are shown in Figure 1.
(a)
2.2. Parameter Selection for Shape Prior It is difficult to select a proper constant 𝜆 for Equation (1) to obtain proper segmentation result. Several shape prior graph cut papers have mentioned that users need to adjust 𝜆 depending on the types of images. Especially for images corrupted by significant noise and intensity inhomogeneities, the need of shape priors at different pixels of an image might vary significantly. In other words, setting a constant value 𝜆 for all pixels on the whole image is not appropriate. Again, we refer to Figure 2 and 3 for the sensitivity of the segmentation to the choice of 𝜆.
3030
(b)
Fig. 1. Examples of shape probablity map. For both (a) and (b), the left one shows the original image while the right one shows the probability map learned from supervised learning [13]. 3.3. Applying Adaptive Shape Prior 3.3.1. Level Set Based Shape Prior Method Freedman et al. [2] introduced to incorporate shape prior template in the form of level-set. In order to incorporate a shape
prior term, they modified the original graph cut energy function by adding a shape term in a similar way to Equation (1): ′
′
𝐸 = (1 − 𝜆 )𝐸𝑖 + 𝜆 𝐸𝑠
(4)
′
where 𝜆 is a constant. Using the idea of a level-set template, they defined shape energy term 𝐸𝑠 in Equation (4) in the following way: ) ( ∑ 𝑝+𝑞 ¯ (5) 𝐸𝑠 = 𝜙 2 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
where 𝜙¯ is a regular, unsigned distance function whose zero level set corresponds to the shape template curve 𝑐¯. To apply adaptive shape prior idea to Freedman et al.’s method, energy function (4) is modified to be: ′ ∑ 𝐸(𝑓 ) = (1 − 𝜆 ) 𝑝∈𝑃 𝐷𝑝 (𝑓𝑝 )+ ′
𝜆[ ∑
∑ (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+
1 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑒(𝛼𝑝 −𝛼𝑞 )2
𝜙¯
( 𝑝+𝑞 ) 2
(6) ]
where 𝛼𝑝 is from the probability map. 3.3.2. Star Shape Prior Another interesting method called star shape prior [3] applied more general shape prior to graph cut segmentation. However, it still shares the common limitations with parameter selection for the shape term. The star shape prior energy function is defined as: ∑ ∑ 𝐸(𝑓 ) = 𝜇 𝑝∈𝑃 𝐷𝑝 (𝑓𝑝 ) + (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+ ∑ (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞
𝑆𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )
(7) where 𝑆𝑝𝑞 is a shape energy term defined as star shape [3]. Energy function (7) appears to have only one parameter 𝜇 to tune, but there is actually a hidden parameter inside shape term 𝑆𝑝𝑞 , which is defined as edge value 𝛽. 𝛽 needs to be chosen appropriately by the user in order to obtain good segmentation result [3]. On the other hand, Peng and Veksler [10] have already designed a parameter selection method to find the right 𝜇 between region and boundary term, by measuring segmentation quality based on different features of segmentation. On top of that, our proposed method solves the remaining issue with shape prior term by adaptively applying the shape prior 𝑆𝑝𝑞 . The adaptive star shape prior makes the process of incorporating shape prior flexible. After applying adaptive shape prior, Equation (7) becomes ∑ ∑ 𝐸(𝑓 ) = 𝑝∈𝑃 𝐷𝑝 (𝑓𝑝 ) + (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑉𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 )+ ∑
1 (𝑝,𝑞)∈𝑁 :𝑓𝑝 ∕=𝑓𝑞 𝑒(𝛼𝑝 −𝛼𝑞 )2
𝑆𝑝𝑞 (𝑓𝑝 , 𝑓𝑞 ) (8)
3031
4. EXPERIMENTAL RESULTS To validate the proposed adaptive shape prior method, we have run experiments on two exisiting graph cut based methods with shape prior detailed in Section 3.3. We use a MATLAB wrapper to interface with the C++ maxflow code by Boykov and Kolmogorov [1]. We have performed our experiments on various types of images including medical images, oil sand images and other star shape images which satisfy the assumption of star shape prior method. To compare our method with the original methods in Section 3.3, we perform experiments based on the same user initializations required from the original methods. Shape prior templates are also provided in the same way for both our method and the original method from [2], so shape alignment step is not considered in the experiments. All 𝛼 probability maps are obtained from a supervised learning algorithm [13] for all images. Visual results are shown in Figure 2 and 3. The original images are shown in column (a), our results are shown in column (e). Results obtained by original methods are shown in columns (b) to (d). The results look convincing. It is clear that our method obtains much better segmentation results without the need to optimize with regard to 𝜆, while the two original graph cut mehtods both need to tune parameter 𝜆. In all cases, our results are better than or similar to the optimal results in the previous studies. 5. CONCLUSIONS AND FUTURE WORK We have proposed an adaptive method for incorporating shape prior into graph cut based segmentation framework to eliminate failure cases in previous approaches in which parameters had to be tuned to fit the image. Adaptive shape prior works by adding the shape term in the energy function based on a probability map. We have shown that the proposed method can be easily applied to various types of graph cut based image segmentation method with shape prior, such as Freedman and Zhang’s original graph cut method with shape prior, and star shape prior method with graph cut. We have validated our method in various types of images and obtained very positive results. Although we need a preprocessing step to obtain the probability map 𝛼, the preprocessing step is straight forward and does not add much additional cost to graph cut segmentation. One major direction of future work is to see if this adaptive shape prior idea could be extended to segmentation algorithms other than graph cut based algorithms. Another major direction for future research is to examine whether more types of shapes can be easily incorporated into the framework. While we have seen that the current method is relative robust compared to other exsiting shape prior methods, it will be interesting to see whether this robustness holds in the case of greater variations in shape and images, and, if not, how the
algorithm may be modified to account for these changes. In addition, applying shape prior locally instead of globally can be another direction to pursue.
[2] D. Freedman and T. Zhang, “Interactive graph cut based segmentation with shape priors,” in IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 755–762. [3] O. Veskler, “Star shape prior for graph-cut image segmentation,” in IEEE European Conference on Computer Vision, 2008, pp. 454–467. [4] P. Das, O. Veskler, V. Zavadsky, and Y. Boykov, “Semiautomatic segmentation with compact shape prior,” in Image and Vision Computing, 2009, vol. 27, pp. 206– 219. [5] G. Slabaugh and G. Unal, “Graph cuts segmentation using an elliptical shape prior,” in International Conference on Image Processing, 2005, vol. 2. [6] M. Kumar, P. Torr, and A. Zisserman, “Obj cut,” in IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 18–25.
Fig. 2. Results from level-set shape prior method [2]. Column (a) shows the original images. Columns (b)-(d) show segmentation results from level-set shape prior method with 𝜆 = 0.1, 0.5 and 0.9. Column (e) shows segmentation results from our adaptive shape prior applied to level-set shape prior method .
[7] N. Vu and B. S. Manjunath, “Shape prior segmentation of multiple objects with graph cuts,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8. [8] J. Malcolm, Y. Rathi, and A. Tannenbaum, “Graph cut segmentation with nonlinear shape priors,” in Proceedings of the 14th IEEE International Conference on Image Processing, 2007, vol. 4, pp. 365–368. [9] V. Lempitsky, P. Kohli, C. Rother, and T. Sharp, “Image segmentation with a bounding box prior,” in IEEE International Conference on Computer Vision, 2009. [10] B. Peng and O. Veskler, “Parameter selection for graph cut based image segmentation.,” in British Machine Vision Conference, 2008. [11] Z. Song, N. Tustison, B. Avants, and J. Gee, “Adaptive graph cuts with tissue priors for brain mri segmentation,” in IEEE International Symposium on Biomedical Imaging, 2006.
Fig. 3. Results from star shape prior method [3]. Column (a) shows original images. Columns (b)-(d) show segmentation results from star shape prior method with 𝜆 = 0.1, 0.5 and 0.8. Column (e) shows segmentation results from our adaptive shape prior applied to star shape prior method.
[12] I. Bar-Yosef, A. Mokeichev, K. Kedem, I. Dinstein, and U. Ehrlich, “Adaptive shape prior for recognition and variational segmentation of degraded historical characters,” in Pattern Recognition, 2009, vol. 42, pp. 3348– 3354. [13] I. Levner, “Data driven object segmentation,” PhD Thesis, Department of Computing Science, University of Alberta, 2008.
6. REFERENCES [1] Y. Boykov and M.-P. Jolly, “Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images,” in IEEE Signal Processing Letters, 2001, vol. 1, pp. 105–112.
3032
[14] A. Levin, D. Lischinski, and Y. Weiss, “A closed-form solution to natural image matting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, pp. 228–242, 2008.
