Automatic Road Sign Detection Method Based on Color Barycenters ...
Automatic Road Sign Detection Method Based on Color Barycenters Hexagon Model Qieshi Zhang and Sei-ichiro Kamata Graduate School of Information, Production and Systems, Waseda University, Japan [email protected] and [email protected] Abstract Road sign detection is one of the major concerned topics in the field of driving safety and intelligent vehicle. In this paper, a novel model based on Color Barycenters Hexagon (CBH) is proposed and used to detect road sign usefully. In CBH model, full color images are calculated the color barycenters and get the barycenters region, then automatic select the idea threshold curves to separate the Region of Interest (ROI) of barycenters aiming to detect the road sign. Because of the practically images have many noise, and the existing color space cannot separate the ROI ideally. The proposed CBH model can thresholding the principal color of ROI and have high robust. With suitably thresholding and operations, road sign on various scene images can be detected.
1. Introduction Road signs are used to regulate traffic, to warn drivers, and provide useful information to help make driving safe and convenient. The aim of road sign detection is to give the caution information to on-board intelligent computing equipments as a major knowledge of driving environment. As a driving assistance system, it is applied in avoiding inattentive sign departure by giving warnings to unaware drivers, offering a safe and comfort driving assistance. It will raise the driving safety by analyzing the road sign and make warning processing. With the technology development, various methods and techniques are proposed to detect road sign information. GPS system can be used to warn some signs, but it needs the wireless network and the information must be update regularity. Another alternative [1] is by using image sequences captured by digital camera for further analysis. Data obtained with digital eyes are friendly to human sight. However, computers deal with such images in a completely
978-1-4244-2175-6/08/$25.00 ©2008 IEEE
different way than human beings, which is the issue of computer vision. In this study, we propose a novel method for road sign detection, which finds the goal region by analysis the color of road scene images in Color Barycenters Hexagon (CBH) model. Analyzing and thresholding the image based on CBH model and compared with the thresholding result in other color space or models [2]-[4], proposed method have obvious advantages.
2. Image thresholding based on CBH model This section will introduce how to threshold the color image by CBH model from analysis created color triangle and its color barycenters region distribution. Here describes how to transform RGB components of RGB 3-D color space to 2-D polar coordinate system, and use multi-threshold to segment the barycenters region of color triangle. By analyzing and processing, it can cluster the color of image to 2~7 colors by 2~7 thresholds for require and the effect is better than traditional thresholding methods.
2.1. Color triangle construction We often use several color spaces like RGB, YCbCr, HSV, HSI etc in image processing. These color spaces use three components to reflect color information, e.g. RGB consist of R, G and B components. This paper tries to transform the 3-D color space to 2-D coordinate system by color triangle (Figure 1). To create the color triangle, a standard 2-D Cartesian coordinate system is used to describe R, G, B values and then transform it to polar coordinate system as (1). 0 1 0 0 0 0
0 0
0 0
√3/2 0 1/2 0 0 0
√3/2 1/2
0 0 0 0 0 0
0 1 0 2 0 2
90° 0 210° 0 330° 0
0 1
0 0
0 0
90° 0
0 0
0 1
0 0
210° 0
0 0
0 0
0 1
330° 0
(1)
2.3. Thresholding based on CBH model
Here R, G, B is the original color value, ( , ), ( , ), ( , ), is the coordinate of R, G, B in Cartesian coordinate system and ( , ), ( , ), ( , ), is the coordinate of R, G, B in polar coordinate system. All the R, G, B components are settled as Figure 1. The range of them is 0, 255 and angles between both of them are equal to 120°. 90°, 210°, 330°,
: : :
information. So those barycenters will in a circular region (L region). Other six color regions reflect the color character of R, G, and B combination [5]. For example in R region, the R value is always larger than B and G values.
0, 255 0, 255 0, 255
(2)
Finally, connect the three apexes to create the color triangle as Figure 1. For different R, G, B values, the shape of triangle is changeable.
2.2. CBH region distribution Because the direction of R, G, B vectors are fixed and the value range is 0, 255 , different R, G, B value will result in the different shape of color triangle and has different barycenters. All possible barycenters construct a hexagon region as Figure 2. For example, the triangle of color 150, 25, 15 , 25, 91, 143 and 218, 200, 143 as Figure 3(a)(b)(c) show and the “ ” reflect the barycenter of correspondence color triangle. In this paper, the hexagon is divided to seven subregions: Red, Green, Blue, Cyan, Magenta, Yellow and achromatic region and they are denoted as R, G, B, C, M, Y and L separately in Figure 2. Observe the relationship between color and its corresponding barycenter position in color triangle, we find that if the R, G, B values are close, it reflects the achromatic
The existing methods cannot classify the color whose barycenter is in L region correctly. But our method regards the L region as one class. So the noises caused by L region will be removed perfectly. Here is the threshold, is the angle, the threshold curve function is: , 0°
360°
(3)
is a constant and its range is [0, 85]. The other six regions are determined by the following formulas: : φ , : φ , : φ , (4) : φ , : φ , : φ , In formula (3) and (4), we set 5, 120° , 180°, 240°, 300° 60°, 360° as the initial value of the thresholdings. and
(a) Original image
(b) Result of (a)
(d)Barycenters histogram
R Y
M
G
B
(c)CBH region distribution
(e)Smoothed (d)
C
Figure 1. Color triangle R
(a)
R
R B
G
Figure 2. CBH model
B
G (b)
B
G (c)
Figure 3. The example of color triangle shape with different color
60°
120°
180°
240°
300°
360°
(f) Thresholds selection in barycenters histogram
Figure 3. Thresholds selection
Based on the separation of the CBH modal, all color in one image can be divided into seven classes. But the results acquired by this way are often not suitable for our segmentation purpose. So this paper proposed an automatic thresholds selection method. Observe the distribution of color barycenters in hexagon region as Figure 4(c), we can see that the intensity of red and blue are stronger than other color, only when , the barycenters are similar and close to each other in the L region. The intensity of color is proportional to the distance away with origin. Through analyzing lots of road signs, we can see that the colors usually locate in R, B and Y region. But it is difficult to find the segmentation borderline of the colors in Figure 4(c). So we transform the Polar coordinate system to Cartesian coordinate system as Figure 4(d) to show the distribution of the barycenters. Figure 4(e) is the barycenters histogram after smoothing. Figure 4(f) shows the magnified picture of (e). And the horizontal axis is ( 0°, 360° ), vertical axis is ( 0, 85 ) and other six vertical color broken-line are color threshold curves , , , and ). ( , Every curve is initialized with the ideal value, same as formula (4). And they can be moved from left 20° to right 20° for best searching. So we can find six valleys respectively in six ranges corresponding to the , the threshold for separating six borderlines. As for the achromatic color, it has a range from 3 to 20 in is equal to the experiment. For different image, average value of six valley values in six regions. In Figure 3, (a) is original image and (b) is the result by our method.
3. Road sign detection based on CBH model In practice, the road scene image sequence is captured from camera fixed onto the front windshield of a vehicle. The RGB color space is used to describe the colors and calculate the color barycenters. After this, use the CBH model to distill the goal region. For reduce the computation time, we use vertical line which is to be left one-third of the width from the right and horizontal line which is to be bottom one-third of
(a) Original image
Vertical line
Horizontal line
(b) Thresholding result
Figure 5. Sample image thresholding
height from the bottom to divide the image into four regions and shown as Figure 5(b). Here we let 426th line as vertical line and 160th row as horizontal lines to divide the image (the image size is 640×480). Because in Japan the car is right drive, only the left-up scene includes the goal regions, so the other regions are ignored to improve computation efficiency. However there still exist some noises. So use open/close operation and connected component labeling (CCL) to denoise and get the final goal region.
3.1. Thresholding by CBH model Based on the proposed CBH model and the method described in section 2.3, the best thresholds can be selected. And because all colors of road signs locates in R, B or Y regions, so we can detect the targets by our thresholds. The result is shown as Figure 3(b).
3.2. Filtering From Figure 4(b), it can be found that there are some noises in our thresholding result. So we use open/close operation to denoise firstly. Then CCL is used to find the connected regions. Because the shapes of road signs are round, triangular etc, we can remove the noises which cannot satisfy the shape characteristic.
3.3. ROI intensifying According to the stages mentioned above, the ideal target regions can be acquired. To show the detected road signs more clearly, the standard color is used to replace the corresponding region color of original image. For example, the color of red ring of road sign in original image is 150, 25, 15 , replace it with standard color 255, 0, 0 .
4. Result and analysis The proposed CBH model can be used to get thresholding result and it is helpful to detect the road signs. To use the CBH model, it is necessary to calculate the color barycenters of the color triangle and select the suitable thresholds for thresholding. After filtering, the ideal result can be acquired. Figure 6 show the segmentation results by the proposed CBH modal and other four color spaces. If the background is simple, the good results can be got using other color spaces; but in the complex situation, those spaces cannot get acceptable results. Because of
existing color space transformation methods only consider one channel values of color space; it cannot reflect the original color characteristic sometimes. For example, we want to distill red region in Figure 6(a), but (c)(e)(f) not only detect the red region of road sign and building, but also detect the black region of car and other objects. According to those characteristics of image, those three color spaces are not suitable. Figure 6(d) use CMKY color space to reflect the goal region, magenta channel is the best, but this channel cannot distinguish red from blue region, so it is not good choice for detecting the targets. Based on the color characteristics of all road signs, we detect the red at first, if nothing is detected, then detect the blue and yellow. From Figure 6, we can see that it is much better for red region detection using our method than others. Figure 7 show the results of different conditions.
5. Conclusion and future work In this study, a novel color model for road sign detection is proposed. The advantages of the proposed model are: 1) it can distinguish the image color into seven classes; 2) it is flexible to use according to the different requirement because we can choose the different number of threshold to describe image and get ideal results; 3) thresholding with threshold curves in CBH model can avoid the influences of bright and dark regions, which usually affect negatively the thresholding result; 4) The proposed method is invariant to size, distance or surroundings. We compare our method with CMYK, RGB, HSV and Lab color spaces and our method got the better results. The future work is to improve the computation efficiency and recognize the contents of detected road sign correctly and show corresponding warning message to driver.
[5] Q. Zhang, S. Kamata and J. Zhang. Color barycenter hexagon model based road sign detection. 2008 International MultiConference of Engineers and Computer Scientists, 1:667-670, March, 2008.
(a) Original image
(b) Proposed method
(c) H channel of HSV color space
(d) M channel of CMYK color space
(e) G channel of RGB color space
(f) L channel of Lab color space
Figure 6. Compare with the thresholding result in different color space
(a) Original image
(b) Result of (a)
(c) Original image
(d) Result of (c)
(e) Original image
(f) Result of (e)
References [1] W.S. Wijesoma, K. R. S. Kodagoda,. A. P. Balasurya, and E. K. Teoh. Laser and camera for road edge and midline detection, Robot Motion and Control, 2001 Proceedings of the Second International Workshop, 269274, October, 2001. [2] T.Y. Sun, S.J. Tsai and V. Chan. HIS color model based lane-marking detection. Proceedings of 2006 IEEE Intelligent Transportation Systems Conference, 11681172, Sepember, 2006. [3] H. Stokman and T. Gevers. Selection and fusion of color models for feature detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1:560-565, June. 2005. [4] A. Broggi, P. Cerri, P. Medici, P. Porta and G. Ghisio. Real time road signs recognition. 2007 IEEE Intelligent Vehicles Symposium, 981-986, June 2007.
Figure 7. Experiment result
1. Introduction Road signs are used to regulate traffic, to warn drivers, and provide useful information to help make driving safe and convenient. The aim of road sign detection is to give the caution information to on-board intelligent computing equipments as a major knowledge of driving environment. As a driving assistance system, it is applied in avoiding inattentive sign departure by giving warnings to unaware drivers, offering a safe and comfort driving assistance. It will raise the driving safety by analyzing the road sign and make warning processing. With the technology development, various methods and techniques are proposed to detect road sign information. GPS system can be used to warn some signs, but it needs the wireless network and the information must be update regularity. Another alternative [1] is by using image sequences captured by digital camera for further analysis. Data obtained with digital eyes are friendly to human sight. However, computers deal with such images in a completely
978-1-4244-2175-6/08/$25.00 ©2008 IEEE
different way than human beings, which is the issue of computer vision. In this study, we propose a novel method for road sign detection, which finds the goal region by analysis the color of road scene images in Color Barycenters Hexagon (CBH) model. Analyzing and thresholding the image based on CBH model and compared with the thresholding result in other color space or models [2]-[4], proposed method have obvious advantages.
2. Image thresholding based on CBH model This section will introduce how to threshold the color image by CBH model from analysis created color triangle and its color barycenters region distribution. Here describes how to transform RGB components of RGB 3-D color space to 2-D polar coordinate system, and use multi-threshold to segment the barycenters region of color triangle. By analyzing and processing, it can cluster the color of image to 2~7 colors by 2~7 thresholds for require and the effect is better than traditional thresholding methods.
2.1. Color triangle construction We often use several color spaces like RGB, YCbCr, HSV, HSI etc in image processing. These color spaces use three components to reflect color information, e.g. RGB consist of R, G and B components. This paper tries to transform the 3-D color space to 2-D coordinate system by color triangle (Figure 1). To create the color triangle, a standard 2-D Cartesian coordinate system is used to describe R, G, B values and then transform it to polar coordinate system as (1). 0 1 0 0 0 0
0 0
0 0
√3/2 0 1/2 0 0 0
√3/2 1/2
0 0 0 0 0 0
0 1 0 2 0 2
90° 0 210° 0 330° 0
0 1
0 0
0 0
90° 0
0 0
0 1
0 0
210° 0
0 0
0 0
0 1
330° 0
(1)
2.3. Thresholding based on CBH model
Here R, G, B is the original color value, ( , ), ( , ), ( , ), is the coordinate of R, G, B in Cartesian coordinate system and ( , ), ( , ), ( , ), is the coordinate of R, G, B in polar coordinate system. All the R, G, B components are settled as Figure 1. The range of them is 0, 255 and angles between both of them are equal to 120°. 90°, 210°, 330°,
: : :
information. So those barycenters will in a circular region (L region). Other six color regions reflect the color character of R, G, and B combination [5]. For example in R region, the R value is always larger than B and G values.
0, 255 0, 255 0, 255
(2)
Finally, connect the three apexes to create the color triangle as Figure 1. For different R, G, B values, the shape of triangle is changeable.
2.2. CBH region distribution Because the direction of R, G, B vectors are fixed and the value range is 0, 255 , different R, G, B value will result in the different shape of color triangle and has different barycenters. All possible barycenters construct a hexagon region as Figure 2. For example, the triangle of color 150, 25, 15 , 25, 91, 143 and 218, 200, 143 as Figure 3(a)(b)(c) show and the “ ” reflect the barycenter of correspondence color triangle. In this paper, the hexagon is divided to seven subregions: Red, Green, Blue, Cyan, Magenta, Yellow and achromatic region and they are denoted as R, G, B, C, M, Y and L separately in Figure 2. Observe the relationship between color and its corresponding barycenter position in color triangle, we find that if the R, G, B values are close, it reflects the achromatic
The existing methods cannot classify the color whose barycenter is in L region correctly. But our method regards the L region as one class. So the noises caused by L region will be removed perfectly. Here is the threshold, is the angle, the threshold curve function is: , 0°
360°
(3)
is a constant and its range is [0, 85]. The other six regions are determined by the following formulas: : φ , : φ , : φ , (4) : φ , : φ , : φ , In formula (3) and (4), we set 5, 120° , 180°, 240°, 300° 60°, 360° as the initial value of the thresholdings. and
(a) Original image
(b) Result of (a)
(d)Barycenters histogram
R Y
M
G
B
(c)CBH region distribution
(e)Smoothed (d)
C
Figure 1. Color triangle R
(a)
R
R B
G
Figure 2. CBH model
B
G (b)
B
G (c)
Figure 3. The example of color triangle shape with different color
60°
120°
180°
240°
300°
360°
(f) Thresholds selection in barycenters histogram
Figure 3. Thresholds selection
Based on the separation of the CBH modal, all color in one image can be divided into seven classes. But the results acquired by this way are often not suitable for our segmentation purpose. So this paper proposed an automatic thresholds selection method. Observe the distribution of color barycenters in hexagon region as Figure 4(c), we can see that the intensity of red and blue are stronger than other color, only when , the barycenters are similar and close to each other in the L region. The intensity of color is proportional to the distance away with origin. Through analyzing lots of road signs, we can see that the colors usually locate in R, B and Y region. But it is difficult to find the segmentation borderline of the colors in Figure 4(c). So we transform the Polar coordinate system to Cartesian coordinate system as Figure 4(d) to show the distribution of the barycenters. Figure 4(e) is the barycenters histogram after smoothing. Figure 4(f) shows the magnified picture of (e). And the horizontal axis is ( 0°, 360° ), vertical axis is ( 0, 85 ) and other six vertical color broken-line are color threshold curves , , , and ). ( , Every curve is initialized with the ideal value, same as formula (4). And they can be moved from left 20° to right 20° for best searching. So we can find six valleys respectively in six ranges corresponding to the , the threshold for separating six borderlines. As for the achromatic color, it has a range from 3 to 20 in is equal to the experiment. For different image, average value of six valley values in six regions. In Figure 3, (a) is original image and (b) is the result by our method.
3. Road sign detection based on CBH model In practice, the road scene image sequence is captured from camera fixed onto the front windshield of a vehicle. The RGB color space is used to describe the colors and calculate the color barycenters. After this, use the CBH model to distill the goal region. For reduce the computation time, we use vertical line which is to be left one-third of the width from the right and horizontal line which is to be bottom one-third of
(a) Original image
Vertical line
Horizontal line
(b) Thresholding result
Figure 5. Sample image thresholding
height from the bottom to divide the image into four regions and shown as Figure 5(b). Here we let 426th line as vertical line and 160th row as horizontal lines to divide the image (the image size is 640×480). Because in Japan the car is right drive, only the left-up scene includes the goal regions, so the other regions are ignored to improve computation efficiency. However there still exist some noises. So use open/close operation and connected component labeling (CCL) to denoise and get the final goal region.
3.1. Thresholding by CBH model Based on the proposed CBH model and the method described in section 2.3, the best thresholds can be selected. And because all colors of road signs locates in R, B or Y regions, so we can detect the targets by our thresholds. The result is shown as Figure 3(b).
3.2. Filtering From Figure 4(b), it can be found that there are some noises in our thresholding result. So we use open/close operation to denoise firstly. Then CCL is used to find the connected regions. Because the shapes of road signs are round, triangular etc, we can remove the noises which cannot satisfy the shape characteristic.
3.3. ROI intensifying According to the stages mentioned above, the ideal target regions can be acquired. To show the detected road signs more clearly, the standard color is used to replace the corresponding region color of original image. For example, the color of red ring of road sign in original image is 150, 25, 15 , replace it with standard color 255, 0, 0 .
4. Result and analysis The proposed CBH model can be used to get thresholding result and it is helpful to detect the road signs. To use the CBH model, it is necessary to calculate the color barycenters of the color triangle and select the suitable thresholds for thresholding. After filtering, the ideal result can be acquired. Figure 6 show the segmentation results by the proposed CBH modal and other four color spaces. If the background is simple, the good results can be got using other color spaces; but in the complex situation, those spaces cannot get acceptable results. Because of
existing color space transformation methods only consider one channel values of color space; it cannot reflect the original color characteristic sometimes. For example, we want to distill red region in Figure 6(a), but (c)(e)(f) not only detect the red region of road sign and building, but also detect the black region of car and other objects. According to those characteristics of image, those three color spaces are not suitable. Figure 6(d) use CMKY color space to reflect the goal region, magenta channel is the best, but this channel cannot distinguish red from blue region, so it is not good choice for detecting the targets. Based on the color characteristics of all road signs, we detect the red at first, if nothing is detected, then detect the blue and yellow. From Figure 6, we can see that it is much better for red region detection using our method than others. Figure 7 show the results of different conditions.
5. Conclusion and future work In this study, a novel color model for road sign detection is proposed. The advantages of the proposed model are: 1) it can distinguish the image color into seven classes; 2) it is flexible to use according to the different requirement because we can choose the different number of threshold to describe image and get ideal results; 3) thresholding with threshold curves in CBH model can avoid the influences of bright and dark regions, which usually affect negatively the thresholding result; 4) The proposed method is invariant to size, distance or surroundings. We compare our method with CMYK, RGB, HSV and Lab color spaces and our method got the better results. The future work is to improve the computation efficiency and recognize the contents of detected road sign correctly and show corresponding warning message to driver.
[5] Q. Zhang, S. Kamata and J. Zhang. Color barycenter hexagon model based road sign detection. 2008 International MultiConference of Engineers and Computer Scientists, 1:667-670, March, 2008.
(a) Original image
(b) Proposed method
(c) H channel of HSV color space
(d) M channel of CMYK color space
(e) G channel of RGB color space
(f) L channel of Lab color space
Figure 6. Compare with the thresholding result in different color space
(a) Original image
(b) Result of (a)
(c) Original image
(d) Result of (c)
(e) Original image
(f) Result of (e)
References [1] W.S. Wijesoma, K. R. S. Kodagoda,. A. P. Balasurya, and E. K. Teoh. Laser and camera for road edge and midline detection, Robot Motion and Control, 2001 Proceedings of the Second International Workshop, 269274, October, 2001. [2] T.Y. Sun, S.J. Tsai and V. Chan. HIS color model based lane-marking detection. Proceedings of 2006 IEEE Intelligent Transportation Systems Conference, 11681172, Sepember, 2006. [3] H. Stokman and T. Gevers. Selection and fusion of color models for feature detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1:560-565, June. 2005. [4] A. Broggi, P. Cerri, P. Medici, P. Porta and G. Ghisio. Real time road signs recognition. 2007 IEEE Intelligent Vehicles Symposium, 981-986, June 2007.
Figure 7. Experiment result
