A New Obstacle Avoidance Method Based on Biped Robot
2362
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
A New Obstacle Avoidance Method Based on Biped Robot Zetao Jiang Nanchang Hangkong University/School of Information Engineering, Nanchang, China Email: [email protected]
Yanru Cui and Qiang Wang Nanchang Hangkong University/School of Information Engineering, Nanchang, China Email: [email protected], [email protected]
Abstract—Currently, in order to avoid obstacle, most robots will choose a new path. This paper proposed a new obstacle avoidance method for biped robots. When lower obstacle exists in the road, we will calculate the obstacle’s three-dimensional information instead of using the traditional method of re-select the path, so that the biped robot will avoid obstacles successfully by crossing the obstacle. In 3-d reconstruction, the stereo matching is complex, and has large time consumption. Therefore, the method in this paper has a combination of color segmentation and height detection, extracts obstacle’s region accurately to reduce detection range of matching, then improves the real-time of reconstruction. Experiments show that the method proposed in this paper is feasible. Index Terms—Biped robot, Color segmentation, Height detection, Stereo matching, Obstacle reconstruction
I.
INTRODUCTION
In recent years, Robotics is a comprehensive new discipline, which focused on engineering technology such as machinery, computer, control, electronics and the latest research in artificial intelligence[1]. In robotics, machine vision which likes human eyes is one of the key technologies. Mobile robot obtains 3-d information about external environment, mainly through the following two ways: laser rangefinder and stereo vision technology[2]. Laser ranging obtains the 3-d distance images in front of robot through laser-scanning sensors. Therefore, we can get obstacle’s information such as shape, height, width, etc[3-4]. However, this method requires relatively complex equipment, and has a bad ability to adapt to the environment, so the practicality is not strong. Stereo vision technology plays an important role in robot navigation applications for its low energy consumption, high reliability, etc[5]. In stereo vision, stereo matching is the most important, its main problem is to find matching points between images. According to different matching primitives, stereo matching can be divided into three types, region matching, phrase Copyright credit, project number: Nature Science Foundation of China (60973096) and Science Foundation of aviation in China(2010ZC56007). Corresponding author: [email protected]
© 2012 ACADEMY PUBLISHER doi:10.4304/jcp.7.10.2362-2367
matching and feature matching[6]. The essence of region matching algorithm is to use related degree of gray information between local windows, it can achieve high accuracy in places where have smooth diversifications and rich details. But it is difficult to choose the window size of matching, and has large compute-intensive and slow calculation speed. Phase matching reflects the signal’s structure information, and has a good inhibitory effect on the high-frequency noise of images. But it exists major problems， such as phase singularities and phase winding. SIFT feature matching does not directly dependent on gray, and has a strong anti-interference, a small compute-intensive and fast calculation speed[7]. In this paper, we used this algorithm, although we can get only sparse disparity field, we can restore its dense disparity map by the existing dense matching method. Currently, the robot can be divided into two categories: wheeled robots and biped robots. When the wheeled robot is moving and there is an obstacle in front of it, we only need to calculate some simple information, such as the distance between obstacle and the wheeled robot, the obstacle width, so that the robot has enough time for a new path planning to avoid a collision with the obstacle. In contrast, biped robot has a high degree of imitation of human characteristics, and can cross over the lower obstacle just like people instead of choosing a new path. We need to consider the biped robot’s step height, step length and other issues, so we can’t achieve its purpose only using traditional obstacle avoidance algorithm. To solve these problems, this paper proposed a new obstacle avoidance method of crossing over the obstacle for biped robot, calculated the obstacle’s real 3-d information, so that we can determine the robot’s step height, step length and other issues in order to avoid a collision with the obstacle. II. GET A REAL 3-D OBSTACLE’S IDEAS AND STEPS Calculating the obstacle’s real 3-d information is to 3-d reconstruct the obstacle. It is that extract the 2-d information of obstacle images as rich as possible, then restore the 3-d information as much as possible, so that achieve the 3-d reconstruction of obstacle. Steps in this
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
2363
paper are as follows. Firstly, in order to extract the obstacle region quickly, reduce the computation of matching, and achieve a better real-time, we will search for a new method about color image segmentation in the analysis of existing methods[8-10]. Secondly, we will use existing methods based on stereo image pair for dense matching[11], restore a dense obstacle region, and calculate the 3-d coordinates of corresponding points by the use of camera matrix, then find the 3-d proportion of obstacle model. Finally, we will detect the obstacle’s real height, and get its real 3-d information according to the 3-d proportion of obstacle model.
strengthen the image segmentation results; Typically, I component is not usually judged by the standard. Therefore, HIS color model will be used in this paper. HIS and RGB color models’ transformation relationship is as follows.
III. EXTRACT REGION OF OBSTACLE BY COLOR SEGMENTATION
B. Color Segmentation Algorithm and Steps The color of ordinary road is very similar. With the light’s change the color of road scene changes, but under the premise that the image is not saturated, the color difference between the road surface and obstacles is always there. Therefore, this paper uses HIS color model which has a smaller sensitivity to light for image segmentation. The purpose is to separate the road and obstacle’s region. Specific steps are as follows: Firstly, choose a M × N obstacle-free area in front of the robot, and calculate the road’s average reference hue HA and saturation SA.
In binocular vision, stereo matching is the most important task. It needs a lot of points to match, and has a huge computation, time consuming, and low real-time. To solve this problem, we should segment the collected images based on color, remove the complex background, and extract the obstacle region. Therefore, in order to reduce the computing time and improve the real-time, we can extract and match feature points only in the target area. A. The Selection of Color Model RGB, YUV, HIS are common color models[12-13]. RGB is the most common color space. After the digital signal processing, the robot’s cameras get a value that refers to each pixel value in RGB space. But the RGB color space is uneven, the visual difference between two colors and two points’ Euclidean distance in RGB color space are not linearly proportional. In different conditions, such as different types of light source and reflective properties of objects, the distribution of the same color’s RGB value is dispersed, and the three components occupied almost the entire color space are interrelated, so it is difficult to find a exact color space threshold. YUV color model, Y component indicates the light intensity, U and V components indicate the color tone[14]. Compared to the RGB color model, the YUV color mode which is independent of light intensity is more suitable for the habit of human eyes, thereby reducing the light and reflection’s impact on the color recognition. Another advantage of YUV color model is that it is a linear transformation relationship with RGB color model. When the robot is working online, it can convert the YUV color model to the RGB color model quickly. So some developers will use this color model as the basis for target recognition. HIS color model is based on visual principles. H component represents the hue, I component represents the brightness, and S component represents the saturation. These three color properties are unrelated, especially the property of H, which has low sensitivity to changes in external light, can reflect the species of colors accurately. For the same color properties of objects, H which has a stable and narrow scope of transformation can be used as a prerequisite for color segmentation; S can be used as auxiliary to determine the conditions in order to © 2012 ACADEMY PUBLISHER
⎧⎪ 1 / 2[( R − G ) + ( R − B )] ⎫⎪ H = cos −1 ⎨ ⎬ ⎪⎩ ( R − G ) 2 + ( R − B )( G − B ) ⎪⎭
S =1−
(R + G + B ) 3
I =
H
S
min( R , G , B ) I
A
A
=
=
M
N
i =1
j =1
∑ ∑
(1)
H (i, j )
M × N M
N
i =1
j =1
∑ ∑
S (i, j )
M × N
(2)
Secondly, select each point (i,j) on the image, and calculate the 4 × 4 neighborhood W of average hue and saturation values.
h (i, j ) =
s (i, j ) =
∑
f (u , v )
( u , v )∈ W
∑
N g (u , v )
( u , v )∈ W
N
(3)
N denotes the number of pixels of the neighborhood W, f(u,v) and g(u,v) represent the hue and saturation of the pixel. Thirdly, determine any pixel (i,j) is obstacle or not, the formula is as follows:
F(i, j) = h(i, j) − HA G(i, j ) = s(i, j) − S A
(4)
When F(i,j) is less than the set threshold HT and G(i,j) is less than the set threshold ST, the pixel (i,j) represents the road, otherwise the obstacle. After such treatment, the obstacle region can be split out, and the whole background is set to black, leaving only the obstacle
2364
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
region. This greatly reduces the workload of the feature points’ extraction and matching in 3-d reconstruction and saves a lot of time.
P in images is dP, and the pixel parallax of the point F in images is dF, the relationship can be expressed as follows.
Δd = dP − dF =
IV. HEIGHT DETECTION AND RECONSTRUCTION OF OBSTACLE A. Preparation Works Binocular vision refers to that, at the same time, we shoot two images for the same object from different angles. In the world coordinate system, a space point on the target object can be mapped to these two images. Feature point matching is to search these two images, and identify each corresponding pixel points, the pixel points are called matching points. As the external environment, lighting and camera distortion and other negative factors will affect the images, it is difficult to find the matching points exactly. Therefore, feature point matching is an important part of binocular vision studies. Scale Invariant Feature Transform (SIFT) algorithm has more widely used and strong match ability in the strength texture feature areas, and remains invariance for image scaling, rotation, and affine transform[15]. So, in this paper, we use SIFT algorithm to extract and match the features. The preparation works mainly include four steps. Firstly, extract features from a pair of images. Secondly, match the features by the traditional matching methods. Thirdly, Random Sample Consensus (RANSAC) algorithm is considered one of the most efficient robust estimation algorithm. Therefore, RANSAC will be used to eliminate the false matches further. Finally, compute the parallax that spatial points in two images. B. The True Height Detection of Obstacle The parallel stereo vision system based on the biped robots is similar to the whole process that the human binocular obtains the object’s depth information. For any object point P in 3-d scene, we get the parallax d of this point first, and calculate the distance OD from this point in optical axis to the camera optical center O. The formula is as follows: (5) Bf
OD =
d
f is the camera focal length, B is the distance between two camera optical centers, d is the parallax of the measured point mapping on the left and right images.
Figure 1. Obstacle height detection.
As shown in Figure1, we can calculate the depth information OD of the obstacle’s top point P and OE of the point F on the ground. The pixel parallax of the point © 2012 ACADEMY PUBLISHER
Bf Bf Bf ⎡ DE⎤ − = OD OE OD⎢⎣ OE⎥⎦
(6)
Seen by the 3-d geometry
DE OE
=
PH H
(7)
We can know that
Bf Δd = OD
⎡ PH ⎤ ⎢H ⎥ ⎣ ⎦
(8)
As Δ d and OD are known, we can deduce the obstacle’s height PH. C.
The Obstacle Model’s 3-d Scale and Recons- truction In order to obtain the 3-d coordinates of the measured object’s feature points, binocular vision sensor should at least obtain two images containing the object’s feature points from different locations. Figure2 shows that two cross cameras observe the same object from different angles. In fact getting two images does not necessarily require two cameras. One camera through a certain movement can also obtain two images by observing a stationary object in different locations.
Figure 2. Dual camera observation of space point.
Figure2 shows, for any space point P, its image point in camera C1 is p1. Because of that the image point in camera C1 of any point on the connection line O1P is p1, we could not get the 3-d position of P from p1. Therefore, from the location of image point p1, we only know that the space point P is located on the connection line O1p1. While observing the space point P by camera C2 at the same time, we can get the depth information of point P because of that the space point P is not only on the connection line O1p1, but also on the connection line O2p2. So the point P is the intersection of two lines O1p1 and O2p2, p1 and p2 is matching points with each other. We must match the feature points on the images first, and find the matching points in order to determine the 3-d location of their corresponding space point. In this paper, the image retains only the obstacle area through the color segmentation. We can restore the dense matching points concentrated in the obstacle area through the existing algorithm based on image dense stereo matching. The space 3-d coordinates of points on the obstacle can be determined through each pair of matching points. The cameras C1 and C2 have been calibrated, and their
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
2365
projection matrices are M1 and M2, then there are the following formulas.
⎡u1 ⎤ ⎡ m11 ⎢ Z c1 ⎢⎢ v1 ⎥⎥ = ⎢ m121 1 ⎢⎣ 1 ⎥⎦ ⎢⎣ m31
1 m12 m122 1 m32
1 m13 m123 1 m33
⎡u 2 ⎤ ⎡ m11 ⎢ 2 Z c 2 ⎢⎢ v2 ⎥⎥ = ⎢ m21 2 ⎢⎣ 1 ⎥⎦ ⎢⎣ m31
m122 2 m22
m132 2 m23
2 m32
2 m33
1
2
⎡X ⎤ 1 ⎤⎢ ⎥ m14 ⎥ Y m124 ⎥ ⎢ ⎥ ⎢ ⎥ 1 ⎥ Z m34 ⎦⎢ 1 ⎥ ⎣ ⎦ ⎡X ⎤ m142 ⎤ ⎢ ⎥ 2 ⎥⎢Y ⎥ m24 ⎥⎢ ⎥ 2 ⎥ Z m34 ⎦⎢ 1 ⎥ ⎣ ⎦
(9)
ground scene. Fig. 4 shows the results segmented by general gray segmentation algorithm. Fig. 5 shows the result segmented by the color segmentation algorithm proposed in this paper. The ordinary gray segmentation algorithm can not satisfactory our needs. However, through the color segmentation algorithm proposed in this paper, we can extract the obstacle region successfully. Compared to the ordinary gray segmentation algorithm, the segmentation results are much better.
(10)
(u1,v1,1) and (u2,v2,1) are the homogeneous coordinates of p1 and p2 in their respective images. (X,Y,Z,1) is the homogeneous coordinates of the point P in world coordinate system. m ijk (k=1,2; i=1,2,3;
Figure 3. Original color image pair.
j=1,2,3,4) is the element located row i column j in the projection matrix Mk. In the formula, Zc1 and Zc2 can be eliminated, then get four linear equations about X,Y,Z: 1 1 1 1 1 1 (u1m31 − m11 )X + (u1m32 − m12 )Y + (u1m33 − m13 )Z 1 1 = m14 −u1m34 1 1 1 (v1m31 − m121) X + (v1m32 − m122)Y + (v1m33 − m123)Z
(11)
1 = m124 − v1m34
Figure 4. Ordinary gray segmentation results.
2 2 2 (u2m31 − m112 )X + (u2m32 − m122 )Y + (u2m33 − m132 )Z 2 = m142 − u2m34
2 2 2 2 2 2 (v2m31 −m21 )X +(v2m32 −m22 )Y +(v2m33 −m23 )Z
(12)
2 2 = m24 −v2m34
We can see from the analytic geometry, the geometric significance of formula 12 and 13 is that the line over O1p1(or O2p2). Due to the space point P is the intersection of line O1p1 and line O2p2, it must satisfy the formula 12 and 13 at the same time. Therefore, we can combine formula 12 with formula 13, then calculate the coordinates (X,Y,Z) of the point P. After finding the 3-d coordinates of all points in the obstacle region, it is not difficult to find the maximum Xmax and minimum Xmin of the coordinate X. So that we can find the length of the obstacle model: LX=Xmax-Xmin. Similarly, we can find the width LY and height LZ, then determine the 3-d proportion of the obstacle model. Finally, the true obstacle’s 3-d information can be restored according to the real obstacle height.
Figure 5. Color segmentation results.
After segmenting the image pairs, this paper will use SIFT algorithm for feature extraction. Fig. 6 shows the results of SIFT feature extraction. The number of feature points is 128.
V. EXPERIMENTAL RESULTS ANALYSIS In order to verify the feasibility of algorithm proposed in this paper. We did the experiments as follows. The experiments are realized in VC 6.0 environment. Fig. 3 shows the original color image pairs of interior © 2012 ACADEMY PUBLISHER
Figure 6. SIFT feature extraction.
2366
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
The next step, we should match the feature points extracted by SIFT algorithm. Fig. 7 shows the results of feature matching. There are 31 pairs of matching points.
Figure 7. Feature matching.
Typically, the initial matching contains much false matches. Therefore, we should eliminate the false matches through RANSAC algorithm. Fig. 8 shows the results after eliminating the false matches. Finally, the number of matching points is 30.
Through measuring the real obstacle, we can know that the obstacle’s 3-d information: the length is 9.85cm, the width is 4.75cm, and the height is 11.98cm. In this paper, we can calculate the obstacle model’s 3-d ratio (0.812:0.403:1) by computing the 3-d coordinates of points in obstacle region, and then we can calculate the real obstacle’s height (12.23cm) by the height detection method. Therefore, the real obstacle 3-d information can be obtained, the length, width, and height are: 9.93cm, 4.93cm, 12.23cm. Because of the color segmentation algorithm can not extract the obstacle region completely and accurately, the camera calibration errors are also exist, there are still errors between the obstacle 3-d information through the method in this paper and the real obstacle 3-d information. However, this small error does not affect the biped robot’s decision. The biped robot can cross the obstacle successfully, avoiding collision with the obstacle. VI. CONCLUSIONS A new obstacle avoidance method which combines the color segmentation with the height detection and reconstructs the obstacle has been proposed in this paper. In order to determine some information (such as robot’s step height, step length) and cross the obstacle successfully, we must calculate the real obstacle 3-d information. Through experimental verification, the proposed method in this paper is feasible. The detection range of match will be reduced by extracting the obstacle region based on the color segmentation, and the time-consuming in whole process of 3-d reconstruction will be also reduced.
Figure 8. RANSAC elimate wrong matches.
As the SIFT algorithm extracts only a sparse image features, this paper will use the image dense stereo matching method based on the region growth. So we can obtain the dense depth map in obstacle region, and reconstruct the obstacle. The results shown in Fig. 9:
ACKNOWLEDGMENT This paper is sponsored by Nature Science Foundation of China (60973096) and Science Foundation of aviation in China(2010ZC56007).
REFERENCES [1]
[2]
[3] Figure 9. Obstacle depth map and 3-d Reconstruction results. [4]
We can compare the experimental results through constructing a table. The experimental data consist of the obstacle’s length, width, and height information. Specific experimental data shown in Table 1: TABLE I. COMPARISON OF THE EXPERIMENTAL DATA Obstacles 3-d Information The method in this paper Real obstacle 3-d information
© 2012 ACADEMY PUBLISHER
Length 9.93cm 9.85cm
Width 4.93cm 4.75cm
Height 12.23cm 11.98cm
[5]
[6]
Pei-yu Jiang. “Study of dynamic walking of biped robot” ,Master’s thesis. Northwestern Polytechnical University, 2004. Wei Cui, Wen-yi Qiang, Xing-lin Chen. “Real time obstacle detection system for biped robot”, Control and Decision, 2004，19(1)：40-43. Simmons R． “The curvature-velocity method for local obstacle avoidance”, International Conference on Robotics and Automation，April，1996． Surmann H，Nüchter A，Hertzberg J． “An autonomous mobile robot with a 3D laser range finder for 3D exploration and digitalization of indoor environments ”, Robotics and Autonomous Systems，2003，45( 3 － 4) : 181-198． Chow Y H, Ronald C. “Obstacle avoidance of legged robot without 3D reconstruction of the surroundings”, Hong Kong: The Chinese University of Hong Kong, 1999. Yu Li. “Obstacle recognition research based on binocular vision”, Master’s thesis, Wuhan Technology University, 2007.
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
S. Allaire, J. Kim, S. Breen, D. Jaffray, and V. Pekar. “Full orientation invariance and improved feature selectivity of 3D SIFT with application to medical image analysis”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008, pages 1-8, June 2008.Allaire S., Kim J., Breen S. Qing-zhong Li, Xian-hua Chen, Wei-kang Gu. “Fast Method for Obstacle Detection Based on Color Stereo Vision”, Computer Science, 2003，30(9):72-75. Kai-yan Lin, Jun-hui Wu, Li-hong Xu. “A Survey on Color Image Segmentation Techniques”, Journal of Image and Graphics, 2005，10(1):1-10. G. Balasubramanian, “Unsupervised Color Image Segmentation Using a Dynamic Color Gradient Thresholding Algorithm”, Thesis (M.S.)-Rochester Institute of Technology, 2008. Bi-na Zheng, Ze-tao Jiang, Min Wu. “3-d in ensional reconstruction based on image dense stereo matching”, Chinese Journal of Stereo Logy and Image Analysis, 2009,14(2):187-191. Zheng Wang, Zeng-qi Sun. “Color Object Recognition Algorithm Based on RoboCup Biped Humanoid Robot”, Journal of University of Jinan, 2007，21：18-22. Asmare, M.H, Asirvadam, V.S, lznita, L. “Color space selection for color image enhancement application,” in ICSAP, 2009. Kelson R.T. Aires, Andre M. Santana, Adelardo A.D. Medeiros “Optical flow using color information preliminary results”. SAC’08 March 1620, 2008, Fortaleza, Cear’a, Brazil Copyright 2008 ACM 9781595937537/08/0003. Lowe D G. “Distinctive Image Features from Scale-Invariant Points ”. International Journal of Computer Vision, 2005, 60(2):91-100.
Zetao Jiang Male, born in 1961 in Jiangxi Province, Professor, Doctor. He is adjunct professor of Nanjing University of Aeronautics and Astronautics, PhD supervisor, academic leader in Jiangxi province. He received the B.S. degree in Beijing Normal University, Beijing, China, in 1986, and received the M.S. degree in Tongji University, Shanghai, China, in 1995, and received the Doctor
© 2012 ACADEMY PUBLISHER
2367
of Philosophy in Northwestern Polytechnical University, Xi’an, China, in 2006. Main research areas were image processing, computer vision, and network information security. He works at School of Information Engineering at Nanchang Hangkong University in Jiangxi. More than 60 papers were published, nearly 30 articles were included in SCI, EI. He presided key technology research project “ Control system based on multi-level security agent”, which was awarded the second class prize in the 2009 annual science and technology progress of Jiangxi province. He presided the research of network and information security defense technology based on digital watermarking, which was awarded the first class prize in science and technology achievements of Jiangxi province. He presided and finished Nature Science Foundation of China (60673055), two subject Foundations of province, an open foundation of measurement and control center of province and more than 10 other subjects. Currently, he is presiding Nature Science Foundation of China (60673055) and Nature Science Foundation of Province, which is in the field of 3d reconstruction. Dr. Jiang, Prof. Zetao is director of Chinese Society for Stereology.
Yanru Cui Female, born in 1987 in Shanxi Province, and received the B.S. degree in Changzhi College, Changzhi, China, in 2009. Currently, she is pursuing the M.S. degree in Nanchang Hangkong University, and the main research area is image processing and computer vision.
Qiang Wang Male, born in 1983 in Hubei Province, and received the B.S. degree in Huazhong University of Science and Technology Wuchang Branch, Wuhan, China, in 2009. Currently, he is pursuing the M.S. degree in Nanchang Hangkong University, and the main research area is image processing and computer vision.
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
A New Obstacle Avoidance Method Based on Biped Robot Zetao Jiang Nanchang Hangkong University/School of Information Engineering, Nanchang, China Email: [email protected]
Yanru Cui and Qiang Wang Nanchang Hangkong University/School of Information Engineering, Nanchang, China Email: [email protected], [email protected]
Abstract—Currently, in order to avoid obstacle, most robots will choose a new path. This paper proposed a new obstacle avoidance method for biped robots. When lower obstacle exists in the road, we will calculate the obstacle’s three-dimensional information instead of using the traditional method of re-select the path, so that the biped robot will avoid obstacles successfully by crossing the obstacle. In 3-d reconstruction, the stereo matching is complex, and has large time consumption. Therefore, the method in this paper has a combination of color segmentation and height detection, extracts obstacle’s region accurately to reduce detection range of matching, then improves the real-time of reconstruction. Experiments show that the method proposed in this paper is feasible. Index Terms—Biped robot, Color segmentation, Height detection, Stereo matching, Obstacle reconstruction
I.
INTRODUCTION
In recent years, Robotics is a comprehensive new discipline, which focused on engineering technology such as machinery, computer, control, electronics and the latest research in artificial intelligence[1]. In robotics, machine vision which likes human eyes is one of the key technologies. Mobile robot obtains 3-d information about external environment, mainly through the following two ways: laser rangefinder and stereo vision technology[2]. Laser ranging obtains the 3-d distance images in front of robot through laser-scanning sensors. Therefore, we can get obstacle’s information such as shape, height, width, etc[3-4]. However, this method requires relatively complex equipment, and has a bad ability to adapt to the environment, so the practicality is not strong. Stereo vision technology plays an important role in robot navigation applications for its low energy consumption, high reliability, etc[5]. In stereo vision, stereo matching is the most important, its main problem is to find matching points between images. According to different matching primitives, stereo matching can be divided into three types, region matching, phrase Copyright credit, project number: Nature Science Foundation of China (60973096) and Science Foundation of aviation in China(2010ZC56007). Corresponding author: [email protected]
© 2012 ACADEMY PUBLISHER doi:10.4304/jcp.7.10.2362-2367
matching and feature matching[6]. The essence of region matching algorithm is to use related degree of gray information between local windows, it can achieve high accuracy in places where have smooth diversifications and rich details. But it is difficult to choose the window size of matching, and has large compute-intensive and slow calculation speed. Phase matching reflects the signal’s structure information, and has a good inhibitory effect on the high-frequency noise of images. But it exists major problems， such as phase singularities and phase winding. SIFT feature matching does not directly dependent on gray, and has a strong anti-interference, a small compute-intensive and fast calculation speed[7]. In this paper, we used this algorithm, although we can get only sparse disparity field, we can restore its dense disparity map by the existing dense matching method. Currently, the robot can be divided into two categories: wheeled robots and biped robots. When the wheeled robot is moving and there is an obstacle in front of it, we only need to calculate some simple information, such as the distance between obstacle and the wheeled robot, the obstacle width, so that the robot has enough time for a new path planning to avoid a collision with the obstacle. In contrast, biped robot has a high degree of imitation of human characteristics, and can cross over the lower obstacle just like people instead of choosing a new path. We need to consider the biped robot’s step height, step length and other issues, so we can’t achieve its purpose only using traditional obstacle avoidance algorithm. To solve these problems, this paper proposed a new obstacle avoidance method of crossing over the obstacle for biped robot, calculated the obstacle’s real 3-d information, so that we can determine the robot’s step height, step length and other issues in order to avoid a collision with the obstacle. II. GET A REAL 3-D OBSTACLE’S IDEAS AND STEPS Calculating the obstacle’s real 3-d information is to 3-d reconstruct the obstacle. It is that extract the 2-d information of obstacle images as rich as possible, then restore the 3-d information as much as possible, so that achieve the 3-d reconstruction of obstacle. Steps in this
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
2363
paper are as follows. Firstly, in order to extract the obstacle region quickly, reduce the computation of matching, and achieve a better real-time, we will search for a new method about color image segmentation in the analysis of existing methods[8-10]. Secondly, we will use existing methods based on stereo image pair for dense matching[11], restore a dense obstacle region, and calculate the 3-d coordinates of corresponding points by the use of camera matrix, then find the 3-d proportion of obstacle model. Finally, we will detect the obstacle’s real height, and get its real 3-d information according to the 3-d proportion of obstacle model.
strengthen the image segmentation results; Typically, I component is not usually judged by the standard. Therefore, HIS color model will be used in this paper. HIS and RGB color models’ transformation relationship is as follows.
III. EXTRACT REGION OF OBSTACLE BY COLOR SEGMENTATION
B. Color Segmentation Algorithm and Steps The color of ordinary road is very similar. With the light’s change the color of road scene changes, but under the premise that the image is not saturated, the color difference between the road surface and obstacles is always there. Therefore, this paper uses HIS color model which has a smaller sensitivity to light for image segmentation. The purpose is to separate the road and obstacle’s region. Specific steps are as follows: Firstly, choose a M × N obstacle-free area in front of the robot, and calculate the road’s average reference hue HA and saturation SA.
In binocular vision, stereo matching is the most important task. It needs a lot of points to match, and has a huge computation, time consuming, and low real-time. To solve this problem, we should segment the collected images based on color, remove the complex background, and extract the obstacle region. Therefore, in order to reduce the computing time and improve the real-time, we can extract and match feature points only in the target area. A. The Selection of Color Model RGB, YUV, HIS are common color models[12-13]. RGB is the most common color space. After the digital signal processing, the robot’s cameras get a value that refers to each pixel value in RGB space. But the RGB color space is uneven, the visual difference between two colors and two points’ Euclidean distance in RGB color space are not linearly proportional. In different conditions, such as different types of light source and reflective properties of objects, the distribution of the same color’s RGB value is dispersed, and the three components occupied almost the entire color space are interrelated, so it is difficult to find a exact color space threshold. YUV color model, Y component indicates the light intensity, U and V components indicate the color tone[14]. Compared to the RGB color model, the YUV color mode which is independent of light intensity is more suitable for the habit of human eyes, thereby reducing the light and reflection’s impact on the color recognition. Another advantage of YUV color model is that it is a linear transformation relationship with RGB color model. When the robot is working online, it can convert the YUV color model to the RGB color model quickly. So some developers will use this color model as the basis for target recognition. HIS color model is based on visual principles. H component represents the hue, I component represents the brightness, and S component represents the saturation. These three color properties are unrelated, especially the property of H, which has low sensitivity to changes in external light, can reflect the species of colors accurately. For the same color properties of objects, H which has a stable and narrow scope of transformation can be used as a prerequisite for color segmentation; S can be used as auxiliary to determine the conditions in order to © 2012 ACADEMY PUBLISHER
⎧⎪ 1 / 2[( R − G ) + ( R − B )] ⎫⎪ H = cos −1 ⎨ ⎬ ⎪⎩ ( R − G ) 2 + ( R − B )( G − B ) ⎪⎭
S =1−
(R + G + B ) 3
I =
H
S
min( R , G , B ) I
A
A
=
=
M
N
i =1
j =1
∑ ∑
(1)
H (i, j )
M × N M
N
i =1
j =1
∑ ∑
S (i, j )
M × N
(2)
Secondly, select each point (i,j) on the image, and calculate the 4 × 4 neighborhood W of average hue and saturation values.
h (i, j ) =
s (i, j ) =
∑
f (u , v )
( u , v )∈ W
∑
N g (u , v )
( u , v )∈ W
N
(3)
N denotes the number of pixels of the neighborhood W, f(u,v) and g(u,v) represent the hue and saturation of the pixel. Thirdly, determine any pixel (i,j) is obstacle or not, the formula is as follows:
F(i, j) = h(i, j) − HA G(i, j ) = s(i, j) − S A
(4)
When F(i,j) is less than the set threshold HT and G(i,j) is less than the set threshold ST, the pixel (i,j) represents the road, otherwise the obstacle. After such treatment, the obstacle region can be split out, and the whole background is set to black, leaving only the obstacle
2364
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
region. This greatly reduces the workload of the feature points’ extraction and matching in 3-d reconstruction and saves a lot of time.
P in images is dP, and the pixel parallax of the point F in images is dF, the relationship can be expressed as follows.
Δd = dP − dF =
IV. HEIGHT DETECTION AND RECONSTRUCTION OF OBSTACLE A. Preparation Works Binocular vision refers to that, at the same time, we shoot two images for the same object from different angles. In the world coordinate system, a space point on the target object can be mapped to these two images. Feature point matching is to search these two images, and identify each corresponding pixel points, the pixel points are called matching points. As the external environment, lighting and camera distortion and other negative factors will affect the images, it is difficult to find the matching points exactly. Therefore, feature point matching is an important part of binocular vision studies. Scale Invariant Feature Transform (SIFT) algorithm has more widely used and strong match ability in the strength texture feature areas, and remains invariance for image scaling, rotation, and affine transform[15]. So, in this paper, we use SIFT algorithm to extract and match the features. The preparation works mainly include four steps. Firstly, extract features from a pair of images. Secondly, match the features by the traditional matching methods. Thirdly, Random Sample Consensus (RANSAC) algorithm is considered one of the most efficient robust estimation algorithm. Therefore, RANSAC will be used to eliminate the false matches further. Finally, compute the parallax that spatial points in two images. B. The True Height Detection of Obstacle The parallel stereo vision system based on the biped robots is similar to the whole process that the human binocular obtains the object’s depth information. For any object point P in 3-d scene, we get the parallax d of this point first, and calculate the distance OD from this point in optical axis to the camera optical center O. The formula is as follows: (5) Bf
OD =
d
f is the camera focal length, B is the distance between two camera optical centers, d is the parallax of the measured point mapping on the left and right images.
Figure 1. Obstacle height detection.
As shown in Figure1, we can calculate the depth information OD of the obstacle’s top point P and OE of the point F on the ground. The pixel parallax of the point © 2012 ACADEMY PUBLISHER
Bf Bf Bf ⎡ DE⎤ − = OD OE OD⎢⎣ OE⎥⎦
(6)
Seen by the 3-d geometry
DE OE
=
PH H
(7)
We can know that
Bf Δd = OD
⎡ PH ⎤ ⎢H ⎥ ⎣ ⎦
(8)
As Δ d and OD are known, we can deduce the obstacle’s height PH. C.
The Obstacle Model’s 3-d Scale and Recons- truction In order to obtain the 3-d coordinates of the measured object’s feature points, binocular vision sensor should at least obtain two images containing the object’s feature points from different locations. Figure2 shows that two cross cameras observe the same object from different angles. In fact getting two images does not necessarily require two cameras. One camera through a certain movement can also obtain two images by observing a stationary object in different locations.
Figure 2. Dual camera observation of space point.
Figure2 shows, for any space point P, its image point in camera C1 is p1. Because of that the image point in camera C1 of any point on the connection line O1P is p1, we could not get the 3-d position of P from p1. Therefore, from the location of image point p1, we only know that the space point P is located on the connection line O1p1. While observing the space point P by camera C2 at the same time, we can get the depth information of point P because of that the space point P is not only on the connection line O1p1, but also on the connection line O2p2. So the point P is the intersection of two lines O1p1 and O2p2, p1 and p2 is matching points with each other. We must match the feature points on the images first, and find the matching points in order to determine the 3-d location of their corresponding space point. In this paper, the image retains only the obstacle area through the color segmentation. We can restore the dense matching points concentrated in the obstacle area through the existing algorithm based on image dense stereo matching. The space 3-d coordinates of points on the obstacle can be determined through each pair of matching points. The cameras C1 and C2 have been calibrated, and their
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
2365
projection matrices are M1 and M2, then there are the following formulas.
⎡u1 ⎤ ⎡ m11 ⎢ Z c1 ⎢⎢ v1 ⎥⎥ = ⎢ m121 1 ⎢⎣ 1 ⎥⎦ ⎢⎣ m31
1 m12 m122 1 m32
1 m13 m123 1 m33
⎡u 2 ⎤ ⎡ m11 ⎢ 2 Z c 2 ⎢⎢ v2 ⎥⎥ = ⎢ m21 2 ⎢⎣ 1 ⎥⎦ ⎢⎣ m31
m122 2 m22
m132 2 m23
2 m32
2 m33
1
2
⎡X ⎤ 1 ⎤⎢ ⎥ m14 ⎥ Y m124 ⎥ ⎢ ⎥ ⎢ ⎥ 1 ⎥ Z m34 ⎦⎢ 1 ⎥ ⎣ ⎦ ⎡X ⎤ m142 ⎤ ⎢ ⎥ 2 ⎥⎢Y ⎥ m24 ⎥⎢ ⎥ 2 ⎥ Z m34 ⎦⎢ 1 ⎥ ⎣ ⎦
(9)
ground scene. Fig. 4 shows the results segmented by general gray segmentation algorithm. Fig. 5 shows the result segmented by the color segmentation algorithm proposed in this paper. The ordinary gray segmentation algorithm can not satisfactory our needs. However, through the color segmentation algorithm proposed in this paper, we can extract the obstacle region successfully. Compared to the ordinary gray segmentation algorithm, the segmentation results are much better.
(10)
(u1,v1,1) and (u2,v2,1) are the homogeneous coordinates of p1 and p2 in their respective images. (X,Y,Z,1) is the homogeneous coordinates of the point P in world coordinate system. m ijk (k=1,2; i=1,2,3;
Figure 3. Original color image pair.
j=1,2,3,4) is the element located row i column j in the projection matrix Mk. In the formula, Zc1 and Zc2 can be eliminated, then get four linear equations about X,Y,Z: 1 1 1 1 1 1 (u1m31 − m11 )X + (u1m32 − m12 )Y + (u1m33 − m13 )Z 1 1 = m14 −u1m34 1 1 1 (v1m31 − m121) X + (v1m32 − m122)Y + (v1m33 − m123)Z
(11)
1 = m124 − v1m34
Figure 4. Ordinary gray segmentation results.
2 2 2 (u2m31 − m112 )X + (u2m32 − m122 )Y + (u2m33 − m132 )Z 2 = m142 − u2m34
2 2 2 2 2 2 (v2m31 −m21 )X +(v2m32 −m22 )Y +(v2m33 −m23 )Z
(12)
2 2 = m24 −v2m34
We can see from the analytic geometry, the geometric significance of formula 12 and 13 is that the line over O1p1(or O2p2). Due to the space point P is the intersection of line O1p1 and line O2p2, it must satisfy the formula 12 and 13 at the same time. Therefore, we can combine formula 12 with formula 13, then calculate the coordinates (X,Y,Z) of the point P. After finding the 3-d coordinates of all points in the obstacle region, it is not difficult to find the maximum Xmax and minimum Xmin of the coordinate X. So that we can find the length of the obstacle model: LX=Xmax-Xmin. Similarly, we can find the width LY and height LZ, then determine the 3-d proportion of the obstacle model. Finally, the true obstacle’s 3-d information can be restored according to the real obstacle height.
Figure 5. Color segmentation results.
After segmenting the image pairs, this paper will use SIFT algorithm for feature extraction. Fig. 6 shows the results of SIFT feature extraction. The number of feature points is 128.
V. EXPERIMENTAL RESULTS ANALYSIS In order to verify the feasibility of algorithm proposed in this paper. We did the experiments as follows. The experiments are realized in VC 6.0 environment. Fig. 3 shows the original color image pairs of interior © 2012 ACADEMY PUBLISHER
Figure 6. SIFT feature extraction.
2366
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
The next step, we should match the feature points extracted by SIFT algorithm. Fig. 7 shows the results of feature matching. There are 31 pairs of matching points.
Figure 7. Feature matching.
Typically, the initial matching contains much false matches. Therefore, we should eliminate the false matches through RANSAC algorithm. Fig. 8 shows the results after eliminating the false matches. Finally, the number of matching points is 30.
Through measuring the real obstacle, we can know that the obstacle’s 3-d information: the length is 9.85cm, the width is 4.75cm, and the height is 11.98cm. In this paper, we can calculate the obstacle model’s 3-d ratio (0.812:0.403:1) by computing the 3-d coordinates of points in obstacle region, and then we can calculate the real obstacle’s height (12.23cm) by the height detection method. Therefore, the real obstacle 3-d information can be obtained, the length, width, and height are: 9.93cm, 4.93cm, 12.23cm. Because of the color segmentation algorithm can not extract the obstacle region completely and accurately, the camera calibration errors are also exist, there are still errors between the obstacle 3-d information through the method in this paper and the real obstacle 3-d information. However, this small error does not affect the biped robot’s decision. The biped robot can cross the obstacle successfully, avoiding collision with the obstacle. VI. CONCLUSIONS A new obstacle avoidance method which combines the color segmentation with the height detection and reconstructs the obstacle has been proposed in this paper. In order to determine some information (such as robot’s step height, step length) and cross the obstacle successfully, we must calculate the real obstacle 3-d information. Through experimental verification, the proposed method in this paper is feasible. The detection range of match will be reduced by extracting the obstacle region based on the color segmentation, and the time-consuming in whole process of 3-d reconstruction will be also reduced.
Figure 8. RANSAC elimate wrong matches.
As the SIFT algorithm extracts only a sparse image features, this paper will use the image dense stereo matching method based on the region growth. So we can obtain the dense depth map in obstacle region, and reconstruct the obstacle. The results shown in Fig. 9:
ACKNOWLEDGMENT This paper is sponsored by Nature Science Foundation of China (60973096) and Science Foundation of aviation in China(2010ZC56007).
REFERENCES [1]
[2]
[3] Figure 9. Obstacle depth map and 3-d Reconstruction results. [4]
We can compare the experimental results through constructing a table. The experimental data consist of the obstacle’s length, width, and height information. Specific experimental data shown in Table 1: TABLE I. COMPARISON OF THE EXPERIMENTAL DATA Obstacles 3-d Information The method in this paper Real obstacle 3-d information
© 2012 ACADEMY PUBLISHER
Length 9.93cm 9.85cm
Width 4.93cm 4.75cm
Height 12.23cm 11.98cm
[5]
[6]
Pei-yu Jiang. “Study of dynamic walking of biped robot” ,Master’s thesis. Northwestern Polytechnical University, 2004. Wei Cui, Wen-yi Qiang, Xing-lin Chen. “Real time obstacle detection system for biped robot”, Control and Decision, 2004，19(1)：40-43. Simmons R． “The curvature-velocity method for local obstacle avoidance”, International Conference on Robotics and Automation，April，1996． Surmann H，Nüchter A，Hertzberg J． “An autonomous mobile robot with a 3D laser range finder for 3D exploration and digitalization of indoor environments ”, Robotics and Autonomous Systems，2003，45( 3 － 4) : 181-198． Chow Y H, Ronald C. “Obstacle avoidance of legged robot without 3D reconstruction of the surroundings”, Hong Kong: The Chinese University of Hong Kong, 1999. Yu Li. “Obstacle recognition research based on binocular vision”, Master’s thesis, Wuhan Technology University, 2007.
JOURNAL OF COMPUTERS, VOL. 7, NO. 10, OCTOBER 2012
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
S. Allaire, J. Kim, S. Breen, D. Jaffray, and V. Pekar. “Full orientation invariance and improved feature selectivity of 3D SIFT with application to medical image analysis”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008, pages 1-8, June 2008.Allaire S., Kim J., Breen S. Qing-zhong Li, Xian-hua Chen, Wei-kang Gu. “Fast Method for Obstacle Detection Based on Color Stereo Vision”, Computer Science, 2003，30(9):72-75. Kai-yan Lin, Jun-hui Wu, Li-hong Xu. “A Survey on Color Image Segmentation Techniques”, Journal of Image and Graphics, 2005，10(1):1-10. G. Balasubramanian, “Unsupervised Color Image Segmentation Using a Dynamic Color Gradient Thresholding Algorithm”, Thesis (M.S.)-Rochester Institute of Technology, 2008. Bi-na Zheng, Ze-tao Jiang, Min Wu. “3-d in ensional reconstruction based on image dense stereo matching”, Chinese Journal of Stereo Logy and Image Analysis, 2009,14(2):187-191. Zheng Wang, Zeng-qi Sun. “Color Object Recognition Algorithm Based on RoboCup Biped Humanoid Robot”, Journal of University of Jinan, 2007，21：18-22. Asmare, M.H, Asirvadam, V.S, lznita, L. “Color space selection for color image enhancement application,” in ICSAP, 2009. Kelson R.T. Aires, Andre M. Santana, Adelardo A.D. Medeiros “Optical flow using color information preliminary results”. SAC’08 March 1620, 2008, Fortaleza, Cear’a, Brazil Copyright 2008 ACM 9781595937537/08/0003. Lowe D G. “Distinctive Image Features from Scale-Invariant Points ”. International Journal of Computer Vision, 2005, 60(2):91-100.
Zetao Jiang Male, born in 1961 in Jiangxi Province, Professor, Doctor. He is adjunct professor of Nanjing University of Aeronautics and Astronautics, PhD supervisor, academic leader in Jiangxi province. He received the B.S. degree in Beijing Normal University, Beijing, China, in 1986, and received the M.S. degree in Tongji University, Shanghai, China, in 1995, and received the Doctor
© 2012 ACADEMY PUBLISHER
2367
of Philosophy in Northwestern Polytechnical University, Xi’an, China, in 2006. Main research areas were image processing, computer vision, and network information security. He works at School of Information Engineering at Nanchang Hangkong University in Jiangxi. More than 60 papers were published, nearly 30 articles were included in SCI, EI. He presided key technology research project “ Control system based on multi-level security agent”, which was awarded the second class prize in the 2009 annual science and technology progress of Jiangxi province. He presided the research of network and information security defense technology based on digital watermarking, which was awarded the first class prize in science and technology achievements of Jiangxi province. He presided and finished Nature Science Foundation of China (60673055), two subject Foundations of province, an open foundation of measurement and control center of province and more than 10 other subjects. Currently, he is presiding Nature Science Foundation of China (60673055) and Nature Science Foundation of Province, which is in the field of 3d reconstruction. Dr. Jiang, Prof. Zetao is director of Chinese Society for Stereology.
Yanru Cui Female, born in 1987 in Shanxi Province, and received the B.S. degree in Changzhi College, Changzhi, China, in 2009. Currently, she is pursuing the M.S. degree in Nanchang Hangkong University, and the main research area is image processing and computer vision.
Qiang Wang Male, born in 1983 in Hubei Province, and received the B.S. degree in Huazhong University of Science and Technology Wuchang Branch, Wuhan, China, in 2009. Currently, he is pursuing the M.S. degree in Nanchang Hangkong University, and the main research area is image processing and computer vision.
