Gravity-Center Template Based Human Face Feature Detection
Gravity-Center Template Based Human Face Feature Detection Jun Miao1, Wen Gao1, Yiqiang Chen1, and Jie Lu 2 1
Digital Media Lab, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100080, P. R. China { jmiao, wgao, yqchen }@ict.ac.cn 2 Institute of Electronic Information Engineering, Tongji University, Shanghai 200092, P. R. China
Abstract. This paper presents a simple and fast technique for
geometrical feature detection of several human face organs such as eyes and mouth. Human face gravity-center template is firstly used for face location, from which position information of face organs such as eyebrows, eyes, nose and mouth are obtained. Then the original image is processed by extracting edges and the regions around the organs are scanned on the edge image to detect out 4 key points which determine the size of the organs. From these key points, eyes and mouth's shape are characterized by fitting curves. The results look well and the procedure is fast. Keywords face detection, edge extraction, face feature detection
1 Introduction Human face and its feature detection is much significant in various applications as human face identification, virtual human face synthesis, and MPEG-4 based human face model coding. As for face image processing , many people have done much work on them. The representative jobs include mosaic technique[1,2], fixed template matching[3], deformable template matching[4], gravity-center template matching [5], “eigenface”[6] or “eigenpicture”[7] scheme, neural network[8,9], and usage of multiple information including color, sound or motion[10,11]. There also have been quite a few techniques reported about face contour and organ geo14metrical feature extraction, such as B-splines[12], adaptive Hough trasmform[13], cost minimiziation[14], deformable template[15,16] , region-based template[17], geometrical model[18]. However, most of them are either complex or time-consuming. Here, we present a simple and fast approach based on the technique of gravity-center template to locate face organs such as eyes and mouth, and detect their key points to characterize their scales and shape.
2 Simplified geometrical model for eyes and mouth The eyes and mouth have similar geometrical configuration as shown in fig. 1. There are four key points, the highest point P1, the lowest point P2, the left corner point P3 and the right corner point P4, which determine the size (a×b) and the arc-like(R1,R2) shape of the organs. Our goal is to detect out these key points to segment face region and concisely describe the features of eyes and mouth.
Fig. 1. Simplified model for eyes and mouth
3 Detection System Architecture We have implemented a facial geometrical feature extraction system for eyes and mouth based on human face gravity-center template technique the author recently suggested [5]. The system consists of there modules : the first one of gravity-center template based human face and its organs location, the second one of four key points extraction, and the last one of geometrical curves fitting. Inputting an original image, it will be processed by mosaicizing, mosaic edge extracting and feature simplifying for gravity-center template matching to locate human face in an unconstrained background and by adaptive Sobel edge extracting for feature points localization. From this information, geometrical shape of eyelids and lips can be easily fitted. Edge Extacting
Original image
Face and Facial Organs Locating
Four Key Points Detecting
Fig. 2. System architecture
Geometrical Curves Fitting
Feature image
3.1 Face and facial organs localization based on face gravity-center template match This procedure will produce the location information about face and its organs such as eyebrows, eyes, nose and mouth, which is necessary for further facial feature detection. We use four face gravity-center templates [5] to detect frontal faces. In these templates, there are 4 to 6 small boxes which correspond to the facial organs: eyebrows, eyes, nose and mouth. Once a face in an image is located, simultaneously, its facial organs can be also located. The three steps b.(Image Mosaicizing), c.(Mosaic Horizontal Edge Extracting) and d.(Feature Simplifying and Gravity-Cenetr Template Matching) in Fig.3 illustrate the procedure of face and its organs localization. The detail description on detecting procedures can be referred in reference [5].
a.
b.
c.
d. Fig. 3. Face and its organs localization
Fig. 4. Face gravity-center templates
3.2 Key feature points extraction From this step, four key points P1, P2, P3 and P4 illustrated in Fig.1 are expected to be extracted. The original image is firstly processed with an adaptive Sobel operator introduced in reference [19] to produce a binary edge image. Fig. 5 shows the result processed by this operator.
Fig. 5. Binary edge image using an adaptive Sobel operator
According to the face organs’ location information obtained from gravity-center template matching, we can scanned in the edge image in areas around those organs to see if current position is one of four key feature point of P1, P2, P3 and P4 showed in Fig.1. If it is true, its coordinates is to be calculated out using a mechanism of medianpass from the coordinates of the points scanned in rows or columns. The procedure is shown in Fig.6.
a.
b.
Fig. 6. Key feature points searching ( e.g. left eye)
In the upper picture of Fig.6 the black area is cut from the area of the left eye in the Fig. 5, in which the position of the cross point of lines X, Y is obtained from the center of the small box corresponding to left eye. And lines X and Y are the initial positions in horizontal and vertical directions to scan and search the top and bottom sides, the left and right sides of the area. Scan starts from the initial X, Y lines and extend upwards and downwards, leftwards and rightwards, respectively in horizontal and vertical lines. A scan towards the one of the four directions will stop when the number of edge pixels it encounters in the scanning line that is orthogonal to the direction the scan towards is less than a threshold such as 3 edge pixels. In this case, the median value pixel is chosen as one key point which belongs to one of P1, P2, P3 and P4.
3.3 Eyelids and lips shapes description According to the simplified model for eyes and mouth’s, from four key feature points: P1, P2, P3 and P4, a upper and a lower circle arc can be generated to fit the upper and the lower eyelids or lips. The effect is shown in Fig.7.
Fig. 7. Eyelids and lips fitting
4
Some experimental examples Fig.8 to Fig.15 are part of the experiment results.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.
Fig. 12.
Fig. 13.
Fig. 14.
Fig. 15.
5 Conclusion and future work Compared with some other detecting schemes such as deformable template technique, the approach introduced here is simple and fast ( 0.28 seconds in average on Pentium550 PC ). Especially for faces with complex background, our system can work as well. During experiments, we still found some detection results of facial feature for those faces with complex texture and poor light conditions are not as well as those for good quality face images. And for the moment, this method can not characterize lips in more details yet. Further work will include eyeball, nose and cheek contour detection and the algorithm is expected to be enhanced to improve the system’s robustness.
References 1. L.D. Harmon, The recognition of faces, Scientific American, 229(5), 71-82, 1973. 2. G. Z. Yang and T. S. Huang, Human face detection in a complex background, Pattern Recognition, 27(1), 43-63, 1994 3. W. Gao and M. B. Liu, A hierarchical approach to human face detection in a complex background, Proceedings of the first International Conference on Multimodal Interface’96, 289-292, 1996 4. A.L. Yuille, Deformable templates for face detection, J. Cogn. neurosci. 3, 59-70, 1991
5. J. Miao, B.C. Yin, K.Q. Wang, et al, A hierarchical multiscale and multiangle system for human face detection in a complex background using gravity-center template, Pattern Recognition, 32(7), 1999 6. M. Turk and A. Pentland, Face recognition using eigenfaces, Proc. IEEE-CSCCVPR, 586-591, 1991 7. M. Kirby and L. Sirovich, Application of the Karhunen-Loeve procedure for the characterization of human faces, IEEE Trans. PAMI, 12(1), 103-108, 1990 8. H.A. Rowley, S. Baluja, and T. Kanade, Neural network-based face detection, IEEE-PAMI, 20(1), 23-38, 1998 9. K.K. Sung and T. Poggio, Example-based learning for view-based human face detection, IEEE-PAMI, 20(1), 39-50, 1998 10.R. Brunelli and D. Falavigna, Person identification using multiple cues, IEEE Trans. PAMI, 17(10), 955-966, 1995 11.C.H. Lee, J.S. Kim and K.H. Park, Automatic human face location in a complex background using motion and color information, Pattern Recognition, 29(11), 1877-1889, 1996 12.C.J. Wu and J.S. Huang, Human faces profiles recognition by computer, Pattern Recognition, 23(3/4), 255-259, 1990 13.X. Li and N. Roeder, Face contour extraction from front-view images, Pattern recognition, 28(8), 1167-1179, 1995 14.K. Lam and H. Yan, Locating and extracting the eye in human face images, Pattern recognition, 29(5), 771-779, 1996 15.A.L. Yuille, D.S. Cohen and P.W. Hallinan, Feature extraction from faces using deformable templates, Proc. IEEE-CS-CCVPR, 104-109, 1989 16.C.L. Huang and C.W. Chen, Human face feature extraction for face interpretation and recognition, Pattern recognition, 25(12), 1435-1444, 1996 17.J.Y. Deng and F. Lai, Region-based template deformation and masking for eyefeature extraction and description, Pattern recognition, 30(3), 403-419, 1997 18.S.H. Jeng, H.Y.M. Liao, et al, Facial feature detection using geometrical face model: an efficient approach, Pattern recognition, 31(3), 273-282, 1998 19.S.Y. Lee, Y.K. Ham and R.H. Park, Recignition of human front faces using knowledge-based feature extraction and neuro-fuzzy algorithm, Pattern recognition, 29(11), 1863-1876, 1996
Digital Media Lab, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100080, P. R. China { jmiao, wgao, yqchen }@ict.ac.cn 2 Institute of Electronic Information Engineering, Tongji University, Shanghai 200092, P. R. China
Abstract. This paper presents a simple and fast technique for
geometrical feature detection of several human face organs such as eyes and mouth. Human face gravity-center template is firstly used for face location, from which position information of face organs such as eyebrows, eyes, nose and mouth are obtained. Then the original image is processed by extracting edges and the regions around the organs are scanned on the edge image to detect out 4 key points which determine the size of the organs. From these key points, eyes and mouth's shape are characterized by fitting curves. The results look well and the procedure is fast. Keywords face detection, edge extraction, face feature detection
1 Introduction Human face and its feature detection is much significant in various applications as human face identification, virtual human face synthesis, and MPEG-4 based human face model coding. As for face image processing , many people have done much work on them. The representative jobs include mosaic technique[1,2], fixed template matching[3], deformable template matching[4], gravity-center template matching [5], “eigenface”[6] or “eigenpicture”[7] scheme, neural network[8,9], and usage of multiple information including color, sound or motion[10,11]. There also have been quite a few techniques reported about face contour and organ geo14metrical feature extraction, such as B-splines[12], adaptive Hough trasmform[13], cost minimiziation[14], deformable template[15,16] , region-based template[17], geometrical model[18]. However, most of them are either complex or time-consuming. Here, we present a simple and fast approach based on the technique of gravity-center template to locate face organs such as eyes and mouth, and detect their key points to characterize their scales and shape.
2 Simplified geometrical model for eyes and mouth The eyes and mouth have similar geometrical configuration as shown in fig. 1. There are four key points, the highest point P1, the lowest point P2, the left corner point P3 and the right corner point P4, which determine the size (a×b) and the arc-like(R1,R2) shape of the organs. Our goal is to detect out these key points to segment face region and concisely describe the features of eyes and mouth.
Fig. 1. Simplified model for eyes and mouth
3 Detection System Architecture We have implemented a facial geometrical feature extraction system for eyes and mouth based on human face gravity-center template technique the author recently suggested [5]. The system consists of there modules : the first one of gravity-center template based human face and its organs location, the second one of four key points extraction, and the last one of geometrical curves fitting. Inputting an original image, it will be processed by mosaicizing, mosaic edge extracting and feature simplifying for gravity-center template matching to locate human face in an unconstrained background and by adaptive Sobel edge extracting for feature points localization. From this information, geometrical shape of eyelids and lips can be easily fitted. Edge Extacting
Original image
Face and Facial Organs Locating
Four Key Points Detecting
Fig. 2. System architecture
Geometrical Curves Fitting
Feature image
3.1 Face and facial organs localization based on face gravity-center template match This procedure will produce the location information about face and its organs such as eyebrows, eyes, nose and mouth, which is necessary for further facial feature detection. We use four face gravity-center templates [5] to detect frontal faces. In these templates, there are 4 to 6 small boxes which correspond to the facial organs: eyebrows, eyes, nose and mouth. Once a face in an image is located, simultaneously, its facial organs can be also located. The three steps b.(Image Mosaicizing), c.(Mosaic Horizontal Edge Extracting) and d.(Feature Simplifying and Gravity-Cenetr Template Matching) in Fig.3 illustrate the procedure of face and its organs localization. The detail description on detecting procedures can be referred in reference [5].
a.
b.
c.
d. Fig. 3. Face and its organs localization
Fig. 4. Face gravity-center templates
3.2 Key feature points extraction From this step, four key points P1, P2, P3 and P4 illustrated in Fig.1 are expected to be extracted. The original image is firstly processed with an adaptive Sobel operator introduced in reference [19] to produce a binary edge image. Fig. 5 shows the result processed by this operator.
Fig. 5. Binary edge image using an adaptive Sobel operator
According to the face organs’ location information obtained from gravity-center template matching, we can scanned in the edge image in areas around those organs to see if current position is one of four key feature point of P1, P2, P3 and P4 showed in Fig.1. If it is true, its coordinates is to be calculated out using a mechanism of medianpass from the coordinates of the points scanned in rows or columns. The procedure is shown in Fig.6.
a.
b.
Fig. 6. Key feature points searching ( e.g. left eye)
In the upper picture of Fig.6 the black area is cut from the area of the left eye in the Fig. 5, in which the position of the cross point of lines X, Y is obtained from the center of the small box corresponding to left eye. And lines X and Y are the initial positions in horizontal and vertical directions to scan and search the top and bottom sides, the left and right sides of the area. Scan starts from the initial X, Y lines and extend upwards and downwards, leftwards and rightwards, respectively in horizontal and vertical lines. A scan towards the one of the four directions will stop when the number of edge pixels it encounters in the scanning line that is orthogonal to the direction the scan towards is less than a threshold such as 3 edge pixels. In this case, the median value pixel is chosen as one key point which belongs to one of P1, P2, P3 and P4.
3.3 Eyelids and lips shapes description According to the simplified model for eyes and mouth’s, from four key feature points: P1, P2, P3 and P4, a upper and a lower circle arc can be generated to fit the upper and the lower eyelids or lips. The effect is shown in Fig.7.
Fig. 7. Eyelids and lips fitting
4
Some experimental examples Fig.8 to Fig.15 are part of the experiment results.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.
Fig. 12.
Fig. 13.
Fig. 14.
Fig. 15.
5 Conclusion and future work Compared with some other detecting schemes such as deformable template technique, the approach introduced here is simple and fast ( 0.28 seconds in average on Pentium550 PC ). Especially for faces with complex background, our system can work as well. During experiments, we still found some detection results of facial feature for those faces with complex texture and poor light conditions are not as well as those for good quality face images. And for the moment, this method can not characterize lips in more details yet. Further work will include eyeball, nose and cheek contour detection and the algorithm is expected to be enhanced to improve the system’s robustness.
References 1. L.D. Harmon, The recognition of faces, Scientific American, 229(5), 71-82, 1973. 2. G. Z. Yang and T. S. Huang, Human face detection in a complex background, Pattern Recognition, 27(1), 43-63, 1994 3. W. Gao and M. B. Liu, A hierarchical approach to human face detection in a complex background, Proceedings of the first International Conference on Multimodal Interface’96, 289-292, 1996 4. A.L. Yuille, Deformable templates for face detection, J. Cogn. neurosci. 3, 59-70, 1991
5. J. Miao, B.C. Yin, K.Q. Wang, et al, A hierarchical multiscale and multiangle system for human face detection in a complex background using gravity-center template, Pattern Recognition, 32(7), 1999 6. M. Turk and A. Pentland, Face recognition using eigenfaces, Proc. IEEE-CSCCVPR, 586-591, 1991 7. M. Kirby and L. Sirovich, Application of the Karhunen-Loeve procedure for the characterization of human faces, IEEE Trans. PAMI, 12(1), 103-108, 1990 8. H.A. Rowley, S. Baluja, and T. Kanade, Neural network-based face detection, IEEE-PAMI, 20(1), 23-38, 1998 9. K.K. Sung and T. Poggio, Example-based learning for view-based human face detection, IEEE-PAMI, 20(1), 39-50, 1998 10.R. Brunelli and D. Falavigna, Person identification using multiple cues, IEEE Trans. PAMI, 17(10), 955-966, 1995 11.C.H. Lee, J.S. Kim and K.H. Park, Automatic human face location in a complex background using motion and color information, Pattern Recognition, 29(11), 1877-1889, 1996 12.C.J. Wu and J.S. Huang, Human faces profiles recognition by computer, Pattern Recognition, 23(3/4), 255-259, 1990 13.X. Li and N. Roeder, Face contour extraction from front-view images, Pattern recognition, 28(8), 1167-1179, 1995 14.K. Lam and H. Yan, Locating and extracting the eye in human face images, Pattern recognition, 29(5), 771-779, 1996 15.A.L. Yuille, D.S. Cohen and P.W. Hallinan, Feature extraction from faces using deformable templates, Proc. IEEE-CS-CCVPR, 104-109, 1989 16.C.L. Huang and C.W. Chen, Human face feature extraction for face interpretation and recognition, Pattern recognition, 25(12), 1435-1444, 1996 17.J.Y. Deng and F. Lai, Region-based template deformation and masking for eyefeature extraction and description, Pattern recognition, 30(3), 403-419, 1997 18.S.H. Jeng, H.Y.M. Liao, et al, Facial feature detection using geometrical face model: an efficient approach, Pattern recognition, 31(3), 273-282, 1998 19.S.Y. Lee, Y.K. Ham and R.H. Park, Recignition of human front faces using knowledge-based feature extraction and neuro-fuzzy algorithm, Pattern recognition, 29(11), 1863-1876, 1996
