Gait Recognition Using Procrustes Shape Analysis and Shape Context
Gait Recognition Using Procrustes Shape Analysis and Shape Context Yuanyuan Zhang1 , Niqing Yang1 , Wei Li1 , Xiaojuan Wu1, , and Qiuqi Ruan2 1
School of Information Science and Engineering, Shandong University, 27 Shanda Nanlu, 250100 Jinan, China [email protected], [email protected], [email protected], [email protected] 2 Institute of Information Science, Beijing Jiaotong University, No.3 Shangyuan Residence Haidian District, 100044 Beijing, China [email protected]
Abstract. This paper proposes a novel algorithm for individual recognition by gait. The method of Procrustes shape analysis is used to produce Procrustes Mean Shape (PMS) as a compressed representation of gait sequence. PMS is adopted as the gait signature in this paper. Instead of using the Procrustes mean shape distance as a similarity measure, we introduce shape context descriptor to measure the similarity between two PMSs. Shape context describes a distribution of all boundary points on a shape with respect to any single boundary point by a histogram of log-polar plot, and offers us a global discriminative characterization of the shape. Standard pattern recognition techniques are used to classify different patterns. The experiments on CASIA Gait Database demonstrate that the proposed method outperforms other algorithms in both classification performance and verification performance. Keywords: Gait recognition, Procrustes shape analysis, shape context descriptor, Procrustes Mean Shape (PMS).
1
Introduction
Among various biometrics like face, iris and fingerprint, gait is a more attractive biometric feature for human identification. Gait signals can be detected from a long distance and measured at low resolution. Therefore, gait can be used in such situations that face or iris information is not available in high enough resolution for recognition. From the perspective of surveillance, gait is a particularly attractive modality. Recently, the study of gait recognition, which concerns recognizing individuals by the way they walk, has received an increasing interest from researchers in the computer vision community. Many contributions have been made to this rapidly developing domain. Gait recognition techniques mainly fall into two categories namely model-based and model-free approaches. The model-based approaches usually model the human
Corresponding author.
H. Zha, R.-i. Taniguchi, and S. Maybank (Eds.): ACCV 2009, Part III, LNCS 5996, pp. 256–265, 2010. c Springer-Verlag Berlin Heidelberg 2010
Gait Recognition Using Procrustes Shape Analysis and Shape Context
257
body structure and extract image features to map them into the structural components of models or to derive motion trajectories of body parts. Bhanu and Han [1] proposed a kinematic-based method to identify individuals. It estimates 3D human walking parameters by performing a least square fit of the 3D kinematic model to the 2D silhouette images. A genetic algorithm is then used for feature selection. The advantage of model-based approaches is that models can handle occlusion and noise better and offer the ability to derive gait features directly from model parameters. They also help to reduce the dimensionality needed to represent the data. However, they suffer from high computational costs. The majority of current approaches are model-free. They typically analyze the image sequence by motion or shape and characterize the whole motion pattern of the human body by a compact representation regardless of the underlying structure. Based on body shape and gait, Lee et al. [2] described a momentbased representation of gait appearance for the purpose of person identification. Sarkar et al. [3] proposed a baseline algorithm for human identification using spatiotemporal correlation of silhouette images. Han and Bhanu [4] proposed a spatiotemporal gait representation called Gait Energy Image to characterize human walking properties. Wang et al. [5] employed a compressed representation of gait sequences and obtained encouraging classification performance. Inspired by Wang [5]’s work, we made further exploration to it in this paper. We use Procrustes shape analysis to produce Procrustes Mean Shape (PMS) of each gait sequence in the database. Quite different and novel, instead of using the Procrustes Mean Shape Distance (MSD) as Wang did, we introduce Shape Context (SC) descriptor [6] to measure the similarity of two PMSs. The experiments on dataset-A and dataset-B in CASIA Gait Database [7] show that SC is an efficient and powerful shape descriptor for similarity measure. We gain favorable performance after combing PMS and SC.
2
Related Work
Intuitively, recognizing people through gait depends greatly on how the silhouette shape of an individual changes over time. Shape is an important cue as it captures a prominent element of an object. Therefore, gait may be considered to be composed of a set of static poses and their temporal variations can be analyzed to obtain distinguishable signatures. Based upon the above consideration, Wang et al. [5] depicted the human shape using the method of Procrustes shape analysis. Pose changes of segmented silhouettes over time are represented as an associated sequence of complex configurations in a two-dimensional (2D) shape space and are further analyzed by the Procrustes shape analysis method to obtain an eigenshape as gait signature. In the field of shape matching and shape similarity measuring, several shape descriptors have been proposed, ranging from moments and Fourier descriptors to Hausdorff distance and the medial axis transform. For a detailed discussion of shape matching techniques, the reader is referred to paper [8]. Belongie and Malik [6] firstly introduced the idea of shape context descriptor. Shape context
258
Y. Zhang et al.
describe a distribution of the remaining boundary points with respect to one point on the boundary. Histogram of a log-polar plot of the shape boundary gives the shape context for that particular boundary point, thus offering a globally discriminative characterization. Their work showed remarkable results involving character recognition using gimpy images. It also had an extensive testing for silhouette images from trademarks, handwritten digits and the COIL dataset.
3
Procrustes Shape Analysis
Procrustes shape analysis [9] is a popular method in directional statistics. It is intended for coping with 2D shapes and provides a good method to find mean shapes. To reduce redundant information, the shape boundary can be easily obtained and stored using a border following algorithm based on connectivity. As we can see in Fig. 1, (xi , yi ) is a random pixel on the boundary. Let the centroid (xc , yc ) of the shape be the origin of the 2D shape space. The two axes Re and Im represent the real and imaginary part of a complex number, respectively. Im
Unwrapping by contour following
( xi , y i )
O
( xC , yC )
Re
Fig. 1. Illustration of computing PMS
As shown in Fig. 1, we can then unwrap each shape anticlockwise into a set of boundary pixel points sampled along its outer-contour in a common complex coordinate system. Each shape can be described as a vector of ordered complex numbers with M elements, u = [u1 , u2 , . . . , uM ]T , ui = (xi − xc ) + j × (yi − yc ). Therefore, each gait sequence will be accordingly converted into an associated sequence of such 2D shape configurations. Given a set of m shapes, we can find their mean shape uˆ by computing the following matrix: Su =
m ui u∗ i
i=1
u∗i ui
,
(1)
where, the superscript ∗ represents the complex conjugation transposition. uˆ is the so-called Procrustes Mean Shape (PMS) which is the dominant eigenvector
Gait Recognition Using Procrustes Shape Analysis and Shape Context 0.1
259
0.1 Sequence 1 Sequence 2 Sequence 3 Sequence 4 Examplar
0.08 0.06
0.06
0.04
0.04
0.02
0.02
0
0
−0.02
−0.02
−0.04
−0.04
−0.06
Sequence 1 Sequence 2 Sequence 3 Sequence 4 Examplar
0.08
−0.06 −0.03 −0.02 −0.01
0
0.01
0.02
0.03
0.04
(a) subject #1 in dataset-A
0.05
−0.03 −0.02 −0.01
0
0.01
0.02
0.03
0.04
0.05
(b) subject #2 in dataset-A
Fig. 2. Two examples of the PMS in dataset-A. Note that the sample points in the PMS depicted here are re-sampled to a visible level for reading convenience.
of Su , i.e., the eigenvector that corresponds to the greatest eigenvalue of Su . Two examples of PMSs are shown in Fig. 2. To compare two different PMSs, the Procrustes Mean Shape Distance (MSD) was used by Wang [5] to measure the similarity between them. Instead of using the MSD, this paper treats the similarity measure totally in a different way as described below.
4
Shape Context
The concept of shape context was originally introduced to measure similarities between shapes and recognize objects. Similar to PMS, a shape is represented by a discrete set of N points, P = {p1 , p2 , . . . , pN }, sampled from the internal or external contours on the object. Assuming contours are piecewise smooth, we can obtain as good an approximation to the underlying continuous shapes as desired by picking N to be sufficiently large. Note that, the point number N here can be either equal to the number M in PMS or not. Consider the set of vectors originating from a random point pi to all other points on the shape. These vectors actually express the configuration of the entire shape relative to the reference point. For the point pi , we compute a coarse histogram hi relative to the coordinates of the remaining N − 1 points, hi (k) = #{q = pi : (q − pi ) ∈ bin(k)},
(2)
where, the symbol # represents number counting, and the histogram is defined to be the shape context (SC) of pi . This formula means counting the number of boundary points within each sector or bin to form the SC. We use bins that are uniform in log-polar space, making the descriptor more sensitive to positions of nearby sample points than to those points farther away. Fig. 3 shows the process of generating the SC. As can be seen in Fig. 3, the SCs of two neighboring points,
Y. Zhang et al.
log r
260
A B log r
θ
log r
θ
C
θ
Fig. 3. Illustration of computing SCs for a PMS. The circles and lines attached on the PMS are the diagram of log-polar histogram bins used in computing the SCs. We use 5 bins for log-radius and 12 bins for θ. Examples of SCs for three points A, B, and C are shown in the right column (Dark = large value).
A and B, are more similar than the SC of point C which is far away. This is because A and B have similar structures relative to the whole shape. Consider a point pi on the first shape and a point qj on the second shape. Let Cij denotes the cost of matching these two points. As SCs are distributions represented as histograms, it is natural to use the χ2 test statistic: K
Cij = C(pi , qj ) =
1 [hi (k) − hj (k)]2 , 2 [hi (k) + hj (k)]
(3)
k=1
where hi (k) and hj (k) denote the K -bin normalized histogram at pi and qj , respectively. Given the cost matrix with elements Cij between all pairs of points pi on the first shape and qj on the second shape, we want to minimize the total cost of matching, H(π) = C(pi , qπ(i) ). (4) i
Obviously, π(·) is a permutation of pi and qj . This is an instance of the square assignment problem, which can be solved in O(N 3 ) time using the Hungarian method [10]. However, we can use the more efficient algorithm of [11] to reduce the computational cost. The input to the assignment problem is a square cost matrix, and the result is a permutation π(·) so that i Ci,π(i) is minimized. We measure the SC distance between two shapes, P and Q, as the symmetric average of the matching costs over best matching pairs, i.e. 1 1 arg min C(p, π(p)), arg min C(q, π(q))), (5) D(P, Q) = min( Np Nq p∈P
q∈Q
where, Np and Nq are point numbers on each shape, respectively. Given such a dissimilarity measure, we can use different classification techniques to recognize objects.
Gait Recognition Using Procrustes Shape Analysis and Shape Context
5
261
Gait Recognition Using PMS and SC
In the scheme of our work, we first use Procrustes shape analysis to find the PMS of each gait sequence as gait signature for recognition. Then SC is adopted as a similarity measure between one PMS and the others to provide evidences for classification. The block diagram in Fig. 4 summarizes the major steps in a gait recognition system combining PMS and SC.
Gait Sequence
Recognition Result
Motion detection Classification
Unwrapping
Similarity Measure
Procrustes
Shape Analysis Shape Context
Database
Fig. 4. The diagram of gait recognition algorithm using PMS and SC
In Fig. 4, the two steps in the dotted rectangle are the major differences to Wang’s [5] work, and they are also the originality of our work. Gait recognition is a traditional pattern classification problem which can be solved by measuring similarities or dissimilarities among gait sequences. We try three different simple classification methods, namely the nearest neighbor classifier (NN), the k-nearest-neighbor classifier (kNN, k=3), and the nearest neighbor classifier with class exemplar (ENN).
6 6.1
Experimental Results Gait Data
Here, we use dataset-A and dataset-B in CASIA Gait Database [7] to verify the effectiveness of the proposed algorithm. Dataset-A (used to be NLPR database) is quite familiar to researchers in gait recognition. It has 80 sequences belonging to 20 pedestrians at three different viewing angles. We use the sequences with a lateral viewing angle here. Dataset-B is a more large-scale and relatively fresh database, and it has 124 different pedestrians (94 males, 30 females). All the subjects were asked to walk naturally on the concrete ground along a straight line in an indoor environment. Each subject walked along the straight line 10 times (6 for normal walking, 2 for walking with a bag, and 2 for walking with a coat). Some examples of this dataset are shown in Fig. 5. Here, we only use 6 times’ normal walk at lateral view that is 744 sequences. We assume that all silhouettes have been extracted from original human walking sequences. Edge detection and segmentation are then used to the image to
262
Y. Zhang et al.
(a) normal walking
(b) walking with a bag (c) walking with a coat
Fig. 5. Sample images in dataset-B
obtain a clear silhouette boundary of the object. The boundary points are sampled to M points. In our experiments, M equals to 360 just as Wang [5] did in his work. To reduce the computational cost of computing shape context, each PMS is re-sampled to 100 points. 6.2
Classification Performance
We use the leave-one-out cross-validation rule to obtain the unbiased estimate of the Correct Classification Rate (CCR). Each time we leave one sequence out as a probe sample and train on the remainder. The CCRs of the original PMS method [5] and the proposed method on both datasets are reported in Table 1. From this table we can see the superiority of our method compared with the original PMS using MSD. Table 1. The classification performance comparison on two datasets Methods
A-NN
A-kNN A-ENN B -NN
Original PMS [5] 71.25% 72.50% 88.75% PMS + SC 88.75% 81.25% 98.75%
B -kNN B -ENN
88.98% 86.69% 91.13% 94.49% 93.15% 97.18%
Another useful classification performance measure is the rank order statistic, which was first introduced by the FERET protocol for the evaluation of face recognition algorithms [12]. It is defined as the cumulative match scores (CMS) that the real class of a test measurement is among its top k matches. The CMS curves on two datasets are shown in Fig. 6. It is noted that the CCR is equivalent to the score when rank = 1. 6.3
Verification Performance
We also estimate False Acceptance Rate (FAR) and False Reject Rate (FRR) via the leave-one-out rule in terms of verification performance. Equal Error Rate (EER) demonstrates the degree of balance between FAR and FRR numerically. The smaller the values of ERRs the better is the performance. The EERs of original PMS using MSD and the proposed method on two datasets are reported in Table 2.
Gait Recognition Using Procrustes Shape Analysis and Shape Context 1
1
0.95
0.98 Cumulative Match Score
Cumulative Match Score
0.9 0.85 0.8 0.75 0.7
Original PMS on A, NN PMS + SC on A, NN Original PMS on A, ENN PMS + SC on A, ENN
0.65 0.6
2
4
6
8
10 12 Rank
14
16
18
(a) CMS curves on dataset-A
0.96 0.94 0.92 0.9 Original PMS on B, NN PMS + SC on B, NN Original PMS on B, ENN PMS + SC on B, ENN
0.88
20
263
0.86
2
4
6
8
10 12 Rank
14
16
18
20
(b) CMS curves on dataset-B
Fig. 6. CMS comparison on two datasets Table 2. The verification performance comparison on two datasets Methods
A-EER B -EER
Original PMS [5] 0.1994 PMS + SC 0.1324
0.1282 0.1081
Table 3. Comparison of recent algorithms on dataset-A Methods
Top1
Top5
Top10
BenAbdelkader [13] 82.50% 93.75% 100.0% Lee [2] 87.50% 98.75% 100.0% Wang [5] 88.75% 96.25% 100.0% Chen [14] 91.25% 96.25% 100.0% Our method 98.75% 100.0% 100.0%
6.4
Comparisons
We also compare the performance of the proposed algorithm with other famous algorithms using the same silhouette data from the dataset-A with a lateral viewing angle. Based on the FERET protocol with rank of 1, 5, and 10, the best results of the algorithms are reported in Table 3, from which we can see that our method compares favorably with others.
7
Discussion
Plenty of experimental results have shown that SC is a rich and powerful shape descriptor for shape matching and object recognition. One may wonder why not directly use SC to represent the contour that the pedestrians generate while walking and obtain similarity measure between one pedestrian and another. That
264
Y. Zhang et al.
is because the processes of finding correspondence and best matching pairs are highly time consuming. It is not an applicable method for a practicable system, especially in the domain of gait recognition which generally needs batch processing. Anyway, Chen and his group [14] have once tried to represent gait sequence by four key stances chosen from one walking cycle. Then they employed SC to generate gait features from those key stances. Their experimental results are listed in Table 3 for comparison. As many work have proved [5], PMS is a excellent way to make a compact representation to gait sequences. PMS uses a single complex vector to represent the structural characteristics of a whole sequence with frame number varying from dozens to hundreds, yet without losing any useful information. This compact characterization makes it possible to take advantage of the descriptive ability of SC just as we did in this paper. On the other hand, SC is a rich and powerful shape descriptor and offers us a global discriminative features. Moreover, SC leads to a robust score for measuring shape similarity to distinguish different objects. By taking advantage of the active points of those two methods, we obtain favorable performance relative to the existing algorithms.
8
Conclusion
In this paper, a novel gait recognition algorithm is exhibited by combining two kinds of shape descriptors, Procrustes shape analysis and shape context descriptor. We take advantage of the compressed representation characteristic of Procrustes shape analysis to represent the continuously pose changing of pedestrian gait sequences. The mean shape of a set of complex vectors is adopted as gait signature for recognition. The shape context, which is a rich and effective shape descriptor, is employed to describe PMSs and to offer a similarity measure of different shapes instead of using MSD. The computational cost of the matching process is largely decreased, and the discriminating power of the PMS is also re-exploited to an encouraging level. Experiments on both small size and large scale datasets have demonstrated the effectiveness and superiority of the proposed algorithm. Acknowledgments. This work is supported by the National Natural Science Foundation of China under grant NO.60675024. Many thanks to Tingting Guo from School of Foreign Languages and Literature, Shandong University for linguistic advice. Portions of the research in this paper use the CASIA Gait Database collected by Institute of Automation, Chinese Academy of Sciences.
References 1. Bhanu, B., Han, J.: Individual Recognition by Kinematic-based Gait Analysis. In: 16th International Conference on Pattern Recognition, pp. 343–346. IEEE Computer Society, Quebec (2002)
Gait Recognition Using Procrustes Shape Analysis and Shape Context
265
2. Lee, L., Grimson, W.E.L.: Gait Analysis for Recognition and Classification. In: International Conference on Automatic Face and Gesture Recognition, pp. 148– 155. IEEE Computer Society, Washington (2002) 3. Sarkar, S., Phillips, P.J., Liu, Z., Vega, I.R., Grother, P., Bowyer, K.W.: The HumanID Gait Challenge Problem: Data Sets, Performance, and Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 27, 162–177 (2005) 4. Han, J., Bhanu, B.: Individual Recognition Using Gait Energy Image. IEEE Trans. Pattern Anal. Mach. Intell. 28, 316–322 (2006) 5. Wang, L., Tan, T., Hu, W., Ning, H.: Automatic Gait Recognition Based on Statistical Shape Analysis. IEEE Trans. Image Process. 12, 1120–1131 (2003) 6. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24, 509–522 (2002) 7. CASIA Gait Database, http://www.sinobiometrics.com 8. Veltkamp, R.C., Latecki, L.J.: Properties and Performance of Shape Similarity Measures. In: IFCS 2006 Conference: Data Science and Classification, pp. 47–56. Springer, Berlin (2006) 9. Kent, J.T.: New Directions in Shape Analysis. Art of Statistical Science: A Tribute to G. S. Watson, pp. 115–127. Wiley, New York (1992) 10. Papadimitriou, C.H., Steiglitz, K.: Combinatorial optimization: algorithms and complexity. Prentice-Hall, Englewood Cliffs (1982) 11. Jonker, J., Volgenant, A.: A Shortest Augmenting Path Algorithm for Dense and Sparse Linear Assignment Problems. Computing 38, 325–440 (1987) 12. Phillips, P.J., Moon, H., Rizvi, S.A., Rauss, P.J.: The FERET Evaluation Methodology for Face Recognition Algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1090–1104 (2000) 13. BenAbdelkader, C., Cutler, R., Davis, L.: Motion-based recognition of people in EigenGait space. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 267–272. IEEE Computer Society, Washington (2002) 14. Chen, S., Ma, T., Huang, W., Gao, Y.: Gait Recognition Based on Shape Context Descriptor (in Chinese). Chinese journal of Pattern Recognition and Artificial Intelligence 20, 794–799 (2007)
School of Information Science and Engineering, Shandong University, 27 Shanda Nanlu, 250100 Jinan, China [email protected], [email protected], [email protected], [email protected] 2 Institute of Information Science, Beijing Jiaotong University, No.3 Shangyuan Residence Haidian District, 100044 Beijing, China [email protected]
Abstract. This paper proposes a novel algorithm for individual recognition by gait. The method of Procrustes shape analysis is used to produce Procrustes Mean Shape (PMS) as a compressed representation of gait sequence. PMS is adopted as the gait signature in this paper. Instead of using the Procrustes mean shape distance as a similarity measure, we introduce shape context descriptor to measure the similarity between two PMSs. Shape context describes a distribution of all boundary points on a shape with respect to any single boundary point by a histogram of log-polar plot, and offers us a global discriminative characterization of the shape. Standard pattern recognition techniques are used to classify different patterns. The experiments on CASIA Gait Database demonstrate that the proposed method outperforms other algorithms in both classification performance and verification performance. Keywords: Gait recognition, Procrustes shape analysis, shape context descriptor, Procrustes Mean Shape (PMS).
1
Introduction
Among various biometrics like face, iris and fingerprint, gait is a more attractive biometric feature for human identification. Gait signals can be detected from a long distance and measured at low resolution. Therefore, gait can be used in such situations that face or iris information is not available in high enough resolution for recognition. From the perspective of surveillance, gait is a particularly attractive modality. Recently, the study of gait recognition, which concerns recognizing individuals by the way they walk, has received an increasing interest from researchers in the computer vision community. Many contributions have been made to this rapidly developing domain. Gait recognition techniques mainly fall into two categories namely model-based and model-free approaches. The model-based approaches usually model the human
Corresponding author.
H. Zha, R.-i. Taniguchi, and S. Maybank (Eds.): ACCV 2009, Part III, LNCS 5996, pp. 256–265, 2010. c Springer-Verlag Berlin Heidelberg 2010
Gait Recognition Using Procrustes Shape Analysis and Shape Context
257
body structure and extract image features to map them into the structural components of models or to derive motion trajectories of body parts. Bhanu and Han [1] proposed a kinematic-based method to identify individuals. It estimates 3D human walking parameters by performing a least square fit of the 3D kinematic model to the 2D silhouette images. A genetic algorithm is then used for feature selection. The advantage of model-based approaches is that models can handle occlusion and noise better and offer the ability to derive gait features directly from model parameters. They also help to reduce the dimensionality needed to represent the data. However, they suffer from high computational costs. The majority of current approaches are model-free. They typically analyze the image sequence by motion or shape and characterize the whole motion pattern of the human body by a compact representation regardless of the underlying structure. Based on body shape and gait, Lee et al. [2] described a momentbased representation of gait appearance for the purpose of person identification. Sarkar et al. [3] proposed a baseline algorithm for human identification using spatiotemporal correlation of silhouette images. Han and Bhanu [4] proposed a spatiotemporal gait representation called Gait Energy Image to characterize human walking properties. Wang et al. [5] employed a compressed representation of gait sequences and obtained encouraging classification performance. Inspired by Wang [5]’s work, we made further exploration to it in this paper. We use Procrustes shape analysis to produce Procrustes Mean Shape (PMS) of each gait sequence in the database. Quite different and novel, instead of using the Procrustes Mean Shape Distance (MSD) as Wang did, we introduce Shape Context (SC) descriptor [6] to measure the similarity of two PMSs. The experiments on dataset-A and dataset-B in CASIA Gait Database [7] show that SC is an efficient and powerful shape descriptor for similarity measure. We gain favorable performance after combing PMS and SC.
2
Related Work
Intuitively, recognizing people through gait depends greatly on how the silhouette shape of an individual changes over time. Shape is an important cue as it captures a prominent element of an object. Therefore, gait may be considered to be composed of a set of static poses and their temporal variations can be analyzed to obtain distinguishable signatures. Based upon the above consideration, Wang et al. [5] depicted the human shape using the method of Procrustes shape analysis. Pose changes of segmented silhouettes over time are represented as an associated sequence of complex configurations in a two-dimensional (2D) shape space and are further analyzed by the Procrustes shape analysis method to obtain an eigenshape as gait signature. In the field of shape matching and shape similarity measuring, several shape descriptors have been proposed, ranging from moments and Fourier descriptors to Hausdorff distance and the medial axis transform. For a detailed discussion of shape matching techniques, the reader is referred to paper [8]. Belongie and Malik [6] firstly introduced the idea of shape context descriptor. Shape context
258
Y. Zhang et al.
describe a distribution of the remaining boundary points with respect to one point on the boundary. Histogram of a log-polar plot of the shape boundary gives the shape context for that particular boundary point, thus offering a globally discriminative characterization. Their work showed remarkable results involving character recognition using gimpy images. It also had an extensive testing for silhouette images from trademarks, handwritten digits and the COIL dataset.
3
Procrustes Shape Analysis
Procrustes shape analysis [9] is a popular method in directional statistics. It is intended for coping with 2D shapes and provides a good method to find mean shapes. To reduce redundant information, the shape boundary can be easily obtained and stored using a border following algorithm based on connectivity. As we can see in Fig. 1, (xi , yi ) is a random pixel on the boundary. Let the centroid (xc , yc ) of the shape be the origin of the 2D shape space. The two axes Re and Im represent the real and imaginary part of a complex number, respectively. Im
Unwrapping by contour following
( xi , y i )
O
( xC , yC )
Re
Fig. 1. Illustration of computing PMS
As shown in Fig. 1, we can then unwrap each shape anticlockwise into a set of boundary pixel points sampled along its outer-contour in a common complex coordinate system. Each shape can be described as a vector of ordered complex numbers with M elements, u = [u1 , u2 , . . . , uM ]T , ui = (xi − xc ) + j × (yi − yc ). Therefore, each gait sequence will be accordingly converted into an associated sequence of such 2D shape configurations. Given a set of m shapes, we can find their mean shape uˆ by computing the following matrix: Su =
m ui u∗ i
i=1
u∗i ui
,
(1)
where, the superscript ∗ represents the complex conjugation transposition. uˆ is the so-called Procrustes Mean Shape (PMS) which is the dominant eigenvector
Gait Recognition Using Procrustes Shape Analysis and Shape Context 0.1
259
0.1 Sequence 1 Sequence 2 Sequence 3 Sequence 4 Examplar
0.08 0.06
0.06
0.04
0.04
0.02
0.02
0
0
−0.02
−0.02
−0.04
−0.04
−0.06
Sequence 1 Sequence 2 Sequence 3 Sequence 4 Examplar
0.08
−0.06 −0.03 −0.02 −0.01
0
0.01
0.02
0.03
0.04
(a) subject #1 in dataset-A
0.05
−0.03 −0.02 −0.01
0
0.01
0.02
0.03
0.04
0.05
(b) subject #2 in dataset-A
Fig. 2. Two examples of the PMS in dataset-A. Note that the sample points in the PMS depicted here are re-sampled to a visible level for reading convenience.
of Su , i.e., the eigenvector that corresponds to the greatest eigenvalue of Su . Two examples of PMSs are shown in Fig. 2. To compare two different PMSs, the Procrustes Mean Shape Distance (MSD) was used by Wang [5] to measure the similarity between them. Instead of using the MSD, this paper treats the similarity measure totally in a different way as described below.
4
Shape Context
The concept of shape context was originally introduced to measure similarities between shapes and recognize objects. Similar to PMS, a shape is represented by a discrete set of N points, P = {p1 , p2 , . . . , pN }, sampled from the internal or external contours on the object. Assuming contours are piecewise smooth, we can obtain as good an approximation to the underlying continuous shapes as desired by picking N to be sufficiently large. Note that, the point number N here can be either equal to the number M in PMS or not. Consider the set of vectors originating from a random point pi to all other points on the shape. These vectors actually express the configuration of the entire shape relative to the reference point. For the point pi , we compute a coarse histogram hi relative to the coordinates of the remaining N − 1 points, hi (k) = #{q = pi : (q − pi ) ∈ bin(k)},
(2)
where, the symbol # represents number counting, and the histogram is defined to be the shape context (SC) of pi . This formula means counting the number of boundary points within each sector or bin to form the SC. We use bins that are uniform in log-polar space, making the descriptor more sensitive to positions of nearby sample points than to those points farther away. Fig. 3 shows the process of generating the SC. As can be seen in Fig. 3, the SCs of two neighboring points,
Y. Zhang et al.
log r
260
A B log r
θ
log r
θ
C
θ
Fig. 3. Illustration of computing SCs for a PMS. The circles and lines attached on the PMS are the diagram of log-polar histogram bins used in computing the SCs. We use 5 bins for log-radius and 12 bins for θ. Examples of SCs for three points A, B, and C are shown in the right column (Dark = large value).
A and B, are more similar than the SC of point C which is far away. This is because A and B have similar structures relative to the whole shape. Consider a point pi on the first shape and a point qj on the second shape. Let Cij denotes the cost of matching these two points. As SCs are distributions represented as histograms, it is natural to use the χ2 test statistic: K
Cij = C(pi , qj ) =
1 [hi (k) − hj (k)]2 , 2 [hi (k) + hj (k)]
(3)
k=1
where hi (k) and hj (k) denote the K -bin normalized histogram at pi and qj , respectively. Given the cost matrix with elements Cij between all pairs of points pi on the first shape and qj on the second shape, we want to minimize the total cost of matching, H(π) = C(pi , qπ(i) ). (4) i
Obviously, π(·) is a permutation of pi and qj . This is an instance of the square assignment problem, which can be solved in O(N 3 ) time using the Hungarian method [10]. However, we can use the more efficient algorithm of [11] to reduce the computational cost. The input to the assignment problem is a square cost matrix, and the result is a permutation π(·) so that i Ci,π(i) is minimized. We measure the SC distance between two shapes, P and Q, as the symmetric average of the matching costs over best matching pairs, i.e. 1 1 arg min C(p, π(p)), arg min C(q, π(q))), (5) D(P, Q) = min( Np Nq p∈P
q∈Q
where, Np and Nq are point numbers on each shape, respectively. Given such a dissimilarity measure, we can use different classification techniques to recognize objects.
Gait Recognition Using Procrustes Shape Analysis and Shape Context
5
261
Gait Recognition Using PMS and SC
In the scheme of our work, we first use Procrustes shape analysis to find the PMS of each gait sequence as gait signature for recognition. Then SC is adopted as a similarity measure between one PMS and the others to provide evidences for classification. The block diagram in Fig. 4 summarizes the major steps in a gait recognition system combining PMS and SC.
Gait Sequence
Recognition Result
Motion detection Classification
Unwrapping
Similarity Measure
Procrustes
Shape Analysis Shape Context
Database
Fig. 4. The diagram of gait recognition algorithm using PMS and SC
In Fig. 4, the two steps in the dotted rectangle are the major differences to Wang’s [5] work, and they are also the originality of our work. Gait recognition is a traditional pattern classification problem which can be solved by measuring similarities or dissimilarities among gait sequences. We try three different simple classification methods, namely the nearest neighbor classifier (NN), the k-nearest-neighbor classifier (kNN, k=3), and the nearest neighbor classifier with class exemplar (ENN).
6 6.1
Experimental Results Gait Data
Here, we use dataset-A and dataset-B in CASIA Gait Database [7] to verify the effectiveness of the proposed algorithm. Dataset-A (used to be NLPR database) is quite familiar to researchers in gait recognition. It has 80 sequences belonging to 20 pedestrians at three different viewing angles. We use the sequences with a lateral viewing angle here. Dataset-B is a more large-scale and relatively fresh database, and it has 124 different pedestrians (94 males, 30 females). All the subjects were asked to walk naturally on the concrete ground along a straight line in an indoor environment. Each subject walked along the straight line 10 times (6 for normal walking, 2 for walking with a bag, and 2 for walking with a coat). Some examples of this dataset are shown in Fig. 5. Here, we only use 6 times’ normal walk at lateral view that is 744 sequences. We assume that all silhouettes have been extracted from original human walking sequences. Edge detection and segmentation are then used to the image to
262
Y. Zhang et al.
(a) normal walking
(b) walking with a bag (c) walking with a coat
Fig. 5. Sample images in dataset-B
obtain a clear silhouette boundary of the object. The boundary points are sampled to M points. In our experiments, M equals to 360 just as Wang [5] did in his work. To reduce the computational cost of computing shape context, each PMS is re-sampled to 100 points. 6.2
Classification Performance
We use the leave-one-out cross-validation rule to obtain the unbiased estimate of the Correct Classification Rate (CCR). Each time we leave one sequence out as a probe sample and train on the remainder. The CCRs of the original PMS method [5] and the proposed method on both datasets are reported in Table 1. From this table we can see the superiority of our method compared with the original PMS using MSD. Table 1. The classification performance comparison on two datasets Methods
A-NN
A-kNN A-ENN B -NN
Original PMS [5] 71.25% 72.50% 88.75% PMS + SC 88.75% 81.25% 98.75%
B -kNN B -ENN
88.98% 86.69% 91.13% 94.49% 93.15% 97.18%
Another useful classification performance measure is the rank order statistic, which was first introduced by the FERET protocol for the evaluation of face recognition algorithms [12]. It is defined as the cumulative match scores (CMS) that the real class of a test measurement is among its top k matches. The CMS curves on two datasets are shown in Fig. 6. It is noted that the CCR is equivalent to the score when rank = 1. 6.3
Verification Performance
We also estimate False Acceptance Rate (FAR) and False Reject Rate (FRR) via the leave-one-out rule in terms of verification performance. Equal Error Rate (EER) demonstrates the degree of balance between FAR and FRR numerically. The smaller the values of ERRs the better is the performance. The EERs of original PMS using MSD and the proposed method on two datasets are reported in Table 2.
Gait Recognition Using Procrustes Shape Analysis and Shape Context 1
1
0.95
0.98 Cumulative Match Score
Cumulative Match Score
0.9 0.85 0.8 0.75 0.7
Original PMS on A, NN PMS + SC on A, NN Original PMS on A, ENN PMS + SC on A, ENN
0.65 0.6
2
4
6
8
10 12 Rank
14
16
18
(a) CMS curves on dataset-A
0.96 0.94 0.92 0.9 Original PMS on B, NN PMS + SC on B, NN Original PMS on B, ENN PMS + SC on B, ENN
0.88
20
263
0.86
2
4
6
8
10 12 Rank
14
16
18
20
(b) CMS curves on dataset-B
Fig. 6. CMS comparison on two datasets Table 2. The verification performance comparison on two datasets Methods
A-EER B -EER
Original PMS [5] 0.1994 PMS + SC 0.1324
0.1282 0.1081
Table 3. Comparison of recent algorithms on dataset-A Methods
Top1
Top5
Top10
BenAbdelkader [13] 82.50% 93.75% 100.0% Lee [2] 87.50% 98.75% 100.0% Wang [5] 88.75% 96.25% 100.0% Chen [14] 91.25% 96.25% 100.0% Our method 98.75% 100.0% 100.0%
6.4
Comparisons
We also compare the performance of the proposed algorithm with other famous algorithms using the same silhouette data from the dataset-A with a lateral viewing angle. Based on the FERET protocol with rank of 1, 5, and 10, the best results of the algorithms are reported in Table 3, from which we can see that our method compares favorably with others.
7
Discussion
Plenty of experimental results have shown that SC is a rich and powerful shape descriptor for shape matching and object recognition. One may wonder why not directly use SC to represent the contour that the pedestrians generate while walking and obtain similarity measure between one pedestrian and another. That
264
Y. Zhang et al.
is because the processes of finding correspondence and best matching pairs are highly time consuming. It is not an applicable method for a practicable system, especially in the domain of gait recognition which generally needs batch processing. Anyway, Chen and his group [14] have once tried to represent gait sequence by four key stances chosen from one walking cycle. Then they employed SC to generate gait features from those key stances. Their experimental results are listed in Table 3 for comparison. As many work have proved [5], PMS is a excellent way to make a compact representation to gait sequences. PMS uses a single complex vector to represent the structural characteristics of a whole sequence with frame number varying from dozens to hundreds, yet without losing any useful information. This compact characterization makes it possible to take advantage of the descriptive ability of SC just as we did in this paper. On the other hand, SC is a rich and powerful shape descriptor and offers us a global discriminative features. Moreover, SC leads to a robust score for measuring shape similarity to distinguish different objects. By taking advantage of the active points of those two methods, we obtain favorable performance relative to the existing algorithms.
8
Conclusion
In this paper, a novel gait recognition algorithm is exhibited by combining two kinds of shape descriptors, Procrustes shape analysis and shape context descriptor. We take advantage of the compressed representation characteristic of Procrustes shape analysis to represent the continuously pose changing of pedestrian gait sequences. The mean shape of a set of complex vectors is adopted as gait signature for recognition. The shape context, which is a rich and effective shape descriptor, is employed to describe PMSs and to offer a similarity measure of different shapes instead of using MSD. The computational cost of the matching process is largely decreased, and the discriminating power of the PMS is also re-exploited to an encouraging level. Experiments on both small size and large scale datasets have demonstrated the effectiveness and superiority of the proposed algorithm. Acknowledgments. This work is supported by the National Natural Science Foundation of China under grant NO.60675024. Many thanks to Tingting Guo from School of Foreign Languages and Literature, Shandong University for linguistic advice. Portions of the research in this paper use the CASIA Gait Database collected by Institute of Automation, Chinese Academy of Sciences.
References 1. Bhanu, B., Han, J.: Individual Recognition by Kinematic-based Gait Analysis. In: 16th International Conference on Pattern Recognition, pp. 343–346. IEEE Computer Society, Quebec (2002)
Gait Recognition Using Procrustes Shape Analysis and Shape Context
265
2. Lee, L., Grimson, W.E.L.: Gait Analysis for Recognition and Classification. In: International Conference on Automatic Face and Gesture Recognition, pp. 148– 155. IEEE Computer Society, Washington (2002) 3. Sarkar, S., Phillips, P.J., Liu, Z., Vega, I.R., Grother, P., Bowyer, K.W.: The HumanID Gait Challenge Problem: Data Sets, Performance, and Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 27, 162–177 (2005) 4. Han, J., Bhanu, B.: Individual Recognition Using Gait Energy Image. IEEE Trans. Pattern Anal. Mach. Intell. 28, 316–322 (2006) 5. Wang, L., Tan, T., Hu, W., Ning, H.: Automatic Gait Recognition Based on Statistical Shape Analysis. IEEE Trans. Image Process. 12, 1120–1131 (2003) 6. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24, 509–522 (2002) 7. CASIA Gait Database, http://www.sinobiometrics.com 8. Veltkamp, R.C., Latecki, L.J.: Properties and Performance of Shape Similarity Measures. In: IFCS 2006 Conference: Data Science and Classification, pp. 47–56. Springer, Berlin (2006) 9. Kent, J.T.: New Directions in Shape Analysis. Art of Statistical Science: A Tribute to G. S. Watson, pp. 115–127. Wiley, New York (1992) 10. Papadimitriou, C.H., Steiglitz, K.: Combinatorial optimization: algorithms and complexity. Prentice-Hall, Englewood Cliffs (1982) 11. Jonker, J., Volgenant, A.: A Shortest Augmenting Path Algorithm for Dense and Sparse Linear Assignment Problems. Computing 38, 325–440 (1987) 12. Phillips, P.J., Moon, H., Rizvi, S.A., Rauss, P.J.: The FERET Evaluation Methodology for Face Recognition Algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1090–1104 (2000) 13. BenAbdelkader, C., Cutler, R., Davis, L.: Motion-based recognition of people in EigenGait space. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 267–272. IEEE Computer Society, Washington (2002) 14. Chen, S., Ma, T., Huang, W., Gao, Y.: Gait Recognition Based on Shape Context Descriptor (in Chinese). Chinese journal of Pattern Recognition and Artificial Intelligence 20, 794–799 (2007)
