Robust Texture Classification by Subsets of Local Binary Patterns Mäenpää Topi, Ojala Timo, Pietikäinen Matti, Soriano Maricor Machine Vision and Media Processing Unit Infotech Oulu and Department of Electrical Engineering P.O.Box 4500, FIN-90014 University of Oulu, Finland {topiolli, skidi, mkp, msoriano}@ee.oulu.fi Abstract Recently, a nonparametric approach to texture analysis has been developed, in which the distributions of simple texture measures based on local binary patterns (LBP) are used for texture description. The basic LBP encodes 256 simple feature detectors in a single 3x3 operator. This paper shows that a properly selected subset of patterns encoded in LBP forms an efficient and robust texture description which can achieve better classification rates in comparison with the whole LBP histogram. Experiments on classification of textures from the Columbia-Utrecht (CURET) database demonstrate the robustness of the approach.
1. Introduction Recently, a nonparametric approach to texture analysis has been developed in which the distributions of simple texture measures based on Local Binary Patterns (LBP) are used for texture description. Very good performance has been obtained in various texture classification and segmentation problems [3,4]. LBP encodes 256 simple feature detectors (e.g. edge, curve/line, corner, curve/line end, spot) at different orientations in a single operator, and the LBP histogram contains the density of each feature over a region. An important point is that, for each pixel, the output of only the best-matching feature detector is utilized. A conventional texture analysis method based on spatial filtering (e.g., Laws’ masks) creates a feature vector for each pixel containing information from both relevant and irrelevant local pattern matches, making this kind of approach less efficient. This paper shows that a major part of the discriminative power of LBP lies in a small subset of patterns encoded in it. The best pattern subsets are determined in two ways presented in Section 2. These subsets of local patterns are found to perform much better than the whole LBP histogram in classification problems involving geometric transformation (tilt) between training and testing images. Most texture classifiers are trained with texture surfaces under some standard position, e.g. surface normal parallel with the camera axis, light source and camera at fixed posi-
tions. However, natural texture facets are 3-D. Hence any deviation from the standard position can cause the texture appearance to change. We have evaluated several texture classifiers trained with textures at the standard position and the same textures with a small tilt from the camera axis as test. We observed that the classification performance deteriorates dramatically if no preprocessing is made to correct for geometric distortions and brightness change associated with the tilting. This paper demonstrates that LBP histograms derived from standard position textures can be used to find the subset of LBP's which is optimal for classifying tilted textures even without preprocessing. Section 3 discusses our experimental results with 16 textures from the Columbia-Utrecht (CURET) database [1]. Finally, Section 4 concludes the paper.
2. LBP operators 2.1. Basic LBP Ojala et al. [3] introduced the Local Binary Pattern (LBP) texture operator shown in Fig. 1. The original 3x3 neighborhood is thresholded by the value of the center pixel. The values of the pixels in the thresholded neighborhood are multiplied by the weights given to the corresponding pixels. Finally, the values of the eight pixels are summed to obtain the LBP number for this neighborhood. In rotation invariant classification, it is advantageous to interpolate the values of the corner pixels in order to obtain a circular sampling of the neighborhood. LBP is by definition invariant against any monotonic gray scale transformation, i.e. as long as the order of pixel values stays the same, the output of the LBP operator remains constant [7]. The 256-bin LBP histogram computed over a region is used for texture description. This histogram characterizes occurrence statistics of simple texture primitives, as each local pattern describes a distinct texture primitive. In our experiments, a log-likelihood measure L(S,M) was used to measure the dissimilarity between sample and model histograms:
L ( S, M ) = –
∑nN = 1 Sn ln M n
(1)
where N is the total number of histogram bins, and Sn and Mn correspond to the sample and model probabilities of bin n, respectively. example
thresholded
6
5
2
1
7
6
1
1
9
8
7
1
0
1
weights
0
1
0
128
1
2
The classification of the optimization set within each iteration was made using a reduced LBP histogram that contained one bin for each pattern chosen so far. The contributions from all the remaining patterns (out of 256) were combined into a single bin. After a predefined number (D) of iterations, the set S contains B distinct pattern sets, from which we must choose the optimal patterns.
4 8
64 32 16
Pattern = 11110001 LBP = 1 + 16 +32 + 64 + 128 = 241 Figure 1. Calculation of the LBP
2.2. LBP subsets Considering the texture primitive detector interpretation of LBP, it is obvious that not all 256 encoded patterns are equally important. Some patterns should remain more stable under geometric image transformations and thus perform better in classification. Searching for the most significant patterns can be carried out in a number of ways. We considered an analytical and a heuristics approach. 2.2.1. Method 1 - beam search Finding the optimal subset of patterns can be a very time consuming operation. If we were to select a pattern set of D or less patterns from the total 256 patterns, we would end up with F different sets, with F defined by: F =
∑iD= 1 -------------------------( 256 – i )!i! 256!
tion set for each s i' ∈ S' • Set L = L + S’ • If d = 1 then break the b loop • End b loop • Sort L according to the obtained classification errors, discarding duplicate li • Set S ={li | i = 1, ..., B} • End d loop
(2)
It can be quite easily seen that finding the optimal subset of patterns is beyond the capabilities of any practical implementation, even for moderate values of D. Therefore, we must accept a good (suboptimal) subset instead of the optimal one. We have adopted a beam search for the task. In this method, we iteratively increase the size of the pattern set up to dimension D and always consider the best B pattern sets produced so far. In the following, a set of pattern sets is denoted by a capital letter (e.g. S), and each pattern set in this set by s i, i = 1, …, S , where |S| denotes the number of items in set S. The set of all patterns is denoted by a. The procedure is as follows: • Create set S so that |S| = 1 and |s1| = 0 • For dimension d = 1 to D • Create set L so that |L| = 0 • For pattern set index b = 1 to B • Construct S’, the set of all possible pattern sets si’, by appending each pattern in a-sb to sb in turn • Obtain the classification errors against the given optimiza-
2.2.2. Method 2 - uniform patterns As a general purpose alternative to the beam search, we propose using a specific set of uniform local binary patterns. The advantage in using a predefined set of patterns is that the time consuming search procedure is not needed. First, we define a measure of nonuniformity U(LBP), which corresponds to the number of transitions in the circular bitwise presentation of the LBP code. For example, patterns 0 (bitwise 00000000) and 255 (bitwise 11111111) have a U value of 0, while patterns 1, 2, 4, 8, 16, 32, 64 and 128 (bitwise 00000001, 00000010, etc.) have a U value of 2, as there are exactly two 0/1 or 1/0 transitions in the bitwise presentation. Similarly, patterns 00000011, 00000111, 00001111, 00011111, 00111111, 01111111, and their circularly bitwise rotated versions have a U value of 2. Other patterns have a U value of at least 4. We argue that the smaller number of spatial transitions (0/1 or 1/0 changes in the circular bitwise presentation of the LBP code) there is in the pattern, i.e. the smaller the nonuniformity measure is, the less likely the pattern is to undergo unwanted changes upon a geometric transformation such as tilting or rotation. Based on this, we propose using those nine uniform patterns that have a U value of at most 2 (00000000, 00000001, 00000011, 00000111, 00001111, 00011111, 00111111, 01111111, 11111111) and their circularly rotated versions for transformation invariant analysis. These nine patterns correspond to 58 of the 256 original unrotated patterns that can occur in the 3x3 neighborhood. Remaining patterns are accumulated into a single bin, resulting in a 59-bin histogram. Using only 58/256 of the pattern information may appear as a waste of information, but this approximation is supported by a very important observation. Namely, the chosen nine uniform patterns seem to contribute most of the spatial patterns present in deterministic microtextures. For example, in a particular set of many Brodatz textures [6] used in our recent study, the nine patterns contributed up to 94% of
the spatial patterns present in the image, averaging 91% [7]. In that study, we also observed that this set of nine uniform patterns sustains rotation extremely well, producing excellent results in three different problems of rotation invariant texture classification. In this study, we analyze their invariance with respect to tilting.
3. Experiments with CURET textures In the experiments, 16 CURET textures shown in Fig. 2 were used [1]. Two versions of the textures were considered: one at 0o tilt (standard position) and the other at 11.25o tilt. Two types of experiments were carried out. First, the textures at standard position were used as testing data to get an impression of how the tilting affects the classification accuracy. Second, the tilted textures were used as optimization and validation data in searching for a subset of LBP patterns that is robust against tilts.
3.1. Nontilted reference In the first experiment, the texture samples in standard position were used for both training and testing. The standard position images were divided into the maximum possible number of non-overlapping 64-by-64 pixel sub-images, half of which were used for training, the other half serving as testing data. In cases, where there was an odd number of subimages, the training set contained the “additional” sample. The average error rate over the validation sets using the whole LBP histogram was 1.48%. With uniform patterns, the accuracy was a little worse, 1.85%.
3.2. Optimization using tilted textures To see the effect of tilting on the classification accuracy, another experiment was carried out. This time the LBP histogram of the whole texture in the standard position was used as a training example, and the maximum number of non-overlapping 64-by-64 pixel subimages was taken from the tilted textures. Half of the subimages were used in optimizing the pattern set, whereas the other half was reserved for validation. In textures with an odd number of sub-windows, the validation set contained one more sample. The classification accuracy over the validation set was calculated using the whole LBP histogram and uniform patterns. The resulting error rates were 11.45% and 11.83%, respectively, showing that no improvement was obtained using the uniform patterns subset. Hence, it seems that uniform patterns do not survive the tilting better than any other patterns. In order to find a subset of local binary patterns that are robust against tilts (in CURET textures), we applied the
beam search procedure with B=6 using the tilted textures as optimization and validation data. The error rate over the optimization set seemed to saturate after 15 iterations to below 3%. After the 15th iteration, the remaining six pattern sets were evaluated against the validation set. The average error rate produced by these sets was 8.21%, which is 3.24 percentage units better than that obtained with the whole histogram. The final pattern set was chosen to include those patterns that occured in at least five of these six sets. There were 14 such patterns (0, 4, 15, 28, 31, 49, 107, 128, 135, 147, 156, 157, 223 and 253), most of which are uniform. This pattern set produced an error rate of 8.40% over the validation set. Table 2 summarizes the results. Table 1. Summary of the error rates in classifying CURET textures. Case
Average error rate
Nontilted, whole histogram
1.48%
Nontilted, uniform patterns Tilted, whole histogram Tilted, uniform patterns Tilted, average over six different 15-pattern sets Tilted, the best 14 patterns
1.85% 11.45% 11.83% 8.21% 8.40%
4. Discussion and conclusions The LBP operator has shown its performance in many problems, but it is still possible to make it even better. Our results show that a major part of the discriminative power of LBP is in a small subset of patterns encoded in LBP. We demonstrated that a small subset of local patterns can perform better than the whole LBP histogram in problems involving geometric transformation (tilt) between training and testing images. A similar conclusion was made in our recent study on rotation invariant texture classification, in which uniform patterns at different orientations were combined [7]. This kind of LBP operator, jointly with a local variance measure, outperformed the existing rotation-invariant texture measures for the same test images that were used in recent papers [2,5]. Every application has its own “optimal” set of patterns. Uniform patterns appear to perform well in many situations, and according to our recent experiments seem to be a preferred choice for rotation and grayscale invariant classification. Different pattern sets for different transformations could be constructed using either heuristics or the forward search method.
5. Acknowledgements The financial support provided by the Academy of Finland and the Graduate School in Electronics, Telecommunications and Automation are gratefully acknowledged.
6. References [1] K. Dana, S. Nayar, B. van Ginneken and J. Koenderink, “Reflectance and Texture of Real-world Surfaces”, Proc. CVPR, 1997, pp. 151-157. http://www.cs.columbia.edu/CAVE/curet/ [2] G.M. Haley and B.S. Manjunath, “Rotation-Invariant Texture Classification Using a Complete Space-Frequency Model”, IEEE Transactions on Image Processing, vol. 8, pp. 255-269, 1999.
[3] T. Ojala and M. Pietikäinen, “Unsupervised Texture Segmentation Using Feature Distributions”, Pattern Recognition, vol. 32, pp. 477-486, 1999. [4] T. Ojala, M. Pietikäinen, and D. Harwood, “A Comparative Study of Texture Measures with Classification Based on Feature Distributions”, Pattern Recognition, vol. 29, pp. 51-59, 1996. [5] R. Porter and N. Canagarajah, “Robust Rotation-Invariant Texture Classification: Wavelet, Gabor Filter and GMRF Based Schemes”, IEE Proc. Vision, Image, and Signal Processing, vol. 144, pp. 180-188, 1997. [6] P. Brodatz, Textures: A Photographic Album for Artists and Designers. New York: Dover, 1966. [7] T. Ojala, M. Pietikäinen and T. Mäenpää, “Gray Scale and Rotation Invariant Texture Classification with Local Binary Patterns”, Proc. ECCV 2000, in press.
Figure 2. CURET textures. The textures include polyester (02), leather (05), sandpaper (06), rough paper (12), roof shingle (14), cork (16), rug_a (18), rug_b (19), styrofoam (20), quarry tile (25), loofah (26), slate_a (33), slate_b (34), brick_a (37), brick_b (41) and concrete (49).