MULTISPECTRAL FUSION FOR INDOOR AND ... - Semantic Scholar
MULTISPECTRAL FUSION FOR INDOOR AND OUTDOOR FACE AUTHENTICATION H. Chang, M. Yi, H. Harishwaran, B. Abidi, A. Koschan, and M. Abidi Imaging, Robotics and Intelligent Systems (IRIS) Lab The University of Tennessee, Knoxville ABSTRACT Face analysis via multispectral imaging is a relatively unexplored territory in face recognition research. The Multispectral, Multimodal and Multi-illuminant IRIS-M3 database was acquired, indoors and outdoors, to promote research in this direction. In the database, each data record has images spanning all bands in the visible spectrum and one thermal image, acquired under different illumination conditions. The spectral power distributions of the lighting sources and daylight conditions are also encoded in the database. Multispectral fused images show improved face recognition performance compared to the visible monochromatic images. Galleries and probes were selected from the indoor and outdoor sections of the database to study the effects of data and decision fusion in the presence of lighting changes. Our experiments were validated by comparing cumulative match characteristics of monochromatic probes against multispectral probes obtained via multispectral fusion by averaging, principal component analysis, wavelet analysis, illumination adjustment and decision level fusion. In this effort, we demonstrate that spectral bands, either individually or fused by different techniques, provide better face recognition results with up to 78% improvement on conventional visible images.
than visible images alone, due to the fact that MSIs enhance certain features that are relatively insignificant in monochromatic visible images [1]. This improvement over the visible is a much desired feature that we hypothesize/expect to yield better recognition rates in the case of face recognition. The main contribution of this paper is to establish that MSIs, individually or in fusion, outperform monochromatic images. Widely used fusion schemes such as wavelet based and principal component based fusions were implemented to compare against new Illumination Adjustment (IA) based fused images and raw MSIs. Since MSIs were unavailable, we took the initiative of acquiring the multispectral, multimodal and multi-illumined IRIS-M3 database [2] to facilitate research progress (ours and peers) in the area of multispectral face recognition. In order to investigate and compare Face Recognition (FR) performances under various illuminations, the luminance and radiance of the illuminants were also collected as part of the data acquisition process. In Figure 1, we show samples from one data record in the IRISM3 database, collected using the data acquisition equipment shown in Figure 2. With the advances in filter technology, tunable programmable filters were used to acquire MSIs by several research groups [3-8]. A tunable filter allows the selection of narrow
1. INTRODUCTION The needs for robust face recognition systems are extensive in a practical world with pressing issues in identity authentication and recognition. A substantial evolution has been made in face recognition research over the last 20 years, especially with the development of powerful face decomposition methods, like the use of eigenfaces. In spite of the diversity of techniques and face recognition tools designed, evaluated and tested, face recognition is not robust enough to be actually deployed in uncalibrated or unconditional backgrounds. Many of the promising methods use images in the visible spectrum. However, very few researchers have exploited raw or fused Multispectral images (MSIs) to improve face recognition. The potential of multispectral imaging has been achieved in many applications, such as remote sensing, data exploitation, medical and health care, and has shown better performance 1-4244-0487-8/06/$20.00 ©2006 IEEE
(a)
(b)
(c)
(d) (e) (f) Figure 1. Sample images in a data record in the IRIS-M3 database; (a) monochromatic image under indoor halogen light, (b) channel 640nm multispectral image under indoor halogen light, (c) monochromatic image under indoor fluorescent light, (d) monochromatic image under outdoor day light, (e) channel 640nm multispectral image outdoor day light, (f) monochromatic image under a different indoor fluorescent lighting.
2006 Biometrics Symposium
band-pass images at preset times. The continuous tuning of wavelengths provides the capability of finer spectral sampling. The Munsell Color Science Laboratory initiated efforts with multispectral images using a Liquid Crystal Tunable Filter (LCTF) over the visible spectrum, especially for high resolution art portrait reconstruction [3], [4]. Hardeberg et al. [5] imaged non-face, rigid objects over the visible spectrum. However, very few researchers have utilized multispectral image fusion to improve face recognition. Pan et al. [6], [7] acquired spectral images over the near-infrared spectrum (0.7-1 µm) and demonstrated that spectral images of faces acquired in the near infrared range can be used to recognize an individual under different poses and expressions. It is evident from the literature that not much research has been done using MS imaging in the visible domain to address the problem of face recognition, especially with respect to changes in illumination conditions. In this paper, visible or visible monochromatic image refers to an image acquired using a commercial-off-the-shelf (COTS) camera and MSIs refers to the narrow-band spectral images throughout the whole visible spectrum from 0.4-0.72 µm. In Section 2, we give a brief description of the IRIS-M3 face database. In Section 3, the fusion approaches are elucidated and the results of experimental comparisons are shown in Section 4. The conclusion is presented in Section 5. 2. THE IRIS-M3 FACE DATABASE To study the effect of using MSIs for face recognition, we collected the IRIS-M3 face database. The IRIS-M3 imaging system includes a multispectral imaging platform, a digital RGB COTS camera, an infrared camera, a spectrometer, a frame grabber and a processing station. The multimodal imaging system is shown in Figure 2(a). The multispectral platform shown in Figure 2(b) comprises a monochrome camera mechanically conjoined with a software-tuned LCTF. The LCTF provides narrow-band filtering with a 7nm Full Width-at-Half-Maximum bandwidth. The IRIS-M3 database has certain significant characteristics. This is a unique database with registered images in the visible, multispectral and thermal modalities, integrated with spectral distributions of the illuminants employed in acquisition. Another noteworthy asset is the larger set of multispectral bands in our database. Previous databases [8] have provided 16 bands per participant and a total of 7 people, while our database has 25 bands per participant and a total of 82 participants. The database was collected in 10 sessions between August 2005 and March 2006. The 82 participants in our database are of different ethnicities, age groups, facial hair characteristics and genders. Figure 3 shows the demographics of these 82 participants. The database is made up of 76% male and 24% female; the ethnic diversity is defined as a collection of 57%
1-4244-0487-8/06/$20.00 ©2006 IEEE
(a) (b) Figure 2. (a) Multi-modal and (b) multispectral imaging systems. Caucasian Asian Female Male
African Descent Indian
(a)
(b)
Figure 3. Demographics of participants in IRIS-M3. (a) Pie chart of male/female participants and, (b) different ethnicities such as Caucasian, Asian, African Descent and Asian Indian in our database.
Caucasian, 23%Asian (Chinese, Japanese, Korean and similar ethnicity), 12% Asian Indian and 8% of African Descent. We acquired IRIS-M3 face database under halogen, fluorescent and day light with shadows [2]. The three illumination scenarios are shown in Figure 4. The quadruple halogen lights with a pair on each side of the participant are shown in Figure 4(a). The second illumination setup was a pair of fluorescent light panels. In Figure 4(c), an example of the outdoor side illumination is shown. 3. FUSION APPROACHES To show the effects of using and fusing MSIs on face recognition, we have implemented a weighted average, illumination adjustment and wavelet fusion algorithms. Rank-based decision level fusion was also tested. The formulation and implementation details are presented herewith. 3. 1. Weighted fusion The image intensity for each pixel p of a monochrome camera in a certain wavelength range, λmin to λmax , can be represented by the image formation model,
(a)
(b)
(c)
Figure 4. Image acquisition setup (a) halogen lighting setup, (b) fluorescent lighting setup and (c) outdoor acquisition scene.
2006 Biometrics Symposium
p=
λ
max R (λ ) L (λ ) S (λ ) dλ , ∫λmin
(1)
where R is the spectral reflectance of the object, L is the spectral power distribution of the illuminant, and S is the spectral response of the CCD array. For a multispectral imaging system, the camera response, p λi corresponding to band i of N-spectral bands and centered at wavelength λi , can be represented as λi , max
∫λi ,min
p λi =
Rλi (λ ) Lλ i (λ ) Sλ i (λ )Tλ i (λ )dλ ,
i
i
i
i
(3)
The pixel value of the weighted fused image, pw , is then
pw = where
C=
∑
N
i =1
wλ
i
1 C
N
∑ wλi ⋅ pλi ,
under L2 by applying the weight wλi = L2,λi L1,λi , which can be represented as p1→ 2,λi = p1,λi wλi . (7) The intensity of the fused image, p1→ 2 , can be written as p1→ 2 =
(2)
where i = 1,2,L N . Tλ i is the spectral transmittance of the LCTF. The camera response is the result of an integration process which can also be calculated in a discrete form as the summation of samples. Because each spectral image is acquired within a very narrow band, we take only one sample of each factor per band. Therefore, the camera response for a spectral image pixel at λi can be written as p λi = Rλ Lλ Sλ Tλ .
Comparing (6) and (7), we can see that the spectral image at λi under L1 can be transformed to the corresponding image
(4)
i =1
. The weights are influenced by the
following information: (1) the transmittance of the filter, (2) the camera response, (3) the skin reflectance and (4) the illumination distribution in the spectral domain. The details of this weighting scheme can be found in [9]. 3. 2. Illumination Adjustment (IA) via data fusion
where C =
∑
1 C
N
∑ wλi p1,λi ,
(8)
i =1 N w . Here, illumination adjustment is i =1 λ i
performed via weighted fusion. The spectral power distributions (SPDs) of the illuminants used in our experiments are shown in Figure 5. 3. 3. Wavelet fusion
Wavelet based methods have been widely used for image fusion. The wavelet transform is a data analysis tool that provides a multi-resolution decomposition of an image. The Haar wavelet-based pixel-level data fusion, as described in [10], is used on different probes. Given two registered images I1 and I 2 , a two dimensional discrete wavelet decomposition is performed on I1 and I 2 , to obtain the wavelet approximation coefficients a1, a2 and detail coefficients d1, d2. Wavelet approximation and detail coefficients of the fused image, af and df, are calculated as follows: a f = Wa1 × a1 + Wa2 × a2 .
(9)
d f = Wd1 × d1 + Wd 2 × d 2 ,
(10)
1
Different light sources have different spectral properties, which information was used for illumination adjustment to improve face recognition rates. The central function of IA is to transform the probe images acquired under a particular illumination to appear as acquired under a different illumination, preferably similar to that of the gallery. Given a particular camera, a filter and an object, the product Fλi = Rλi S λi Tλi remains the same. In this case, the camera response has a direct relationship to the incident illumination. The camera response, p1,λi , in band λi
(a)
(b)
acquired using a particular illumination (denoted by L1 ), can be represented as p1,λi = Fλi L1, λi , (5) where L1,λi is the value of the spectral power density (SPD) of L1 at λi . The camera response, p2,λi , acquired under another illumination, L2, can be represented as p2, λi = Fλi L2, λi . (6)
1-4244-0487-8/06/$20.00 ©2006 IEEE
(c) (d) Figure 5. Normalized spectral power distribution of (a) halogen light, (b) day light and, (c,d) different fluorescent light setups used in the acquisition of the IRIS-M3 database.
2006 Biometrics Symposium
3. 4. Rank-based decision level fusion (RDLF)
Rank-based Decision level fusion is also implemented. The ranks of each individual in two or more probe sets are obtained and the lowest rank is chosen for each individual. These decisions are cumulated to form the CMC curve of the rank-based decision level fusion. 4. EXPERIMENTAL RESULTS
The goal of this effort is to identify single band MSI or fused MSIs that provide a better recognition performance than monochromatic images, especially when the gallery and probes are acquired under different illuminations. To support our expectations, averaging, PCA fusion, wavelet fusion, IA, and decision level fusion are implemented on the MSIs in IRIS-M3 database. PCA fusion, a widely used fusion method was implemented as described in [10]. We have compared the first rank characteristics of the fused images with those of the monochromatic images and single channel MSIs using Face-It®, a renowned face recognition engine [11], as the evaluation engine in these experiments. To perform a numerical comparison of various CMC curves, we define a mapping operation which maps the multi-index CMC curve into a single number. We name this many-to-one mapping as the CMC measure, which is expressed by,
QCMC =
∑i =1 M
Ci ⋅
1 i
(11)
where M is the gallery size, i represents the rank, and Ci is the percentage of probe images that can be correctly identified at rank i. In our implementation, M=10, as it is considered that rank matches beyond 10 are insignificant. Often face recognition tests comparing many probe sets with a single gallery result in very similar CMC curves, with the rank percentages intersecting and crossing over other CMC curves. These curves are difficult to compare visually and numerically. This necessitates a metric such as the CMC measure. In the following experiments, the first rank recognition rates and the CMC measures of different experiments are compared and the corresponding CMC plots are also presented. The data for this effort was acquired over an academic period of two semesters. The data was imaged indoors and outdoors and acquired under different illuminants as abbreviated in Table 1.
1-4244-0487-8/06/$20.00 ©2006 IEEE
Table 1. Different sessions, modalities, illuminations and their corresponding abbreviations in the IRIS-M3 Face database. Session
1 1 2 2 3 3 3
Imaging Modality Illumination 1st semester Monochromatic Halogen MSI Halogen Monochromatic Fluorescent MSI Fluorescent 2nd semester Monochromatic Fluorescent Monochromatic Day light MSI Day light
Abbreviation
S1H_BW S1H_C S2F_BW S2F_C S3F_BW S3D_BW S3D_C
4.1. FR using within-semester data and different illuminations
In the first set of experiments, monochromatic images acquired under fluorescent illuminant (S2F_BW) are compared against different raw and fused probe sets acquired under halogen illumination. The data used in these tests are: Gallery: S2F_BW Visible monochromatic images Probe 0: S1H_BW Visible monochromatic images Probe 1: S1H_C650 Single channel MS images Probe 2: S1H_C650 and S1H_BW fused via wavelet fusion Probe 3: IA fused images
In these experiments, we compare the CMC measures of different probe sets comprised of raw or fused MSIs. The gallery images were acquired under a fluorescent illumination and the probe images were acquired under halogen illumination. The gallery and probe images are typically acquired within the same month. Experiments were conducted with all bands being tested, individually and in fusion with other bands and the monochromatic image, against a fixed gallery. Empirically, we established that band 650 nm, used individually or fused with the monochromatic image, had a slightly better recognition performance than Probe 0. Probe 2 (wavelet fused images) has the highest CMC measure. All fused images show a slight improvement in overall recognition over the monochromatic images as shown in Figure. 6. CMC Measures (%)
where Wa1 , Wa 2 and Wd1 , Wd 2 are weights determined empirically. The weights are chosen such that Wa1 + Wa2 =1, Wa1 = Wd1 and Wa2 = Wd2. The two dimensional discrete wavelet inverse transform is then performed to obtain the fused image.
97 96
95.92
96.14
Probe 0
Probe 1
96.38 95.96
95 94 Probe 2
Probe 3
Figure 6. Comparison of CMC measures of select probes in the first set of experiments.
2006 Biometrics Symposium
4.2. FR using between-semester data under different illuminations
In the second set of experiments, visible monochromatic images acquired in the second semester under the fluorescent illuminant (S3H_BW) are compared against different raw and fused probe sets acquired under halogen illumination. The data used in these tests are: Gallery: S3F_BW Visible monochromatic images (2nd semester) Probe 0: S1H_BW Visible monochromatic images Probe 1: S1H_C640 Single channel MSIs Probe 2: Wavelet fusion of S1H_C640 and S1H_C610 Probe 3: Wavelet fusion of Probe 0 and Probe 1 Probe 4: PCA fused images (All MSIs) Probe 5: Averaged images (All MSIs) Probe 6: Images (All MSIs) fused using weighted fusion Probe 7: Images (All MSIs) fused using IA fusion Probe 8: Decision level fusion using Ranks of Probes 0 and 1
CMC Measures (%)
Face recognition tests were performed on the probe sets described above against the gallery. It was observed from Figure 7 that within the probe sets 0-7 (data fusion) that wavelet fused images (Probe 3) gave the best face recognition performance followed by single channel MSIs (Probe 1). The decision level fusion gave the best performance and the identification rate of it at Rank 1 reached 100%. The CMC curves were shown in Figure 8 and some examples of the probe images of one subject were shown in Figure 9.
100 98 96 94 92
100 97.14 93.74
94.29 92.38
93.81
(a)
(b)
(c)
(d) (e) (f) Figure 9. Samples of select probes in the second set of experiments. (a) Probe 1: S1H_C640, (b) Probe 2: wavelet fusion of two channels, (c) Probe 3: wavelet fusion of probe 0 and probe 1, (d) Probe 5: averaged fusion (e) Probe 6: weighted fusion (f) Probe 7: IA fusion.
4.3. FR using between-semester indoor and outdoor data
In Sections 4.2 and 4.3, the experiments involved a large time separation between gallery and probe images and it is the case of most practical face recognition situations. In the third set of experiments, visible monochromatic images acquired in the first semester under halogen illumination (S1H_BW) are compared against different raw and fused probe sets acquired under side sunny day light in the second semester. The data used in these tests are: Gallery: S1H_BW Visible monochromatic images (1st semester) Probe 0: S3D_BW Visible monochromatic images Probe 1: S3D_C700 Single channel MS images Probe 2: Wavelet fusion of S1H_C700 and S1H_C660 Probe 3: Wavelet fusion of Probe 0 and Probe 1 Probe 4: PCA fused images (All MSIs) Probe 5: Averaged images (All MSIs) Probe 6: Images (All MSIs) fused using weighted fusion Probe 7: Images (All MSIs) fused using IA fusion Probe 8: Decision level fusion
Probe 0 Probe 1 Probe 3 Probe6 Probe 7 Probe 8
CMC Measures(%)
Figure 7. Comparison of CMC measures of select probes in the second set of experiments.
From CMC measures in Figure 10 and CMC curves in Figure 11, it is observed that single channel, wavelet, weighted, IA and decision fusions all provide higher recognition rates than monochromatic images (compared in Rank 1). The weights used for weighted fusion are obtained by multiplying the spectral power distribution of the illuminant, which is part of the IRIS-M3 database and the transmittance of the LCTF, provided by the manufacturer. Also, wavelet fusion of two single or several band images
Figure 8. CMC plots of select probes in the second set of experiments.
1-4244-0487-8/06/$20.00 ©2006 IEEE
83.18 78 68 58 48
64.35 53.21
58.49
59.7
52.68
Probe 0 Probe 1 Probe 3 Probe 6 Probe 7 Probe 8
Figure 10. Comparison of CMC measures of select probes in the third set of experiments.
2006 Biometrics Symposium
(580-720nm) with the monochromatic image, outperforms using monochromatic images alone. The identification rate at Rank 1 of Probe 8 (decision fusion) is of 71.43%, while for Probe 0 (monochromatic outdoor) this rate is only of 40%. A few examples of fused images are shown in Figure 12. 5. SUMMARY AND CONCLUSIONS
Face recognition using narrow band MSIs in the visible spectrum was a relatively unexplored territory in face related research. To promote research in this direction, a digital database of 2624 images consisting of visible, multispectral and thermal images was built. Acquisition was performed indoors and outdoors and the spectral power distributions of the illuminants were acquired as part of the database. MSIs have been fused by averaging, PCA, wavelet, weighted, IA and decision fusions. Face recognition tests were conducted between different galleries and probe sets acquired under different illuminations and with various time gaps. the respective CMC curves and measures were obtained. The recognition performance of wavelet fused and IA fused
Figure 11. The CMC plots of select probes in the third set of experiments.
(a)
(b)
(c)
(d) (e) (f) Figure 12. Samples of some of the probe images in the third set of experiments (a) Probe 1: S1H_C700 (b) Probe 2: wavelet fusion of two channels (c) Probe 3: wavelet fusion of probe 0 and probe 1. (d) Probe 5: Averaged fusion (e) Probe 6: weighted fusion (f) Probe 7: IA fusion.
1-4244-0487-8/06/$20.00 ©2006 IEEE
images were found to always outperform monochromatic images. The recognition performance was better even in the case of large time separation between gallery and probe data, which simulates real world scenario. An increase of up to 78% for Rank 1 in face recognition was obtained for outdoor probes with shadows. We also observed that there was at least one spectral band in each set of experiments that had better face recognition performance than the conventional monochromatic images. To advance this technology, efforts are currently spent in developing a datadriven frame work for the selection of such spectral bands. ACKNOWLEDGEMENTS
This work was supported by the DOE University Research program in Robotics under grant #DOE-DEFG0286NE37968 and NSF-CITeR grant #01-598B-UT. 6. REFERENCES [1] M. D. Fairchild, M. R. Rosen, and G. M. Johnson, "Spectral and metameric color imaging", Technical Report, Munsell Color Science Laboratory, 2001. [2] H. Chang, H. Harishwaran, M. Yi, A. Koschan, B. Abidi, and M. Abidi, "An indoor and outdoor, Multimodal, Multispectral and Multi-illuminant database for face recognition”, IEEE proc. CVPR2006, workshop on multi-model biometrics, pp. 5461, 2006. [3] F. H. Imai and R. S. Berns, “High-resolution multispectral image archives: a hybrid approach,” Proceedings of IS&T/ SID Sixth Color Imaging Conference, pp. 224-227, 1998. [4] F. H. Imai, M. R. Rosen, and R. S. Berns, “Multi-spectral imaging of a van Gogh’s self-portrait at the National Gallery of Art, Washington, D.C.,” Proceedings of IS&T PICS, pp. 185-189, 2001. [5] J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral image capture using a liquid crystal tunable filters,” Optical Engineering, Vol. 41, No. 10, pp. 2532-2548, 2002. [6] Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Face recognition in hyperspectral images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 12, pp. 1552-1560, 2003. [7] Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Hyperspectral face recognition under variable outdoor illumination,” Proceedings of SPIE, Vol. 5425, pp. 520-529, 2004. [8] M. Rosen and X. Jiang, “Lippmann2000: A spectral image database under construction,” Proceedings of the International Symposium on Multispectral Imaging and Color reproduction for Digital Archives, pp. 117-122, 1999. [9] H. Chang, A. Koschan, B. Abidi, and M. Abidi, “Physicsbased fusion of multispectral data for improved face recognition,” IEEE Proceedings of International Conference on Pattern Recognition, 2006 [10] R. Gonzalez, R. Woods, Digital Image Processing, Prentice Hall, 1st ed. 2004 [11] P. J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi, and M. Bone, "Face Recognition Vendor Test 2002, Evaluation Report".
2006 Biometrics Symposium
than visible images alone, due to the fact that MSIs enhance certain features that are relatively insignificant in monochromatic visible images [1]. This improvement over the visible is a much desired feature that we hypothesize/expect to yield better recognition rates in the case of face recognition. The main contribution of this paper is to establish that MSIs, individually or in fusion, outperform monochromatic images. Widely used fusion schemes such as wavelet based and principal component based fusions were implemented to compare against new Illumination Adjustment (IA) based fused images and raw MSIs. Since MSIs were unavailable, we took the initiative of acquiring the multispectral, multimodal and multi-illumined IRIS-M3 database [2] to facilitate research progress (ours and peers) in the area of multispectral face recognition. In order to investigate and compare Face Recognition (FR) performances under various illuminations, the luminance and radiance of the illuminants were also collected as part of the data acquisition process. In Figure 1, we show samples from one data record in the IRISM3 database, collected using the data acquisition equipment shown in Figure 2. With the advances in filter technology, tunable programmable filters were used to acquire MSIs by several research groups [3-8]. A tunable filter allows the selection of narrow
1. INTRODUCTION The needs for robust face recognition systems are extensive in a practical world with pressing issues in identity authentication and recognition. A substantial evolution has been made in face recognition research over the last 20 years, especially with the development of powerful face decomposition methods, like the use of eigenfaces. In spite of the diversity of techniques and face recognition tools designed, evaluated and tested, face recognition is not robust enough to be actually deployed in uncalibrated or unconditional backgrounds. Many of the promising methods use images in the visible spectrum. However, very few researchers have exploited raw or fused Multispectral images (MSIs) to improve face recognition. The potential of multispectral imaging has been achieved in many applications, such as remote sensing, data exploitation, medical and health care, and has shown better performance 1-4244-0487-8/06/$20.00 ©2006 IEEE
(a)
(b)
(c)
(d) (e) (f) Figure 1. Sample images in a data record in the IRIS-M3 database; (a) monochromatic image under indoor halogen light, (b) channel 640nm multispectral image under indoor halogen light, (c) monochromatic image under indoor fluorescent light, (d) monochromatic image under outdoor day light, (e) channel 640nm multispectral image outdoor day light, (f) monochromatic image under a different indoor fluorescent lighting.
2006 Biometrics Symposium
band-pass images at preset times. The continuous tuning of wavelengths provides the capability of finer spectral sampling. The Munsell Color Science Laboratory initiated efforts with multispectral images using a Liquid Crystal Tunable Filter (LCTF) over the visible spectrum, especially for high resolution art portrait reconstruction [3], [4]. Hardeberg et al. [5] imaged non-face, rigid objects over the visible spectrum. However, very few researchers have utilized multispectral image fusion to improve face recognition. Pan et al. [6], [7] acquired spectral images over the near-infrared spectrum (0.7-1 µm) and demonstrated that spectral images of faces acquired in the near infrared range can be used to recognize an individual under different poses and expressions. It is evident from the literature that not much research has been done using MS imaging in the visible domain to address the problem of face recognition, especially with respect to changes in illumination conditions. In this paper, visible or visible monochromatic image refers to an image acquired using a commercial-off-the-shelf (COTS) camera and MSIs refers to the narrow-band spectral images throughout the whole visible spectrum from 0.4-0.72 µm. In Section 2, we give a brief description of the IRIS-M3 face database. In Section 3, the fusion approaches are elucidated and the results of experimental comparisons are shown in Section 4. The conclusion is presented in Section 5. 2. THE IRIS-M3 FACE DATABASE To study the effect of using MSIs for face recognition, we collected the IRIS-M3 face database. The IRIS-M3 imaging system includes a multispectral imaging platform, a digital RGB COTS camera, an infrared camera, a spectrometer, a frame grabber and a processing station. The multimodal imaging system is shown in Figure 2(a). The multispectral platform shown in Figure 2(b) comprises a monochrome camera mechanically conjoined with a software-tuned LCTF. The LCTF provides narrow-band filtering with a 7nm Full Width-at-Half-Maximum bandwidth. The IRIS-M3 database has certain significant characteristics. This is a unique database with registered images in the visible, multispectral and thermal modalities, integrated with spectral distributions of the illuminants employed in acquisition. Another noteworthy asset is the larger set of multispectral bands in our database. Previous databases [8] have provided 16 bands per participant and a total of 7 people, while our database has 25 bands per participant and a total of 82 participants. The database was collected in 10 sessions between August 2005 and March 2006. The 82 participants in our database are of different ethnicities, age groups, facial hair characteristics and genders. Figure 3 shows the demographics of these 82 participants. The database is made up of 76% male and 24% female; the ethnic diversity is defined as a collection of 57%
1-4244-0487-8/06/$20.00 ©2006 IEEE
(a) (b) Figure 2. (a) Multi-modal and (b) multispectral imaging systems. Caucasian Asian Female Male
African Descent Indian
(a)
(b)
Figure 3. Demographics of participants in IRIS-M3. (a) Pie chart of male/female participants and, (b) different ethnicities such as Caucasian, Asian, African Descent and Asian Indian in our database.
Caucasian, 23%Asian (Chinese, Japanese, Korean and similar ethnicity), 12% Asian Indian and 8% of African Descent. We acquired IRIS-M3 face database under halogen, fluorescent and day light with shadows [2]. The three illumination scenarios are shown in Figure 4. The quadruple halogen lights with a pair on each side of the participant are shown in Figure 4(a). The second illumination setup was a pair of fluorescent light panels. In Figure 4(c), an example of the outdoor side illumination is shown. 3. FUSION APPROACHES To show the effects of using and fusing MSIs on face recognition, we have implemented a weighted average, illumination adjustment and wavelet fusion algorithms. Rank-based decision level fusion was also tested. The formulation and implementation details are presented herewith. 3. 1. Weighted fusion The image intensity for each pixel p of a monochrome camera in a certain wavelength range, λmin to λmax , can be represented by the image formation model,
(a)
(b)
(c)
Figure 4. Image acquisition setup (a) halogen lighting setup, (b) fluorescent lighting setup and (c) outdoor acquisition scene.
2006 Biometrics Symposium
p=
λ
max R (λ ) L (λ ) S (λ ) dλ , ∫λmin
(1)
where R is the spectral reflectance of the object, L is the spectral power distribution of the illuminant, and S is the spectral response of the CCD array. For a multispectral imaging system, the camera response, p λi corresponding to band i of N-spectral bands and centered at wavelength λi , can be represented as λi , max
∫λi ,min
p λi =
Rλi (λ ) Lλ i (λ ) Sλ i (λ )Tλ i (λ )dλ ,
i
i
i
i
(3)
The pixel value of the weighted fused image, pw , is then
pw = where
C=
∑
N
i =1
wλ
i
1 C
N
∑ wλi ⋅ pλi ,
under L2 by applying the weight wλi = L2,λi L1,λi , which can be represented as p1→ 2,λi = p1,λi wλi . (7) The intensity of the fused image, p1→ 2 , can be written as p1→ 2 =
(2)
where i = 1,2,L N . Tλ i is the spectral transmittance of the LCTF. The camera response is the result of an integration process which can also be calculated in a discrete form as the summation of samples. Because each spectral image is acquired within a very narrow band, we take only one sample of each factor per band. Therefore, the camera response for a spectral image pixel at λi can be written as p λi = Rλ Lλ Sλ Tλ .
Comparing (6) and (7), we can see that the spectral image at λi under L1 can be transformed to the corresponding image
(4)
i =1
. The weights are influenced by the
following information: (1) the transmittance of the filter, (2) the camera response, (3) the skin reflectance and (4) the illumination distribution in the spectral domain. The details of this weighting scheme can be found in [9]. 3. 2. Illumination Adjustment (IA) via data fusion
where C =
∑
1 C
N
∑ wλi p1,λi ,
(8)
i =1 N w . Here, illumination adjustment is i =1 λ i
performed via weighted fusion. The spectral power distributions (SPDs) of the illuminants used in our experiments are shown in Figure 5. 3. 3. Wavelet fusion
Wavelet based methods have been widely used for image fusion. The wavelet transform is a data analysis tool that provides a multi-resolution decomposition of an image. The Haar wavelet-based pixel-level data fusion, as described in [10], is used on different probes. Given two registered images I1 and I 2 , a two dimensional discrete wavelet decomposition is performed on I1 and I 2 , to obtain the wavelet approximation coefficients a1, a2 and detail coefficients d1, d2. Wavelet approximation and detail coefficients of the fused image, af and df, are calculated as follows: a f = Wa1 × a1 + Wa2 × a2 .
(9)
d f = Wd1 × d1 + Wd 2 × d 2 ,
(10)
1
Different light sources have different spectral properties, which information was used for illumination adjustment to improve face recognition rates. The central function of IA is to transform the probe images acquired under a particular illumination to appear as acquired under a different illumination, preferably similar to that of the gallery. Given a particular camera, a filter and an object, the product Fλi = Rλi S λi Tλi remains the same. In this case, the camera response has a direct relationship to the incident illumination. The camera response, p1,λi , in band λi
(a)
(b)
acquired using a particular illumination (denoted by L1 ), can be represented as p1,λi = Fλi L1, λi , (5) where L1,λi is the value of the spectral power density (SPD) of L1 at λi . The camera response, p2,λi , acquired under another illumination, L2, can be represented as p2, λi = Fλi L2, λi . (6)
1-4244-0487-8/06/$20.00 ©2006 IEEE
(c) (d) Figure 5. Normalized spectral power distribution of (a) halogen light, (b) day light and, (c,d) different fluorescent light setups used in the acquisition of the IRIS-M3 database.
2006 Biometrics Symposium
3. 4. Rank-based decision level fusion (RDLF)
Rank-based Decision level fusion is also implemented. The ranks of each individual in two or more probe sets are obtained and the lowest rank is chosen for each individual. These decisions are cumulated to form the CMC curve of the rank-based decision level fusion. 4. EXPERIMENTAL RESULTS
The goal of this effort is to identify single band MSI or fused MSIs that provide a better recognition performance than monochromatic images, especially when the gallery and probes are acquired under different illuminations. To support our expectations, averaging, PCA fusion, wavelet fusion, IA, and decision level fusion are implemented on the MSIs in IRIS-M3 database. PCA fusion, a widely used fusion method was implemented as described in [10]. We have compared the first rank characteristics of the fused images with those of the monochromatic images and single channel MSIs using Face-It®, a renowned face recognition engine [11], as the evaluation engine in these experiments. To perform a numerical comparison of various CMC curves, we define a mapping operation which maps the multi-index CMC curve into a single number. We name this many-to-one mapping as the CMC measure, which is expressed by,
QCMC =
∑i =1 M
Ci ⋅
1 i
(11)
where M is the gallery size, i represents the rank, and Ci is the percentage of probe images that can be correctly identified at rank i. In our implementation, M=10, as it is considered that rank matches beyond 10 are insignificant. Often face recognition tests comparing many probe sets with a single gallery result in very similar CMC curves, with the rank percentages intersecting and crossing over other CMC curves. These curves are difficult to compare visually and numerically. This necessitates a metric such as the CMC measure. In the following experiments, the first rank recognition rates and the CMC measures of different experiments are compared and the corresponding CMC plots are also presented. The data for this effort was acquired over an academic period of two semesters. The data was imaged indoors and outdoors and acquired under different illuminants as abbreviated in Table 1.
1-4244-0487-8/06/$20.00 ©2006 IEEE
Table 1. Different sessions, modalities, illuminations and their corresponding abbreviations in the IRIS-M3 Face database. Session
1 1 2 2 3 3 3
Imaging Modality Illumination 1st semester Monochromatic Halogen MSI Halogen Monochromatic Fluorescent MSI Fluorescent 2nd semester Monochromatic Fluorescent Monochromatic Day light MSI Day light
Abbreviation
S1H_BW S1H_C S2F_BW S2F_C S3F_BW S3D_BW S3D_C
4.1. FR using within-semester data and different illuminations
In the first set of experiments, monochromatic images acquired under fluorescent illuminant (S2F_BW) are compared against different raw and fused probe sets acquired under halogen illumination. The data used in these tests are: Gallery: S2F_BW Visible monochromatic images Probe 0: S1H_BW Visible monochromatic images Probe 1: S1H_C650 Single channel MS images Probe 2: S1H_C650 and S1H_BW fused via wavelet fusion Probe 3: IA fused images
In these experiments, we compare the CMC measures of different probe sets comprised of raw or fused MSIs. The gallery images were acquired under a fluorescent illumination and the probe images were acquired under halogen illumination. The gallery and probe images are typically acquired within the same month. Experiments were conducted with all bands being tested, individually and in fusion with other bands and the monochromatic image, against a fixed gallery. Empirically, we established that band 650 nm, used individually or fused with the monochromatic image, had a slightly better recognition performance than Probe 0. Probe 2 (wavelet fused images) has the highest CMC measure. All fused images show a slight improvement in overall recognition over the monochromatic images as shown in Figure. 6. CMC Measures (%)
where Wa1 , Wa 2 and Wd1 , Wd 2 are weights determined empirically. The weights are chosen such that Wa1 + Wa2 =1, Wa1 = Wd1 and Wa2 = Wd2. The two dimensional discrete wavelet inverse transform is then performed to obtain the fused image.
97 96
95.92
96.14
Probe 0
Probe 1
96.38 95.96
95 94 Probe 2
Probe 3
Figure 6. Comparison of CMC measures of select probes in the first set of experiments.
2006 Biometrics Symposium
4.2. FR using between-semester data under different illuminations
In the second set of experiments, visible monochromatic images acquired in the second semester under the fluorescent illuminant (S3H_BW) are compared against different raw and fused probe sets acquired under halogen illumination. The data used in these tests are: Gallery: S3F_BW Visible monochromatic images (2nd semester) Probe 0: S1H_BW Visible monochromatic images Probe 1: S1H_C640 Single channel MSIs Probe 2: Wavelet fusion of S1H_C640 and S1H_C610 Probe 3: Wavelet fusion of Probe 0 and Probe 1 Probe 4: PCA fused images (All MSIs) Probe 5: Averaged images (All MSIs) Probe 6: Images (All MSIs) fused using weighted fusion Probe 7: Images (All MSIs) fused using IA fusion Probe 8: Decision level fusion using Ranks of Probes 0 and 1
CMC Measures (%)
Face recognition tests were performed on the probe sets described above against the gallery. It was observed from Figure 7 that within the probe sets 0-7 (data fusion) that wavelet fused images (Probe 3) gave the best face recognition performance followed by single channel MSIs (Probe 1). The decision level fusion gave the best performance and the identification rate of it at Rank 1 reached 100%. The CMC curves were shown in Figure 8 and some examples of the probe images of one subject were shown in Figure 9.
100 98 96 94 92
100 97.14 93.74
94.29 92.38
93.81
(a)
(b)
(c)
(d) (e) (f) Figure 9. Samples of select probes in the second set of experiments. (a) Probe 1: S1H_C640, (b) Probe 2: wavelet fusion of two channels, (c) Probe 3: wavelet fusion of probe 0 and probe 1, (d) Probe 5: averaged fusion (e) Probe 6: weighted fusion (f) Probe 7: IA fusion.
4.3. FR using between-semester indoor and outdoor data
In Sections 4.2 and 4.3, the experiments involved a large time separation between gallery and probe images and it is the case of most practical face recognition situations. In the third set of experiments, visible monochromatic images acquired in the first semester under halogen illumination (S1H_BW) are compared against different raw and fused probe sets acquired under side sunny day light in the second semester. The data used in these tests are: Gallery: S1H_BW Visible monochromatic images (1st semester) Probe 0: S3D_BW Visible monochromatic images Probe 1: S3D_C700 Single channel MS images Probe 2: Wavelet fusion of S1H_C700 and S1H_C660 Probe 3: Wavelet fusion of Probe 0 and Probe 1 Probe 4: PCA fused images (All MSIs) Probe 5: Averaged images (All MSIs) Probe 6: Images (All MSIs) fused using weighted fusion Probe 7: Images (All MSIs) fused using IA fusion Probe 8: Decision level fusion
Probe 0 Probe 1 Probe 3 Probe6 Probe 7 Probe 8
CMC Measures(%)
Figure 7. Comparison of CMC measures of select probes in the second set of experiments.
From CMC measures in Figure 10 and CMC curves in Figure 11, it is observed that single channel, wavelet, weighted, IA and decision fusions all provide higher recognition rates than monochromatic images (compared in Rank 1). The weights used for weighted fusion are obtained by multiplying the spectral power distribution of the illuminant, which is part of the IRIS-M3 database and the transmittance of the LCTF, provided by the manufacturer. Also, wavelet fusion of two single or several band images
Figure 8. CMC plots of select probes in the second set of experiments.
1-4244-0487-8/06/$20.00 ©2006 IEEE
83.18 78 68 58 48
64.35 53.21
58.49
59.7
52.68
Probe 0 Probe 1 Probe 3 Probe 6 Probe 7 Probe 8
Figure 10. Comparison of CMC measures of select probes in the third set of experiments.
2006 Biometrics Symposium
(580-720nm) with the monochromatic image, outperforms using monochromatic images alone. The identification rate at Rank 1 of Probe 8 (decision fusion) is of 71.43%, while for Probe 0 (monochromatic outdoor) this rate is only of 40%. A few examples of fused images are shown in Figure 12. 5. SUMMARY AND CONCLUSIONS
Face recognition using narrow band MSIs in the visible spectrum was a relatively unexplored territory in face related research. To promote research in this direction, a digital database of 2624 images consisting of visible, multispectral and thermal images was built. Acquisition was performed indoors and outdoors and the spectral power distributions of the illuminants were acquired as part of the database. MSIs have been fused by averaging, PCA, wavelet, weighted, IA and decision fusions. Face recognition tests were conducted between different galleries and probe sets acquired under different illuminations and with various time gaps. the respective CMC curves and measures were obtained. The recognition performance of wavelet fused and IA fused
Figure 11. The CMC plots of select probes in the third set of experiments.
(a)
(b)
(c)
(d) (e) (f) Figure 12. Samples of some of the probe images in the third set of experiments (a) Probe 1: S1H_C700 (b) Probe 2: wavelet fusion of two channels (c) Probe 3: wavelet fusion of probe 0 and probe 1. (d) Probe 5: Averaged fusion (e) Probe 6: weighted fusion (f) Probe 7: IA fusion.
1-4244-0487-8/06/$20.00 ©2006 IEEE
images were found to always outperform monochromatic images. The recognition performance was better even in the case of large time separation between gallery and probe data, which simulates real world scenario. An increase of up to 78% for Rank 1 in face recognition was obtained for outdoor probes with shadows. We also observed that there was at least one spectral band in each set of experiments that had better face recognition performance than the conventional monochromatic images. To advance this technology, efforts are currently spent in developing a datadriven frame work for the selection of such spectral bands. ACKNOWLEDGEMENTS
This work was supported by the DOE University Research program in Robotics under grant #DOE-DEFG0286NE37968 and NSF-CITeR grant #01-598B-UT. 6. REFERENCES [1] M. D. Fairchild, M. R. Rosen, and G. M. Johnson, "Spectral and metameric color imaging", Technical Report, Munsell Color Science Laboratory, 2001. [2] H. Chang, H. Harishwaran, M. Yi, A. Koschan, B. Abidi, and M. Abidi, "An indoor and outdoor, Multimodal, Multispectral and Multi-illuminant database for face recognition”, IEEE proc. CVPR2006, workshop on multi-model biometrics, pp. 5461, 2006. [3] F. H. Imai and R. S. Berns, “High-resolution multispectral image archives: a hybrid approach,” Proceedings of IS&T/ SID Sixth Color Imaging Conference, pp. 224-227, 1998. [4] F. H. Imai, M. R. Rosen, and R. S. Berns, “Multi-spectral imaging of a van Gogh’s self-portrait at the National Gallery of Art, Washington, D.C.,” Proceedings of IS&T PICS, pp. 185-189, 2001. [5] J. Y. Hardeberg, F. Schmitt, and H. Brettel, “Multispectral image capture using a liquid crystal tunable filters,” Optical Engineering, Vol. 41, No. 10, pp. 2532-2548, 2002. [6] Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Face recognition in hyperspectral images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 12, pp. 1552-1560, 2003. [7] Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Hyperspectral face recognition under variable outdoor illumination,” Proceedings of SPIE, Vol. 5425, pp. 520-529, 2004. [8] M. Rosen and X. Jiang, “Lippmann2000: A spectral image database under construction,” Proceedings of the International Symposium on Multispectral Imaging and Color reproduction for Digital Archives, pp. 117-122, 1999. [9] H. Chang, A. Koschan, B. Abidi, and M. Abidi, “Physicsbased fusion of multispectral data for improved face recognition,” IEEE Proceedings of International Conference on Pattern Recognition, 2006 [10] R. Gonzalez, R. Woods, Digital Image Processing, Prentice Hall, 1st ed. 2004 [11] P. J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi, and M. Bone, "Face Recognition Vendor Test 2002, Evaluation Report".
2006 Biometrics Symposium
