System Model and Performance Evaluation of ... - OSA Publishing
JW1A.3.pdf
Imaging and Applied Optics Technical Digest © 2012 OSA
System Model and Performance Evaluation of Spectrally Coded Plenoptic Camera Lingfei Meng and Kathrin Berkner Ricoh Innovations, Inc., 2882 Sand Hill Rd., Menlo Park, CA {meng,berkner}@rii.ricoh.com
Abstract: We introduce an end-to-end imaging system model for a spectrally coded plenoptic camera. The model includes a system-dependent spectral demultiplexing algorithm and is used to evaluate spectral quality and classification performance of a spectrally coded plenoptic camera. © 2012 Optical Society of America
OCIS codes: 110.1758, 110.4234, 100.3010.
1.
Introduction
Plenoptic camera architectures are designed to capture a 4D light field of the scene and have been used for different applications, such as digital refocusing and depth estimation. Several studies modified the plenoptic camera architecture to collect multispectral images in a single snapshot by inserting a filter array in the pupil plane of the main lens [1,2], passing the light at different wavelength. The light is then focused on the microlens array, which is mounted in front of a detector array. The microlens array projects the multispectral light onto the detector, producing an image that contains multispectral information of the scene. A framework has been presented in [3] to optimize the layout of a spectral filter mask. However, the system performance of such a spectrally coded plenoptic camera in terms of spectral information reconstruction and classification application has not been systematically studied. In this paper we introduce an end-to-end imaging system model for a spectrally coded plenoptic camera. The system model (Fig. 1) consists of source model, optics model, detector model, and spectral reconstruction algorithm. Finally, task-specific performance metrics are defined to evaluate the spectral quality and classification performance. 2.
Formulation of System Model
Our proposed system model for a spectrally-coded plenoptic camera includes a statistic model of the source, a geometric model of the optical response of a plenoptic camera, a detector model, and reconstruction algorithms. 2.1.
Source model
In the source model the radiance reflected from the object is estimated and its first and second order statistics are computed. The inputs to the source model are irradiance of light source E(λ ), reflectance of object R(λ ), and light incident angle θi . The irradiance and reflectance are wavelength (λ ) dependent. The object surface is assumed to be Lambertian in our current model. The radiance reflected from an object surface is calculated as L(λ ) = π1 E(λ )R(λ ) cos(θi )+Nsource , where Nsource is the source noise due to the texture variation of the object surface. The mean and covariance of the reflected radiance are then calculated and further propagated to a camera model. Camera Model Reflectance
0.5
Skin 1 Skin 2
λ
0.3
y
0.1
450
550
650
x
Wavelength [nm]
Source
Optics
Detector
Spectral Demultiplexing
Performance Metrics
Figure 1. End-to-end imaging system model of spectrally coded plenoptic camera.
JW1A.3.pdf
2.2.
Imaging and Applied Optics Technical Digest © 2012 OSA
Camera model
The radiance L(λ ) originating from an object point passed through optics is converted to a digital output signal. Considering a spectrally coded plenoptic camera, the light passed through its ith filter in the aperture is calculated as ˆ τo L(λ )Ao Ad λct · · ρi (λ )dλ · ge , (1) bi = r2 h·c ∆λ
where Ao is the area of a sub-aperture occupied by the ith filter, r is the distance between main lens and detector, τo is the optical transmittance, ρ(λ ) is the system spectral sensitivity, Ad is the pixel area, t is the exposure time, ∆λ is the filter bandwidth, λc is the center wavelength, h is the Planck’s constant, c is the speed of light, and ge is the sensor gain [4]. For a single object point the intensity measured at each sensor location depends on the optical response function of the system. In this paper we will calculate a geometric approximation to the optical response function using ray transfer matrices. The ray transfer matrix for a plenoptic imaging system is computed as [3] R{tρi } = S[z3 ]L[− f1 ]S[z2 ]L[− F1 ; ρi ]S[z1 ], where S represents a ray transfer matrix for free space, L represents a ray transλ λ fer matrix for thin lens, z1 is the object distance from main lens, z2 is the distance between main lens and microlens, z3 is the distance between microlens and detector array, Fλ is the focal length of main lens, and fλ is the focal length of microlens. Tracing of a light path through the system can then be performed by multiplying this matrix with a vector representing the light ray originating from an object point. Using this ray tracing, a geometric approximation of the optical response of the system can be obtained. We calculate the fraction of the number of rays that hit a sensor pixel, considering an aperture mask in the main lens containing several filters. Each sensor pixel measurement can be thought of as a linear combination of the spectral intensities xi = ∑Nj=1 fi, j b j , where the coefficients fi, j model the fraction of the number of rays passing through N filters arriving at sensor location i. The measurements for all the sensor pixels in a super-pixel behind a lenslet form an intensity, modeled as x = Fb + N photon + Nsystem ,
(2)
where x = [x1 , · · · , xM ]T is a vector containing plenoptic sensor data in each super-pixel, F is an M × N fraction matrix containing the coefficients fi, j , b = [b1 , · · · , bN ]T is a vector containing the spectral intensity values, N photon is the signal-dependent shot noise, and Nsystem is the signal independent noise, such as read noise and quantization noise. The fraction matrix F models multiplexing of the different spectral responses due to chromatic aberration of the optical system and sensor pixelation. 2.3.
Spectral Reconstruction Algorithms
A spectral feature vector is extracted from the plenoptic sensor data vector x. The spectral information contained in the sensor pixels is contaminated due the spectral crosstalk. In this paper spectral reconstruction algorithms are used to reconstruct N spectral features that correspond to the N spectral filters inserted in the aperture mask. A linear transformation can be implemented to extract the spectral feature vector as y = Φx = Φ(Fb + N photon + Nsystem ),
(3)
where y = [y1 , · · · , yN ]T is the extracted feature vector, and Φ is an N × M spectral reconstruction matrix. Three different approaches are used in this paper for spectral reconstruction. 1) Averaging contiguous sub-pixels that are considered as collecting light from the same spectral filter. The average is taken based on a user-defined mask on each super-pixel [1]. 2) Extracting a single pixel in each cell of the mask that has the maximum response to the corresponding spectral filter [2]. 3) We propose a system-dependent spectral demultiplexing algorithm. The output ˆ The spectral features are extracted by taking signal is demultiplexed based on a calibrated system response matrix F. ˆ i.e., Φ = (Fˆ T F) ˆ −1 F. ˆ a pseudoinverse of F, 2.4.
Performance Metrics
The spectral reconstruction quality is evaluated from two different perspectives: the spectral reconstruction accuracy and the signal-to-noise ratio (SNR) of the extracted spectral features. For spectral reconstruction accuracy a spectral reconstruction error is used to evaluate how the crosstalk caused by the optical system paffects the accuracy of the extracted spectral information. The spectral reconstruction error can be found as Err = E[(y − b)2 ]. An averaged SNR of the extracted spectral features can be calculated as SNR =
JW1A.3.pdf
Imaging and Applied Optics Technical Digest © 2012 OSA
y¯ feature data, and σyi isaccuracy the standard of the feature data ∑Ni=1 σyii , where y¯i is the mean of the extracted Based on the BD the classification Pa can bedeviation estimated using on a empirically derived high order equation as introduced in [6]. associated with sensor noise. To evaluate the spectral classification performance of the camera we employ two metrics to measure the separability 3. Simulation Results of statistical distributions, the Fisher discriminant ratio (FDR) and the Bhattacharyya distance (BD) [5]. Based on the Simulation experiments have been performed to evaluate the performance of a spectrally coded plenoptic camera BD the classification accuracy Pa can be estimated using on an empirically equation as introduced inreconstruction [6]. using our proposed system model. For derived spectral reconstruction quality the spectral error and the SNR of 1 N
extracted spectral features were calculated based on Monte Carlo experiments. In the source model a Tungsten light bulb was used as the light source. Reflectance data was measured on skin using a spectrometer. In the camera model a spectrally coded plenoptic camera with a 3 ⇥ 3 square filter layout design was considered. The outputs of the camera model are digital (DN) withof12a bit depth. Thecoded system response matrix Fˆ was using estimated when the object was Simulation experiments have been performed to evaluate thenumbers performance spectrally plenoptic camera presented at a distance of z = 300mm from the main lens. The spectral reconstruction error and SNR of extracted 1 our proposed system model. In the source model a Tungsten light bulb was used as the light source. Reflectance data features based on the three different spectral reconstruction algorithms are compared in Table 2. Other results are was measured on skin using a spectrometer. shown In theincamera model a spectrally plenoptic camera a 3It × Table 2 when the object distance z1coded shows certain variability (Dz =with 25mm). can3be seen that the average square filter layout design was considered. The outputs of the model are accuracy, digital numbers (DN) 12 leads bit to very low SNR. The method shows very poorcamera spectral reconstruction and the single pixel with extraction demultiplexing presents lowpresented spectral reconstruction error of andzmaintains high SNR. It is also noticed that the depth. The system response matrix Fˆ was estimated when method the object was at a distance 1 = 300mm from performance of demultiplexing method varies when the object is located at different distance.
3.
Simulation Results
the main lens. The spectral reconstruction error and SNR of extracted features based on the three different spectral reconstruction algorithms are calculated based on Monte Carlo experiments and compared in Table 1. Other results are reconstruction quality comparison. shown in Table 1 when the object distance z1 shows certain variabilityTable (∆z2.=Spectral 25mm). It can be seen that the average Metric Pixel SNR. Demultiplexing method shows very poor spectral reconstruction accuracy, and the singleObject pixelDistance extraction Average leads to Single very low The z1 243.48 7.06 2.80 demultiplexing method presents low spectral reconstructionErr error and maintains high SNR. It is also noticed that the [DN] z1 Dz 254.74 6.94 26.54 performance of demultiplexing method varies when the object is located at different distance. z1 + Dz 234.40 6.95 16.19 Based on the BD the classification accuracy Pa can be estimated using on a empirically derived high order equation z1 reflectance 45.76 as introduced in [6]. The classification performance was evaluated for two object classes. The skin data was37.61 collected on 45.34 two SNR [dB] z1 Dz 45.89 37.64 45.35 objects, and its statistics were calculated and provided as inputs to the source model. The FDR, BD, and Pa based 3. Simulation Results z1 + Dz 45.86 37.56 45.46 on the single extraction andperformance demultiplexing methods are camera compared in Table 2. For benchmark comparison the Simulation experiments have beenpixel performed to evaluate the of a spectrally coded plenoptic using our proposed system model. performance For spectral reconstruction qualitysimulated the spectral reconstruction error and the SNR of classification was also forThe a color filter array (CFA) design, which The hasskin a mosaic of data 9 was collected on two classification performance wasarchitecture evaluated for two object classes. reflectance extracted spectral features were calculated based on Monte Carlo experiments. In the source model a Tungsten light objects, and its statistics wereoutperforms calculated and provided as inputs toextraction the source model. The FDR, BD, and Pa based spectral filters placed on the pixel sensor. The demultiplexing method the single pixel approach, bulb was used as the light source. Reflectance data was measured on skin using a spectrometer. In the camera model a on the The single pixel and demultiplexing methods are compared in Table 3. For benchmark comparison the spectrally coded plenoptic higher camera with a 3 ⇥is 3 square filter layout design considered. outputs of extraction the camera because SNR achieved based onwasdemultiplexing. The CFA design gives better classification accuracy, because classification performance was also simulated for a color filter array (CFA) architecture design, which has a mosaic of 9 model are digital numbers (DN) with 12 bit depth. The system response matrix Fˆ was estimated when the object was it does not introduce any spectral cross talk and each pixel integrates light through the whole aperture which presented at a distance of z1 = 300mm from the main lens. The spectral reconstruction error and SNR of extracted spectral filters placed on the pixel sensor.passed The demultiplexing method outperforms the single pixel extraction approach, features basedgives on the three different spectral reconstructioncoded algorithms are compared in Table 2. Other are classification because higher SNR results is achieved based on demultiplexing. The CFAwhich design gives better classification accuracy, because higher signal. Spectrally plenoptic camera presents good performance is comparable shown in Table 2 when the object distance z1 shows certain variability (Dz = 25mm). It cannot be seen that theany average it does introduce spectral cross talkcustomizing and each pixel spectral integrates light through the whole aperture which the design, andaccuracy, meanwhile provides larger flexibility and filterpassed arrays. method showsto very poorCFA spectral reconstruction and the single pixel extraction leads to very lowfor SNR.changing The
gives higher SNR. Spectrally coded plenoptic camera presents good classification performance which is comparable
demultiplexing method presents low spectral reconstruction error and maintains high SNR. It is also noticed that the the CFA design, and meanwhile provides larger flexibility on customizing spectral filters. performance of demultiplexing method varies when the object is located at differenttodistance.
Table 1. Spectral reconstruction quality comparison. Table 2. Spectral reconstruction quality comparison.
Metric Err [DN] SNR [dB]
Object Distance z1 z1 Dz z1 + Dz z1 z1 Dz z1 + Dz
Average 243.48 254.74 234.40 45.76 45.89 45.86
Single Pixel 7.06 6.94 6.95 37.61 37.64 37.56
Demultiplexing 2.80 26.54 16.19 45.34 45.35 45.46
Table 2. Classification performance comparison. Table 3. Classification performance comparison.
Metric FDR BD Pa
CFA 5.10 1.54 95.45%
Single Pixel 1.87 0.49 82.08%
Demultiplexing 5.04 1.30 93.82%
4. Conclusions
The classification performance was evaluated for two object classes. The skin reflectance data was collected on two We The haveFDR, presented imaging system model for a spectrally coded plenoptic camera. In addition we objects, and its statistics were calculated and provided as inputs to the source model. BD, andanPa end-to-end based on the single pixel extraction and demultiplexing methods are compared in Table 3. For benchmark comparison the spectral demultiplexing algorithm. Based on the system model and the defined perintroduced a system-dependent 4. Conclusions classification performance was also simulated for a color filter array (CFA) architecture design, which has awe mosaic of 9 the spectral reconstruction algorithms and classification performance of a spectrally formance metrics evaluated spectral filters placed on the pixel sensor. The demultiplexing method outperforms the single pixel extraction approach, coded plenoptic camera. The performance of a spectrally coded plenoptic was found to be comparable to the CFA because higherWe SNRhave is achieved based on demultiplexing. The CFA design gives better classification accuracy, introduced an end-to-end imaging system model for abecause spectrally coded plenoptic camera, including a novel architecture design. it does not introduce any spectral cross talk and each pixel integrates light passed through the whole aperture which system-dependent spectral demultiplexing algorithm. This model was used to evaluate the spectral and classification gives higher SNR. Spectrally coded plenoptic camera presents good classification performance which is comparable to the CFA design, and meanwhileof provides larger flexibility on customizing spectralof filters. performance the camera. The performance a spectrally coded plenoptic camera was found to be comparable to
the CFA architecture design, while providing flexibility for changing spectral filters. Table 3. Classification performance comparison.
CFA Single Pixel Demultiplexing ReferencesMetric FDR 5.10 1.87 5.04 BD 1.54 0.49 1.30 1. R. Horstmeyer et al., "Flexible multimodal Pa 95.45% 82.08% 93.82%
camera using a light field architecture," in ICCP (2009). 2. D. Cavanaugh et al. "VNIR hypersensor camera system", in Proc. of SPIE, (2009), vol. 6966. 4. Conclusions 3. K. Berkner and S. Shroff, "Optimization of spectrally coded mask for multi-modal plenoptic camera," in Computational Optical Sensing and Imaging, OSAcamera. Technical Digest (CD) (Optical Society of America, 2011). We have presented an end-to-end imaging system model for a spectrally coded plenoptic In addition we introduced a system-dependent spectral demultiplexing on the system model and the defined per4. J. P. Kerekes and J. E.algorithm. Baum,Based "Spectral imaging system analytical model for subpixel object detection," IEEE formance metrics we evaluated the spectral reconstruction algorithms and classification performance of a spectrally ona spectrally Geoscience and Remote 40, to5,thepp. 1088-1101 (2002). coded plenoptic camera.Transactions The performance of coded plenoptic was foundSensing, to be comparable CFA architecture design.5. R. O. Duda, P. E. Hart, D. G. Stork, "Pattern classification," John Wiley & Sons, New York, 2001. 6. E. Choi et al., “Feature extraction based on the Bhattacharyya distance," Patt. Recog. 36, 1703-1709 (2003).
Imaging and Applied Optics Technical Digest © 2012 OSA
System Model and Performance Evaluation of Spectrally Coded Plenoptic Camera Lingfei Meng and Kathrin Berkner Ricoh Innovations, Inc., 2882 Sand Hill Rd., Menlo Park, CA {meng,berkner}@rii.ricoh.com
Abstract: We introduce an end-to-end imaging system model for a spectrally coded plenoptic camera. The model includes a system-dependent spectral demultiplexing algorithm and is used to evaluate spectral quality and classification performance of a spectrally coded plenoptic camera. © 2012 Optical Society of America
OCIS codes: 110.1758, 110.4234, 100.3010.
1.
Introduction
Plenoptic camera architectures are designed to capture a 4D light field of the scene and have been used for different applications, such as digital refocusing and depth estimation. Several studies modified the plenoptic camera architecture to collect multispectral images in a single snapshot by inserting a filter array in the pupil plane of the main lens [1,2], passing the light at different wavelength. The light is then focused on the microlens array, which is mounted in front of a detector array. The microlens array projects the multispectral light onto the detector, producing an image that contains multispectral information of the scene. A framework has been presented in [3] to optimize the layout of a spectral filter mask. However, the system performance of such a spectrally coded plenoptic camera in terms of spectral information reconstruction and classification application has not been systematically studied. In this paper we introduce an end-to-end imaging system model for a spectrally coded plenoptic camera. The system model (Fig. 1) consists of source model, optics model, detector model, and spectral reconstruction algorithm. Finally, task-specific performance metrics are defined to evaluate the spectral quality and classification performance. 2.
Formulation of System Model
Our proposed system model for a spectrally-coded plenoptic camera includes a statistic model of the source, a geometric model of the optical response of a plenoptic camera, a detector model, and reconstruction algorithms. 2.1.
Source model
In the source model the radiance reflected from the object is estimated and its first and second order statistics are computed. The inputs to the source model are irradiance of light source E(λ ), reflectance of object R(λ ), and light incident angle θi . The irradiance and reflectance are wavelength (λ ) dependent. The object surface is assumed to be Lambertian in our current model. The radiance reflected from an object surface is calculated as L(λ ) = π1 E(λ )R(λ ) cos(θi )+Nsource , where Nsource is the source noise due to the texture variation of the object surface. The mean and covariance of the reflected radiance are then calculated and further propagated to a camera model. Camera Model Reflectance
0.5
Skin 1 Skin 2
λ
0.3
y
0.1
450
550
650
x
Wavelength [nm]
Source
Optics
Detector
Spectral Demultiplexing
Performance Metrics
Figure 1. End-to-end imaging system model of spectrally coded plenoptic camera.
JW1A.3.pdf
2.2.
Imaging and Applied Optics Technical Digest © 2012 OSA
Camera model
The radiance L(λ ) originating from an object point passed through optics is converted to a digital output signal. Considering a spectrally coded plenoptic camera, the light passed through its ith filter in the aperture is calculated as ˆ τo L(λ )Ao Ad λct · · ρi (λ )dλ · ge , (1) bi = r2 h·c ∆λ
where Ao is the area of a sub-aperture occupied by the ith filter, r is the distance between main lens and detector, τo is the optical transmittance, ρ(λ ) is the system spectral sensitivity, Ad is the pixel area, t is the exposure time, ∆λ is the filter bandwidth, λc is the center wavelength, h is the Planck’s constant, c is the speed of light, and ge is the sensor gain [4]. For a single object point the intensity measured at each sensor location depends on the optical response function of the system. In this paper we will calculate a geometric approximation to the optical response function using ray transfer matrices. The ray transfer matrix for a plenoptic imaging system is computed as [3] R{tρi } = S[z3 ]L[− f1 ]S[z2 ]L[− F1 ; ρi ]S[z1 ], where S represents a ray transfer matrix for free space, L represents a ray transλ λ fer matrix for thin lens, z1 is the object distance from main lens, z2 is the distance between main lens and microlens, z3 is the distance between microlens and detector array, Fλ is the focal length of main lens, and fλ is the focal length of microlens. Tracing of a light path through the system can then be performed by multiplying this matrix with a vector representing the light ray originating from an object point. Using this ray tracing, a geometric approximation of the optical response of the system can be obtained. We calculate the fraction of the number of rays that hit a sensor pixel, considering an aperture mask in the main lens containing several filters. Each sensor pixel measurement can be thought of as a linear combination of the spectral intensities xi = ∑Nj=1 fi, j b j , where the coefficients fi, j model the fraction of the number of rays passing through N filters arriving at sensor location i. The measurements for all the sensor pixels in a super-pixel behind a lenslet form an intensity, modeled as x = Fb + N photon + Nsystem ,
(2)
where x = [x1 , · · · , xM ]T is a vector containing plenoptic sensor data in each super-pixel, F is an M × N fraction matrix containing the coefficients fi, j , b = [b1 , · · · , bN ]T is a vector containing the spectral intensity values, N photon is the signal-dependent shot noise, and Nsystem is the signal independent noise, such as read noise and quantization noise. The fraction matrix F models multiplexing of the different spectral responses due to chromatic aberration of the optical system and sensor pixelation. 2.3.
Spectral Reconstruction Algorithms
A spectral feature vector is extracted from the plenoptic sensor data vector x. The spectral information contained in the sensor pixels is contaminated due the spectral crosstalk. In this paper spectral reconstruction algorithms are used to reconstruct N spectral features that correspond to the N spectral filters inserted in the aperture mask. A linear transformation can be implemented to extract the spectral feature vector as y = Φx = Φ(Fb + N photon + Nsystem ),
(3)
where y = [y1 , · · · , yN ]T is the extracted feature vector, and Φ is an N × M spectral reconstruction matrix. Three different approaches are used in this paper for spectral reconstruction. 1) Averaging contiguous sub-pixels that are considered as collecting light from the same spectral filter. The average is taken based on a user-defined mask on each super-pixel [1]. 2) Extracting a single pixel in each cell of the mask that has the maximum response to the corresponding spectral filter [2]. 3) We propose a system-dependent spectral demultiplexing algorithm. The output ˆ The spectral features are extracted by taking signal is demultiplexed based on a calibrated system response matrix F. ˆ i.e., Φ = (Fˆ T F) ˆ −1 F. ˆ a pseudoinverse of F, 2.4.
Performance Metrics
The spectral reconstruction quality is evaluated from two different perspectives: the spectral reconstruction accuracy and the signal-to-noise ratio (SNR) of the extracted spectral features. For spectral reconstruction accuracy a spectral reconstruction error is used to evaluate how the crosstalk caused by the optical system paffects the accuracy of the extracted spectral information. The spectral reconstruction error can be found as Err = E[(y − b)2 ]. An averaged SNR of the extracted spectral features can be calculated as SNR =
JW1A.3.pdf
Imaging and Applied Optics Technical Digest © 2012 OSA
y¯ feature data, and σyi isaccuracy the standard of the feature data ∑Ni=1 σyii , where y¯i is the mean of the extracted Based on the BD the classification Pa can bedeviation estimated using on a empirically derived high order equation as introduced in [6]. associated with sensor noise. To evaluate the spectral classification performance of the camera we employ two metrics to measure the separability 3. Simulation Results of statistical distributions, the Fisher discriminant ratio (FDR) and the Bhattacharyya distance (BD) [5]. Based on the Simulation experiments have been performed to evaluate the performance of a spectrally coded plenoptic camera BD the classification accuracy Pa can be estimated using on an empirically equation as introduced inreconstruction [6]. using our proposed system model. For derived spectral reconstruction quality the spectral error and the SNR of 1 N
extracted spectral features were calculated based on Monte Carlo experiments. In the source model a Tungsten light bulb was used as the light source. Reflectance data was measured on skin using a spectrometer. In the camera model a spectrally coded plenoptic camera with a 3 ⇥ 3 square filter layout design was considered. The outputs of the camera model are digital (DN) withof12a bit depth. Thecoded system response matrix Fˆ was using estimated when the object was Simulation experiments have been performed to evaluate thenumbers performance spectrally plenoptic camera presented at a distance of z = 300mm from the main lens. The spectral reconstruction error and SNR of extracted 1 our proposed system model. In the source model a Tungsten light bulb was used as the light source. Reflectance data features based on the three different spectral reconstruction algorithms are compared in Table 2. Other results are was measured on skin using a spectrometer. shown In theincamera model a spectrally plenoptic camera a 3It × Table 2 when the object distance z1coded shows certain variability (Dz =with 25mm). can3be seen that the average square filter layout design was considered. The outputs of the model are accuracy, digital numbers (DN) 12 leads bit to very low SNR. The method shows very poorcamera spectral reconstruction and the single pixel with extraction demultiplexing presents lowpresented spectral reconstruction error of andzmaintains high SNR. It is also noticed that the depth. The system response matrix Fˆ was estimated when method the object was at a distance 1 = 300mm from performance of demultiplexing method varies when the object is located at different distance.
3.
Simulation Results
the main lens. The spectral reconstruction error and SNR of extracted features based on the three different spectral reconstruction algorithms are calculated based on Monte Carlo experiments and compared in Table 1. Other results are reconstruction quality comparison. shown in Table 1 when the object distance z1 shows certain variabilityTable (∆z2.=Spectral 25mm). It can be seen that the average Metric Pixel SNR. Demultiplexing method shows very poor spectral reconstruction accuracy, and the singleObject pixelDistance extraction Average leads to Single very low The z1 243.48 7.06 2.80 demultiplexing method presents low spectral reconstructionErr error and maintains high SNR. It is also noticed that the [DN] z1 Dz 254.74 6.94 26.54 performance of demultiplexing method varies when the object is located at different distance. z1 + Dz 234.40 6.95 16.19 Based on the BD the classification accuracy Pa can be estimated using on a empirically derived high order equation z1 reflectance 45.76 as introduced in [6]. The classification performance was evaluated for two object classes. The skin data was37.61 collected on 45.34 two SNR [dB] z1 Dz 45.89 37.64 45.35 objects, and its statistics were calculated and provided as inputs to the source model. The FDR, BD, and Pa based 3. Simulation Results z1 + Dz 45.86 37.56 45.46 on the single extraction andperformance demultiplexing methods are camera compared in Table 2. For benchmark comparison the Simulation experiments have beenpixel performed to evaluate the of a spectrally coded plenoptic using our proposed system model. performance For spectral reconstruction qualitysimulated the spectral reconstruction error and the SNR of classification was also forThe a color filter array (CFA) design, which The hasskin a mosaic of data 9 was collected on two classification performance wasarchitecture evaluated for two object classes. reflectance extracted spectral features were calculated based on Monte Carlo experiments. In the source model a Tungsten light objects, and its statistics wereoutperforms calculated and provided as inputs toextraction the source model. The FDR, BD, and Pa based spectral filters placed on the pixel sensor. The demultiplexing method the single pixel approach, bulb was used as the light source. Reflectance data was measured on skin using a spectrometer. In the camera model a on the The single pixel and demultiplexing methods are compared in Table 3. For benchmark comparison the spectrally coded plenoptic higher camera with a 3 ⇥is 3 square filter layout design considered. outputs of extraction the camera because SNR achieved based onwasdemultiplexing. The CFA design gives better classification accuracy, because classification performance was also simulated for a color filter array (CFA) architecture design, which has a mosaic of 9 model are digital numbers (DN) with 12 bit depth. The system response matrix Fˆ was estimated when the object was it does not introduce any spectral cross talk and each pixel integrates light through the whole aperture which presented at a distance of z1 = 300mm from the main lens. The spectral reconstruction error and SNR of extracted spectral filters placed on the pixel sensor.passed The demultiplexing method outperforms the single pixel extraction approach, features basedgives on the three different spectral reconstructioncoded algorithms are compared in Table 2. Other are classification because higher SNR results is achieved based on demultiplexing. The CFAwhich design gives better classification accuracy, because higher signal. Spectrally plenoptic camera presents good performance is comparable shown in Table 2 when the object distance z1 shows certain variability (Dz = 25mm). It cannot be seen that theany average it does introduce spectral cross talkcustomizing and each pixel spectral integrates light through the whole aperture which the design, andaccuracy, meanwhile provides larger flexibility and filterpassed arrays. method showsto very poorCFA spectral reconstruction and the single pixel extraction leads to very lowfor SNR.changing The
gives higher SNR. Spectrally coded plenoptic camera presents good classification performance which is comparable
demultiplexing method presents low spectral reconstruction error and maintains high SNR. It is also noticed that the the CFA design, and meanwhile provides larger flexibility on customizing spectral filters. performance of demultiplexing method varies when the object is located at differenttodistance.
Table 1. Spectral reconstruction quality comparison. Table 2. Spectral reconstruction quality comparison.
Metric Err [DN] SNR [dB]
Object Distance z1 z1 Dz z1 + Dz z1 z1 Dz z1 + Dz
Average 243.48 254.74 234.40 45.76 45.89 45.86
Single Pixel 7.06 6.94 6.95 37.61 37.64 37.56
Demultiplexing 2.80 26.54 16.19 45.34 45.35 45.46
Table 2. Classification performance comparison. Table 3. Classification performance comparison.
Metric FDR BD Pa
CFA 5.10 1.54 95.45%
Single Pixel 1.87 0.49 82.08%
Demultiplexing 5.04 1.30 93.82%
4. Conclusions
The classification performance was evaluated for two object classes. The skin reflectance data was collected on two We The haveFDR, presented imaging system model for a spectrally coded plenoptic camera. In addition we objects, and its statistics were calculated and provided as inputs to the source model. BD, andanPa end-to-end based on the single pixel extraction and demultiplexing methods are compared in Table 3. For benchmark comparison the spectral demultiplexing algorithm. Based on the system model and the defined perintroduced a system-dependent 4. Conclusions classification performance was also simulated for a color filter array (CFA) architecture design, which has awe mosaic of 9 the spectral reconstruction algorithms and classification performance of a spectrally formance metrics evaluated spectral filters placed on the pixel sensor. The demultiplexing method outperforms the single pixel extraction approach, coded plenoptic camera. The performance of a spectrally coded plenoptic was found to be comparable to the CFA because higherWe SNRhave is achieved based on demultiplexing. The CFA design gives better classification accuracy, introduced an end-to-end imaging system model for abecause spectrally coded plenoptic camera, including a novel architecture design. it does not introduce any spectral cross talk and each pixel integrates light passed through the whole aperture which system-dependent spectral demultiplexing algorithm. This model was used to evaluate the spectral and classification gives higher SNR. Spectrally coded plenoptic camera presents good classification performance which is comparable to the CFA design, and meanwhileof provides larger flexibility on customizing spectralof filters. performance the camera. The performance a spectrally coded plenoptic camera was found to be comparable to
the CFA architecture design, while providing flexibility for changing spectral filters. Table 3. Classification performance comparison.
CFA Single Pixel Demultiplexing ReferencesMetric FDR 5.10 1.87 5.04 BD 1.54 0.49 1.30 1. R. Horstmeyer et al., "Flexible multimodal Pa 95.45% 82.08% 93.82%
camera using a light field architecture," in ICCP (2009). 2. D. Cavanaugh et al. "VNIR hypersensor camera system", in Proc. of SPIE, (2009), vol. 6966. 4. Conclusions 3. K. Berkner and S. Shroff, "Optimization of spectrally coded mask for multi-modal plenoptic camera," in Computational Optical Sensing and Imaging, OSAcamera. Technical Digest (CD) (Optical Society of America, 2011). We have presented an end-to-end imaging system model for a spectrally coded plenoptic In addition we introduced a system-dependent spectral demultiplexing on the system model and the defined per4. J. P. Kerekes and J. E.algorithm. Baum,Based "Spectral imaging system analytical model for subpixel object detection," IEEE formance metrics we evaluated the spectral reconstruction algorithms and classification performance of a spectrally ona spectrally Geoscience and Remote 40, to5,thepp. 1088-1101 (2002). coded plenoptic camera.Transactions The performance of coded plenoptic was foundSensing, to be comparable CFA architecture design.5. R. O. Duda, P. E. Hart, D. G. Stork, "Pattern classification," John Wiley & Sons, New York, 2001. 6. E. Choi et al., “Feature extraction based on the Bhattacharyya distance," Patt. Recog. 36, 1703-1709 (2003).
