Dual-coded Compressive Hyperspectral Imaging
Dual-coded Compressive Hyperspectral Imaging Xing Lin1, Gordon Wetzstein2, Yebin Liu1 and Qionghai Dai1 1Tsinghua
University,
2MIT
Media lab
Lin, X., Wetzstein, G., Liu, Y., & Dai, Q., Optics Letters, 2014, 39(7), 2044-2047. Lin, X., Wetzstein, G., Liu, Y., & Dai, Q., Computational Optical Sensing and Imaging, 2014, CW3C.5.
Motivations
Conventional Camera
RGB Color Image
2
Motivations
400 nm~1000 nm
SOC710 Hyperspectral Camera
Hyperspectral Images
3
Motivations Intensity
Green
Wavelength (nm)
Intensity
Yellow
400 nm~1000 nm
Hyperspectral Images Wavelength (nm)
Spectrum of Individual Regions 4
Overview of Our Method Scene
Dual-coded Compressive Hyper-spectral Imager
Spectral Modulation Spatial Modulation 5
Overview of Our Method
6
Overview of Our Method
7
Overview of Our Method
8
Hyperspectral Images
9
Applications
Astronomy
Remote Sensing
Solar Physics
Medical Diagnosis
Computer Vision
Computer Graphic 10
Related Work Coded Aperture(CASSI)
Pinhole-mask
[Porter et al. 1987]
x
y
[Arguello et al. 2013]
[Basedow et al. 1995] [Gehm et al. 2007] [Wagadarikar et al. 2009]
[Schechner et al. 2002] [Yamaguchi et al. 2006]
[Ford et al. 2001]
Image Replicating Imaging Spectrometer (IRIS). al. 2009] 11 [Harvey [Du et al.et2005]
Dual-coded Compressive Hyperspectral Imaging Scene
h(x, l )
Relay Lens
i (x) h(x, ) m(x, ) d
Diffraction Grating
Relay Lens Spectral Plane Sensor
Dynamic Modulation
i(x) 12
Prototype Camera Schematic c
b
a
Imaging Scene
c
Objective Lens
Sensor a b
c b a
Band-pass Filter
Lcos
Diffraction Grating Relay Lens
Relay Lens
DMD 13
Prototype Camera Objective Lens Senso r Band-pass Filter
Diffraction Relay Lens Grating Relay Lens
DMD
Lcos
14
Programmable Spatially-varying Color Filtering Single Exposure Bayer Filter Response Curve Spectral Modulation Spatial Modulation (Local Pixel Region 22 )
Coded Image
Color Demosaicing Result
15
Multiplexed Hyperspectral Imaging Single Exposure Spectral Modulation (Hadamard Coding)
... ... 16
Spatial Modulation (Local Pixel Region 4 4)
Multiplexed Image
400nm
420nm
440nm
460nm
480nm
500nm
520nm
540nm
560nm
580nm
600nm
620nm
640nm
660nm
680nm
700nm
De-multiplexing & Inverse Result 16
Compressive Hyperspectral Imaging Single Exposure Spectral Random Modulation
... 31 ... 16
Spatial Random Modulation
540nm
1
3 Dual Random Coded Image
Full Resolution Hyperspectral Images 17
=
...
...
Over-complete HS Dictionary Learning
Spatially Varying Color Spectrum Coding Dual Sparse Reconstruction Coded Sampling
3D Hyper-spectral Images
...
2D Sensor Image
Compressive Hyperspectral Imaging Architecture
18
Compressibility of HS Images 700nm 420nm y 410nm 400nm
x
...
y
x
HS Datacube
Slice of HS 19
Over-complete HS Dictionary Learning
…
Sparse
Coding
Hyper-spectral(HS) Database
Training HS Patches
Learned HS Atoms
20
PSNR in dB
Compressibility of HS Images
Compression Ratio in % Coefficients
Compressibility of HS Images in Different Bases 21
400nm
3D HS patches ensing & Reconstruction Compression W/15 Coeffs (Compr. Sensing-BPDN) (Sparse Coding-Lasso)
700nm
Compressibility of HS Images 3D DCT
3D HS atoms
22
Dual Coded Sampling Scene
Hyper-spectral Images: h(x, l )
Relay Lens
Modulation Pattern: m(x, l )
x
Diffractio n Grating Relay Lens Projection
Spectral Plane Sensor
Sensor Image: i ( x )
Modulation of Hyper-spectral Images Ray Optics of SSCSI
23
Dual Coded Sampling F2
F3
Modulation Pattern: m(x, l )
= Measurement Matrix
Hyper-spectral Images
Sensor Image
F1
Hyper-spectral Images: h(x, l )
x
Projection
Sensor Image: i ( x )
Modulation of Hyper-spectral Images 24
Sparse Reconstruction
Sparse Representative F2
F3
Unknow Sparse Coefficients
Sensor Image
F1
= Measurement Matrix
Over-complete HS Dictionary Learning (Color Coded)
25
Compressive HS Imaging of a Color Checker Scene
Dual Random Coded Image
Synthetic Color Image
High-resolution Hyperspectral Images with 31 Wavebands
26
Compressive HS Imaging of a Color Checker Scene
Dual Random Coded Image
Synthetic Color Image
Spectral Profile of Individual Regions
27
Compare to CASSI
Coded Aperture Image
PSNR=26.3dB
Coded Aperture Hyperspectral Imaging
Dual Random Coded Image
PSNR=31.9dB
Dual Coded Hyperspectral Imaging
28
Future Work Dual Coded Mask
Spatial-Spectral Encoded Mask
Coded Aperture
Prism & Mask
Sensor & Mask
Mask Sensor
Relay Lens Diffraction Grating
Relay Lens Mask Sensor & Mask
Coded Spectrum
Mask Sensor
Lenslet Array
Coded Lenslet Array
… Mask Sensor
Lenslet Sensor
Mask Lenslet Sensor
29
Dual-coded Compressive Hyperspectral Imaging (DCSI) Programmable Spatially-varying Color Filtering Multiplexed Hyperspectral Imaging Compressive Hyperspectral Imaging
Questions?
30
University,
2MIT
Media lab
Lin, X., Wetzstein, G., Liu, Y., & Dai, Q., Optics Letters, 2014, 39(7), 2044-2047. Lin, X., Wetzstein, G., Liu, Y., & Dai, Q., Computational Optical Sensing and Imaging, 2014, CW3C.5.
Motivations
Conventional Camera
RGB Color Image
2
Motivations
400 nm~1000 nm
SOC710 Hyperspectral Camera
Hyperspectral Images
3
Motivations Intensity
Green
Wavelength (nm)
Intensity
Yellow
400 nm~1000 nm
Hyperspectral Images Wavelength (nm)
Spectrum of Individual Regions 4
Overview of Our Method Scene
Dual-coded Compressive Hyper-spectral Imager
Spectral Modulation Spatial Modulation 5
Overview of Our Method
6
Overview of Our Method
7
Overview of Our Method
8
Hyperspectral Images
9
Applications
Astronomy
Remote Sensing
Solar Physics
Medical Diagnosis
Computer Vision
Computer Graphic 10
Related Work Coded Aperture(CASSI)
Pinhole-mask
[Porter et al. 1987]
x
y
[Arguello et al. 2013]
[Basedow et al. 1995] [Gehm et al. 2007] [Wagadarikar et al. 2009]
[Schechner et al. 2002] [Yamaguchi et al. 2006]
[Ford et al. 2001]
Image Replicating Imaging Spectrometer (IRIS). al. 2009] 11 [Harvey [Du et al.et2005]
Dual-coded Compressive Hyperspectral Imaging Scene
h(x, l )
Relay Lens
i (x) h(x, ) m(x, ) d
Diffraction Grating
Relay Lens Spectral Plane Sensor
Dynamic Modulation
i(x) 12
Prototype Camera Schematic c
b
a
Imaging Scene
c
Objective Lens
Sensor a b
c b a
Band-pass Filter
Lcos
Diffraction Grating Relay Lens
Relay Lens
DMD 13
Prototype Camera Objective Lens Senso r Band-pass Filter
Diffraction Relay Lens Grating Relay Lens
DMD
Lcos
14
Programmable Spatially-varying Color Filtering Single Exposure Bayer Filter Response Curve Spectral Modulation Spatial Modulation (Local Pixel Region 22 )
Coded Image
Color Demosaicing Result
15
Multiplexed Hyperspectral Imaging Single Exposure Spectral Modulation (Hadamard Coding)
... ... 16
Spatial Modulation (Local Pixel Region 4 4)
Multiplexed Image
400nm
420nm
440nm
460nm
480nm
500nm
520nm
540nm
560nm
580nm
600nm
620nm
640nm
660nm
680nm
700nm
De-multiplexing & Inverse Result 16
Compressive Hyperspectral Imaging Single Exposure Spectral Random Modulation
... 31 ... 16
Spatial Random Modulation
540nm
1
3 Dual Random Coded Image
Full Resolution Hyperspectral Images 17
=
...
...
Over-complete HS Dictionary Learning
Spatially Varying Color Spectrum Coding Dual Sparse Reconstruction Coded Sampling
3D Hyper-spectral Images
...
2D Sensor Image
Compressive Hyperspectral Imaging Architecture
18
Compressibility of HS Images 700nm 420nm y 410nm 400nm
x
...
y
x
HS Datacube
Slice of HS 19
Over-complete HS Dictionary Learning
…
Sparse
Coding
Hyper-spectral(HS) Database
Training HS Patches
Learned HS Atoms
20
PSNR in dB
Compressibility of HS Images
Compression Ratio in % Coefficients
Compressibility of HS Images in Different Bases 21
400nm
3D HS patches ensing & Reconstruction Compression W/15 Coeffs (Compr. Sensing-BPDN) (Sparse Coding-Lasso)
700nm
Compressibility of HS Images 3D DCT
3D HS atoms
22
Dual Coded Sampling Scene
Hyper-spectral Images: h(x, l )
Relay Lens
Modulation Pattern: m(x, l )
x
Diffractio n Grating Relay Lens Projection
Spectral Plane Sensor
Sensor Image: i ( x )
Modulation of Hyper-spectral Images Ray Optics of SSCSI
23
Dual Coded Sampling F2
F3
Modulation Pattern: m(x, l )
= Measurement Matrix
Hyper-spectral Images
Sensor Image
F1
Hyper-spectral Images: h(x, l )
x
Projection
Sensor Image: i ( x )
Modulation of Hyper-spectral Images 24
Sparse Reconstruction
Sparse Representative F2
F3
Unknow Sparse Coefficients
Sensor Image
F1
= Measurement Matrix
Over-complete HS Dictionary Learning (Color Coded)
25
Compressive HS Imaging of a Color Checker Scene
Dual Random Coded Image
Synthetic Color Image
High-resolution Hyperspectral Images with 31 Wavebands
26
Compressive HS Imaging of a Color Checker Scene
Dual Random Coded Image
Synthetic Color Image
Spectral Profile of Individual Regions
27
Compare to CASSI
Coded Aperture Image
PSNR=26.3dB
Coded Aperture Hyperspectral Imaging
Dual Random Coded Image
PSNR=31.9dB
Dual Coded Hyperspectral Imaging
28
Future Work Dual Coded Mask
Spatial-Spectral Encoded Mask
Coded Aperture
Prism & Mask
Sensor & Mask
Mask Sensor
Relay Lens Diffraction Grating
Relay Lens Mask Sensor & Mask
Coded Spectrum
Mask Sensor
Lenslet Array
Coded Lenslet Array
… Mask Sensor
Lenslet Sensor
Mask Lenslet Sensor
29
Dual-coded Compressive Hyperspectral Imaging (DCSI) Programmable Spatially-varying Color Filtering Multiplexed Hyperspectral Imaging Compressive Hyperspectral Imaging
Questions?
30
