Illumination Recovery From Image With Cast ... - Semantic Scholar

Illumination Recovery From Image With Cast Shadows Via Sparse Representation IEEE Transactions on Image Processing, Vol. 20, No. 8, 2011 Xue Mei, Haibin Ling, and David W.Jacobs Presented by Ran Shu

School of Electrical Engineering and Computer Science Kyungpook National Univ.

Abstract  Using

sparse representation

─ Recovering illumination from image with cast shadows • Complicacy of image with casting shadows • Difference in capturing shadows and diffusive image

─ Diffusive image • Approximating by low-dimensional linear subspaces

─ Shadows • Sparse representation ─  1 - regularized least-square formulation » Nonnegative constraints

─ Experiments on synthetic and real data

Introduction  Illumination

recovery

─ Recovering illumination distribution from appearance of objects • Augmented reality ─ Virtual objects matching existing image ─ Convincing shadows on real scene rendered with recovered illumination

─ Casting shadows onto scene • Occlusion of incoming light ─ Determining the 3-D shape of objects ─ Containing the illumination information

─ Illumination recovery from image with cast shadows • Capturing geometry and albedos • Sparse representation ─ Efficient estimation ─ Dimensionality reduction ─ Efficient modeling

Related work  Previous

methods for shadows

─ Shadows casting on Lambertian surface • Approximating by low dimensional linear subspace • Providing useful information in recovering lighting ─ Occlusion of lighting

• Estimating the illumination distribution • Estimating the illumination ─ Detecting shadows of a scene with textured surfaces ─ Obtaining coarse 3-D information

─ Sparsity of cast shadows • Recovering illumination distribution from image brightness inside shadows ─ An adaptive sampling framework » Using a smaller number of sampling directions

• Recovering lighting from cast shadows with sparse representation ─ Haar wavelet basis capturing lighting sparsely

 Shadow

image represented by a few point lights

─ Reflected light field as a convolution of lighting and BRDF • Spherical harmonics • Directional light sources using raytracing

─ Breaking environment map into a set of directional light for image synthesis • Structured importance sampling

─ Approximating illumination from point lights • Using a strongly sublinear algorithm

Example and motivation  Directional

lights approximating lighting

─ Highly compressible ─ Perceptual loss hardly noticeable

Fig. 1. Flagpole rendered with: (a) one directional source, (b) two directional sources, (c) four directional sources, (d) eight directional sources, (e) 16 directional sources, (f) and 32 directional sources. The shadows are lighter as the number of directional sources increases.

Modeling images with cast shadows  Lighting

model

─ Given geometry of the scene ─ Ignoring specular reflections and effects of saturated pixels I   x( )I dir ( )d , S

x( )  0

S

x  

I dir  

(1)

I dir    S

─ For practical reasons N

I   xk I k k 1

(2)

 Lighting

decomposition

─ Low-frequency components • Using a spherical harmonic basis

─ High-frequency components • Using a sparse set of components ─ Each representing high-frequency part of a single directional source

– Low frequency • Projecting image onto 9-D spherical harmonic N





I   xk Iˆk  Ik , k 1

xk  0

(3)

• Separating low-frequency from high-frequency N

N

I  xk Iˆk  xk Ik , k1

k1

xk  0

(4)

• Projecting I to spherical harmonic sub-space N

I  Iˆ   xk Ik , k 1

xk  0

(5)

– High-frequency • Sparse set of components arg min || Ax  I ||2 , x

A=  I1I2 ......IN 

x=  x1 ,......, xN 

T

xk  0

(6)

 Low

dimensional approximation

─ Reducing computation of linear system • Low-dimensional approximation by applying PCA d ─ Mean vector   ─ Projection matrix W 

m d





xˆ  arg min || W  A    1T  x  W I   ||2 x

arg min || A x  I  ||2 ， 

xk  0

x

where  denotes the cross product, 1  N 1 is a vector of 1’s, A W A    1T and I  W I   denote the PCA transformation of A and I, respectively.









(7)

1 -Regularized least squares 0 ,1, 2 -Regularized

least squares

─ Preventing overfitting 2 or Tikhonov regularization arg min || Ax  I  ||2  || x ||2 , x

xk  0

(8)

2 norm of     and            x where                              denotes the   || x ||2   kN1 xk2    0 is the regularization parameter. 1/2

─ Solving low-complexity recovery of unknow vector x • Formulating with 0 -regularization arg min || A  x  I  ||2   || x ||0 ， x

xk  0

(9)

─ Considering computational cost arg min || A x  I  ||2  || x ||1 ,

xk  0

x

(10)

N || x ||   where 1 k 1 xk denotes the  1 norm of x and   0 is the regularization parameter.

─ Sparse vector x is nonnegative N

arg min || A x  I  ||2    xk , 

x

k 1

xk  0

─  1-regularization giving much sparse representation

(11)

Fig.2. Recovered coefficients and (b) 2-regularized LS

x

from (a)  1-regularized LS

Fig. 3. Light probes [8] used to generate our synthetic dataset: (a) kitchen, (b)grace, (c) campus, and (d) building. The light probes are sphere maps and shown in low dynamic range for display purposes.

 Sparse

representation for inverse lighting

─ Requirement • Input image I • Geometry of scene

─ Solving 1 -regularized LS via interior-point method • Preconditioned conjugate gradients (PCG)

1)

Obtain N directional source images I1,I2,.......IN

2)

Create first nine spherical harmonic images

3)

Project each directional source image Ik to 9-D spherical harmonic subspace and obtain the corresponding residual directional source image  I1I2 ......IN  .

4)

Normalize Ik such that Ik

5)

Project the query image I to the spherical harmonic subspace and obtain the residual image I .

6)

Apply PCA to A and obtain the projection matrix W and mean vector  .

7)

 1 and form matrix A  I1I2......IN 

2





A W  A   1T  and I W I   .

8)

Solve the 1 -regularized least-squares problem with nonnegativity constraints(11).

9)

Render the scene with the spherical harmonic lighting plus recovered sparse set of directional light sources.

Experiments  Conducting

two kinds of experiments

─ Illumination recovery • Recovering lighting from images and rendering scene with recovered lighting

─ Illumination transfer on both synthetic and real data • Applying recovered illumination from one scene to other scene

 Experimental

setup

─ Data • Synthetic data ─ Two synthetic scenes with four different lighting environment maps

• Real data ─ Photos of three real objects printed by using a 3-D printer from predesigned CAD models

─ Methods for comparison • Spherical Harmonics (SH) • Non-negative linear (NNL) ─ Minimizing H  I ,   0 ,where H is the matrix whose columns are directional source images

• Semidefinite programming (SDP) » Applying to perform constrained optimization

• Haar wavelets(Haar) » Illumination distribution is mapped to a cube and 2-D Haar wavelet basis elements are used in each face of cube » Linear combination of the wavelet basis function i  ,  and  ijkl  ,   L  ,   

 i

where

  c i  i  ,    

 d  ,       i , j , k ,l ijkl j , k ,l 

c i and d i , j , k , l are coefficients of the corresponding basis functions.

(12)

─ Evaluation criterion • Accuracy » Using rms errors of pixel values » Input image I 

d

and recovery image I 

» Defining rms as r  I , I  

where

I  I 2 / d 2

d is the number of pixels in the image

• Runtime » Preprocessing time » Time for solving lighting recovery algorithm » Rendering scene with recovered lighting

d

 Experiments

with synthetic data

─ Synthetic images • Two synthetic scenes ─ Under flexible casting shadow conditions

• Four synthetic images ─ Rending scene with high dynamic environment maps

• Consisting of teacup and plant

Fig. 5. Experiments on synthetic images rendered from the teacup scene. (a) Truth, (b) SH, (c) NNL1, (d) NNL2, (e) SDP, (f) Haar, and (g) Proposed

• Consisting of one table and four chairs ─ Four rendering scene using nine methods

Fig. 6. Experiments on synthetic images rendered from the table-chair scene. (a) Truth. (b) SH. (c) NNL1. (d) NNL2. (e) SDP. (f) Haar. (g) Proposed.

─ Experiment • Using POV-ray raytracer ─ Generating directional images

• Obtaining directions ─ Uniformly sampling the upper hemisphere

• Generating directional source images ─ Rendering with one directional light

• Using images to compute nine harmonic images ─ Each with lighting consisting of a single spherical harmonic

• Coefficient of integrated image ─ Intensity value in spherical harmonica function

• First nine harmonic images

Fig. 4. First nine harmonic images from a 3-D model of one teacup (top) and a table with four chairs (bottom).

• Illumination transfering experiments ─ Applying the recovered illumination from teacup to tablechair

Fig. 7. Illumination transfer experiments on synthetic images. (a) Ground truth. (b) SH. (c) NNL1. (d) NNL2. (e) SDP. (f) Haar. (g) Proposed.

• Texting data on synthectic teacup and table-chair Table 1. Rms errors and average runtime on synthetic teacup

Table 2. Rms errors and average runtime on table-chair

• Texting data on illumination transfering errors of synthetic Table 3. Illumination transfer errors on synthetic images

 Experiments

with real data

– Real data • Chair 1,Chari 2,Couch

Fig. 8. (a)–(b) Two of the three 3-D models used for generating real objects. The marked feature points in color are used for registration. (c) The three real objects printed out from 3-D models by a 3-D printer, from left to right: chair1, couch, and chair2.

─ Registration • Rotating objects • Translating to the camera coordinate system • Projecting onto the image plane

─ Experiments • Illumination recovery experiments » Applying all algorithms to real image

• Recovering illumination on real data ─ Chair1,chair2,couch

Fig. 10. Illumination recovery experiments on real images. (a) Ground truth. (b) SH. (c) NNL. (d) SDP. (e) Haar. (f) Proposed.

• Illumination transfer experiments

Fig. 11. Illumination transfer experiments on real images. (a) Ground truth. (b) SH. (c) NNL. (d) SDP. (e) Haar. (f) Proposed.

• First nine harmonic images

Fig. 9. First nine harmonic images created from more than 3000 directional source images derived from a 3-D model of the chair1.

• Texting data on real image Table 4. Illumination recovery errors and runtimes on real image

• Texting data illumination transfering errors of real Table 5. Illumination transfer errors on real images

─ Discussion • Specular texture ─ Not ideally Lambertian ─ Registration of objects to 3-D models

• Using random projection matrix ─ Sampling from a zero-mean normal distribution ─ Each row is normalized to unit length

Fig. 12. Illumination recovery experiments using the random projection matrix. (a) Teacup. (b) Chair1.

 Sparsity

evaluation

─ Approximating by sparse representation

Fig. 13. Improvement in accuracy by adding directional sources. RMS versus the number of directional sources for a synthetic image (teacup, second row of Fig. 5) rendered with a grace light probe (left figure) and a real image (chair2, middle left of Fig. 10) under natural indoor lighting (right figure).

Conclusion  Sparse

representation method

─ Solving infinity dimensionality of the subspace •  1-regularized least-square

─ Efficient and fast solution • Most significant directional source for Estimation

─ Accuracy and speed • Real data • Synthetic data

 Wavelet

function

– Limited time – Average value

– Wavelet function and scaling function f x 

 c  k  j0

k



j0 , k

 x     d j  k  j ,k  x  j  j0

k

– Haar wavelet functions 0  t  1/ 2

1   (t )   1  0 

1/ 2  t  1

1

 (t )

0

1

1 2

1

ˆ ( )  i

4



ei / 2 sin 2  / 4 

 Inverse

lighting using Haar wavelets

– Motivation • Compact supports of basis functions – Support of a function at nonzero region

• Sparsity – Wavelet coefficients are either zero or negligible » Fewer than 1% coefficients representing natural illumination

• Orthonormality – Reducing computation

– Proposed method • Computing coefficients under constraints • Representing illumination of each incident direction up to resolution L  ,   

 i

  c i  i  ,    

 d  ,       i , j , k ,l ijkl j , k ,l 

Fig.13. The basis functions take 1, 0, and -1 in white, gray, and black regions. Normalization constants are ignored for display.

(12)

 Adaptive

sampling method using cost function

– Approximating Image brightness

Fig.14. Each pixel provides a linear equation

– Adaptive sampling of radiance distribution • Using cost function

Fig.15. Subdivision of sampling directions

U  L1 , L2   diff  L1 , L2    min  L1 , L2  angle  L1 , L2  where diff  L1 , L2  is the radiance difference between L1 , L2 , min  L1 , L2  gives the smaller radiance of L1 , L2 , angle  L1 , L2  is the angle between direction to L1 , L2 ,  is manually specified parameter determing the relative weight

 Nine

spherical harmoincs

– Estimation of illumination distribution using image irradiance

• Nine basis harmonic images

m blm  x, y     x, y  AY l l   x, y  ,   x, y  

– Spherical harmonics basis function

Yl

m

 ,   

 2n  1  l  m ! 4  l  m  !

Pl m (cos  )eim

» where Pl mare the associated Legendre functions

Pl ( z )  m

1  z 2 

m /2

2n n !

l d l m 2 z 1    dz l  m

» Heisenberg Uncertainty Principle Y00  Y21e  3

1 3 3 3 1 5 z , Y11e  x, Y11o  y, Y20  , Y10  3z 2  1 ,  4 4 4 2 4 4 5 5 3 5 5 xz , Y21o  3 yz , Y22e  x 2  y 2  , Y22o  3 xy,  12 12 2 12 12

• Source images – Using a linear combination of spherical harmonics 2

l

I  x, y     Lml blm  x, y  l  0 m  l