Realistic Inverse Lighting from a Single 2D Image ... - Semantic Scholar
Realistic Inverse Lighting from a Single 2D Image of a Face, Taken under Unknown and Complex Lighting
Davoud Shahlaei, Volker Blanz May 2015, Ljubljana
Inverse Rendering input
CMU PIE
●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
3D Morphable Model (3DMM) (Blanz & Vetter, SIGGRAPH'99)
α1
+ α2
+ α3
+ α4
+ ...
β1
+ β2
+ β3
+ β4
+ ...
Shape and Texture Vectors
Inverse Rendering with 3DMM Framework input
CMU PIE
3DMM so far ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
CMU PIE
3DMM so far ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
proposed ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
proposed ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
3DMM so far ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
3DMM
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
So Far 3DMM Lighting
●
Ambient + one directional light
●
Ad hoc Phong model and Texture
Enhanced 3DMM Lighting
●
Ambient + one directional light
●
Ad hoc Phong model and Texture
100 light sources No ambient
Enhanced 3DMM Lighting
●
Ambient + one directional light
100 light sources No ambient
●
Ad hoc Phong model and Texture
Measured BRDF of Skin From Weyrich et al. 2006
Superposition L1
L2
Superposition L1
L2
L1
L2
Superposition L1
L1
L2
+
Pixel-wise addition
=
L2
Illumination Cone
...
Under all the possible lighting situations
Light Stage
Paul Debevec, Light Stage at USC - ICT
Virtual Light Stage
Virtual Light Stage L1
L2
Step 0: input
Step 1: input
3D Morphable Model
Shape
Texture
Step 2: input
3D Morphable Model
Shape
Texture
Light Sources
Step 2: input
3D Morphable Model
Shape
Light Sources
Texture
...
Synthetic Illumination Cone
...
⃗ C1
⃗ C2
⃗ C3
Superposition for Synthetic Illumination Cone
⃗I ≈ C α ⃗ Input image
Coefficient vector Synthetic cone images
(⃗ C1 ⃗ C2 ⃗ C3
. ..) = C
Superposition for Synthetic Illumination Cone
⃗I ≈ C α ⃗
∀ i α i≥0
When solved:
αi is the intensity of the light Li
Single input
Pipeline: 3DMM fitting
Synthetic illumination cone
⃗I ≈ C α ⃗
Superposition Regularized Non-Negative Least Squares
Lighting estimation
Scene reconstruction
α ⃗ ≥0
Illuminate novel objects
De-illumination
g
De-Illumination
De-Illumination
Specular reflectance
Defuse reflectance
Cast shadows
Novel Objects
Lighting Map
g
Photo by: Barrie Spence
CMU PIE
CMU PIE
CMU PIE
CMU PIE
CMU PIE
CMU PIE
CMU PIE
Thank You
Davoud Shahlaei, Volker Blanz May 2015, Ljubljiana
Davoud Shahlaei, Volker Blanz May 2015, Ljubljana
Inverse Rendering input
CMU PIE
●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
3D Morphable Model (3DMM) (Blanz & Vetter, SIGGRAPH'99)
α1
+ α2
+ α3
+ α4
+ ...
β1
+ β2
+ β3
+ β4
+ ...
Shape and Texture Vectors
Inverse Rendering with 3DMM Framework input
CMU PIE
3DMM so far ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
CMU PIE
3DMM so far ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
proposed ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
proposed ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Inverse Lighting with 3DMM Framework input
Photo by: Barrie Spence
3DMM so far ●
3D Geometry
●
Reflectance
●
Lighting
●
Camera
●
Color Balance
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
Photo by: Barrie Spence
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
Challenges
3DMM
●
Specular Highlights
●
Grazing angles (Fresnel)
●
Multiple lighting directions
●
Colorful light
●
Cast shadows
●
Harsh illumination
So Far 3DMM Lighting
●
Ambient + one directional light
●
Ad hoc Phong model and Texture
Enhanced 3DMM Lighting
●
Ambient + one directional light
●
Ad hoc Phong model and Texture
100 light sources No ambient
Enhanced 3DMM Lighting
●
Ambient + one directional light
100 light sources No ambient
●
Ad hoc Phong model and Texture
Measured BRDF of Skin From Weyrich et al. 2006
Superposition L1
L2
Superposition L1
L2
L1
L2
Superposition L1
L1
L2
+
Pixel-wise addition
=
L2
Illumination Cone
...
Under all the possible lighting situations
Light Stage
Paul Debevec, Light Stage at USC - ICT
Virtual Light Stage
Virtual Light Stage L1
L2
Step 0: input
Step 1: input
3D Morphable Model
Shape
Texture
Step 2: input
3D Morphable Model
Shape
Texture
Light Sources
Step 2: input
3D Morphable Model
Shape
Light Sources
Texture
...
Synthetic Illumination Cone
...
⃗ C1
⃗ C2
⃗ C3
Superposition for Synthetic Illumination Cone
⃗I ≈ C α ⃗ Input image
Coefficient vector Synthetic cone images
(⃗ C1 ⃗ C2 ⃗ C3
. ..) = C
Superposition for Synthetic Illumination Cone
⃗I ≈ C α ⃗
∀ i α i≥0
When solved:
αi is the intensity of the light Li
Single input
Pipeline: 3DMM fitting
Synthetic illumination cone
⃗I ≈ C α ⃗
Superposition Regularized Non-Negative Least Squares
Lighting estimation
Scene reconstruction
α ⃗ ≥0
Illuminate novel objects
De-illumination
g
De-Illumination
De-Illumination
Specular reflectance
Defuse reflectance
Cast shadows
Novel Objects
Lighting Map
g
Photo by: Barrie Spence
CMU PIE
CMU PIE
CMU PIE
CMU PIE
CMU PIE
CMU PIE
CMU PIE
Thank You
Davoud Shahlaei, Volker Blanz May 2015, Ljubljiana
