Motion Detail Preserving Optical Flow Estimation - VideoLectures
Motion Detail Preserving Optical Flow Estimation Li Xu1, Jiaya Jia1, Yasuyuki Matsushita2 1
The Chinese University of Hong Kong 2 Microsoft Research Asia
Conventional Optical Flow
• Middlebury Benchmark [Baker et al. 07] • Dominant Scheme: Coarse-to-Fine Warping
Large Displacement Optical Flow
• Region Matching [Brox et al. 09, 10] • Discrete Local Search [Steinbrucker et al. 09]
Both Large and Small Motion Exist
• Capture large motion • Preserve sub-pixel accuracy
Our Work • Framework – Extended coarse-to-fine motion estimation for both large and small displacement optical flow
• Model – A new data term to selectively combine constraints
• Solver – Efficient numerical solver for discrete-continuous optimization
Outline • Framework – Extended coarse-to-fine motion estimation for both large and small displacement optical flow
• Model – A new data term to selectively combine constraints
• Solver – Efficient numerical solver for discrete-continuous optimization
The Multi-scale Problem
The Multi-scale Problem
Ground truth
Ground truth
Ground truth
The Multi-scale Problem
Ground truth
Ground truth
Ground truth
Ground truth Estimate
…
Ground truth
Ground truth
Estimate
Estimate
The Multi-scale Problem • Large discrepancy between initial values and optimal motion vectors • Our solution – Improve flow initialization to reduce the reliance on the initialization from coarser levels
Extended Flow Initialization • Sparse feature matching for each level
Extended Flow Initialization • Identify missing motion vectors
Extended Flow Initialization • Identify missing motion vectors
Extended Flow Initialization
…
Extended Flow Initialization
Fuse
…
Outline • Framework: extended initialization for coarseto-fine motion estimation
• Model: selective data term • Efficient numerical solver
Data Constraints • Color constancy DI (u, x)=
I 2 (x + u) − I1 (x)
• Gradient constancy D∇I (u, = x) τ ∇I 2 (x + u) − ∇I1 ( x)
• Average 1 1 = ED (u, x) ∑ DI (u, x) + D∇I (u, x) 2 x 2
Problems • Pixels moving out of shadow • Color constancy is violated DI (u p1 , p1 ) = 6.63
u p1: ground truth motion of p1
• Gradient constancy holds D∇I (u p1 , p1 ) = 0.32
• Average: 1 DI (u p1 , p1 ) + D∇I (u p1 , p1 ) ) = 3.48 ( 2
Problems • Pixels undergoing rotational motion • Color constancy holds DI (u p2 , p 2 ) = 0.29
u p2: ground truth motion of p2
• Gradient constancy is violated D∇I (u p2 , p 2 ) = 4.20
• Average: 1 DI (u p2 , p 2 ) + D∇I (u p2 , p 2 ) ) = 2.24 ( 2
Our Proposal • Selectively combine the constraints = = ED (u,α )
(u,x) + (1 (1 − α (x)) (x)) D ∑ α (x) D (u,x) I
x
where α (x) : Ζ 2 {0,1}
∇I
(u,x)
Comparisons 5
4,5
4
A A E
color gradient
3,5
average 3
ours
2,5
2
RubberWhale
Urban2
Outline • Framework: extended initialization for coarse to fine motion estimation • Model: selective data term
∑ α (x) D (u, x) + (1 − α (x)) D I
x
• Efficient numerical solver
∇I
(u, x)
Energy Functions and Solver • Total energy = E (u,α )
∑ α (x) D (u, x) + (1 − α (x)) D I
∇I
(u, x) + λ S (∇u, x)
x
• Probability of a particular state of the system 1 − β E (u, α ) P(u, α ) = e Z
Mean Field Approximation • Partition function Z =∑
∑ α
e − β E (u, α )
{u} { = 0,1}
• Sum over all possible values of α = ∑e
− βλ
∑ S (∇u,x ) x
e
∑α (x ) DI (u, x) + (1−α (x )) D∇I (u, x) x
{ = 0,1}
{u }
...
= ∑e
∑ α
−β
−β{
∑ λ S (∇u,x ) − β ln( e− β DI ( u, x ) + e− β D∇I ( u, x ) )} 1
x
{u}
The effective potential Eeff (u) [Geiger & Girosi, 1989]
E (u)= eff
∑ λ S (∇u,x) − x
1
β
ln(e − β DI (u, x ) + e − β D∇I (u, x ) )
• Optimal condition (Euler-Lagrange equations) α ( x)
1 − α ( x)
e β D∇I (u,x ) e β DI (u,x ) ∂ D (u, x) + β D∇I (u, ∂ D (u, x) β D∇I (u,x ) (u,x x) β DI (u,x ) u I β DI (u,x ) u ∇I e e +e +e − λ div(∂ ∇u S (∇u,x)) = = 0
• It decomposes to α (x) =
{
1 1 + e β ( DI (u,x ) − D∇I (u,x ))
α (x)∂ u DI (u, x) + (1 − α (x))∂ u D∇I (u, x) − λ div(∂ ∇u S (∇u,x)) = 0
α (x) =
{
1 1 + e β ( DI (u,x ) − D∇I (u,x ))
α (x)∂ u DI (u, x) + (1 − α (x))∂ u D∇I (u, x) − λ div(∂ ∇u S (∇u,x)) = 0
Algorithm Skeleton α (x) =
{
1 1 + e β ( DI (u,x ) − D∇I (u,x ))
α (x)∂ u DI (u, x) + (1 − α (x))∂ u D∇I (u, x) − λ div(∂ ∇u S (∇u,x)) = 0
• For each level • Extended Flow Initialization (QPBO) • Continuous Minimization (Iterative reweight) – Update α – Compute flow field (Variable Splitting)
Results
Difference
Averaging constraints
Ours
Middlebury Dataset EPE=0.74
Results from Different Steps
Coarse-to-fine
Extended coarse-to-fine
EPE=0.15
EPE=0.24
Large Displacement
Overlaid Input
Large Displacement • Motion Estimates
Coarse-to-fine
Our Result
Warping Result
Comparison • Motion Magnitude Maps
LDOP [Brox et al. 09 ] [Steinbrucker et al. 09]
Ours
More Results
Overlaid Input
Conventional Coarse-to-fine
Our Result
More Results
Overlaid Input
Coarse-to-fine
Our Result
Conclusion • Extended initialization (Framework) • Selective data term (Model) • Efficient numerical scheme (Solver) • Limitations – Featureless motion details – Large occlusions
Thank you!
Running Time • 640x480 image pair • Standard PC with Intel Core2 Quad @2.4GHz • Extended Flow Initialization • Continuous Minimization
~3 mins < 40s
The Chinese University of Hong Kong 2 Microsoft Research Asia
Conventional Optical Flow
• Middlebury Benchmark [Baker et al. 07] • Dominant Scheme: Coarse-to-Fine Warping
Large Displacement Optical Flow
• Region Matching [Brox et al. 09, 10] • Discrete Local Search [Steinbrucker et al. 09]
Both Large and Small Motion Exist
• Capture large motion • Preserve sub-pixel accuracy
Our Work • Framework – Extended coarse-to-fine motion estimation for both large and small displacement optical flow
• Model – A new data term to selectively combine constraints
• Solver – Efficient numerical solver for discrete-continuous optimization
Outline • Framework – Extended coarse-to-fine motion estimation for both large and small displacement optical flow
• Model – A new data term to selectively combine constraints
• Solver – Efficient numerical solver for discrete-continuous optimization
The Multi-scale Problem
The Multi-scale Problem
Ground truth
Ground truth
Ground truth
The Multi-scale Problem
Ground truth
Ground truth
Ground truth
Ground truth Estimate
…
Ground truth
Ground truth
Estimate
Estimate
The Multi-scale Problem • Large discrepancy between initial values and optimal motion vectors • Our solution – Improve flow initialization to reduce the reliance on the initialization from coarser levels
Extended Flow Initialization • Sparse feature matching for each level
Extended Flow Initialization • Identify missing motion vectors
Extended Flow Initialization • Identify missing motion vectors
Extended Flow Initialization
…
Extended Flow Initialization
Fuse
…
Outline • Framework: extended initialization for coarseto-fine motion estimation
• Model: selective data term • Efficient numerical solver
Data Constraints • Color constancy DI (u, x)=
I 2 (x + u) − I1 (x)
• Gradient constancy D∇I (u, = x) τ ∇I 2 (x + u) − ∇I1 ( x)
• Average 1 1 = ED (u, x) ∑ DI (u, x) + D∇I (u, x) 2 x 2
Problems • Pixels moving out of shadow • Color constancy is violated DI (u p1 , p1 ) = 6.63
u p1: ground truth motion of p1
• Gradient constancy holds D∇I (u p1 , p1 ) = 0.32
• Average: 1 DI (u p1 , p1 ) + D∇I (u p1 , p1 ) ) = 3.48 ( 2
Problems • Pixels undergoing rotational motion • Color constancy holds DI (u p2 , p 2 ) = 0.29
u p2: ground truth motion of p2
• Gradient constancy is violated D∇I (u p2 , p 2 ) = 4.20
• Average: 1 DI (u p2 , p 2 ) + D∇I (u p2 , p 2 ) ) = 2.24 ( 2
Our Proposal • Selectively combine the constraints = = ED (u,α )
(u,x) + (1 (1 − α (x)) (x)) D ∑ α (x) D (u,x) I
x
where α (x) : Ζ 2 {0,1}
∇I
(u,x)
Comparisons 5
4,5
4
A A E
color gradient
3,5
average 3
ours
2,5
2
RubberWhale
Urban2
Outline • Framework: extended initialization for coarse to fine motion estimation • Model: selective data term
∑ α (x) D (u, x) + (1 − α (x)) D I
x
• Efficient numerical solver
∇I
(u, x)
Energy Functions and Solver • Total energy = E (u,α )
∑ α (x) D (u, x) + (1 − α (x)) D I
∇I
(u, x) + λ S (∇u, x)
x
• Probability of a particular state of the system 1 − β E (u, α ) P(u, α ) = e Z
Mean Field Approximation • Partition function Z =∑
∑ α
e − β E (u, α )
{u} { = 0,1}
• Sum over all possible values of α = ∑e
− βλ
∑ S (∇u,x ) x
e
∑α (x ) DI (u, x) + (1−α (x )) D∇I (u, x) x
{ = 0,1}
{u }
...
= ∑e
∑ α
−β
−β{
∑ λ S (∇u,x ) − β ln( e− β DI ( u, x ) + e− β D∇I ( u, x ) )} 1
x
{u}
The effective potential Eeff (u) [Geiger & Girosi, 1989]
E (u)= eff
∑ λ S (∇u,x) − x
1
β
ln(e − β DI (u, x ) + e − β D∇I (u, x ) )
• Optimal condition (Euler-Lagrange equations) α ( x)
1 − α ( x)
e β D∇I (u,x ) e β DI (u,x ) ∂ D (u, x) + β D∇I (u, ∂ D (u, x) β D∇I (u,x ) (u,x x) β DI (u,x ) u I β DI (u,x ) u ∇I e e +e +e − λ div(∂ ∇u S (∇u,x)) = = 0
• It decomposes to α (x) =
{
1 1 + e β ( DI (u,x ) − D∇I (u,x ))
α (x)∂ u DI (u, x) + (1 − α (x))∂ u D∇I (u, x) − λ div(∂ ∇u S (∇u,x)) = 0
α (x) =
{
1 1 + e β ( DI (u,x ) − D∇I (u,x ))
α (x)∂ u DI (u, x) + (1 − α (x))∂ u D∇I (u, x) − λ div(∂ ∇u S (∇u,x)) = 0
Algorithm Skeleton α (x) =
{
1 1 + e β ( DI (u,x ) − D∇I (u,x ))
α (x)∂ u DI (u, x) + (1 − α (x))∂ u D∇I (u, x) − λ div(∂ ∇u S (∇u,x)) = 0
• For each level • Extended Flow Initialization (QPBO) • Continuous Minimization (Iterative reweight) – Update α – Compute flow field (Variable Splitting)
Results
Difference
Averaging constraints
Ours
Middlebury Dataset EPE=0.74
Results from Different Steps
Coarse-to-fine
Extended coarse-to-fine
EPE=0.15
EPE=0.24
Large Displacement
Overlaid Input
Large Displacement • Motion Estimates
Coarse-to-fine
Our Result
Warping Result
Comparison • Motion Magnitude Maps
LDOP [Brox et al. 09 ] [Steinbrucker et al. 09]
Ours
More Results
Overlaid Input
Conventional Coarse-to-fine
Our Result
More Results
Overlaid Input
Coarse-to-fine
Our Result
Conclusion • Extended initialization (Framework) • Selective data term (Model) • Efficient numerical scheme (Solver) • Limitations – Featureless motion details – Large occlusions
Thank you!
Running Time • 640x480 image pair • Standard PC with Intel Core2 Quad @2.4GHz • Extended Flow Initialization • Continuous Minimization
~3 mins < 40s
