A MAXIMUM LIKELIHOOD ESTIMATOR FOR ... - Semantic Scholar
A MAXIMUM LIKELIHOOD ESTIMATOR FOR CHOOSING THE REGULARIZATION PARAMETERS IN GLOBAL OPTICAL FLOW METHODS Kai Krajsek and Rudolf Mester J. W. Goethe University Visual Sensorics and Information Processing Lab Robert Mayer Str.2-4, D-60054 Frankfurt am Main, Germany ABSTRACT Global optical flow estimation methods based on variational calculus contain a regularization parameter which controls the tradeoff between the different constraints on the optical flow field. The counterpart to the regularization parameter are the hyper-parameters in the Bayesian framework [1]. These hyper-parameters have distinct physical meanings and thus can be inferred from the observable data. We derive a combined marginal maximum likelihood /maximum a posteriori (MML/MAP) estimator for simultaneously estimating hyper-parameters and optical flow for all differential variational approaches directly from the observed signal without any prior knowledge of the optical flow. Experiments demonstrate the performance of this optimization technique and show that the choice of the regularization parameter is an essential key-point in order to obtain precise motion estimation. 1. INTRODUCTION In this contribution, we develop a combined marginal maximum likelihood/maximum a posteriori (MML/MAP) estimator for simultaneous estimation of hyper-parameters and optical flow field. The optimal hyper-parameters are estimated without any prior knowledge or assumption of the current optical flow field. Only few attention has been focused on the optimal choice of the regularization parameter in global motion estimation techniques in literature so far (the authors are only aware of one contribution [2]). Some authors even assume that the optimal choice of the regularization parameter is of minor importance [3]. In contradiction to this assumption, obviously is the different value of the regularization parameter which has been proposed by different authors (0.5 in [4], 100 in [5] and 3000 in [3]) and the dependency of the motion estimate on the regularization parameter (as demonstrated in [5, 2]). But the proposal of a certain value of the regularization parameter is meaningless since its optimal value depends on image statistics, on the image noise statistics as well as Work has been supported by DFG ME 1796/5 - 3 and DAAD D/05/26027.
on the statistics of the optical flow field [1]. In this contribution we show how to optimize the regularization parameter for motion estimation techniques without any prior information of the current image sequence. Experiments demonstrate the performance of this optimization technique and show that the choice of the regularization parameter is an essential keypoint in order to obtain precise motion estimation. 2. DIFFERENTIAL MOTION ESTIMATION The general principle behind all differential approaches to motion estimation is that the conservation of some local image characteristic throughout its temporal evolution is reflected in terms of differential-geometric entities on the space-time signal s(x), x = (x, y, t)T . In its simplest form, the assumed conservation of brightness along the motion trajectory through space-time leads to the well-known brightness constancy constraint equation (BCCE), where g denotes the gradient of the gray value signal s, uh = (ux , uy , 1) the homogenous form of the direction of motion and u = (ux , uy )T the optical flow field gT uh = 0 .
(1)
Since it is fundamentally impossible to solve for uh by a single linear equation (aperture problem), additional constraints have to be considered. Whereas local methods minimize an error function over a local area V ⊂ A, global methods [4, 6, 7] estimate the optical flow field by minimizing an error functional (or error function if u is considered on a discrete grid) over the whole region of interest in space-time. The necessary additional constraint is incorporated by a regularization term ρ(u) (ρ denotes an operator acting on u) imposing supplementary information on the solution, e.g. the optical flow field should be smooth and should not vary abruptly [4]. The regularization parameter λ specifies the influence of the regularization term ρ(u) relative to the data term ψ gT uh , (ψ=real positive function). The optical flow field is estimated by minimizing Z J(u) = ψ gT uh + λρ(u) dx (2) A
with respect to the optical flow field u. 3. BAYESIAN MOTION ESTIMATION In a Bayesian formulation (see e.g. [8]), the optical flow is estimated via a probability density function pdf which connects the observable signal or its gradient with the entity of interest, the optical flow. In order to design such a pdf, we assume a regular grid in space-time considering only signal values and optical flow vectors on the knots of the grid. Since N knots in space-time are isomorphic to the Euclidian space IRN , the signal and the optical flow field can be expressed by a set of vectors s ∈ IRN and u ∈ IR2N . The gradients w of the optical flow components u as well as the gradients g of the signal components s can be written in a compact matrix vector equation w = Hu ∈ IR6N , g = Ps ∈ IR3N . Prior knowledge about u is incorporated into the estimation framework via the prior pdf p(u). The maximum a posteriori (MAP) estimator infers the optical flow field by maximizing the posterior pdf p(u|g) or minimizing its negative logarithm. Using Bayes’ law, this leads to ˆ = u
arg min {− ln(p(g|u)) − ln(p(u))} . u
(3)
The term in the bracket on the right side of equ.(3) is denoted as the objective function L. For Gibbs fields with the partition functions ZL (α), Zp (β), the energies JL and Jp and the corresponding hyper-parameters α, β, the objective function becomes L = JL (α) + Jp (β) + ln (ZL (α)Zp (β)) .
(4)
After discussing the explicit form of the likelihood energy and the prior energy for the case of motion estimation, we develop a method for optimizing the hyper-parameters directly from the observable data. Since all variational methods are equivalent to a corresponding Bayesian formulation [1] and the regularization parameter corresponds to the ratio of the hyper-parameters α, β, our approach allows to optimize the regularization parameters of global optical flow methods in general. 4. LIKELIHOOD FUNCTIONS AND PRIOR DISTRIBUTIONS FOR MOTION ESTIMATION In Bayesian estimation, the relation between the observed data g and the optical flow field u has to be established by the likelihood function p(g|u). If the optical flow field can be assumed to be constant within spatial neighborhoods Vj and the errors within each of these regions are identical independent distributed, the likelihood function can be derived as an expression depending on the structure tensors Cj and the hyperparameter α which itself depends on the noise of the gradients [1] PN T 1 e−α j=1 ψ(uhj Cgj uhj ) . (5) p(gt |u, gs ) = ZL (α)
Note, that if we consider only one BCCE at each pixel and a Gaussian gradient noise distribution (ψ(x) = x), the likelihood function corresponds to the data term of the Horn and Schunk approach [4]. For the non-Gaussian case, the likelihood function corresponds to the data term proposed by Black et al. [7] and if we further consider more BCCE’s in a local neighborhood, the likelihood function corresponds to the data term proposed in [3]. The prior pdf encodes our prior information/assumption of the optical flow field. The prior pdf corresponding to the smoothness assumption reads (β := (βx , βy )) p(u) =
PN 2 2 1 e− j=1 (βx ψ(wxj )+βy ψ(wyj )) Zp (β)
.
(6)
In contrast to the regularization terms usually used [3], we explicitly consider also the non-isotropic case (βx 6= βy ). 5. MAXIMUM LIKELIHOOD HYPER-PARAMETER ESTIMATION In this section we develop the likelihood estimator which allows us to infer the hyper-parameters directly from the observable data g. A large number of techniques (e.g. see [9] and references therein) have been developed for hyperparameter estimation but, as mentioned in the introduction, only one [2] has been derived for motion estimation so far. The drawback of the method of Ng et al. is its computational cost: a full search in parameter space is necessary to obtain the optimal regularization parameter. Furthermore, only one hyper-parameter can be estimated which corresponds to the prior hyper-parameter β = βx = βy in our approach. Thus it is necessary to know the hyper-parameter α which depends on the noise of the gradient field [1]. On the contrary our approach allows to estimate all hyper-parameters directly from the observable data. However, if the noise distribution is known, α can be computed [1] and it is only necessary to estimate β. Furthermore, the number of hyper-parameters which can be estimated is not limited in our approach which makes it also possible to consider anisotropic prior distributions. Based on the approach of Zhou et al. [9], we derive a combined marginalized maximum likelihood/ maximum a posteriori (MML/MAP) estimator for the case of motion estimation which estimates hyper-parameters and the optical flow field simultaneously. The main idea of our approach is to approximate the joint pdf p(u, g; α, β) of the gradient field g and the optical flow field u by a Gaussian distribution (if it is not already Gaussian), such that an analytic integration (marginalization) over the optical flow field can be performed. Using Bayes’ law, the joint pdf can be obtained from the prior pdf p(u; β) and the likelihood function p(g|u; α). The result is the approximated likelihood function of the hyperparameters for the current gradient field p(g; α, β) Z p(g; α, β) = p(u, g; α, β)du . (7)
The hyper-parameters are estimated by minimizing the negative logarithm of likelihood function p(g; α, β) with respect to α and β for the present realization of the gradient field g. As already mentioned, the integral in equ.(7) is only analytically solvable for the Gaussian case. In order to also handle the non-Gaussian case for exponential joint pdfs, we develop its exponent in a Taylor series up to second order around its minimum with respect to u. Let Q(ˆ u, α, β) denote the Hessian of the posterior energy J(u, α, β) and A(u, α), B(u, β) the Hessians of the likelihood and prior energy , respectively. The optical flow field which minimizes J(u, α, β) is in fact ˆ of the optical flow field for fixed α, β. the MAP estimator u ˆ = Q(ˆ With the notation Jˆ = J(ˆ u, α, β) and Q u, α, β) the approximated posterior energy reads 1 ˆ (u − u ˆ )T Q ˆ) . J(u, α, β) ≈ Jˆ + (u − u 2
(8)
Inserting the approximated posterior pdf in equ.(7) and integrating over u yields the likelihood function of the gradient field (2π)N p(g; α, β) = exp −Jˆ . (9) 12 ˆ ZL (α)Zp (β) Q Since the computation of the determinant |Q(ˆ u, α, β)| is not feasible for usual image sequence sizes, an approximation has to be performed. In analogy to the pseudo-likelihood approximation, we neglect the interaction between different neighborhood optical flow field components, i.e. B becomes block diagonal. In the Gaussian case, Aj are the spatial structure tensors at positions j. Furthermore, the determinant of Q(ˆ u, α, β) factorizes into the product of determinants of Qj (ˆ u, α, β) = Aj + diag {βx , βy }
(10)
and the individual determinants of the 2 × 2 matrices can be easily computed |Qj | = |Aj | + (a11j βy + a22j βx ) + βx βy .
(11)
In the general case, the approximated objective function for the hyper-parameters reads N 1 X ˆ L(ˆ u, α, β) ∝ Jˆ + ln Qj + ln (ZL Zp ) . 2 j=1
(12)
ˆ itself depends on the hyper-parameters {α, β} we Since u have to apply an iterative scheme for estimating u, α and β simultaneously n o uk+1 = arg min J(u, αk , β k ) (13) u n o αk+1 = arg min L(uk , α, β k ) α β k+1 = arg min L(uk , αk , β) . β
Note that the first step in the iterative scheme is nothing but the usual optical flow estimation for fixed hyper-parameters as used usually in global approaches. The second and third term in equ.(12) which distinguishes our objective function from others, enables the simultaneous hyper-parameter and motion estimation. 6. EXPERIMENTAL RESULTS We demonstrate the performance of our MML/MAP estimator by some examples showing the usefulness of optimizing the hyper-parameter instead of choosing them ad hoc. As motion estimators we applied the classical Horn and Schunk (HS) approach [4], the robust counterpart proposed by Black at al. [7] as well as the linear (L-) and non-linear (NL-) 3DCLG-estimator proposed in [3]. For all these techniques the relation between the regularization parameter and the hyperparameters reads λ = β/α for the isotropic prior (β = βx = βy ). We used four image sequences, together with their true optical flow : ’Yosemite’ (without clouds), ’Diverging Tree’, ’Office’ and the ’Marble’ sequence. All remaining free parameters, like average volume, filter-size etc. have been optimized to the image sequences under examination. For performance evaluation, the average angular error (AAE) [5] was computed. For all image sequences and all motion estimation techniques, the MML/MAP approach estimated the optimal hyper-parameters, e.g. the estimated AAE lies near the minimum when compared with a full search in the hyperparameter space. Comparing our results with the performance of these estimators in literature (e.g. Yosemite: AAE: 1.45◦ for our (MML/NL-3D-CLG); AAE: 1.46◦ for the pure NL3D-CLG estimator presented in [3] where the regularization parameter has been tuned by the known ground truth and AAE: 1.77◦ for our MML/HS approach; AAE: 11.26◦ for the pure HS presented in [5] for an ad hoc chosen regularization parameter) shows that our results outperform other motion estimators where the regularization parameter is chosen ad hoc and is in the same range of those approaches where the regularization parameters have been tuned by the known ground truth. In order to demonstrate the usefulness of non-isotropic priors we consider two image sequences with quite different statistics of the gradients of the optical flow components (figure 1 (b) and (d)). These image sequences have been generated by taking two sub-image sequences of the Yosemite sequence. For an image sequence as in (a) where the norm of the gradient of one optical flow component is much larger in one direction an anisotropic prior is more suitable and our experimental results (AAE: 5.72◦ for the anisotropic prior; AAE: 7.08◦ for the isotropic prior using the NL-3D-CLG estimator) support this statement. For the case where the norm of gradients of both optical flow components are in the same range (figure 1(c)), the estimated hyper-parameters βx , βy are also in the same range and the resulting AAE (AAE: 1.74◦ for the isotropic as well as for the anisotropic prior) is equal
pdfs like those proposed in [10] into our framework.
1
1
8. REFERENCES 0 0
(a)
0 0
1
(b)
(c)
1
(d)
Fig. 1. (a): First image of the sub-image sequence: Yosemite Sequence [75:125;1:50], (b): scatter plot of the square of gradients of the computed optical flow components, the horizontal axis depict |∇ux |2 , the vertical axis |∇uy |2 in arbitrary units (c): as in (a) but [75:125;151:200], (d): as in (b) but for the image sequence in (c) to the case of the MAP/MML estimate with an isotropic prior. Another class of image sequences where the optimal choice of the hyper-parameters are essential for a precise motion estimation is the case of image sequence containing scenes with slowly varying illumination (such that the constant brightness assumption approximatively holds). Since the hyperparameters depend substantially on the first order statistic [1] of the image sequence, i.e. the gray value distribution, one gets only optimal results if one adapts the hyper-parameters to the current images. One might argue to scale all considered images such that the first order statistics keeps nearly constant. But this is not realistic in real world applications. Consider for example a scene with overall change of brightness but one or more objects keep constant brightness, such as in a traffic scene. The lights of the vehicles keep constant while the illumination of the background might change. In order to show the dependency on the first order statistics of the image we multiplied each gray value of the ’Office’ sequence by 0.1. The MML/HS approach applied to the original image sequence resulted in an AAE=3.5◦ , the MML/HS estimator applied to the scaled image resulted in an AAE=3.5◦ whereas we obtained an AAE=10.4◦ for the pure HS estimator with the optimized regularization parameter of the original not scaled ’Office’ sequence. Thus, keeping a fixed regularization parameter can lead to large errors in motion estimation scenes with varying first order statistics. 7. SUMMARY AND CONCLUSION
[1] Kai Krajsek and Rudolf Mester, “On the equivalence of variational and statistical differential motion estimation,” in Proc. IEEE SouthWest Symposium on Image Analysis and Interpretation, 2006. [2] Lydia Ng and Victor Solo, “A data-driven method for choosing smoothing parameters in optical flow problems,” in Proc. International Conference on Image Processing, Santa Barbara, California, USA, 1997, pp. 360– 363. [3] A. Bruhn, J. Weickert, and C. Schn¨orr, “Lucas/Kanade meets Horn/Schunk: Combining local and global optical flow methods,” Int. J. Comput. Vision, vol. 61, no. 3, 2005. [4] B. Horn and B. Schunck, “Determining optical flow,” Artificial Intelligence, vol. 17, pp. 185–204, 1981. [5] J. L. Barron, D. J. Fleet, and S. S. Beauchemin, “Performance of optical flow techniques,” Int. Journal of Computer Vision, vol. 12, pp. 43–77, 1994. [6] Joachim Weickert and Christoph Schn¨orr, “A theoretical framework for convex regularizers in pde-based computation of image motion,” Int. J. Comput. Vision, vol. 45, no. 3, pp. 245–264, 2001. [7] M. J. Black and P. Anandan, “A framework for the robust estimation of optical flow,” in Proc. Fourth International Conf. on Computer Vision, (ICCV93), Berlin, Germany, 1993, pp. 231–236. [8] E.P Simoncelli, E. H. Adelson, and D. J. Heeger, “Probability distribution of optical flow,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, 1991, pp. 310–315. [9] Z. Zhou, R. Leahy, and J. Qi, “Approximate maximum likelihood hyperparameter estimation for Gibbs priors,” IEEE Trans. Image Processing, vol. 6, no. 6, pp. 844– 861, 1997.
In this contribution, we present a combined MML/MAP es[10] Stefan Roth and Michael J. Black, “On the spatial statistimator for simultaneously estimating hyper-parameters and tics of optical flow.,” in ICCV, 2005, pp. 42–49. optical flow directly from the observed signal without any prior knowledge of the optical flow. Experiments show that our MML/MAP estimator delivers the optimal hyper-parameters and show the need for optimizing the hyper-parameters to each image sequence for precise motion estimation. Since there are still free parameters left in global motion estimation techniques like the filter size, our future work will focus on deriving optimization schemes for these still free parameters. Another task will be the incorporation of more complex prior
on the statistics of the optical flow field [1]. In this contribution we show how to optimize the regularization parameter for motion estimation techniques without any prior information of the current image sequence. Experiments demonstrate the performance of this optimization technique and show that the choice of the regularization parameter is an essential keypoint in order to obtain precise motion estimation. 2. DIFFERENTIAL MOTION ESTIMATION The general principle behind all differential approaches to motion estimation is that the conservation of some local image characteristic throughout its temporal evolution is reflected in terms of differential-geometric entities on the space-time signal s(x), x = (x, y, t)T . In its simplest form, the assumed conservation of brightness along the motion trajectory through space-time leads to the well-known brightness constancy constraint equation (BCCE), where g denotes the gradient of the gray value signal s, uh = (ux , uy , 1) the homogenous form of the direction of motion and u = (ux , uy )T the optical flow field gT uh = 0 .
(1)
Since it is fundamentally impossible to solve for uh by a single linear equation (aperture problem), additional constraints have to be considered. Whereas local methods minimize an error function over a local area V ⊂ A, global methods [4, 6, 7] estimate the optical flow field by minimizing an error functional (or error function if u is considered on a discrete grid) over the whole region of interest in space-time. The necessary additional constraint is incorporated by a regularization term ρ(u) (ρ denotes an operator acting on u) imposing supplementary information on the solution, e.g. the optical flow field should be smooth and should not vary abruptly [4]. The regularization parameter λ specifies the influence of the regularization term ρ(u) relative to the data term ψ gT uh , (ψ=real positive function). The optical flow field is estimated by minimizing Z J(u) = ψ gT uh + λρ(u) dx (2) A
with respect to the optical flow field u. 3. BAYESIAN MOTION ESTIMATION In a Bayesian formulation (see e.g. [8]), the optical flow is estimated via a probability density function pdf which connects the observable signal or its gradient with the entity of interest, the optical flow. In order to design such a pdf, we assume a regular grid in space-time considering only signal values and optical flow vectors on the knots of the grid. Since N knots in space-time are isomorphic to the Euclidian space IRN , the signal and the optical flow field can be expressed by a set of vectors s ∈ IRN and u ∈ IR2N . The gradients w of the optical flow components u as well as the gradients g of the signal components s can be written in a compact matrix vector equation w = Hu ∈ IR6N , g = Ps ∈ IR3N . Prior knowledge about u is incorporated into the estimation framework via the prior pdf p(u). The maximum a posteriori (MAP) estimator infers the optical flow field by maximizing the posterior pdf p(u|g) or minimizing its negative logarithm. Using Bayes’ law, this leads to ˆ = u
arg min {− ln(p(g|u)) − ln(p(u))} . u
(3)
The term in the bracket on the right side of equ.(3) is denoted as the objective function L. For Gibbs fields with the partition functions ZL (α), Zp (β), the energies JL and Jp and the corresponding hyper-parameters α, β, the objective function becomes L = JL (α) + Jp (β) + ln (ZL (α)Zp (β)) .
(4)
After discussing the explicit form of the likelihood energy and the prior energy for the case of motion estimation, we develop a method for optimizing the hyper-parameters directly from the observable data. Since all variational methods are equivalent to a corresponding Bayesian formulation [1] and the regularization parameter corresponds to the ratio of the hyper-parameters α, β, our approach allows to optimize the regularization parameters of global optical flow methods in general. 4. LIKELIHOOD FUNCTIONS AND PRIOR DISTRIBUTIONS FOR MOTION ESTIMATION In Bayesian estimation, the relation between the observed data g and the optical flow field u has to be established by the likelihood function p(g|u). If the optical flow field can be assumed to be constant within spatial neighborhoods Vj and the errors within each of these regions are identical independent distributed, the likelihood function can be derived as an expression depending on the structure tensors Cj and the hyperparameter α which itself depends on the noise of the gradients [1] PN T 1 e−α j=1 ψ(uhj Cgj uhj ) . (5) p(gt |u, gs ) = ZL (α)
Note, that if we consider only one BCCE at each pixel and a Gaussian gradient noise distribution (ψ(x) = x), the likelihood function corresponds to the data term of the Horn and Schunk approach [4]. For the non-Gaussian case, the likelihood function corresponds to the data term proposed by Black et al. [7] and if we further consider more BCCE’s in a local neighborhood, the likelihood function corresponds to the data term proposed in [3]. The prior pdf encodes our prior information/assumption of the optical flow field. The prior pdf corresponding to the smoothness assumption reads (β := (βx , βy )) p(u) =
PN 2 2 1 e− j=1 (βx ψ(wxj )+βy ψ(wyj )) Zp (β)
.
(6)
In contrast to the regularization terms usually used [3], we explicitly consider also the non-isotropic case (βx 6= βy ). 5. MAXIMUM LIKELIHOOD HYPER-PARAMETER ESTIMATION In this section we develop the likelihood estimator which allows us to infer the hyper-parameters directly from the observable data g. A large number of techniques (e.g. see [9] and references therein) have been developed for hyperparameter estimation but, as mentioned in the introduction, only one [2] has been derived for motion estimation so far. The drawback of the method of Ng et al. is its computational cost: a full search in parameter space is necessary to obtain the optimal regularization parameter. Furthermore, only one hyper-parameter can be estimated which corresponds to the prior hyper-parameter β = βx = βy in our approach. Thus it is necessary to know the hyper-parameter α which depends on the noise of the gradient field [1]. On the contrary our approach allows to estimate all hyper-parameters directly from the observable data. However, if the noise distribution is known, α can be computed [1] and it is only necessary to estimate β. Furthermore, the number of hyper-parameters which can be estimated is not limited in our approach which makes it also possible to consider anisotropic prior distributions. Based on the approach of Zhou et al. [9], we derive a combined marginalized maximum likelihood/ maximum a posteriori (MML/MAP) estimator for the case of motion estimation which estimates hyper-parameters and the optical flow field simultaneously. The main idea of our approach is to approximate the joint pdf p(u, g; α, β) of the gradient field g and the optical flow field u by a Gaussian distribution (if it is not already Gaussian), such that an analytic integration (marginalization) over the optical flow field can be performed. Using Bayes’ law, the joint pdf can be obtained from the prior pdf p(u; β) and the likelihood function p(g|u; α). The result is the approximated likelihood function of the hyperparameters for the current gradient field p(g; α, β) Z p(g; α, β) = p(u, g; α, β)du . (7)
The hyper-parameters are estimated by minimizing the negative logarithm of likelihood function p(g; α, β) with respect to α and β for the present realization of the gradient field g. As already mentioned, the integral in equ.(7) is only analytically solvable for the Gaussian case. In order to also handle the non-Gaussian case for exponential joint pdfs, we develop its exponent in a Taylor series up to second order around its minimum with respect to u. Let Q(ˆ u, α, β) denote the Hessian of the posterior energy J(u, α, β) and A(u, α), B(u, β) the Hessians of the likelihood and prior energy , respectively. The optical flow field which minimizes J(u, α, β) is in fact ˆ of the optical flow field for fixed α, β. the MAP estimator u ˆ = Q(ˆ With the notation Jˆ = J(ˆ u, α, β) and Q u, α, β) the approximated posterior energy reads 1 ˆ (u − u ˆ )T Q ˆ) . J(u, α, β) ≈ Jˆ + (u − u 2
(8)
Inserting the approximated posterior pdf in equ.(7) and integrating over u yields the likelihood function of the gradient field (2π)N p(g; α, β) = exp −Jˆ . (9) 12 ˆ ZL (α)Zp (β) Q Since the computation of the determinant |Q(ˆ u, α, β)| is not feasible for usual image sequence sizes, an approximation has to be performed. In analogy to the pseudo-likelihood approximation, we neglect the interaction between different neighborhood optical flow field components, i.e. B becomes block diagonal. In the Gaussian case, Aj are the spatial structure tensors at positions j. Furthermore, the determinant of Q(ˆ u, α, β) factorizes into the product of determinants of Qj (ˆ u, α, β) = Aj + diag {βx , βy }
(10)
and the individual determinants of the 2 × 2 matrices can be easily computed |Qj | = |Aj | + (a11j βy + a22j βx ) + βx βy .
(11)
In the general case, the approximated objective function for the hyper-parameters reads N 1 X ˆ L(ˆ u, α, β) ∝ Jˆ + ln Qj + ln (ZL Zp ) . 2 j=1
(12)
ˆ itself depends on the hyper-parameters {α, β} we Since u have to apply an iterative scheme for estimating u, α and β simultaneously n o uk+1 = arg min J(u, αk , β k ) (13) u n o αk+1 = arg min L(uk , α, β k ) α β k+1 = arg min L(uk , αk , β) . β
Note that the first step in the iterative scheme is nothing but the usual optical flow estimation for fixed hyper-parameters as used usually in global approaches. The second and third term in equ.(12) which distinguishes our objective function from others, enables the simultaneous hyper-parameter and motion estimation. 6. EXPERIMENTAL RESULTS We demonstrate the performance of our MML/MAP estimator by some examples showing the usefulness of optimizing the hyper-parameter instead of choosing them ad hoc. As motion estimators we applied the classical Horn and Schunk (HS) approach [4], the robust counterpart proposed by Black at al. [7] as well as the linear (L-) and non-linear (NL-) 3DCLG-estimator proposed in [3]. For all these techniques the relation between the regularization parameter and the hyperparameters reads λ = β/α for the isotropic prior (β = βx = βy ). We used four image sequences, together with their true optical flow : ’Yosemite’ (without clouds), ’Diverging Tree’, ’Office’ and the ’Marble’ sequence. All remaining free parameters, like average volume, filter-size etc. have been optimized to the image sequences under examination. For performance evaluation, the average angular error (AAE) [5] was computed. For all image sequences and all motion estimation techniques, the MML/MAP approach estimated the optimal hyper-parameters, e.g. the estimated AAE lies near the minimum when compared with a full search in the hyperparameter space. Comparing our results with the performance of these estimators in literature (e.g. Yosemite: AAE: 1.45◦ for our (MML/NL-3D-CLG); AAE: 1.46◦ for the pure NL3D-CLG estimator presented in [3] where the regularization parameter has been tuned by the known ground truth and AAE: 1.77◦ for our MML/HS approach; AAE: 11.26◦ for the pure HS presented in [5] for an ad hoc chosen regularization parameter) shows that our results outperform other motion estimators where the regularization parameter is chosen ad hoc and is in the same range of those approaches where the regularization parameters have been tuned by the known ground truth. In order to demonstrate the usefulness of non-isotropic priors we consider two image sequences with quite different statistics of the gradients of the optical flow components (figure 1 (b) and (d)). These image sequences have been generated by taking two sub-image sequences of the Yosemite sequence. For an image sequence as in (a) where the norm of the gradient of one optical flow component is much larger in one direction an anisotropic prior is more suitable and our experimental results (AAE: 5.72◦ for the anisotropic prior; AAE: 7.08◦ for the isotropic prior using the NL-3D-CLG estimator) support this statement. For the case where the norm of gradients of both optical flow components are in the same range (figure 1(c)), the estimated hyper-parameters βx , βy are also in the same range and the resulting AAE (AAE: 1.74◦ for the isotropic as well as for the anisotropic prior) is equal
pdfs like those proposed in [10] into our framework.
1
1
8. REFERENCES 0 0
(a)
0 0
1
(b)
(c)
1
(d)
Fig. 1. (a): First image of the sub-image sequence: Yosemite Sequence [75:125;1:50], (b): scatter plot of the square of gradients of the computed optical flow components, the horizontal axis depict |∇ux |2 , the vertical axis |∇uy |2 in arbitrary units (c): as in (a) but [75:125;151:200], (d): as in (b) but for the image sequence in (c) to the case of the MAP/MML estimate with an isotropic prior. Another class of image sequences where the optimal choice of the hyper-parameters are essential for a precise motion estimation is the case of image sequence containing scenes with slowly varying illumination (such that the constant brightness assumption approximatively holds). Since the hyperparameters depend substantially on the first order statistic [1] of the image sequence, i.e. the gray value distribution, one gets only optimal results if one adapts the hyper-parameters to the current images. One might argue to scale all considered images such that the first order statistics keeps nearly constant. But this is not realistic in real world applications. Consider for example a scene with overall change of brightness but one or more objects keep constant brightness, such as in a traffic scene. The lights of the vehicles keep constant while the illumination of the background might change. In order to show the dependency on the first order statistics of the image we multiplied each gray value of the ’Office’ sequence by 0.1. The MML/HS approach applied to the original image sequence resulted in an AAE=3.5◦ , the MML/HS estimator applied to the scaled image resulted in an AAE=3.5◦ whereas we obtained an AAE=10.4◦ for the pure HS estimator with the optimized regularization parameter of the original not scaled ’Office’ sequence. Thus, keeping a fixed regularization parameter can lead to large errors in motion estimation scenes with varying first order statistics. 7. SUMMARY AND CONCLUSION
[1] Kai Krajsek and Rudolf Mester, “On the equivalence of variational and statistical differential motion estimation,” in Proc. IEEE SouthWest Symposium on Image Analysis and Interpretation, 2006. [2] Lydia Ng and Victor Solo, “A data-driven method for choosing smoothing parameters in optical flow problems,” in Proc. International Conference on Image Processing, Santa Barbara, California, USA, 1997, pp. 360– 363. [3] A. Bruhn, J. Weickert, and C. Schn¨orr, “Lucas/Kanade meets Horn/Schunk: Combining local and global optical flow methods,” Int. J. Comput. Vision, vol. 61, no. 3, 2005. [4] B. Horn and B. Schunck, “Determining optical flow,” Artificial Intelligence, vol. 17, pp. 185–204, 1981. [5] J. L. Barron, D. J. Fleet, and S. S. Beauchemin, “Performance of optical flow techniques,” Int. Journal of Computer Vision, vol. 12, pp. 43–77, 1994. [6] Joachim Weickert and Christoph Schn¨orr, “A theoretical framework for convex regularizers in pde-based computation of image motion,” Int. J. Comput. Vision, vol. 45, no. 3, pp. 245–264, 2001. [7] M. J. Black and P. Anandan, “A framework for the robust estimation of optical flow,” in Proc. Fourth International Conf. on Computer Vision, (ICCV93), Berlin, Germany, 1993, pp. 231–236. [8] E.P Simoncelli, E. H. Adelson, and D. J. Heeger, “Probability distribution of optical flow,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, 1991, pp. 310–315. [9] Z. Zhou, R. Leahy, and J. Qi, “Approximate maximum likelihood hyperparameter estimation for Gibbs priors,” IEEE Trans. Image Processing, vol. 6, no. 6, pp. 844– 861, 1997.
In this contribution, we present a combined MML/MAP es[10] Stefan Roth and Michael J. Black, “On the spatial statistimator for simultaneously estimating hyper-parameters and tics of optical flow.,” in ICCV, 2005, pp. 42–49. optical flow directly from the observed signal without any prior knowledge of the optical flow. Experiments show that our MML/MAP estimator delivers the optimal hyper-parameters and show the need for optimizing the hyper-parameters to each image sequence for precise motion estimation. Since there are still free parameters left in global motion estimation techniques like the filter size, our future work will focus on deriving optimization schemes for these still free parameters. Another task will be the incorporation of more complex prior
