Variational Bayes and Mean Field Approximations ... - Semantic Scholar
Variational Bayes and Mean Field Approximations for Markov Field Unsupervised Estimation Ali Mohammad-Djafari and Hacheme Ayasso ` Groupe Problemes Inverses ` Laboratoire des Signaux et Systemes(UMR 8506 CNRS - SUPELEC - Univ Paris Sud 11) ´ Supelec, Plateau de Moulon, 91192 Gif-sur-Yvette, FRANCE. E-mail: [email protected], [email protected] MLSP2009, 2-4 September 2009, Grenoble, France
Absract
We consider the problem of parameter estimation of Markovian models where the exact computation of the partition function is not possible or computationally too expensive with MCMC methods. The main idea is then to approximate the expression of the likelihood by a simpler one where we can either have an analytical expression or compute it more efficiently. We consider two approaches: Variational Bayes Approximation (VBA) and Mean Field Approximation (MFA) and study the properties of such approximations and their effects on the estimation of the parameters.
Introduction
Generalized Gaussian (GG): X X β E(x) = |xi − xj |
Gibbs-Markov fields: 1 exp (−λE(x)) , p(x|λ) = Zp (λ) Energy: X Φc (xc ) E(x) =
i
Entropic (I-Distribution family): First kind: E(x) =
E(x) =
X
Maximum likelihood (ML) estimation of λ: ˆ λ {ln p(x|λ)} MV = arg max λ = arg max {− ln Z (λ) − λE(x)} λ
∂λ
j∈V(i)
X X i
exp (−λE(x)) dx
−→
xj ln
xj xi
ln(q(xi )) ∝ −λ − (xj − xi ).
xi ln
j∈V(i)
xi xj
− (xi − xj ).
Potts and Ising models: X X δ(xi − xj ), E(x) = − i
j∈V(i)
Variational Approximations Basics Main objective: Approximate p by simpler q ∈ Q minimizing
= E(x)
Variational energies and entropy Main objective: Approximate p by q minimizing
X
hxj iqj ln
j∈V(i)
with µ ˜i = Entropic 2:
Second kind:
C set of cliques, Φc (.) potential Partition function:Z
−∂ ln Z (λ)
X X i
c∈C
Zp (λ) =
j∈V(i)
VBA, MFA Expressions ˜i ) with Gaussian: q (x ) = N ( µ ˜ , v i i i 1 P 1 ˜i = |V|λ µ ˜ i = |V| j∈V(i) µ ˜ j and v Entropic 1: hxj iqj xi
− xi + hxj iqj
j∈V(i) hxj iqj
P
ln(q(xi )) ∝ −λ
X
j∈V(i)
xi ln e
xi + xi − hxj iqj −hln xj iq j
hln xj iqj with µ ˜i = e P PK −λ j∈V(i) qj (xj =k ) Potts: qi (xi = k ) ∝ k =1 e −
P
j∈V(i)
Simulations ◮Estimation quality Ising
Entropic
q(x) = arg min KL(q : p). q∈Q
and use it for all posterior computations.
Kullback-Leibler divergence KL(q|p) =< − ln
q p
>q = −
Z
q(x) ln
X
q(x) p(x)
dx,
Variational free energy: F (q) = U(q) − H(q) Variational average energy: Z q(x) E(x)dx U(q) =< E(x) >q = X
Variational entropy: H(q) =< − ln q >q = −
Z
q(x) ln q(x)dx X
Two main relations: Variational average energy: U(q) = − ln Z (λ)+ < ln p >q Variational free energy: F (q) = − ln Z (λ) + KL(q|p) = FHelmoltz + KL(q|p).
The main inequality: F (q) ≥ FHelmoltz, with equality when q = p.
Main conclusion:
Minimizing F (q) is a good way to compute FHelmoltz = − ln Z and use it where necessary.
Markov fields in imaging systems x = {x(ri ), ri ∈ R} represent the pixels of an image x(r) ri spatial position of the pixel or voxel number i. Markov fields considered: XX Φi (xi , xj ), E(x) = i
j∈Vi
Expression: p(x|λ) ∝
Y i
p (xi |xj , j ∈ V(i))
No constraint: R Q = {q : q = 1} −→ q = p Separable: P Q = {q : q(x) = j qj (xj )} −→ VBA Parametric: Q = {q : q(x) = qθ (x)} −→ Parametric VBA Separable and parametric −→ MFA Variational Bayes Approximation The solution has the form: 1 qi (xi ) = exp −λ hE(x)iQj6=i qj Zi (λ) Needs the computation of hE(x)iQj6=i qj Analytic expression if p(x) is in Exponential family In our application only parametric form is usable iterative Mean Field Approximation We impose the form as: Y q (xi |a, bj , j ∈ V(i)) q(x|λ) = i Y X exp −a Φ(xi − bj ) , ∝ i
j∈V(i)
Parameteric optimization. sub-optimal solution iterative
◮Computational cost
Conclusions Gauss-Markov-Potts are useful prior models for images incorporating regions and contours Bayesian computation needs often approximations (Laplace, MCMC, Variational Bayes) Application in different CT systems as well as other inverse problems References C. A. Bouman and K. D. Sauer, IEEE Trans. Image Process, vol. 2, pp. 296–310, July 1993. ´ X. Descombes, PhD thesis, Ecole Nationale ´ ´ ecommunications, ´ Superieure des Tel Paris, France, December 1993. J. Giovannelli, Research report, Bordeaux1 univercity, 2009
Absract
We consider the problem of parameter estimation of Markovian models where the exact computation of the partition function is not possible or computationally too expensive with MCMC methods. The main idea is then to approximate the expression of the likelihood by a simpler one where we can either have an analytical expression or compute it more efficiently. We consider two approaches: Variational Bayes Approximation (VBA) and Mean Field Approximation (MFA) and study the properties of such approximations and their effects on the estimation of the parameters.
Introduction
Generalized Gaussian (GG): X X β E(x) = |xi − xj |
Gibbs-Markov fields: 1 exp (−λE(x)) , p(x|λ) = Zp (λ) Energy: X Φc (xc ) E(x) =
i
Entropic (I-Distribution family): First kind: E(x) =
E(x) =
X
Maximum likelihood (ML) estimation of λ: ˆ λ {ln p(x|λ)} MV = arg max λ = arg max {− ln Z (λ) − λE(x)} λ
∂λ
j∈V(i)
X X i
exp (−λE(x)) dx
−→
xj ln
xj xi
ln(q(xi )) ∝ −λ − (xj − xi ).
xi ln
j∈V(i)
xi xj
− (xi − xj ).
Potts and Ising models: X X δ(xi − xj ), E(x) = − i
j∈V(i)
Variational Approximations Basics Main objective: Approximate p by simpler q ∈ Q minimizing
= E(x)
Variational energies and entropy Main objective: Approximate p by q minimizing
X
hxj iqj ln
j∈V(i)
with µ ˜i = Entropic 2:
Second kind:
C set of cliques, Φc (.) potential Partition function:Z
−∂ ln Z (λ)
X X i
c∈C
Zp (λ) =
j∈V(i)
VBA, MFA Expressions ˜i ) with Gaussian: q (x ) = N ( µ ˜ , v i i i 1 P 1 ˜i = |V|λ µ ˜ i = |V| j∈V(i) µ ˜ j and v Entropic 1: hxj iqj xi
− xi + hxj iqj
j∈V(i) hxj iqj
P
ln(q(xi )) ∝ −λ
X
j∈V(i)
xi ln e
xi + xi − hxj iqj −hln xj iq j
hln xj iqj with µ ˜i = e P PK −λ j∈V(i) qj (xj =k ) Potts: qi (xi = k ) ∝ k =1 e −
P
j∈V(i)
Simulations ◮Estimation quality Ising
Entropic
q(x) = arg min KL(q : p). q∈Q
and use it for all posterior computations.
Kullback-Leibler divergence KL(q|p) =< − ln
q p
>q = −
Z
q(x) ln
X
q(x) p(x)
dx,
Variational free energy: F (q) = U(q) − H(q) Variational average energy: Z q(x) E(x)dx U(q) =< E(x) >q = X
Variational entropy: H(q) =< − ln q >q = −
Z
q(x) ln q(x)dx X
Two main relations: Variational average energy: U(q) = − ln Z (λ)+ < ln p >q Variational free energy: F (q) = − ln Z (λ) + KL(q|p) = FHelmoltz + KL(q|p).
The main inequality: F (q) ≥ FHelmoltz, with equality when q = p.
Main conclusion:
Minimizing F (q) is a good way to compute FHelmoltz = − ln Z and use it where necessary.
Markov fields in imaging systems x = {x(ri ), ri ∈ R} represent the pixels of an image x(r) ri spatial position of the pixel or voxel number i. Markov fields considered: XX Φi (xi , xj ), E(x) = i
j∈Vi
Expression: p(x|λ) ∝
Y i
p (xi |xj , j ∈ V(i))
No constraint: R Q = {q : q = 1} −→ q = p Separable: P Q = {q : q(x) = j qj (xj )} −→ VBA Parametric: Q = {q : q(x) = qθ (x)} −→ Parametric VBA Separable and parametric −→ MFA Variational Bayes Approximation The solution has the form: 1 qi (xi ) = exp −λ hE(x)iQj6=i qj Zi (λ) Needs the computation of hE(x)iQj6=i qj Analytic expression if p(x) is in Exponential family In our application only parametric form is usable iterative Mean Field Approximation We impose the form as: Y q (xi |a, bj , j ∈ V(i)) q(x|λ) = i Y X exp −a Φ(xi − bj ) , ∝ i
j∈V(i)
Parameteric optimization. sub-optimal solution iterative
◮Computational cost
Conclusions Gauss-Markov-Potts are useful prior models for images incorporating regions and contours Bayesian computation needs often approximations (Laplace, MCMC, Variational Bayes) Application in different CT systems as well as other inverse problems References C. A. Bouman and K. D. Sauer, IEEE Trans. Image Process, vol. 2, pp. 296–310, July 1993. ´ X. Descombes, PhD thesis, Ecole Nationale ´ ´ ecommunications, ´ Superieure des Tel Paris, France, December 1993. J. Giovannelli, Research report, Bordeaux1 univercity, 2009
