Expectation-Maximization Bernoulli-Gaussian Approximate Message ...
Expectation-Maximization Bernoulli-Gaussian Approximate Message Passing Asilomar 2011 Jeremy Vila and Philip Schniter The Ohio State University Department of Electrical and Computer Engineering [email protected] and [email protected]
November 8th, 2011
This work has been supported in part by NSF-I/UCRC grant IIP-0968910, by NSF grant CCF-1018368, and by DARPA/ONR grant N66001-10-1-4090.
CS Problem Statement Recover a signal from undersampled measurements x ∈ RN
y = Ax + w
y, w ∈ RM
M 0.65! Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
11 / 19
Noisy Signal Recovery We now compare EM-BG-GAMP to state-of-the-art CS algorithms for noisy signal recovery using normalized MSE. For BG signals, fix N = 1000, K = 100, SNR = 25dB and vary M. genie Lasso
The other “Bayesian” approaches, BCS and SBL, exhibit the next best performance.
SL0 T-MSBL BCS EM-BG-GAMP
−5
NMSE [dB]
EM-BG-GAMP outperforms the other algorithms for all meaningful M/N.
−10 −15 −20 −25 0.2
0.25
0.3
0.35
0.4
0.45
0.5
M/N Noisy Bernoulli-Gaussian recovery NMSE.
Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
12 / 19
Noisy Signal Recovery (cont.) We also see excellent NMSE for other K -sparse distributions: 0
genie Lasso SL0 T-MSBL BCS EM-BG-GAMP
−5
−5
NMSE [dB]
NMSE [dB]
0
−10 −15 genie Lasso
−20
SL0
−10 −15 −20 −25
T-MSBL
−25
−30
BCS EM-BG-GAMP
−30 0.2
0.25
0.3
0.35
0.4
0.45
0.5
−35 0.2
M/N
0.25
0.3
0.35
0.4
0.45
0.5
M/N
Noisy Bernoulli-Rademacher recovery NMSE.
Noisy Bernoulli recovery NMSE.
For Bernoulli signals especially, EM-BG-GAMP exhibits a huge improvement over the other algorithms.
Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
13 / 19
Algorithm Complexity We now compare algorithm complexity. Fix M = 0.5N, K = 0.1N, SNR = 25dB, and vary N. Results averaged over 50 iterations. 2
−20
NMSE [dB]
Runtime [sec]
10
1
10
T-MSBL BCS SL0
0
10
EM-BG-GAMP
T-MSBL BCS SL0 EM-BG-GAMP SPGL1 SPGL1 fft EM-BG-GAMP fft
−25
−30
SPGL1 SPGL1 fft EM-BG-GAMP fft −1
10
3
10
4
5
10
10
6
10
−35 3 10
Noisy Bernoulli-Rademacher recovery time.
4
5
10
10
6
10
N
N
Noisy Bernoulli-Rademacher recovery NMSE.
For large N, EM-BG-AMP has state-of-the-art complexity. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
14 / 19
EM-BG-GAMP Limitations EM-BG-GAMP is outperformed by genie-LASSO and SL0 with a non-compressible Student’s-t signal. genie Lasso
NMSE [dB]
−5
SL0 T-MSBL BCS EM-BG-GAMP
−6 −7 −8 −9 −10 0.3
0.35
0.4
0.45
0.5
0.55
0.6
M/N Noisy Student’s-t recovery NMSE.
Interestingly, the algorithms that performed best for sparse signals performed the worse for the Student’s-t. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
15 / 19
Conclusions We proposed an extension of BG-AMP wherein the signal and noise distributional parameters were automatically learned via the EM algorithm. Advantages of EM-BG-AMP State-of-the-art NMSE performance for a wide class of signal/matrix types. State-of-the-art complexity scaling as problem dimensions get large. No tuning parameters.
Limitations of EM-BG-AMP If the true signal/noise pdfs cannot be well matched by BG/Gaussian priors, then performance may suffer.
To address this limitation, we are working on a Gaussian-Mixture version (EM-GM-AMP) with automatic selection of the mixture order. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
16 / 19
EM-GM-GAMP Teaser Our new EM-GM-GAMP algorithm may alleviate the shortcomings seen in recovering a non-compressible Student’s-t signal. EM-GM-GAMP genie Lasso
NMSE [dB]
−5
SL0 T-MSBL BCS EM-BG-GAMP
−6 −7 −8 −9 −10 0.2
0.25
0.3
0.35
0.4
0.45
0.5
M/N Noisy Student’s-t recovery NMSE.
Details coming soon. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
17 / 19
Matlab code is publicly available at http://ece.osu.edu/~vilaj/EMBGAMP/EMBGAMP.html
Thanks!
Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
18 / 19
Explicit Results GAMP outputs: xˆ
=
π(ˆ r , ν r ; q) γ(ˆ r , ν r ; q)
νx
=
2 π(ˆ r , ν r ; q) β(ˆ r , ν r ; q) + |γ(ˆ r , ν r ; q)|2 − π(ˆ r , ν r ; q) |γ(ˆ r , ν r ; q)|2 ,
where p(s = 1|y ) , π(ˆ r , ν r ; q) ,
1 1+
r ;θ,φ+ν r ) −1 λ N (ˆ 1−λ N (ˆ r ;0,ν r )
ˆ r /ν r + θ/φ 1/ν r + 1/φ 1 var(x|y , s = 1) , β(ˆ r , ν r ; q) , . 1/ν r + 1/φ E [x|y , s = 1] , γ(ˆ r , ν r ; q) ,
EM updates: λi+1 =
φi+1 =
N 1 X π(ˆ rn , νnr ; qi ). N n=1
1 λi+1 N
N X n=1
Jeremy Vila and Philip Schniter (OSU)
θ i+1 =
1 λi+1 N
N X
π(ˆ rn , νnr ; qi )γ(ˆ rn , νnr ; qi )
n=1
2 π(ˆ rn , νnr , qi ) θ i − γ(ˆ rn , νnr ; qi ) + β(ˆ rn , νnr ; qi ) EM-BG-AMP
November 8th, 2011
19 / 19
November 8th, 2011
This work has been supported in part by NSF-I/UCRC grant IIP-0968910, by NSF grant CCF-1018368, and by DARPA/ONR grant N66001-10-1-4090.
CS Problem Statement Recover a signal from undersampled measurements x ∈ RN
y = Ax + w
y, w ∈ RM
M 0.65! Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
11 / 19
Noisy Signal Recovery We now compare EM-BG-GAMP to state-of-the-art CS algorithms for noisy signal recovery using normalized MSE. For BG signals, fix N = 1000, K = 100, SNR = 25dB and vary M. genie Lasso
The other “Bayesian” approaches, BCS and SBL, exhibit the next best performance.
SL0 T-MSBL BCS EM-BG-GAMP
−5
NMSE [dB]
EM-BG-GAMP outperforms the other algorithms for all meaningful M/N.
−10 −15 −20 −25 0.2
0.25
0.3
0.35
0.4
0.45
0.5
M/N Noisy Bernoulli-Gaussian recovery NMSE.
Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
12 / 19
Noisy Signal Recovery (cont.) We also see excellent NMSE for other K -sparse distributions: 0
genie Lasso SL0 T-MSBL BCS EM-BG-GAMP
−5
−5
NMSE [dB]
NMSE [dB]
0
−10 −15 genie Lasso
−20
SL0
−10 −15 −20 −25
T-MSBL
−25
−30
BCS EM-BG-GAMP
−30 0.2
0.25
0.3
0.35
0.4
0.45
0.5
−35 0.2
M/N
0.25
0.3
0.35
0.4
0.45
0.5
M/N
Noisy Bernoulli-Rademacher recovery NMSE.
Noisy Bernoulli recovery NMSE.
For Bernoulli signals especially, EM-BG-GAMP exhibits a huge improvement over the other algorithms.
Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
13 / 19
Algorithm Complexity We now compare algorithm complexity. Fix M = 0.5N, K = 0.1N, SNR = 25dB, and vary N. Results averaged over 50 iterations. 2
−20
NMSE [dB]
Runtime [sec]
10
1
10
T-MSBL BCS SL0
0
10
EM-BG-GAMP
T-MSBL BCS SL0 EM-BG-GAMP SPGL1 SPGL1 fft EM-BG-GAMP fft
−25
−30
SPGL1 SPGL1 fft EM-BG-GAMP fft −1
10
3
10
4
5
10
10
6
10
−35 3 10
Noisy Bernoulli-Rademacher recovery time.
4
5
10
10
6
10
N
N
Noisy Bernoulli-Rademacher recovery NMSE.
For large N, EM-BG-AMP has state-of-the-art complexity. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
14 / 19
EM-BG-GAMP Limitations EM-BG-GAMP is outperformed by genie-LASSO and SL0 with a non-compressible Student’s-t signal. genie Lasso
NMSE [dB]
−5
SL0 T-MSBL BCS EM-BG-GAMP
−6 −7 −8 −9 −10 0.3
0.35
0.4
0.45
0.5
0.55
0.6
M/N Noisy Student’s-t recovery NMSE.
Interestingly, the algorithms that performed best for sparse signals performed the worse for the Student’s-t. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
15 / 19
Conclusions We proposed an extension of BG-AMP wherein the signal and noise distributional parameters were automatically learned via the EM algorithm. Advantages of EM-BG-AMP State-of-the-art NMSE performance for a wide class of signal/matrix types. State-of-the-art complexity scaling as problem dimensions get large. No tuning parameters.
Limitations of EM-BG-AMP If the true signal/noise pdfs cannot be well matched by BG/Gaussian priors, then performance may suffer.
To address this limitation, we are working on a Gaussian-Mixture version (EM-GM-AMP) with automatic selection of the mixture order. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
16 / 19
EM-GM-GAMP Teaser Our new EM-GM-GAMP algorithm may alleviate the shortcomings seen in recovering a non-compressible Student’s-t signal. EM-GM-GAMP genie Lasso
NMSE [dB]
−5
SL0 T-MSBL BCS EM-BG-GAMP
−6 −7 −8 −9 −10 0.2
0.25
0.3
0.35
0.4
0.45
0.5
M/N Noisy Student’s-t recovery NMSE.
Details coming soon. Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
17 / 19
Matlab code is publicly available at http://ece.osu.edu/~vilaj/EMBGAMP/EMBGAMP.html
Thanks!
Jeremy Vila and Philip Schniter (OSU)
EM-BG-AMP
November 8th, 2011
18 / 19
Explicit Results GAMP outputs: xˆ
=
π(ˆ r , ν r ; q) γ(ˆ r , ν r ; q)
νx
=
2 π(ˆ r , ν r ; q) β(ˆ r , ν r ; q) + |γ(ˆ r , ν r ; q)|2 − π(ˆ r , ν r ; q) |γ(ˆ r , ν r ; q)|2 ,
where p(s = 1|y ) , π(ˆ r , ν r ; q) ,
1 1+
r ;θ,φ+ν r ) −1 λ N (ˆ 1−λ N (ˆ r ;0,ν r )
ˆ r /ν r + θ/φ 1/ν r + 1/φ 1 var(x|y , s = 1) , β(ˆ r , ν r ; q) , . 1/ν r + 1/φ E [x|y , s = 1] , γ(ˆ r , ν r ; q) ,
EM updates: λi+1 =
φi+1 =
N 1 X π(ˆ rn , νnr ; qi ). N n=1
1 λi+1 N
N X n=1
Jeremy Vila and Philip Schniter (OSU)
θ i+1 =
1 λi+1 N
N X
π(ˆ rn , νnr ; qi )γ(ˆ rn , νnr ; qi )
n=1
2 π(ˆ rn , νnr , qi ) θ i − γ(ˆ rn , νnr ; qi ) + β(ˆ rn , νnr ; qi ) EM-BG-AMP
November 8th, 2011
19 / 19
