Noisy Signal Recovery via Iterative Reweighted L1 ... - Semantic Scholar
Introduction
L1-Minimization
Reweighted L1
Main Results
Noisy Signal Recovery via Iterative Reweighted L1-Minimization Deanna Needell UC Davis / Stanford University
Asilomar SSC, November 2009
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
What? Theorem 1 I have injured vocal chords and can’t speak loudly. I’m not nervous, it’s just the voice!
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Setup 1 2 3
Suppose x is an unknown signal in Rd . Design measurement matrix Φ : Rd → Rm . Collect noisy measurements u = Φx + e.
u =
4 5
x + e
Φ
Problem: Reconstruct signal x from measurements u Wait, isn’t this impossible? def
Assume x is s-sparse: kxk0 = | supp(x)| ≤ s ≪ d. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Setup 1 2 3
Suppose x is an unknown signal in Rd . Design measurement matrix Φ : Rd → Rm . Collect noisy measurements u = Φx + e.
u =
4 5
x + e
Φ
Problem: Reconstruct signal x from measurements u Wait, isn’t this impossible? def
Assume x is s-sparse: kxk0 = | supp(x)| ≤ s ≪ d. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Setup 1 2 3
Suppose x is an unknown signal in Rd . Design measurement matrix Φ : Rd → Rm . Collect noisy measurements u = Φx + e.
u =
4 5
x + e
Φ
Problem: Reconstruct signal x from measurements u Wait, isn’t this impossible? def
Assume x is s-sparse: kxk0 = | supp(x)| ≤ s ≪ d. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Applications Compressive Imaging Computational Biology Medical Imaging Astronomy Geophysical Data Analysis Compressive Radar Many more (see www.dsp.ece.rice.edu)
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How can we reconstruct? Obvious way: Suppose the matrix Φ is one-to-one on the set of sparse vectors and e = 0. Set xˆ = argmin ||z||0
such that Φz = u.
Then xˆ = x! Bad news: This would require a search through numerically feasible.
D. Needell
d s
subspaces! Not
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How can we reconstruct? Obvious way: Suppose the matrix Φ is one-to-one on the set of sparse vectors and e = 0. Set xˆ = argmin ||z||0
such that Φz = u.
Then xˆ = x! Bad news: This would require a search through numerically feasible.
D. Needell
d s
subspaces! Not
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How else can we reconstruct?
Geometric Idea Minimizing the ℓ0 -ball is too hard, so let’s try a different one. Our favorites... Least Squares L1-Minimization (using Linear Programming) Which one?
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How else can we reconstruct?
Geometric Idea Minimizing the ℓ0 -ball is too hard, so let’s try a different one. Our favorites... Least Squares L1-Minimization (using Linear Programming) Which one?
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Which one?
x* x
x = x*
Figure: Minimizing the ℓ2 versus the ℓ1 balls. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
What do we assume about Φ? Restricted Isometry Property (RIP) The sth restricted isometry constant of Φ is the smallest δs such that (1 − δs )kxk2 ≤ kΦxk2 ≤ (1 + δs )kxk2
whenever kxk0 ≤ s.
For Gaussian or Bernoulli measurement matrices, with high probability δs ≤ c < 1 when m & s log d. Random Fourier and others with fast multiply have similar property. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
What do we assume about Φ? Restricted Isometry Property (RIP) The sth restricted isometry constant of Φ is the smallest δs such that (1 − δs )kxk2 ≤ kΦxk2 ≤ (1 + δs )kxk2
whenever kxk0 ≤ s.
For Gaussian or Bernoulli measurement matrices, with high probability δs ≤ c < 1 when m & s log d. Random Fourier and others with fast multiply have similar property. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Proven Results
L1-Minimization [Cand`es-Tao] Assume √ that the measurement matrix Φ satisfies the RIP with δ2s < 2 − 1. Then every s-sparse vector x can be exactly recovered from its measurements u = Φx as a unique solution to the linear optimization problem: xˆ = argmin ||z||1
D. Needell
such that Φz = u.
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Numerical Results Percentage of flat signals recovered with BP, d=256 100 90
Percentage recovered
80
s=4 s=12
70
s=20 60
s=28 s=36
50 40 30 20 10 0
0
50
100 150 Measurements m
200
250
Figure: The percentage of sparse flat signals exactly recovered by Basis Pursuit as a function of the number of measurements m in dimension d = 256 for various levels of sparsity s. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Numerical Results Recovery Error from BP on Perturbed Measurements (by Gaussian noise), d=256 14
12 s=4 s=12
Average Error
10
s=20 s=28
8
s=36
6
4
2
0 60
80
100
120
140 160 180 Measurements m
200
220
240
Figure: The recovery error of L1-Minimization under perturbed measurements (kek2 = 0.5) as a function of the number of measurements m in dimension d = 256 for various levels of sparsity s. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What if we are close?
Suppose we recover xˆ ≈ x
Most likely, this means xˆi ≈ xi
In particular, xˆi is small/large when xi is small/large Weighted L1 xˆ
(2)
d X zi = argmin xˆi z i =1
D. Needell
such that
kΦz − uk2 ≤ ε
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What if we are close?
Suppose we recover xˆ ≈ x
Most likely, this means xˆi ≈ xi
In particular, xˆi is small/large when xi is small/large Weighted L1 xˆ
(2)
d X zi = argmin xˆi z i =1
D. Needell
such that
kΦz − uk2 ≤ ε
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What if we are close?
Suppose we recover xˆ ≈ x
Most likely, this means xˆi ≈ xi
In particular, xˆi is small/large when xi is small/large Weighted L1 xˆ
(2)
d X zi = argmin xˆi + a z i =1
D. Needell
such that
kΦz − uk2 ≤ ε
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Weighted Geometry x = x* x
x*
Figure: The geometry of the weighted ℓ1 -ball.
Noise-free case: In cases where xˆ 6= x, we should have that xˆ(2) is closer to x, or even equal. Noisy case: This implies xˆ(2) should be closer to x than xˆ was. Can we repeat this again? D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Reweighted ℓ1 -minimization (RWL1) Input: Measurement vector u ∈ Rm , stability parameter a Output: Reconstructed vector xˆ Initialize Set the weights wi = 1 for i = 1 . . . d. Approximate Solve the reweighted ℓ1 -minimization problem: xˆ = argmin xˆ∈Rd
d X i =1
wi xˆi subject to kΦˆ x − uk2 ≤ ε.
Update Reset the weights: wi =
D. Needell
1 . |ˆ xi | + a
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results
Sup norm errors after 1 iteration of (reweighted) BP
Sup norm errors after nine iterations of reweighted BP
1.4
1.4
1.2
1.2
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
100
200
300
400
500
0
0
100
200
300
400
500
Figure: ℓ∞ -norm error for reweighted L1 in the noise-free case
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results
Figure: Probability of reconstruction [Cand`es-Wakin-Boyd].
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results with noise Improvement by reweighted L1 after 9 iterations for d=256 m=128 s=30 with decreasing e 150
Reconstruction error for d=256 m=128 s=30 with decreasing e 6 One iteration 9 Iterations 5
100 Number of trials
Error
4
3
2
50
1
0
0
100
200
300
400
500
Trials
0 0.1
0.2
0.3
0.4 0.5 Improvement factor
0.6
0.7
0.8
Figure: Improvements in the ℓ2 reconstruction error using reweighted ℓ1 -minimization versus standard ℓ1 -minimization for sparse Gaussian signals.
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results with noise Reconstruction error for d=256 m=128 s=30 with decreasing e One iteration
5
Improvement by reweighted L1 after 9 iterations for d=256 m=128 s=30 with decreasing e 140
9 Iterations
4.5
120
4 100 Number of trials
Error
3.5 3 2.5
80
60
2 40 1.5 20
1 0.5
0
100
200
300
400
500
Trials
0 0.1
0.2
0.3
0.4
0.5 0.6 0.7 Improvement factor
0.8
0.9
1
Figure: Improvements in the ℓ2 reconstruction error using reweighted ℓ1 -minimization versus standard ℓ1 -minimization for sparse Bernoulli signals.
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Observations
The noiseless case suggests that an ℓ∞ -norm bound may be required for RWL1 to succeed. In the noisy case it is clear that we cannot recover signal coordinates that are below some threshold. If each iteration of RWL1 improves the error, perhaps we should take a → 0. (Recall wi = |ˆxi 1|+a ).
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Observations
The noiseless case suggests that an ℓ∞ -norm bound may be required for RWL1 to succeed. In the noisy case it is clear that we cannot recover signal coordinates that are below some threshold. If each iteration of RWL1 improves the error, perhaps we should take a → 0. (Recall wi = |ˆxi 1|+a ).
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Observations
The noiseless case suggests that an ℓ∞ -norm bound may be required for RWL1 to succeed. In the noisy case it is clear that we cannot recover signal coordinates that are below some threshold. If each iteration of RWL1 improves the error, perhaps we should take a → 0. (Recall wi = |ˆxi 1|+a ).
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Main Results
RWL1 - Sparse case [N]
√ Assume Φ satisfies the RIP with δ2s ≤ δ where δ < 2 − 1. Let x be an s-sparse vector with noisy measurements u = Φx + e where kek2 ≤ ε. Assume the smallest nonzero coordinate µ of x satisfies 4αε . Then the limiting approximation from reweighted µ ≥ 1−ρ ℓ1 -minimization satisfies
where C ′′ =
2α 1+ρ ,
ρ=
√
kx − xˆk2 ≤ C ′′ ε,
2δ 1−δ
and α =
D. Needell
√ 2 1+δ 1−δ .
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Extension RWL1 - non-sparse extension [N]
√ Assume Φ satisfies the RIP with δ2s ≤ 2 − 1. Let x be an arbitrary vector with noisy measurements u = Φx + e where kek2 ≤ ε. Assume the smallest nonzero coordinate µ of xs satisfies 0 √1 µ ≥ 4αε 1−ρ , where ε0 = 1.2(kx − xs k2 + s kx − xs k1 ) + ε. Then the limiting approximation from reweighted ℓ1 -minimization satisfies kx − xˆk2 ≤ and kx − xˆk2 ≤
4.1α kx − xs/2 k1 √ +ε , 1+ρ s
kx − xs k1 2.4α √ kx − xs k2 + +ε , 1+ρ s
where ρ and α are as before. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Theoretical Results
3
2
1
0
(b) Number of iterations til convergence, M=10, e=0.1 6
Iterations til Convergence
Iterations til Convergence
(a) Number of iterations til convergence, M=10, e=0.01 4
0
0.1
0.2 δ value
10 8 6 4 2 0.05
0.1 0.15 δ value
0.2
3 2 1
0.25
0
0.1
0.2 δ value
0.3
(d) Number of iterations til convergence, M=10, e=1 30
Iterations til Convergence
Iterations til Convergence
12
0
4
0
0.3
(c) Number of iterations til convergence, M=10, e=0.5
0
5
25 20 15 10 5 0
0
0.02
0.04 δ value
0.06
0.08
Figure: Number of iterations required for theoretical error bounds to reach limiting theoretical error when (a) µ = 10, ε = 0.01, (b) µ = 10, ε = 0.1, (c) µ = 10, ε = 0.5, (d) µ = 10, ε = 1.0. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Recent work
Wipf-Nagarajan elaborate on convergence and show connections to reweighted ℓ2 -minimization. Wipf-Nagarajan also show that a non-separable variant has desirable properties. Xu-Khajehnejad-Avestimehr-Hassibi provide a theoretical foundation for the analysis of RWL1 and show that for a nontrivial class of signals, a variant of RWL1 indeed can improve upon L1 in the noiseless case.
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Thank you
For more information E-mail: [email protected] Web: http://www-stat.stanford.edu/~dneedell References: Candes, Wakin, Boyd, “Enhancing sparsity by reweighted ℓ1 minimization”, J. Fourier Anal. Appl., 14 877-905. Needell, “Noisy signal recovery via iterative reweighted L1-minimization,” 2009. Wipf and Nagarajan, “Solving Sparse Linear Inverse Problems: Analysis of Reweighted ℓ1 and ℓ2 Methods,” J. of Selected Topics in Signal Processing, Special Issue on Compressive Sensing, 2010. Xu, Khajehnejad, Avestimehr, and Hassibi, “Breaking through the Thresholds: an Analysis for Iterative Reweighted ℓ1 Minimization via the Grassmann Angle Framework,” Proc. Allerton Conference on Communication, Control, and computing, Sept. 2009. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
L1-Minimization
Reweighted L1
Main Results
Noisy Signal Recovery via Iterative Reweighted L1-Minimization Deanna Needell UC Davis / Stanford University
Asilomar SSC, November 2009
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
What? Theorem 1 I have injured vocal chords and can’t speak loudly. I’m not nervous, it’s just the voice!
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Setup 1 2 3
Suppose x is an unknown signal in Rd . Design measurement matrix Φ : Rd → Rm . Collect noisy measurements u = Φx + e.
u =
4 5
x + e
Φ
Problem: Reconstruct signal x from measurements u Wait, isn’t this impossible? def
Assume x is s-sparse: kxk0 = | supp(x)| ≤ s ≪ d. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Setup 1 2 3
Suppose x is an unknown signal in Rd . Design measurement matrix Φ : Rd → Rm . Collect noisy measurements u = Φx + e.
u =
4 5
x + e
Φ
Problem: Reconstruct signal x from measurements u Wait, isn’t this impossible? def
Assume x is s-sparse: kxk0 = | supp(x)| ≤ s ≪ d. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Setup 1 2 3
Suppose x is an unknown signal in Rd . Design measurement matrix Φ : Rd → Rm . Collect noisy measurements u = Φx + e.
u =
4 5
x + e
Φ
Problem: Reconstruct signal x from measurements u Wait, isn’t this impossible? def
Assume x is s-sparse: kxk0 = | supp(x)| ≤ s ≪ d. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Applications Compressive Imaging Computational Biology Medical Imaging Astronomy Geophysical Data Analysis Compressive Radar Many more (see www.dsp.ece.rice.edu)
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How can we reconstruct? Obvious way: Suppose the matrix Φ is one-to-one on the set of sparse vectors and e = 0. Set xˆ = argmin ||z||0
such that Φz = u.
Then xˆ = x! Bad news: This would require a search through numerically feasible.
D. Needell
d s
subspaces! Not
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How can we reconstruct? Obvious way: Suppose the matrix Φ is one-to-one on the set of sparse vectors and e = 0. Set xˆ = argmin ||z||0
such that Φz = u.
Then xˆ = x! Bad news: This would require a search through numerically feasible.
D. Needell
d s
subspaces! Not
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How else can we reconstruct?
Geometric Idea Minimizing the ℓ0 -ball is too hard, so let’s try a different one. Our favorites... Least Squares L1-Minimization (using Linear Programming) Which one?
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
How else can we reconstruct?
Geometric Idea Minimizing the ℓ0 -ball is too hard, so let’s try a different one. Our favorites... Least Squares L1-Minimization (using Linear Programming) Which one?
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
Which one?
x* x
x = x*
Figure: Minimizing the ℓ2 versus the ℓ1 balls. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
What do we assume about Φ? Restricted Isometry Property (RIP) The sth restricted isometry constant of Φ is the smallest δs such that (1 − δs )kxk2 ≤ kΦxk2 ≤ (1 + δs )kxk2
whenever kxk0 ≤ s.
For Gaussian or Bernoulli measurement matrices, with high probability δs ≤ c < 1 when m & s log d. Random Fourier and others with fast multiply have similar property. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Problem Background
What do we assume about Φ? Restricted Isometry Property (RIP) The sth restricted isometry constant of Φ is the smallest δs such that (1 − δs )kxk2 ≤ kΦxk2 ≤ (1 + δs )kxk2
whenever kxk0 ≤ s.
For Gaussian or Bernoulli measurement matrices, with high probability δs ≤ c < 1 when m & s log d. Random Fourier and others with fast multiply have similar property. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Proven Results
L1-Minimization [Cand`es-Tao] Assume √ that the measurement matrix Φ satisfies the RIP with δ2s < 2 − 1. Then every s-sparse vector x can be exactly recovered from its measurements u = Φx as a unique solution to the linear optimization problem: xˆ = argmin ||z||1
D. Needell
such that Φz = u.
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Numerical Results Percentage of flat signals recovered with BP, d=256 100 90
Percentage recovered
80
s=4 s=12
70
s=20 60
s=28 s=36
50 40 30 20 10 0
0
50
100 150 Measurements m
200
250
Figure: The percentage of sparse flat signals exactly recovered by Basis Pursuit as a function of the number of measurements m in dimension d = 256 for various levels of sparsity s. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What about noise? Noisy Formulation For a non-sparse vector x with noisy measurements u = Φx + e, xˆ = argmin ||z||1
such that kΦz − uk2 ≤ ε.
(1)
L1-Minimization [Cand`es-Romberg-Tao] Let Φ √ be a measurement matrix satisfying the RIP with δ2s < 2 − 1. Then for any arbitrary signal and corrupted measurements u = Φx + e with kek2 ≤ ε, the solution xˆ to (1) satisfies kx − xs k1 √ . kˆ x − xk2 ≤ Cs · ε + Cs′ · s √ Note: As δ2s → 2 − 1, Cs , Cs′ → ∞!! D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Numerical Results Recovery Error from BP on Perturbed Measurements (by Gaussian noise), d=256 14
12 s=4 s=12
Average Error
10
s=20 s=28
8
s=36
6
4
2
0 60
80
100
120
140 160 180 Measurements m
200
220
240
Figure: The recovery error of L1-Minimization under perturbed measurements (kek2 = 0.5) as a function of the number of measurements m in dimension d = 256 for various levels of sparsity s. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What if we are close?
Suppose we recover xˆ ≈ x
Most likely, this means xˆi ≈ xi
In particular, xˆi is small/large when xi is small/large Weighted L1 xˆ
(2)
d X zi = argmin xˆi z i =1
D. Needell
such that
kΦz − uk2 ≤ ε
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What if we are close?
Suppose we recover xˆ ≈ x
Most likely, this means xˆi ≈ xi
In particular, xˆi is small/large when xi is small/large Weighted L1 xˆ
(2)
d X zi = argmin xˆi z i =1
D. Needell
such that
kΦz − uk2 ≤ ε
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
What if we are close?
Suppose we recover xˆ ≈ x
Most likely, this means xˆi ≈ xi
In particular, xˆi is small/large when xi is small/large Weighted L1 xˆ
(2)
d X zi = argmin xˆi + a z i =1
D. Needell
such that
kΦz − uk2 ≤ ε
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Results
Weighted Geometry x = x* x
x*
Figure: The geometry of the weighted ℓ1 -ball.
Noise-free case: In cases where xˆ 6= x, we should have that xˆ(2) is closer to x, or even equal. Noisy case: This implies xˆ(2) should be closer to x than xˆ was. Can we repeat this again? D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Reweighted ℓ1 -minimization (RWL1) Input: Measurement vector u ∈ Rm , stability parameter a Output: Reconstructed vector xˆ Initialize Set the weights wi = 1 for i = 1 . . . d. Approximate Solve the reweighted ℓ1 -minimization problem: xˆ = argmin xˆ∈Rd
d X i =1
wi xˆi subject to kΦˆ x − uk2 ≤ ε.
Update Reset the weights: wi =
D. Needell
1 . |ˆ xi | + a
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results
Sup norm errors after 1 iteration of (reweighted) BP
Sup norm errors after nine iterations of reweighted BP
1.4
1.4
1.2
1.2
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
100
200
300
400
500
0
0
100
200
300
400
500
Figure: ℓ∞ -norm error for reweighted L1 in the noise-free case
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results
Figure: Probability of reconstruction [Cand`es-Wakin-Boyd].
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results with noise Improvement by reweighted L1 after 9 iterations for d=256 m=128 s=30 with decreasing e 150
Reconstruction error for d=256 m=128 s=30 with decreasing e 6 One iteration 9 Iterations 5
100 Number of trials
Error
4
3
2
50
1
0
0
100
200
300
400
500
Trials
0 0.1
0.2
0.3
0.4 0.5 Improvement factor
0.6
0.7
0.8
Figure: Improvements in the ℓ2 reconstruction error using reweighted ℓ1 -minimization versus standard ℓ1 -minimization for sparse Gaussian signals.
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Numerical Results with noise Reconstruction error for d=256 m=128 s=30 with decreasing e One iteration
5
Improvement by reweighted L1 after 9 iterations for d=256 m=128 s=30 with decreasing e 140
9 Iterations
4.5
120
4 100 Number of trials
Error
3.5 3 2.5
80
60
2 40 1.5 20
1 0.5
0
100
200
300
400
500
Trials
0 0.1
0.2
0.3
0.4
0.5 0.6 0.7 Improvement factor
0.8
0.9
1
Figure: Improvements in the ℓ2 reconstruction error using reweighted ℓ1 -minimization versus standard ℓ1 -minimization for sparse Bernoulli signals.
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Observations
The noiseless case suggests that an ℓ∞ -norm bound may be required for RWL1 to succeed. In the noisy case it is clear that we cannot recover signal coordinates that are below some threshold. If each iteration of RWL1 improves the error, perhaps we should take a → 0. (Recall wi = |ˆxi 1|+a ).
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Observations
The noiseless case suggests that an ℓ∞ -norm bound may be required for RWL1 to succeed. In the noisy case it is clear that we cannot recover signal coordinates that are below some threshold. If each iteration of RWL1 improves the error, perhaps we should take a → 0. (Recall wi = |ˆxi 1|+a ).
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Reweighted L1-Minimization
Observations
The noiseless case suggests that an ℓ∞ -norm bound may be required for RWL1 to succeed. In the noisy case it is clear that we cannot recover signal coordinates that are below some threshold. If each iteration of RWL1 improves the error, perhaps we should take a → 0. (Recall wi = |ˆxi 1|+a ).
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Main Results
RWL1 - Sparse case [N]
√ Assume Φ satisfies the RIP with δ2s ≤ δ where δ < 2 − 1. Let x be an s-sparse vector with noisy measurements u = Φx + e where kek2 ≤ ε. Assume the smallest nonzero coordinate µ of x satisfies 4αε . Then the limiting approximation from reweighted µ ≥ 1−ρ ℓ1 -minimization satisfies
where C ′′ =
2α 1+ρ ,
ρ=
√
kx − xˆk2 ≤ C ′′ ε,
2δ 1−δ
and α =
D. Needell
√ 2 1+δ 1−δ .
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Remarks
Without noise, this result coincides with previous results on L1. √ The key improvement: As δ → 2 − 1, C ′′ remains bounded.
The error bound is the limiting bound, but a recursive relation in the proof gives exact improvements per iteration. We show in practice it is attained quite quickly. For signals whose smallest non-zero coefficient µ does not satisfy the condition of the theorem, we may apply the theorem to those coefficients that do satisfy this requirement, and treat the others as noise...
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Extension RWL1 - non-sparse extension [N]
√ Assume Φ satisfies the RIP with δ2s ≤ 2 − 1. Let x be an arbitrary vector with noisy measurements u = Φx + e where kek2 ≤ ε. Assume the smallest nonzero coordinate µ of xs satisfies 0 √1 µ ≥ 4αε 1−ρ , where ε0 = 1.2(kx − xs k2 + s kx − xs k1 ) + ε. Then the limiting approximation from reweighted ℓ1 -minimization satisfies kx − xˆk2 ≤ and kx − xˆk2 ≤
4.1α kx − xs/2 k1 √ +ε , 1+ρ s
kx − xs k1 2.4α √ kx − xs k2 + +ε , 1+ρ s
where ρ and α are as before. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Theoretical Results
3
2
1
0
(b) Number of iterations til convergence, M=10, e=0.1 6
Iterations til Convergence
Iterations til Convergence
(a) Number of iterations til convergence, M=10, e=0.01 4
0
0.1
0.2 δ value
10 8 6 4 2 0.05
0.1 0.15 δ value
0.2
3 2 1
0.25
0
0.1
0.2 δ value
0.3
(d) Number of iterations til convergence, M=10, e=1 30
Iterations til Convergence
Iterations til Convergence
12
0
4
0
0.3
(c) Number of iterations til convergence, M=10, e=0.5
0
5
25 20 15 10 5 0
0
0.02
0.04 δ value
0.06
0.08
Figure: Number of iterations required for theoretical error bounds to reach limiting theoretical error when (a) µ = 10, ε = 0.01, (b) µ = 10, ε = 0.1, (c) µ = 10, ε = 0.5, (d) µ = 10, ε = 1.0. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Main Results
Recent work
Wipf-Nagarajan elaborate on convergence and show connections to reweighted ℓ2 -minimization. Wipf-Nagarajan also show that a non-separable variant has desirable properties. Xu-Khajehnejad-Avestimehr-Hassibi provide a theoretical foundation for the analysis of RWL1 and show that for a nontrivial class of signals, a variant of RWL1 indeed can improve upon L1 in the noiseless case.
D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
Introduction
L1-Minimization
Reweighted L1
Main Results
Thank you
For more information E-mail: [email protected] Web: http://www-stat.stanford.edu/~dneedell References: Candes, Wakin, Boyd, “Enhancing sparsity by reweighted ℓ1 minimization”, J. Fourier Anal. Appl., 14 877-905. Needell, “Noisy signal recovery via iterative reweighted L1-minimization,” 2009. Wipf and Nagarajan, “Solving Sparse Linear Inverse Problems: Analysis of Reweighted ℓ1 and ℓ2 Methods,” J. of Selected Topics in Signal Processing, Special Issue on Compressive Sensing, 2010. Xu, Khajehnejad, Avestimehr, and Hassibi, “Breaking through the Thresholds: an Analysis for Iterative Reweighted ℓ1 Minimization via the Grassmann Angle Framework,” Proc. Allerton Conference on Communication, Control, and computing, Sept. 2009. D. Needell
Noisy Signal Recovery via Iterative Reweighted L1-Minimization
