A Family of Robust Algorithms Exploiting Sparsity ... - Semantic Scholar
572
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
A Family of Robust Algorithms Exploiting Sparsity in Adaptive Filters Leonardo Rey Vega, Student Member, IEEE, Hernán Rey, Jacob Benesty, Senior Member, IEEE, and Sara Tressens
Abstract—We introduce a new family of algorithms to exploit sparsity in adaptive filters. It is based on a recently introduced new framework for designing robust adaptive filters. It results from minimizing a certain cost function subject to a time-dependent constraint on the norm of the filter update. Although in general this problem does not have a closed-form solution, we propose an approximate one which is very close to the optimal solution. We take a particular algorithm from this family and provide some theoretical results regarding the asymptotic behavior of the algorithm. Finally, we test it in different environments for system identification and acoustic echo cancellation applications. Index Terms—Acoustic echo cancellation, adaptive filtering, impulsive noise, robust filtering, sparse systems.
I. INTRODUCTION N the last several years, several authors have paid attention to the problem of system identification of sparse systems [1]–[6]. These systems have the property of concentrating most of their energy in a small fraction of their coefficients. In principle, using different dynamics for updating each coefficient might result in an improvement of the initial speed of convergence without compromising the steady-state behavior. In [1], a detection guided normalized least-mean-square (NLMS) algorithm is proposed. However, if the signal-to-background-noise ratio (SBNR) is high, the steady-state behavior might be severely compromised. In [3], a different approach allows the inclusion of a priori information on the system. Nevertheless, the performance of the resulting algorithms might be compromised in the presence of large perturbations. Only a few attempts have been done [4] to exploit sparsity while performing robustly (in the sense of “slightly sensitive to large perturbations”). In this paper, we use a recently introduced framework for the design of robust adaptive filters [7]. As a result, a family
I
Manuscript received July 11, 2008; revised October 24, 2008. Current version published March 10, 2009. This work was supported in part by the Universidad de Buenos Aires under project UBACYT I005. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jingdong Chen. L. Rey Vega is with the Department of Electronics and CONICET, Universidad de Buenos Aires, 1063 Buenos Aires, Argentina (e-mail: [email protected]). H. Rey is with the Instituto de Ingeniería Biomédica (FIUBA) and CONICET, Universidad de Buenos Aires, 1063 Buenos Aires, Argentina (e-mail: hrey@fi. uba.ar). J. Benesty is with the INRS-EMT, Université du Québec, Montréal, Québec, H5A 1K6, Canada (e-mail: [email protected]). S. Tressens is with the Department of Electronics, Universidad de Buenos Aires, 1063 Buenos Aires, Argentina (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TASL.2008.2010156
of robust adaptive filters suitable for sparse system identification arises. Particularly, we use the ideas of the improved proportionate NLMS (IPNLMS) algorithm [5]. The proposed algorithm can also be interpreted as a variable step-size IPNLMS. We present some theoretical results regarding the asymptotic behavior of the algorithm. Its performance is then tested under several scenarios in system identification and in acoustic echo cancellation problems. Finally, we present the notation used throughout the paper. be an unknown Let linear finite-impulse response system. The input vector at time , , passes through the . This output is observed, but it is system leading to usually corrupted by a noise , which will be considered addi, where stands tive. In many practical situations, for the background measurement noise and is an impulsive gives an output . noise. Thus, each input We want to find , an estimate of . This filter receives the . same input, leading to an output error The misalignment vector is . We also define and the a priori error the a posteriori error . II. NEW FAMILY OF ROBUST ADAPTIVE FILTERS We propose to design a new family of adaptive filters using the framework introduced in [7]. In order to avoid a degradation of the system performance after a large noise sample perturbs it, the energy of the filter update is constrained at each iteration. This can be formally stated as (1) where is a positive sequence. Its choice will influence the dynamics of the algorithm but in any case, (1) guarantees that any noise sample can perturb the square norm of the filter update . Next, a cost function is required. by at most the amount Following suggestions in [6], we minimize (2) subject to the constraint (1). Here, the matrices and are any positive definite matrices. Different choices of these matrices will lead to different algorithms. These matrices could dewhich would make the parameter space to be a pend on Riemannian space, i.e., a curved differentiable manifold where the distance properties are not uniform along the space. The use of Riemannian manifolds could be exploited when some a priori information about the true system is known [3], [8], [9]. Such a
1558-7916/$25.00 © 2009 IEEE Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
573
case could be when we know that the subjacent true system is sparse. It is easy to show that the unconstrained problem given only by the cost function (2) has a unique minimum given by
and as the where is a Lagrange multiplier. Defining -components of and and as the -diagonal element and setting to zero the gradient of with respect of of , we can write
(3)
(10)
With this in mind and in order to minimize subject to (1), we should investigate two situations. Case 1) The minimum of the unconstrained problem is contained in the hypersphere given by (1). In this case, the solution to the constrained problem is given by (3). This case takes place if and only if (4)
(6)
(11) -order polynomial equation, which is impossible to This is a . The closed form solution for solve in close form when corresponds to the roots of a quartic equation, which are is of no very difficult to compute. Moreover, the case could be very large and the use practical interest. Typically, of numerical root finding algorithms in real-time could be not an option for finding the Lagrange multiplier. This is because of the high computational load and the precision required in the computation of the roots of (11). For that reason, the problem (7) should be solved using other numerical techniques as numerical gradient search techniques [11]. However, the implementation of these numerical algorithms in real-time problems could be precluded, especially if is large. Therefore, we propose the approximate solution:
, the
(12)
Case 2) The minimum of the unconstrained problem is not contained in the hypersphere given by (1). This is the situation when (4) is not satisfied. Defining as (5) the cost function can be rewritten as
where . Defining problem to be solved can be put as
subject to
This is the optimal value of which obviously depends on . The optimal value of the Lagrange multiplier is one of the roots of the following polynomial:
(7)
This is a quadratically constrained quadratic program. These kinds of problems are well studied [10]. However, in general they do not admit a closed form solution. Let us define (8) is a diagonal positive definite matrix. The rationale where is the same as in [2]. Although for using this structure for the choice of will be made more explicit in Section III, we can say that the choice of the diagonal elements of this matrix will be proportional to the amplitude of the correspondent entry . This allows different adaptation dynamics for the coof efficients according to their influence in the adaptive filter. This could increase the convergence speed of the adaptive filter algorithm when the subjacent true system is sparse, because the stronger and most significant coefficients will converge faster. With (8) in (7), we can write the Lagrange function
Clearly, . If is small enough, (12) will be close to the optimal solution of problem (7) (since the optimal and suboptimal solutions must lie inside the hypersphere of ra). It is true that other suboptimal solutions could be dius proposed, each of them leading to a different algorithm. For exalso ample, the vector with all its entries equal to satisfy the constraint and it is close to the optimal one. However, this solution will not perform as well as (12). The proposed solution in (12) has the same direction than the update in (3) (with and given in (8), which is the solution of the choices of the unconstrained problem. It is clear then that this update is a good one, preserving the direction that minimize (2) but with , which provides the norm of the update constrained to robustness. It also has to be mentioned that solution (12) is an easily implementable one, which only requires an appropriate normalization of the update given in (3). As we will see later, the results of Fig. 2 also validate the use of (12). Replacing (8) in (3), (4), and (11), leads to the family of robust adaptive filters
(9) Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
(13)
574
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
The only thing that remains is the choice of the delta sequence. Following the ideas in [7], we choose
(14) The memory factor
can be chosen as (15) Fig. 1. Impulse response of a sparse system.
where is an integer-valued parameter that depends on the correlation of the input. The initial condition of the delta sequence , with and can be also parameterized as standing for the power of the input and observed output signals, respectively. From (13), the proposed algorithm can be interpreted as a robust variable step-size version of the normalized natural gradient algorithm [12]. When the condition (4) is satisfied, i.e., there is a low chance of having a large noise sample contamiis not too large, the step nating the error signal as long as size is equal to one. If this is not the case, then . Using the same ideas of [7], it can be proved that the sequence (14) is strictly decreasing towards zero, regardless of the values and . Therefore, the adaptive step size is not only proof viding the algorithm with a switching mechanism in order to perform robustly against the noise. As the error decreases during the adaptation, the variable step-size will also let the algorithm to further decrease the error when the update (3) is no longer capable of doing so. As a consequence of this fact, under certain hypotheses, we will show in Section IV that the algorithm converges in the mean-square sense to the true system. will lead to different algorithms with Different choices of the potential of exploiting the sparsity of the system (PNLMS [2], IPNLMS [5], and other variants [6]). It is important to note , the update (13) reduces to the RVSS-NLMS that if presented in [7]. In Section III, we will make a particular choice and simulate how close is the proposed solution (12) to of the optimal one in (7). III. PARTICULAR SELECTION OF From all the possible , we choose a scaled version of the one associated with the IPNLMS [5]. This matrix is diagonal, with the th element of its diagonal computed as
(14) and (16) is referred as the robust variable step-size IPNLMS (RVSS-IPNLMS). Now we compare the proposed suboptimal solution, in terms of (2), with the optimal one. We define the SBNR as SBNR
(17)
where and stand for the power of the system output and background noise, respectively. We use an AR1(0.8) input dB. The true system is chosen as the signal and SBNR first 158 taps of the measured impulse response shown in Fig. 1. At each time step where the condition (4) is satisfied, the solution (3) is optimal. However, when it is not, we use the filter estimate after the update (12) as an initial condition in a numerical gradient optimization algorithm to compute the optimal estimate (7). The resulting values of the cost function were very close (not shown). We also compare how well the algorithms estimate the unknown system. In this case, the RVSS-IPNLMS is run in parallel with the optimal algorithm. As a measure of performance, we use the normalized mismatch expressed in dB (18) In Fig. 2, we show that both algorithms present an almost identical mismatch. Although they seem to be close to convergence, the curves actually continue going down very slowly until they reach the machine precision in accordance with the results of Section IV. This supports the use of the approximate update that can be satisfactory implemented in practice, which is not the case of the optimal solution. IV. MEAN-SQUARE STEADY-STATE BEHAVIOR
(16) where is a small positive constant required at the beginning of the adaptation when typically . The reason to use matrix of the IPNLMS is that it the scaled version of the can be regularized with the same constant as its NLMS coun, the IPNLMS is equivalent to terpart. In fact, when the NLMS. In the sequel, we set in all the cases [5]. The advantage of the IPNLMS is that it works well even if the system is not sparse [5]. The resulting algorithm using (13),
We are interested in the mean-square steady-state behavior , where is the true time-invariant system. of We will assume that the noise sequence is i.i.d., zero-mean and it is independent of the input regressors , which belong to a zero-mean stationary process with a strictly positive-definite . These are reasonable and correlation matrix standard assumptions. From (13) it is easy to show that
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
(19)
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
With the choice of
575
given in (16), it is not difficult to show (25)
is large, the quanFor a large number of inputs of interest, if will be small. In fact, in the important case of tity spherically invariant random processes (SIRP) [13], which are relevant in speech processes applications [14], it can be shown that (26) This implies that the variance of
Fig. 2. Mismatch (in dB). AR1(0.8) input. SBNR realization is shown without any averaging.
Multiplying this equation by
= 20 dB. = 2. A single
and using (13), we obtain
(27) will be very small when is sufficiently large. In this case, we can make the following approximation:
(28) where (20) Taking the expectation of the squared norm on both sides
(29) In this way, assuming that
is large, we can write (21) as
(21) The parameter is typically close to one. This means that (13) is . Then the variance the result of low-pass filtering of the random variable would be small enough to assume that
(30) Observing that telescoping series, setting of
constitutes a , and assuming the existence
(22)
(31) (23) Let us analyze the asymptotic behavior of is not difficult to show that
Let us analyze the following quantity: (24)
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
. First of all, it
(32)
576
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
where
. We define
tion was used and validated through simulations in [15]. Putting in terms of and (44)
(33) (34) Clearly, if
Thanks to the Gaussianity of theorem [16] in order to obtain
and
we can use Price’s
, we have (35)
(45) where
. It is not difficult to show that
Using the ideas developed in [7] it can be shown that (36)
(46)
i.e., and behave asymptotically as . This imbehaves in the same manner when is large. plies that Hence
where the expectation is taken with respect to the noise distribution which could be arbitrary. We can also write the following:
(37) (47) At the same time we have (38)
where defined as
is the trace operator and
is a diagonal matrix
(48)
Therefore
Using the standard assumption that the input regressors are independent we can put (49)
(39) where
, and
This means that we can break (31) in two series and using the fact that [see (51)]
(50) . The same holds for It is clear that the expression in (49). Then it is clear that
(40) , it is clear
Assuming that that
(41)
(51) Using these facts and (38) we should have
This is because
(52) (42) where
It is clear that
when
and
. We define (53) (43)
If is large and under certain mixing condition on the input, it and are Gaussian. This assumpcan be assumed that
for every value of . In fact, if , (53) is equal , the probability density function of the noise in to
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
. This means that we should have any of the two following situations:
577
RVSS-NLMS [7] with the same parameters as the proposed algorithm. Finally, we simulate a robust version of the IPNLMS (RIPNLMS) based on [4]
(54) (55) (57)
It is not difficult to show that either (54) or (55) imply the following:
(58) (56) This is a very interesting result which states that under the hypotheses taken, after a sufficiently long time and independently of and , the adaptive filter converges to the true system in a mean-square sense. It should be noted that this result, which is valid under the considered assumptions, does not depend on the existence of moments of any order of the noise distribution. Therefore, this result would be valid for the important case where the noise comes from an -stable distribution. V. PRACTICAL CONSIDERATIONS We discuss certain algorithm implementation issues. A regu. We chose larization constant is added to the quantity . A major issue should be considered carefully. As the proposed delta sequence goes asymptotically towards zero, although the algorithm becomes more robust against perturbations, it also loses its tracking ability. For this reason, the nonstationary control methods introduced in [7] are included, although other schemes might be used. The objective is to detect changes in the true system. See [7] for a detailed description of the parameters and their role. It is clear that as these nonstationary control methods are less used, the actual asymptotic performance of the algorithm will be closer to that predicted in Section IV.
where
is a parameter and
is computed as
The scaling factor has an initial value and is never al. We actually simulate: lowed to become smaller than and leading to the same steady-state RIPNLMS2, with when no impulsive noise (or double as the NLMS with as RIPNLMS2 talk) is present; RIPNLMS3, with the same and chosen to give the same steady state as the one in the RVSS-IPNLMS when no impulsive noise (or double talk) is and the value of that present; and RIPNLMS1, with leads to the same steady state as the one in the RVSS-IPNLMS when no impulsive noise (or double talk) is present. We also want to test the nonstationary control. As a measure of its performance we compute for each simulation (59) is the second largest value of ( and are dewhere fined in [7]). In all the simulations (except in Figs. 7 and 8), a sudden change is introduced at a certain time step by multi. In these cases, is acplying the system coefficients by complished near the time of the sudden change, while is acis related to the complished at any other time. The value of threshold (defined in [7]) while that of gives an idea of the reliability of the detection.
VI. SIMULATION RESULTS The system is taken from a measured impulse response with (Fig. 1). The 90% of its energy is allocated length in 38 taps and the 99.9% in 525 taps. If the adaptive filter sets the other small coefficients to zero, it will produce a systematic error of 30 dB, which sets a lower bound on the mismatch performance, even if the SBNR is higher than 30 dB. This is the kind of problem that the scheme [1] might suffer. in each case. We use the The adaptive filter length is set to mismatch, defined in (18), as a measure of performance. The plots are the result of a single realization for all the algorithms without any additional smoothing (unless otherwise stated). is added to the system A zero-mean Gaussian white noise output so that a certain SBNR, defined in (17), is achieved. All the algorithms are regularized with the constant discussed in the previous section. The behavior of the RVSS-IPNLMS is compared with other strategies. We simulate an NLMS with a value chosen to give the same steady state as the one in the RVSS-IPNLMS when no impulsive noise/double talk is present. We also include the
A. System Identification Under Impulsive Noise The input process is a correlated AR1 with pole in 0.9. In addition to the background noise , the impulsive noise could also be added to the output signal . The impulsive noise is , where is a Bernoulli process with generated as and is a zero-mean probability of success . The RVSS-IPNLMS and Gaussian with power RVSS-NLMS use the nonstationary control method 1 described in [7]. In Fig. 3, it can be seen that the RVSS-IPNLMS has the same initial speed of convergence as the one of RIPNLMS2 but with a 10-dB lower steady state. The NLMS has the worst speed of convergence. To compare the gain with respect to the RVSSNLMS we look at the number of iterations to reach the steady(in this case, it is state level of the NLMS algorithm with 41 dB). Then, the RVSS-IPNLMS needs 35% less iterations than the RVSS-NLMS. The RIPNLMS1 and RIPNLMS3 have a worse speed of convergence and a very bad tracking ability. In fact, the tracking performance of the proposed algorithm is
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
578
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
=
= 00:5. No = 10 M = 13098.
Fig. 3. Mismatch (in dB). AR1(0.9) input. SBNR 40 dB. impulsive noise. .V M. V : V . . . Steady state: 51 dB (except for the RIPNLMS2).
R = 67792
=3
=2 0
= 0 75
= 0:1. The other parameters are the same as M = 14443 R = 56147.
Fig. 4. Mismatch (in dB). p . in Fig. 3.
almost the same as the one of the RIPNLMS2 (the best of the tested algorithms). . Fig. 4 Now we include impulsive noise with shows that the NLMS presents a very bad performance. The RVSS-IPNLMS and the RVSS-NLMS show almost the same behavior as in Fig. 3, loosing only 1 dB in steady state. The RIPNLMS1 performs similarly to the RVSS-NLMS before the sudden change. The RIPNLMS2 and RIPNLMS3 worsen their speed of convergence and loose 18 and 3 dB, respectively, compared to the steady state of the RVSS-IPNLMS. The three RIPNLMS algorithms recover very poorly to the sudden change. This shows the advantage of the proposed algorithm when an impulse since it does not use the IPNLMS with of noise is present or when the mismatch is low enough (and further gain can be accomplished).
= 10 dB. The other parameters are the M = 28 R = 86. Steady-state: 021 dB (except for the
Fig. 5. Mismatch (in dB). SBNR same as in Fig. 3. . RIPNLMS2).
= 0:1. The other parameters = M = 26 R = 113
Fig. 6. Mismatch (in dB). SBNR 10 dB. p are the same as in Fig. 3. . .
In Figs. 5 and 6, we repeat the simulations but now with a low SBNR of 10 dB. All the parameters are the same as in the high SBNR scenarios. When no impulsive noise is present, the RVSS-IPNLMS has 8 dB less in steady state than the RIPNLMS2, while showing the same initial speed of convergence and tracking performance. When compared with the RVSS-NLMS, the RVSS-IPNLMS requires 55% less iterations to reach the 11 dB level. When the impulsive noise is added, the same conclusions as in the high SBNR scenario can be drawn. In the four cases studied in Figs. 3–6, the nonstationary control presents a good performance. The RVSS-IPNLMS can recover well from a sudden change, with a speed close to the one and appear on the capof the RIPNLMS2. The values of tion of each figure. The sudden change was reliably identified as it is revealed by the large values of .
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
=
Fig. 7. Mismatch (in dB). Speech input. SBNR 25 dB. No double talk. .T : . .C M. V M. : . .V : . : .
M==21 2 =R 2=51 12= 10
=5
=1
=2
= 0 95
B. Acoustic Echo Cancellation With Double-Talk Situations In echo cancellation applications, a double-talk detector (DTD) is used to suppress adaptation during periods of simultaneous far- and near-end signals. We use the simple Geigel DTD [17]. The Geigel DTD declares double-talk if
579
Fig. 8. ERLE (in dB). Same scenario as in Fig. 7.
state. Of course, this is not in contradiction with the results of Section IV. This is because the results in that section are valid when the input is stationary and its correlation matrix is strictly positive definite which is not the case considered here. To overcome this issue, we just change the way of analyzing the algorithms performance. We define the echo return loss enhancement (ERLE)
(60) (61) are the samples of the far-end signal and are the where samples of the far-end signal filtered by the acoustic impulse response and possibly contaminated by a background noise and a near-end signal. An important detail is that in the following simulations the filter update is not stopped even when double talk is declared. This is to test the robust performance of the algorithms. When double talk is declared, the RIPNLMS algorithm does not perform the adaptation of the scale factor. The RVSS-IPNLMS and RVSS-NLMS use the nonstationary control method 2 described in [7]. The far-end and near-end signals are speech sampled at 8 kHz. The parameters of the DTD are and . In Fig. 7, neither double talk nor a sudden change are present. It seems strange that in this case the NLMS and RVSS-NLMS show a better steady state than the RIPNLMS3 and RVSS-IPNLMS, respectively. We may think that the parameters were chosen wrongly, but it is not the case. This problem appears due to the nature of the speech signal. Although not shown here, the input signal presents almost zero energy in the low-frequency region of the spectrum. On the other hand, during the initial convergence the IPNLMS-like update is highly nonlinear. Since in this case the filter is not being updated in the direction of the input vector, the matrix might generate an update direction that falls in the nonexcited region of the input spectrum. This might in turn generate an error in the filter estimate from which the algorithm can never recovered from. The final effect will be an increase in steady
where is a moving average with a span of 8000 samples. This quantity relates the power of the system output with the one of the a priori error. Since both are related to the input signal, any nonexcited region of the input spectrum will have no impact on the ERLE. Now we can see in Fig. 8 that the RVSS-IPNLMS and RIPNLMS2 have the fastest speed of convergence. The RIPNLMS2 has a steady-state ERLE 8 dB worse than the other algorithms and among these ones, the RVSS-IPNLMS converges 30 s in advance. In Fig. 9, double talk and a sudden change are included. After 8 s of adaptation, the near-end signal appears for a period of 3 s. The proportion of detections during the double-talk situation was 34% while the proportion of false alarms when no doubletalk was present was 9.7%. After passing through the DTD, the power of the near-end signal was reduced about 5.3 times. The SBNR is 25 dB while the signal to total noise ratio (STNR), i.e.,
is set to 0 dB, where is the power of the near-end signal before passing through the DTD. Clearly, the proposed algorithm shows the best initial convergence, robust behavior during the double-talk situation and recovery from the sudden change.
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
580
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
M = 241 R = 469 =
Fig. 9. ERLE (in dB). STNR . . Fig. 7.
0 dB. The other parameters are the same as in
VII. CONCLUSION In this paper, we introduced a family of algorithms to exploit sparsity in adaptive filters based on a recently introduced new framework for designing robust adaptive filters. The result is a variable step-size version of the natural gradient algorithm. A leads to the proposed RVSS-IPNLMS particular choice of algorithm. We provide a theoretical analysis showing that, under certain reasonable assumptions, the adaptive filter converges in the mean square to the real system. When tested in different scenarios on a system identification setup, it outperformed the other strategies no matter if the SBNR was low or high and even in the presence of impulsive noise. Under an acoustic echo cancellation scenario, it behaves robustly during double-talk situations without compromising neither the initial speed of convergence nor the tracking ability.
[5] J. Benesty and S. L. Gay, “An improved PNLMS algorithm,” in Proc. IEEE ICASSP, 2002, pp. 1881–1884. [6] Y. Huang, J. Benesty, and J. Chen, Acoustic MIMO Signal Processing. Berlin, Germany: Springer-Verlag, 2006. [7] L. R. Vega, H. Rey, J. Benesty, and S. Tressens, “A new robust variable step-size NLMS algorithm,” IEEE Trans. Signal Process., vol. 56, no. 5, pp. 1878–1893, May 2008. [8] R. E. Mahoney and R. C. Williamson, “Prior knowledge and preferential structures in gradient descent learning algorithms,” J. Mach. Learn. Res., vol. 1, pp. 311–355, Sep. 2001. [9] T. Abrudan, J. Eriksson, and V. Koivunen, “Efficient riemannian algorithms for optimization under unitary matrix constraint,” in Proc. IEEE ICASSP, 2008, pp. 2353–2356. [10] S. Lucidi, L. Palagi, and M. Roma, “On some properties of quadratic programs with a convex quadratic constraint,” SIAM J. Optim., vol. 8, pp. 105–122, 1998. [11] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004. [12] S. Amari, “Natural gradient works efficiently in learning,” Neural Comput., vol. 10, pp. 251–276, Feb. 1998. [13] K. Yao, “A representation theorem and its applications to spherically-invariant random processes,” IEEE Trans. Trans. Inf. Theory, vol. IT-19, no. 5, pp. 600–608, Sep. 1973. [14] H. Brehm and W. Stammler, “Description and generation of spherically invariant speech-model signals,” Signal Process., vol. 12, pp. 119–141, Mar. 1987. [15] T. Al-Naffouri and A. Sayed, “Transient analysis of adaptive filters with error nonlinearities,” IEEE Trans. Signal Process., vol. 51, no. 3, pp. 653–663, Mar. 2003. [16] R. Price, “A useful theorem for nonlinear devices having Gaussian inputs,” IRE Trans. Inf. Theory, vol. IT-4, no. 2, pp. 69–72, Jun. 1958. [17] D. L. Duttweiler, “A twelve channel digital echo canceler,” IEEE Trans. Commun., vol. COM-26, no. 5, pp. 647–653, May 1978.
Leonardo Rey Vega (S’08) was born in Buenos Aires, Argentina, in 1979. He received the B.Eng. degree in electronic engineering from University of Buenos Aires in 2004, where he is currently pursuing the Ph.D. degree. Since 2004, he has been a Research Assistant with the Department of Electronics, University of Buenos Aires. His research interests include adaptive filtering theory and statistical signal processing.
REFERENCES [1] J. Homer, I. Mareels, and C. Hoang, “Enhanced detection-guided NLMS estimation of sparse FIR-modeled signal channels,” IEEE Trans. Circuits Syst. I, vol. 53, no. 8, pp. 1783–1791, Aug. 2006. [2] D. L. Duttweiler, “Proportionate normalized least mean square adaptation in echo cancelers,” IEEE Trans. Speech Audio Process., vol. 8, no. 5, pp. 508–518, Sep. 2000. [3] R. K. Martin, W. A. Sethares, R. C. Williamson, and C. R. Johnson, Jr, “Exploiting sparsity in adaptive filters,” IEEE Trans. Signal Process., vol. 50, no. 8, pp. 1883–1894, Aug. 2002. [4] T. Gänsler, S. L. Gay, M. M. Sondhi, and J. Benesty, “Double-talk robust fast converging algorithms for network echo cancellation,” IEEE Trans. Speech Audio Process., vol. 8, no. 6, pp. 656–663, Nov. 2000.
Hernán Rey was born in Buenos Aires, Argentina, in 1978. He received the B.Eng. degree in electronic engineering from the University of Buenos Aires in 2002 and the Ph.D. degree from the University of Buenos Aires in 2009. Since 2002, he has been a Research Assistant with the Department of Electronics, University of Buenos Aires. He is currently a Postdoctoral Researcher at the Institute of Biomedical Engineering, University of Buenos Aires. His research interests include adaptive filtering theory, neural networks, and computational neuroscience.
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
Jacob Benesty (M’98–SM’04) was born in 1963. He received the M.S. degree in microwaves from Pierre & Marie Curie University, France, in 1987, and the Ph.D. degree in control and signal processing from Orsay University, Orsay, France, in April 1991. During the Ph.D. degree (from November 1989 to April 1991), he worked on adaptive filters and fast algorithms at the Centre National d’Etudes des Telecommunications (CNET), Paris, France. From January 1994 to July 1995, he worked at Telecom Paris University on multichannel adaptive filters and acoustic echo cancellation. From October 1995 to May 2003, he was first a Consultant and then a Member of the Technical Staff at Bell Laboratories, Murray Hill, NJ. In May 2003, he joined the University of Quebec, INRS-EMT, Montreal, QC, Canada, as an Associate Professor. His research interests are in signal processing, acoustic signal processing, and multimedia communications. He was a member of the editorial board of the EURASIP Journal on Applied Signal Processing and was the cochair of the 1999 International Workshop on Acoustic Echo and Noise Control. He coauthored the books Acoustic MIMO Signal Processing (Springer-Verlag, 2006) and Advances in Network and Acoustic Echo Cancellation (Springer-Verlag, 2001). He is also a coeditor/coauthor of the books Speech Enhancement (Spinger-Verlag, 2005), Audio Signal Processing for Next Generation Multimedia Communication Systems (Kluwer, 2004), Adaptive Signal Processing: Applications to Real-World Problems (Springer-Verlag, 2003), and coustic Signal Processing for Telecommunication” (Kluwer, 2000). Dr. Benesty received the 2001 Best Paper Award from the IEEE Signal Processing Society.
581
Sara Tressens was born in Buenos Aires, Argentina. She received the degree of Electrical Engineering from the University of Buenos Aires in 1967. From 1967 to 1982, she was with the Institute of Biomedical Engineering, University of Buenos Aires, where she became Assistant Professor in 1977 and worked in the area of speech recognition, digital communication, and adaptive filtering. From 1980 to 1981, she worked in the Laboratoire des Signaux et Systemes, Gif-sur Yvette, France. From 1982 to 1993, she was with the National Research Center in Telecommunications (CNET), Issy les Moulineaux, France. Her research interests were in the areas of spectral estimation and spatial array processing. Since 1994, she has been an Associated Professor at the University de Buenos Aires. Her primary research interests include adaptive filtering, communications, and spectral analysis. Ms. Tressens received the 1990 Best Paper Award from the IEEE Signal Processing Society.
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
A Family of Robust Algorithms Exploiting Sparsity in Adaptive Filters Leonardo Rey Vega, Student Member, IEEE, Hernán Rey, Jacob Benesty, Senior Member, IEEE, and Sara Tressens
Abstract—We introduce a new family of algorithms to exploit sparsity in adaptive filters. It is based on a recently introduced new framework for designing robust adaptive filters. It results from minimizing a certain cost function subject to a time-dependent constraint on the norm of the filter update. Although in general this problem does not have a closed-form solution, we propose an approximate one which is very close to the optimal solution. We take a particular algorithm from this family and provide some theoretical results regarding the asymptotic behavior of the algorithm. Finally, we test it in different environments for system identification and acoustic echo cancellation applications. Index Terms—Acoustic echo cancellation, adaptive filtering, impulsive noise, robust filtering, sparse systems.
I. INTRODUCTION N the last several years, several authors have paid attention to the problem of system identification of sparse systems [1]–[6]. These systems have the property of concentrating most of their energy in a small fraction of their coefficients. In principle, using different dynamics for updating each coefficient might result in an improvement of the initial speed of convergence without compromising the steady-state behavior. In [1], a detection guided normalized least-mean-square (NLMS) algorithm is proposed. However, if the signal-to-background-noise ratio (SBNR) is high, the steady-state behavior might be severely compromised. In [3], a different approach allows the inclusion of a priori information on the system. Nevertheless, the performance of the resulting algorithms might be compromised in the presence of large perturbations. Only a few attempts have been done [4] to exploit sparsity while performing robustly (in the sense of “slightly sensitive to large perturbations”). In this paper, we use a recently introduced framework for the design of robust adaptive filters [7]. As a result, a family
I
Manuscript received July 11, 2008; revised October 24, 2008. Current version published March 10, 2009. This work was supported in part by the Universidad de Buenos Aires under project UBACYT I005. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jingdong Chen. L. Rey Vega is with the Department of Electronics and CONICET, Universidad de Buenos Aires, 1063 Buenos Aires, Argentina (e-mail: [email protected]). H. Rey is with the Instituto de Ingeniería Biomédica (FIUBA) and CONICET, Universidad de Buenos Aires, 1063 Buenos Aires, Argentina (e-mail: hrey@fi. uba.ar). J. Benesty is with the INRS-EMT, Université du Québec, Montréal, Québec, H5A 1K6, Canada (e-mail: [email protected]). S. Tressens is with the Department of Electronics, Universidad de Buenos Aires, 1063 Buenos Aires, Argentina (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TASL.2008.2010156
of robust adaptive filters suitable for sparse system identification arises. Particularly, we use the ideas of the improved proportionate NLMS (IPNLMS) algorithm [5]. The proposed algorithm can also be interpreted as a variable step-size IPNLMS. We present some theoretical results regarding the asymptotic behavior of the algorithm. Its performance is then tested under several scenarios in system identification and in acoustic echo cancellation problems. Finally, we present the notation used throughout the paper. be an unknown Let linear finite-impulse response system. The input vector at time , , passes through the . This output is observed, but it is system leading to usually corrupted by a noise , which will be considered addi, where stands tive. In many practical situations, for the background measurement noise and is an impulsive gives an output . noise. Thus, each input We want to find , an estimate of . This filter receives the . same input, leading to an output error The misalignment vector is . We also define and the a priori error the a posteriori error . II. NEW FAMILY OF ROBUST ADAPTIVE FILTERS We propose to design a new family of adaptive filters using the framework introduced in [7]. In order to avoid a degradation of the system performance after a large noise sample perturbs it, the energy of the filter update is constrained at each iteration. This can be formally stated as (1) where is a positive sequence. Its choice will influence the dynamics of the algorithm but in any case, (1) guarantees that any noise sample can perturb the square norm of the filter update . Next, a cost function is required. by at most the amount Following suggestions in [6], we minimize (2) subject to the constraint (1). Here, the matrices and are any positive definite matrices. Different choices of these matrices will lead to different algorithms. These matrices could dewhich would make the parameter space to be a pend on Riemannian space, i.e., a curved differentiable manifold where the distance properties are not uniform along the space. The use of Riemannian manifolds could be exploited when some a priori information about the true system is known [3], [8], [9]. Such a
1558-7916/$25.00 © 2009 IEEE Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
573
case could be when we know that the subjacent true system is sparse. It is easy to show that the unconstrained problem given only by the cost function (2) has a unique minimum given by
and as the where is a Lagrange multiplier. Defining -components of and and as the -diagonal element and setting to zero the gradient of with respect of of , we can write
(3)
(10)
With this in mind and in order to minimize subject to (1), we should investigate two situations. Case 1) The minimum of the unconstrained problem is contained in the hypersphere given by (1). In this case, the solution to the constrained problem is given by (3). This case takes place if and only if (4)
(6)
(11) -order polynomial equation, which is impossible to This is a . The closed form solution for solve in close form when corresponds to the roots of a quartic equation, which are is of no very difficult to compute. Moreover, the case could be very large and the use practical interest. Typically, of numerical root finding algorithms in real-time could be not an option for finding the Lagrange multiplier. This is because of the high computational load and the precision required in the computation of the roots of (11). For that reason, the problem (7) should be solved using other numerical techniques as numerical gradient search techniques [11]. However, the implementation of these numerical algorithms in real-time problems could be precluded, especially if is large. Therefore, we propose the approximate solution:
, the
(12)
Case 2) The minimum of the unconstrained problem is not contained in the hypersphere given by (1). This is the situation when (4) is not satisfied. Defining as (5) the cost function can be rewritten as
where . Defining problem to be solved can be put as
subject to
This is the optimal value of which obviously depends on . The optimal value of the Lagrange multiplier is one of the roots of the following polynomial:
(7)
This is a quadratically constrained quadratic program. These kinds of problems are well studied [10]. However, in general they do not admit a closed form solution. Let us define (8) is a diagonal positive definite matrix. The rationale where is the same as in [2]. Although for using this structure for the choice of will be made more explicit in Section III, we can say that the choice of the diagonal elements of this matrix will be proportional to the amplitude of the correspondent entry . This allows different adaptation dynamics for the coof efficients according to their influence in the adaptive filter. This could increase the convergence speed of the adaptive filter algorithm when the subjacent true system is sparse, because the stronger and most significant coefficients will converge faster. With (8) in (7), we can write the Lagrange function
Clearly, . If is small enough, (12) will be close to the optimal solution of problem (7) (since the optimal and suboptimal solutions must lie inside the hypersphere of ra). It is true that other suboptimal solutions could be dius proposed, each of them leading to a different algorithm. For exalso ample, the vector with all its entries equal to satisfy the constraint and it is close to the optimal one. However, this solution will not perform as well as (12). The proposed solution in (12) has the same direction than the update in (3) (with and given in (8), which is the solution of the choices of the unconstrained problem. It is clear then that this update is a good one, preserving the direction that minimize (2) but with , which provides the norm of the update constrained to robustness. It also has to be mentioned that solution (12) is an easily implementable one, which only requires an appropriate normalization of the update given in (3). As we will see later, the results of Fig. 2 also validate the use of (12). Replacing (8) in (3), (4), and (11), leads to the family of robust adaptive filters
(9) Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
(13)
574
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
The only thing that remains is the choice of the delta sequence. Following the ideas in [7], we choose
(14) The memory factor
can be chosen as (15) Fig. 1. Impulse response of a sparse system.
where is an integer-valued parameter that depends on the correlation of the input. The initial condition of the delta sequence , with and can be also parameterized as standing for the power of the input and observed output signals, respectively. From (13), the proposed algorithm can be interpreted as a robust variable step-size version of the normalized natural gradient algorithm [12]. When the condition (4) is satisfied, i.e., there is a low chance of having a large noise sample contamiis not too large, the step nating the error signal as long as size is equal to one. If this is not the case, then . Using the same ideas of [7], it can be proved that the sequence (14) is strictly decreasing towards zero, regardless of the values and . Therefore, the adaptive step size is not only proof viding the algorithm with a switching mechanism in order to perform robustly against the noise. As the error decreases during the adaptation, the variable step-size will also let the algorithm to further decrease the error when the update (3) is no longer capable of doing so. As a consequence of this fact, under certain hypotheses, we will show in Section IV that the algorithm converges in the mean-square sense to the true system. will lead to different algorithms with Different choices of the potential of exploiting the sparsity of the system (PNLMS [2], IPNLMS [5], and other variants [6]). It is important to note , the update (13) reduces to the RVSS-NLMS that if presented in [7]. In Section III, we will make a particular choice and simulate how close is the proposed solution (12) to of the optimal one in (7). III. PARTICULAR SELECTION OF From all the possible , we choose a scaled version of the one associated with the IPNLMS [5]. This matrix is diagonal, with the th element of its diagonal computed as
(14) and (16) is referred as the robust variable step-size IPNLMS (RVSS-IPNLMS). Now we compare the proposed suboptimal solution, in terms of (2), with the optimal one. We define the SBNR as SBNR
(17)
where and stand for the power of the system output and background noise, respectively. We use an AR1(0.8) input dB. The true system is chosen as the signal and SBNR first 158 taps of the measured impulse response shown in Fig. 1. At each time step where the condition (4) is satisfied, the solution (3) is optimal. However, when it is not, we use the filter estimate after the update (12) as an initial condition in a numerical gradient optimization algorithm to compute the optimal estimate (7). The resulting values of the cost function were very close (not shown). We also compare how well the algorithms estimate the unknown system. In this case, the RVSS-IPNLMS is run in parallel with the optimal algorithm. As a measure of performance, we use the normalized mismatch expressed in dB (18) In Fig. 2, we show that both algorithms present an almost identical mismatch. Although they seem to be close to convergence, the curves actually continue going down very slowly until they reach the machine precision in accordance with the results of Section IV. This supports the use of the approximate update that can be satisfactory implemented in practice, which is not the case of the optimal solution. IV. MEAN-SQUARE STEADY-STATE BEHAVIOR
(16) where is a small positive constant required at the beginning of the adaptation when typically . The reason to use matrix of the IPNLMS is that it the scaled version of the can be regularized with the same constant as its NLMS coun, the IPNLMS is equivalent to terpart. In fact, when the NLMS. In the sequel, we set in all the cases [5]. The advantage of the IPNLMS is that it works well even if the system is not sparse [5]. The resulting algorithm using (13),
We are interested in the mean-square steady-state behavior , where is the true time-invariant system. of We will assume that the noise sequence is i.i.d., zero-mean and it is independent of the input regressors , which belong to a zero-mean stationary process with a strictly positive-definite . These are reasonable and correlation matrix standard assumptions. From (13) it is easy to show that
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
(19)
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
With the choice of
575
given in (16), it is not difficult to show (25)
is large, the quanFor a large number of inputs of interest, if will be small. In fact, in the important case of tity spherically invariant random processes (SIRP) [13], which are relevant in speech processes applications [14], it can be shown that (26) This implies that the variance of
Fig. 2. Mismatch (in dB). AR1(0.8) input. SBNR realization is shown without any averaging.
Multiplying this equation by
= 20 dB. = 2. A single
and using (13), we obtain
(27) will be very small when is sufficiently large. In this case, we can make the following approximation:
(28) where (20) Taking the expectation of the squared norm on both sides
(29) In this way, assuming that
is large, we can write (21) as
(21) The parameter is typically close to one. This means that (13) is . Then the variance the result of low-pass filtering of the random variable would be small enough to assume that
(30) Observing that telescoping series, setting of
constitutes a , and assuming the existence
(22)
(31) (23) Let us analyze the asymptotic behavior of is not difficult to show that
Let us analyze the following quantity: (24)
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
. First of all, it
(32)
576
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
where
. We define
tion was used and validated through simulations in [15]. Putting in terms of and (44)
(33) (34) Clearly, if
Thanks to the Gaussianity of theorem [16] in order to obtain
and
we can use Price’s
, we have (35)
(45) where
. It is not difficult to show that
Using the ideas developed in [7] it can be shown that (36)
(46)
i.e., and behave asymptotically as . This imbehaves in the same manner when is large. plies that Hence
where the expectation is taken with respect to the noise distribution which could be arbitrary. We can also write the following:
(37) (47) At the same time we have (38)
where defined as
is the trace operator and
is a diagonal matrix
(48)
Therefore
Using the standard assumption that the input regressors are independent we can put (49)
(39) where
, and
This means that we can break (31) in two series and using the fact that [see (51)]
(50) . The same holds for It is clear that the expression in (49). Then it is clear that
(40) , it is clear
Assuming that that
(41)
(51) Using these facts and (38) we should have
This is because
(52) (42) where
It is clear that
when
and
. We define (53) (43)
If is large and under certain mixing condition on the input, it and are Gaussian. This assumpcan be assumed that
for every value of . In fact, if , (53) is equal , the probability density function of the noise in to
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
. This means that we should have any of the two following situations:
577
RVSS-NLMS [7] with the same parameters as the proposed algorithm. Finally, we simulate a robust version of the IPNLMS (RIPNLMS) based on [4]
(54) (55) (57)
It is not difficult to show that either (54) or (55) imply the following:
(58) (56) This is a very interesting result which states that under the hypotheses taken, after a sufficiently long time and independently of and , the adaptive filter converges to the true system in a mean-square sense. It should be noted that this result, which is valid under the considered assumptions, does not depend on the existence of moments of any order of the noise distribution. Therefore, this result would be valid for the important case where the noise comes from an -stable distribution. V. PRACTICAL CONSIDERATIONS We discuss certain algorithm implementation issues. A regu. We chose larization constant is added to the quantity . A major issue should be considered carefully. As the proposed delta sequence goes asymptotically towards zero, although the algorithm becomes more robust against perturbations, it also loses its tracking ability. For this reason, the nonstationary control methods introduced in [7] are included, although other schemes might be used. The objective is to detect changes in the true system. See [7] for a detailed description of the parameters and their role. It is clear that as these nonstationary control methods are less used, the actual asymptotic performance of the algorithm will be closer to that predicted in Section IV.
where
is a parameter and
is computed as
The scaling factor has an initial value and is never al. We actually simulate: lowed to become smaller than and leading to the same steady-state RIPNLMS2, with when no impulsive noise (or double as the NLMS with as RIPNLMS2 talk) is present; RIPNLMS3, with the same and chosen to give the same steady state as the one in the RVSS-IPNLMS when no impulsive noise (or double talk) is and the value of that present; and RIPNLMS1, with leads to the same steady state as the one in the RVSS-IPNLMS when no impulsive noise (or double talk) is present. We also want to test the nonstationary control. As a measure of its performance we compute for each simulation (59) is the second largest value of ( and are dewhere fined in [7]). In all the simulations (except in Figs. 7 and 8), a sudden change is introduced at a certain time step by multi. In these cases, is acplying the system coefficients by complished near the time of the sudden change, while is acis related to the complished at any other time. The value of threshold (defined in [7]) while that of gives an idea of the reliability of the detection.
VI. SIMULATION RESULTS The system is taken from a measured impulse response with (Fig. 1). The 90% of its energy is allocated length in 38 taps and the 99.9% in 525 taps. If the adaptive filter sets the other small coefficients to zero, it will produce a systematic error of 30 dB, which sets a lower bound on the mismatch performance, even if the SBNR is higher than 30 dB. This is the kind of problem that the scheme [1] might suffer. in each case. We use the The adaptive filter length is set to mismatch, defined in (18), as a measure of performance. The plots are the result of a single realization for all the algorithms without any additional smoothing (unless otherwise stated). is added to the system A zero-mean Gaussian white noise output so that a certain SBNR, defined in (17), is achieved. All the algorithms are regularized with the constant discussed in the previous section. The behavior of the RVSS-IPNLMS is compared with other strategies. We simulate an NLMS with a value chosen to give the same steady state as the one in the RVSS-IPNLMS when no impulsive noise/double talk is present. We also include the
A. System Identification Under Impulsive Noise The input process is a correlated AR1 with pole in 0.9. In addition to the background noise , the impulsive noise could also be added to the output signal . The impulsive noise is , where is a Bernoulli process with generated as and is a zero-mean probability of success . The RVSS-IPNLMS and Gaussian with power RVSS-NLMS use the nonstationary control method 1 described in [7]. In Fig. 3, it can be seen that the RVSS-IPNLMS has the same initial speed of convergence as the one of RIPNLMS2 but with a 10-dB lower steady state. The NLMS has the worst speed of convergence. To compare the gain with respect to the RVSSNLMS we look at the number of iterations to reach the steady(in this case, it is state level of the NLMS algorithm with 41 dB). Then, the RVSS-IPNLMS needs 35% less iterations than the RVSS-NLMS. The RIPNLMS1 and RIPNLMS3 have a worse speed of convergence and a very bad tracking ability. In fact, the tracking performance of the proposed algorithm is
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
578
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
=
= 00:5. No = 10 M = 13098.
Fig. 3. Mismatch (in dB). AR1(0.9) input. SBNR 40 dB. impulsive noise. .V M. V : V . . . Steady state: 51 dB (except for the RIPNLMS2).
R = 67792
=3
=2 0
= 0 75
= 0:1. The other parameters are the same as M = 14443 R = 56147.
Fig. 4. Mismatch (in dB). p . in Fig. 3.
almost the same as the one of the RIPNLMS2 (the best of the tested algorithms). . Fig. 4 Now we include impulsive noise with shows that the NLMS presents a very bad performance. The RVSS-IPNLMS and the RVSS-NLMS show almost the same behavior as in Fig. 3, loosing only 1 dB in steady state. The RIPNLMS1 performs similarly to the RVSS-NLMS before the sudden change. The RIPNLMS2 and RIPNLMS3 worsen their speed of convergence and loose 18 and 3 dB, respectively, compared to the steady state of the RVSS-IPNLMS. The three RIPNLMS algorithms recover very poorly to the sudden change. This shows the advantage of the proposed algorithm when an impulse since it does not use the IPNLMS with of noise is present or when the mismatch is low enough (and further gain can be accomplished).
= 10 dB. The other parameters are the M = 28 R = 86. Steady-state: 021 dB (except for the
Fig. 5. Mismatch (in dB). SBNR same as in Fig. 3. . RIPNLMS2).
= 0:1. The other parameters = M = 26 R = 113
Fig. 6. Mismatch (in dB). SBNR 10 dB. p are the same as in Fig. 3. . .
In Figs. 5 and 6, we repeat the simulations but now with a low SBNR of 10 dB. All the parameters are the same as in the high SBNR scenarios. When no impulsive noise is present, the RVSS-IPNLMS has 8 dB less in steady state than the RIPNLMS2, while showing the same initial speed of convergence and tracking performance. When compared with the RVSS-NLMS, the RVSS-IPNLMS requires 55% less iterations to reach the 11 dB level. When the impulsive noise is added, the same conclusions as in the high SBNR scenario can be drawn. In the four cases studied in Figs. 3–6, the nonstationary control presents a good performance. The RVSS-IPNLMS can recover well from a sudden change, with a speed close to the one and appear on the capof the RIPNLMS2. The values of tion of each figure. The sudden change was reliably identified as it is revealed by the large values of .
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
=
Fig. 7. Mismatch (in dB). Speech input. SBNR 25 dB. No double talk. .T : . .C M. V M. : . .V : . : .
M==21 2 =R 2=51 12= 10
=5
=1
=2
= 0 95
B. Acoustic Echo Cancellation With Double-Talk Situations In echo cancellation applications, a double-talk detector (DTD) is used to suppress adaptation during periods of simultaneous far- and near-end signals. We use the simple Geigel DTD [17]. The Geigel DTD declares double-talk if
579
Fig. 8. ERLE (in dB). Same scenario as in Fig. 7.
state. Of course, this is not in contradiction with the results of Section IV. This is because the results in that section are valid when the input is stationary and its correlation matrix is strictly positive definite which is not the case considered here. To overcome this issue, we just change the way of analyzing the algorithms performance. We define the echo return loss enhancement (ERLE)
(60) (61) are the samples of the far-end signal and are the where samples of the far-end signal filtered by the acoustic impulse response and possibly contaminated by a background noise and a near-end signal. An important detail is that in the following simulations the filter update is not stopped even when double talk is declared. This is to test the robust performance of the algorithms. When double talk is declared, the RIPNLMS algorithm does not perform the adaptation of the scale factor. The RVSS-IPNLMS and RVSS-NLMS use the nonstationary control method 2 described in [7]. The far-end and near-end signals are speech sampled at 8 kHz. The parameters of the DTD are and . In Fig. 7, neither double talk nor a sudden change are present. It seems strange that in this case the NLMS and RVSS-NLMS show a better steady state than the RIPNLMS3 and RVSS-IPNLMS, respectively. We may think that the parameters were chosen wrongly, but it is not the case. This problem appears due to the nature of the speech signal. Although not shown here, the input signal presents almost zero energy in the low-frequency region of the spectrum. On the other hand, during the initial convergence the IPNLMS-like update is highly nonlinear. Since in this case the filter is not being updated in the direction of the input vector, the matrix might generate an update direction that falls in the nonexcited region of the input spectrum. This might in turn generate an error in the filter estimate from which the algorithm can never recovered from. The final effect will be an increase in steady
where is a moving average with a span of 8000 samples. This quantity relates the power of the system output with the one of the a priori error. Since both are related to the input signal, any nonexcited region of the input spectrum will have no impact on the ERLE. Now we can see in Fig. 8 that the RVSS-IPNLMS and RIPNLMS2 have the fastest speed of convergence. The RIPNLMS2 has a steady-state ERLE 8 dB worse than the other algorithms and among these ones, the RVSS-IPNLMS converges 30 s in advance. In Fig. 9, double talk and a sudden change are included. After 8 s of adaptation, the near-end signal appears for a period of 3 s. The proportion of detections during the double-talk situation was 34% while the proportion of false alarms when no doubletalk was present was 9.7%. After passing through the DTD, the power of the near-end signal was reduced about 5.3 times. The SBNR is 25 dB while the signal to total noise ratio (STNR), i.e.,
is set to 0 dB, where is the power of the near-end signal before passing through the DTD. Clearly, the proposed algorithm shows the best initial convergence, robust behavior during the double-talk situation and recovery from the sudden change.
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
580
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 4, MAY 2009
M = 241 R = 469 =
Fig. 9. ERLE (in dB). STNR . . Fig. 7.
0 dB. The other parameters are the same as in
VII. CONCLUSION In this paper, we introduced a family of algorithms to exploit sparsity in adaptive filters based on a recently introduced new framework for designing robust adaptive filters. The result is a variable step-size version of the natural gradient algorithm. A leads to the proposed RVSS-IPNLMS particular choice of algorithm. We provide a theoretical analysis showing that, under certain reasonable assumptions, the adaptive filter converges in the mean square to the real system. When tested in different scenarios on a system identification setup, it outperformed the other strategies no matter if the SBNR was low or high and even in the presence of impulsive noise. Under an acoustic echo cancellation scenario, it behaves robustly during double-talk situations without compromising neither the initial speed of convergence nor the tracking ability.
[5] J. Benesty and S. L. Gay, “An improved PNLMS algorithm,” in Proc. IEEE ICASSP, 2002, pp. 1881–1884. [6] Y. Huang, J. Benesty, and J. Chen, Acoustic MIMO Signal Processing. Berlin, Germany: Springer-Verlag, 2006. [7] L. R. Vega, H. Rey, J. Benesty, and S. Tressens, “A new robust variable step-size NLMS algorithm,” IEEE Trans. Signal Process., vol. 56, no. 5, pp. 1878–1893, May 2008. [8] R. E. Mahoney and R. C. Williamson, “Prior knowledge and preferential structures in gradient descent learning algorithms,” J. Mach. Learn. Res., vol. 1, pp. 311–355, Sep. 2001. [9] T. Abrudan, J. Eriksson, and V. Koivunen, “Efficient riemannian algorithms for optimization under unitary matrix constraint,” in Proc. IEEE ICASSP, 2008, pp. 2353–2356. [10] S. Lucidi, L. Palagi, and M. Roma, “On some properties of quadratic programs with a convex quadratic constraint,” SIAM J. Optim., vol. 8, pp. 105–122, 1998. [11] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004. [12] S. Amari, “Natural gradient works efficiently in learning,” Neural Comput., vol. 10, pp. 251–276, Feb. 1998. [13] K. Yao, “A representation theorem and its applications to spherically-invariant random processes,” IEEE Trans. Trans. Inf. Theory, vol. IT-19, no. 5, pp. 600–608, Sep. 1973. [14] H. Brehm and W. Stammler, “Description and generation of spherically invariant speech-model signals,” Signal Process., vol. 12, pp. 119–141, Mar. 1987. [15] T. Al-Naffouri and A. Sayed, “Transient analysis of adaptive filters with error nonlinearities,” IEEE Trans. Signal Process., vol. 51, no. 3, pp. 653–663, Mar. 2003. [16] R. Price, “A useful theorem for nonlinear devices having Gaussian inputs,” IRE Trans. Inf. Theory, vol. IT-4, no. 2, pp. 69–72, Jun. 1958. [17] D. L. Duttweiler, “A twelve channel digital echo canceler,” IEEE Trans. Commun., vol. COM-26, no. 5, pp. 647–653, May 1978.
Leonardo Rey Vega (S’08) was born in Buenos Aires, Argentina, in 1979. He received the B.Eng. degree in electronic engineering from University of Buenos Aires in 2004, where he is currently pursuing the Ph.D. degree. Since 2004, he has been a Research Assistant with the Department of Electronics, University of Buenos Aires. His research interests include adaptive filtering theory and statistical signal processing.
REFERENCES [1] J. Homer, I. Mareels, and C. Hoang, “Enhanced detection-guided NLMS estimation of sparse FIR-modeled signal channels,” IEEE Trans. Circuits Syst. I, vol. 53, no. 8, pp. 1783–1791, Aug. 2006. [2] D. L. Duttweiler, “Proportionate normalized least mean square adaptation in echo cancelers,” IEEE Trans. Speech Audio Process., vol. 8, no. 5, pp. 508–518, Sep. 2000. [3] R. K. Martin, W. A. Sethares, R. C. Williamson, and C. R. Johnson, Jr, “Exploiting sparsity in adaptive filters,” IEEE Trans. Signal Process., vol. 50, no. 8, pp. 1883–1894, Aug. 2002. [4] T. Gänsler, S. L. Gay, M. M. Sondhi, and J. Benesty, “Double-talk robust fast converging algorithms for network echo cancellation,” IEEE Trans. Speech Audio Process., vol. 8, no. 6, pp. 656–663, Nov. 2000.
Hernán Rey was born in Buenos Aires, Argentina, in 1978. He received the B.Eng. degree in electronic engineering from the University of Buenos Aires in 2002 and the Ph.D. degree from the University of Buenos Aires in 2009. Since 2002, he has been a Research Assistant with the Department of Electronics, University of Buenos Aires. He is currently a Postdoctoral Researcher at the Institute of Biomedical Engineering, University of Buenos Aires. His research interests include adaptive filtering theory, neural networks, and computational neuroscience.
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
REY VEGA et al.: FAMILY OF ROBUST ALGORITHMS EXPLOITING SPARSITY IN ADAPTIVE FILTERS
Jacob Benesty (M’98–SM’04) was born in 1963. He received the M.S. degree in microwaves from Pierre & Marie Curie University, France, in 1987, and the Ph.D. degree in control and signal processing from Orsay University, Orsay, France, in April 1991. During the Ph.D. degree (from November 1989 to April 1991), he worked on adaptive filters and fast algorithms at the Centre National d’Etudes des Telecommunications (CNET), Paris, France. From January 1994 to July 1995, he worked at Telecom Paris University on multichannel adaptive filters and acoustic echo cancellation. From October 1995 to May 2003, he was first a Consultant and then a Member of the Technical Staff at Bell Laboratories, Murray Hill, NJ. In May 2003, he joined the University of Quebec, INRS-EMT, Montreal, QC, Canada, as an Associate Professor. His research interests are in signal processing, acoustic signal processing, and multimedia communications. He was a member of the editorial board of the EURASIP Journal on Applied Signal Processing and was the cochair of the 1999 International Workshop on Acoustic Echo and Noise Control. He coauthored the books Acoustic MIMO Signal Processing (Springer-Verlag, 2006) and Advances in Network and Acoustic Echo Cancellation (Springer-Verlag, 2001). He is also a coeditor/coauthor of the books Speech Enhancement (Spinger-Verlag, 2005), Audio Signal Processing for Next Generation Multimedia Communication Systems (Kluwer, 2004), Adaptive Signal Processing: Applications to Real-World Problems (Springer-Verlag, 2003), and coustic Signal Processing for Telecommunication” (Kluwer, 2000). Dr. Benesty received the 2001 Best Paper Award from the IEEE Signal Processing Society.
581
Sara Tressens was born in Buenos Aires, Argentina. She received the degree of Electrical Engineering from the University of Buenos Aires in 1967. From 1967 to 1982, she was with the Institute of Biomedical Engineering, University of Buenos Aires, where she became Assistant Professor in 1977 and worked in the area of speech recognition, digital communication, and adaptive filtering. From 1980 to 1981, she worked in the Laboratoire des Signaux et Systemes, Gif-sur Yvette, France. From 1982 to 1993, she was with the National Research Center in Telecommunications (CNET), Issy les Moulineaux, France. Her research interests were in the areas of spectral estimation and spatial array processing. Since 1994, she has been an Associated Professor at the University de Buenos Aires. Her primary research interests include adaptive filtering, communications, and spectral analysis. Ms. Tressens received the 1990 Best Paper Award from the IEEE Signal Processing Society.
Authorized licensed use limited to: Inst Natl de la Recherche Scientific EMT. Downloaded on March 18, 2009 at 08:10 from IEEE Xplore. Restrictions apply.
