Variable step-size LMS algorithm with a quotient form
ARTICLE IN PRESS Signal Processing 89 (2009) 67–76
Contents lists available at ScienceDirect
Signal Processing journal homepage: www.elsevier.com/locate/sigpro
Variable step-size LMS algorithm with a quotient form Shengkui Zhao a,, Zhihong Man b, Suiyang Khoo a, Hong Ren Wu c a
School of Computer Engineering, Nanyang Technological University (NTU), Nanyang Avenue, Singapore 639798, Singapore School of Engineering, Monash University (Malaysia), Jalan Lagoon Selatan, Bandar Sunway, Petaling Jaya 46150, Malaysia c School of Electrical and Computer Engineering, RMIT University, VIC 3001, Australia b
a r t i c l e in fo
abstract
Article history: Received 21 January 2008 Received in revised form 14 July 2008 Accepted 14 July 2008 Available online 25 July 2008
An improved robust variable step-size least mean square (LMS) algorithm is developed in this paper. Unlike many existing approaches, we adjust the variable step-size using a quotient form of filtered versions of the quadratic error. The filtered estimates of the error are based on exponential windows, applying different decaying factors for the estimations in the numerator and denominator. The new algorithm, called more robust variable step-size (MRVSS), is able to reduce the sensitivity to the power of the measurement noise, and improve the steady-state performance for comparable transient behavior, with negligible increase in the computational cost. The mean convergence, the steady-state performance and the mean step-size behavior of the MRVSS algorithm are studied under a slow time-varying system model, which can be served as guidelines for the design of MRVSS algorithm in practical applications. Simulation results are demonstrated to corroborate the analytic results, and to compare MRVSS with the existing representative approaches. Superior properties of the MRVSS algorithm are indicated. & 2008 Elsevier B.V. All rights reserved.
Keywords: Least mean square (LMS) Variable step-size System identification
1. Introduction The least mean square (LMS) [1] adaptive algorithm is simple, robust, and has been used in numerous applications in the area of signal processing. To implement LMS in practice, the parameter of step-size that governs the convergence speed and steady-state excess mean-square error (EMSE) must be properly selected. It is shown that the rate of convergence is proportional to the step-size, while the steady-state EMSE is inversely proportional to the step-size [2,3]. When the fixed step-size LMS algorithm is used, the tradeoff between the rate of convergence and the steady-state EMSE must be considered. The problem of the fixed step-size LMS algorithm has motivated many researchers to approach to a time Corresponding author.
E-mail addresses: [email protected] (S. Zhao), [email protected] (Z. Man), [email protected] (S. Khoo), [email protected] (H.R. Wu). 0165-1684/$ - see front matter & 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.sigpro.2008.07.013
varying step-size. The main task of using a variable stepsize (VSS) approach is to speed up the rate of convergence using large step-size at the initial stages, and reduce the estimation error with small step-size in steady-state. Numerous VSS adjustment schemes have been proposed [6–19] to meet the conflicting requirements of fast convergence speed and low steady-state EMSE or misadjustment with increase in computational cost. All these schemes are shown to be based on a specific criterion. Different criteria usually lead to distinct performance improvements and limitations. For instance, the schemes based on the squared instantaneous error [8], on the timeaveraged estimate of the error autocorrelation at adjacent time [9], on the sign of the instantaneous error [12], and on the estimate of gradient power [7] have low-computational cost, but their steady-state performances are highly dependent on the measurement noise power level. As a result, these algorithms suffer from the applications where low signal-to-noise ratios (SNR) present. The schemes introduced in [16,17] were shown to be largely
ARTICLE IN PRESS 68
S. Zhao et al. / Signal Processing 89 (2009) 67–76
insensitive to the power of measurement noise, but the increases in computational cost are significant compared to the fixed step-size LMS algorithm for real-time applications. Recently, it was demonstrated in [20] that the VSS LMS algorithm developed by Kwong and Johnston in [8] is probably the best low-complexity VSS LMS algorithm available in the literature if well designed, except for its limited robustness to the measurement noise power. More recently, a modified version of the VSS algorithm called noise resilient variable step-size (NRVSS) LMS algorithm was presented in [18,19]. Compared with the VSS algorithm, the NRVSS algorithm was shown to achieve the same transient behavior and less sensitivity to the measurement noise for small increase in the computational cost. Since the optimal design of the NRVSS algorithm requires the reference signal power, it is specially indicated for the applications where estimation of the input signal power is possible. In this paper, we propose a more general VSS LMS algorithm where the VSS adjustment is based on the quotient of filtered quadratic error. We shall demonstrate that the newly proposed algorithm called more robust variable step-size (MRVSS) LMS algorithm can be designed to achieve lower steady-state EMSE or misadjustment, faster tracking property in nonstationary environment, and equal performance for the application of noise cancelation where the desired noiseless signal is correlated in time, when compared to NRVSS. Since the parameter setting of the MRVSS algorithm is free from estimation of the reference signal power, more applications should be found. This paper is organized as follows. In Section 2, the techniques of the VSS algorithm [8] and NRVSS algorithm [18,19] are introduced first, and then the proposed MRVSS algorithm is formulated. Next, we provide an analysis on the mean convergence of the MRVSS algorithm in Section 3, and discuss the steady-state performance of the MRVSS algorithm in Section 4. In Section 5, the mean stepsize behavior of the MRVSS algorithm is investigated. Section 6 discusses the design guidelines for new added constant parameters, and Section 7 presents extensive computer simulations. Finally, Section 8 concludes this paper.
2. Algorithm formulation Consider a general adaptive filtering problem of system identification, where xðnÞ is input signal, w0 is the vector consisting of the coefficients of the unknown system, wðnÞ is the vector consisting of the coefficients of the adaptive filter that models the unknown system, eo ðnÞ denotes the measurement noise, which is assumed to be zero mean and statistically independent of xðnÞ, and dðnÞ represents the desired response consisting of system output plus the measurement noise. A VSS LMS algorithm uses the following weight vector update equation: wðn þ 1Þ ¼ wðnÞ þ mðnÞeðnÞxðnÞ,
(1)
where mðnÞ is a scalar VSS, eðnÞ is the instantaneous estimation error given by eðnÞ ¼ dðnÞ wT ðnÞxðnÞ, the
superscript T denotes matrix transposition, and xðnÞ is the tap-delayed input vector. The VSS algorithm [8] uses the following VSS adjustment scheme: VSS:
mðn þ 1Þ ¼ aVSS mðnÞ þ gVSS e2 ðnÞ,
(2)
where aVSS and gVSS are the constant parameters, which are chosen as 0oaVSS o1 and gVSS 40. To ensure the stability and desired steady-state performance of the algorithm, the VSS is usually bounded as mmin pmðnÞp mmax . The extensive studies in [8,18–20] show that the VSS algorithm can achieve the mean step-size behavior that is the closest to the optimal step-size behavior for stationary environments. The main complaint about the VSS algorithm goes to its intrinsic large sensitivity to the measurement noise power. This high sensitivity could be explained by the analytic results of mean step-size and steady-state misadjustment derived in [8] where n is assumed to equal infinity, which are listed as follows: VSS:
E½mð1Þ ¼
M VSS
gVSS ðJmin þ Jex ð1ÞÞ , 1 aVSS
gVSS ð3 aVSS ÞðJmin þ Jex ð1ÞÞ trðRÞ, 1 a2VSS
(3)
(4)
where Jmin denotes the minimum MSE, which is equal to the power of measurement noise, Jex ð1Þ denotes the steady-state EMSE, which is given by J ex ð1Þ ¼ trfRKVSS ð1Þg with KVSS ðnÞ ¼ E½eðnÞeT ðnÞ, eðnÞ ¼ w0 wðnÞ the weight error vector, and trðRÞ the trace of the input correlation matrix R9E½xðnÞxT ðnÞ. We would like to write R as R ¼ Q KQ T with Q the unitary eigenvector matrix and K the diagonal eigenvalue matrix, which will be used in our latter analysis. It is seen from (3) and (4) that the mean step-size and steady-state misadjustment of VSS algorithm are proportional to the power of measurement noise. It indicates that the VSS algorithm will produce large steady-state EMSE or misadjustment in the applications where low SNR presents. When the power of measurement noise increases, the mean stepsize of the VSS algorithm also increases, which may diverge the algorithm for a large increase in the noise power. The work presented in [18,19] addressed this problem with the following step-size adjustment scheme: NRVSS: mðn þ 1Þ ¼ aNRVSS mðnÞ þ gNRVSS ½kxT ðnÞxðnÞ 1e2 ðnÞ, (5) where the constant parameter settings, aNRVSS ¼ aVSS and gNRVSS ¼ NgVSS =2, are provided in [18,19], and the newly added parameter k is optimally designed as k ¼ 1=ðr x NÞ with r x the input power and N the filter length. The absolute value of mðnÞ in (5) is used for the weight vector update. It was shown in [18,19] that the NRVSS algorithm can achieve the same initial transient behavior, whereas less sensitivity to the noise power when compared to VSS algorithm. In this paper, we would like to further address the problem of the VSS algorithm by proposing a new approach to adjust the VSS based on a quotient of filtered
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
quadratic error. The proposed scheme is listed as follows:
mðn þ 1Þ ¼ amðnÞ þ gyðnÞ,
MRVSS: Pn
yðnÞ ¼ Pni¼0
ai e2 ðn iÞ
j 2 j¼0 b e ðn
jÞ
,
(6)
(7)
where a and b are decaying factors used for the exponential windows in numerator and denominator of (7) and bounded as 0oaobo1. The constant parameter settings, a ¼ aVSS and g ¼ gVSS =N, should be recommended such that the MRVSS algorithm achieves the almost optimal mean step-size behavior as the VSS algorithm for stationary environments according to the design guidelines of [20]. For nonstationary environments, we shall demonstrate that the parameter g in (6) could be optimally designed and small a should be chosen for good tracking performance. The use of a quotient form in (6) and (7) serves as two objectives. First, the quotient is expected for a smoothing decrease of the step-size, where the transient behavior of the proposed VSS in stationary environment may be described by a reformulation of (7) as follows: AðnÞ aAðn 1Þ þ e2 ðnÞ ¼ yðnÞ ¼ BðnÞ bBðn 1Þ þ e2 ðnÞ a e2 ðnÞ . yðn 1Þ þ b bBðn 1Þ
(8)
Note that in derivation of (8), we have neglected the value of e2 ðnÞ in the denominator, since compared to e2 ðnÞ the error cumulant bBðn 1Þ becomes much larger during adaptation because the decaying factor b is very close to one, which will be discussed in Section 6. Assuming that the initial step-size mð0Þ is set to be mmax for initial fast transient behavior, from (8) we may observe that since the ratio of e2 ðnÞ and bBðn 1Þ decreases following the decrease of the power of output error, the VSS mðnÞ should also decrease from its maximum value following the decrease of yðnÞ. Second, as shown in the analysis given in the latter sections of the paper, in steady-state, the EMSE of the algorithm should be much smaller compared to the power of measurement noise. This implies that the measurement noise dominates the numerator and denominator of (7). In statistic sense, the power of measurement noise in the numerator and denominator could be canceled out, leaving the steady-state mean stepsize determined only by the constant parameters. Therefore, the decaying factors a and b could be designed beforehand for a desired steady-state mean step-size level. Comparing the computational complexities of the NRVSS and MRVSS algorithms, we find that the MRVSS algorithm requires the same number of multiplications/ divisions and additions per iteration by using the recursive forms of AðnÞ and BðnÞ in (8). Note that there is a division computation in the MRVSS algorithm, which usually requires some more processing time compared to additions or multiplications. Both of the algorithms are considered to be low complexity and real-time algorithms. Comparing the parameter design simplicity of NRVSS and MRVSS added on the VSS algorithm, the NRVSS algorithm
69
adds one new design parameter k, which could be optimally designed with the reference signal power known, and the MRVSS algorithm adds two new decaying factors a and b, which could provide a flexible design as discussed in latter sections and demonstrated in simulations. In the following sections, we shall study the convergence and performance of the MRVSS algorithm in both stationary and nonstationary environments, and the design guidelines of the decaying factors a and b. 3. Convergence in mean sense For the problem of system identification being considered in this paper, we assume that the desired response of the system arises from the model dðnÞ ¼ wT0 xðnÞ þ eo ðnÞ.
(9)
We would like to consider an unknown plant whose weight vector follows a random walk process [2]: w0 ðn þ 1Þ ¼ w0 ðnÞ þ rðnÞ,
(10)
where rðnÞ represents a driving vector of zero-mean white sequence with a diagonal autocorrelation matrix of s2r I, and I is an identity matrix. It is known that for s2r ¼ 0, the unknown plant is considered as time invariant and the adaptive filters do not need to perform a tracking task. In this paper, we consider a general case of s2r . Using (1) and (10), the transient behavior of the weight vector error, eðnÞ, can be modeled as
eðn þ 1Þ ¼ eðnÞ mðnÞeðnÞxðnÞ þ rðnÞ.
(11)
Consider that the output error can be reformed as eðnÞ ¼ eo ðnÞ þ xT ðnÞeðnÞ.
(12)
Substituting (12) into (11) and applying the unitary transformation Q T on both sides of (11) yields T
^ x^ ðnÞvðnÞ mðnÞeo ðnÞxðnÞ ^ þ r^ ðnÞ, vðn þ 1Þ ¼ ½I mðnÞxðnÞ (13) ^ where r^ ðnÞ; xðnÞ, and vðnÞ are the transformed versions of rðnÞ; xðnÞ, and eðnÞ by Q T, respectively. To proceed further, we introduce the independence assumption. That is, the input data vectors xð1Þ; xð2Þ; . . . ; xðnÞ are mutually independent, and xðnÞ is statistically independent of the previous samples of the system response dðnÞ. We also require the following averaging principle commonly used in [4,5,8,9]: T
T
^ x^ ðnÞ E½mðnÞE½xðnÞ ^ x^ ðnÞ. E½mðnÞxðnÞ
(14)
This is approximately true since the step-size mðnÞ varies slowly around its mean value when compared to eðnÞ and xðnÞ for small g. Therefore, this justifies the independence assumption of mðnÞ and m2 ðnÞ with eðnÞ; xðnÞ and wðnÞ. Based on above assumptions, Eq. (13) is stable in the mean sense if and only if the eigenvalues of the matrix
ARTICLE IN PRESS 70
S. Zhao et al. / Signal Processing 89 (2009) 67–76 T
^ x^ ðnÞ are all within the unit circle. DðnÞ9E½I mðnÞxðnÞ This may be solved as follows:
In the steady-state, we can assume that limn!1 cðn þ 1Þ ¼ limn!1 cðnÞ. From (19), we then obtain
det½DðnÞ ZI ¼ detfI E½mðnÞK ZIg
cð1Þ ¼ ½I Lð1Þ1 fE½m2 ð1ÞJ min k þ s2r 1g,
¼
N Y
ð1 Z E½mðnÞli Þ ¼ 0,
(15)
i¼1
where Z denotes the eigenvalues of DðnÞ; li is the ith eigenvalue of the input correlation matrix R. To ensure all the solutions of (15) within the unit circle, we only need to find the conditions to make sure both the smallest and the largest solutions to be within the unit circle. That is 1o1 E½mðnÞlmax o1 E½mðnÞlmin o1,
where n is assumed to equal infinity. Using (22) into the steady-state excess MSE given by Jex ð1Þ ¼ tr½RKð1Þ ¼ kT cð1Þ, we obtain J ex ð1Þ ¼ kT ½I Lð1Þ1 fE½m2 ð1ÞJmin k þ s2r 1g.
0oE½mðnÞo
lmax
.
J ex ð1Þ ¼ kT fq kE½m2 ð1ÞkT g1 fE½m2 ð1ÞJmin k þ s2r 1g ¼ kT fq1 þ q1 kðE½m2 ð1Þ1 kT q1 kÞ1 kT q1 g
(16)
(17)
The condition in (17) guarantees the stability of the MRVSS algorithm in the mean sense for nonstationary environments. For the special case of s2r ¼ 0, the algorithm is guaranteed to be convergent in the mean sense. Alternative derivations of condition (17) could be found in [8,9]. For the case of a constant step-size mðnÞ, condition (17) is the same as the condition for mean convergence of the fixed step-size LMS algorithm as shown in [4,5]. 4. Analysis of steady-state performance In this section, we shall study the steady-state performance of MRVSS algorithm in terms of the misadjustment. To proceed further, besides the independent assumption we need to further assume that the input signal xðnÞ and the desired response dðnÞ are jointly Gaussian. Postmultiplying both sides of (13) by vT ðn þ 1Þ and then taking the expectations yields
fE½m2 ð1ÞJmin k þ s2r 1g
2
2
þ 2E½m ðnÞKCðnÞK þ E½m ðnÞK trðKCðnÞÞ þ E½m2 ðnÞJmin K þ s2r I,
¼ f1 E½m2 ð1ÞkT q1 kg1 fE½m2 ð1ÞJmin kT q1 k þ s2r kT q1 1g.
(18)
where CðnÞ ¼ E½vðnÞvT ðnÞ denotes the transformed weight error covariance of eðnÞ. Note that in the derivation and simplification of (18), the Gaussian moment factoring theorem [2] was used. Eq. (18) gives the transient behavior of the transformed weight error covariance. Now, let cðnÞ be the vector with the entries of the diagonal elements of CðnÞ, let k be the column vector of eigenvalues of R, and let 1 be a column vector of 1’s with the same length as k. Following the analyzed results in [5,8], from (18) we have cðn þ 1Þ ¼ LðnÞcðnÞ þ E½m2 ðnÞJ min k þ s2r 1,
(19)
LðnÞ ¼ diag½r1 ; r2 ; . . . ; rN þ E½m2 ðnÞkkT ,
(20)
ri ¼ 1 2E½mðnÞli þ 2E½m2 ðnÞl2i .
(21)
(24)
It is easy to obtain the following equalities:
kT q1 k ¼
kT q1 1 ¼
N X
li
i¼1
2E½mð1Þ 2E½m2 ð1Þli
N X i¼1
,
(25)
1 . 2E½mð1Þ 2E½m2 ð1Þli
(26)
Substituting the equalities of (25) and (26) into (24) yields 9 8 P > > E½m2 ð1Þli > > > > J min N = < i¼1 2E½mð1Þ 2E½m2 ð1Þli J ex ð1Þ ¼ 2 > > PN E½m ð1Þli > > > > ; :1 i¼1 2E½mð1Þ 2E½m2 ð1Þli 9 8 2 PN > > sr > > > > = < i¼1 2E½mð1Þ 2E½m2 ð1Þli . (27) þ 2 > > P E½m ð1Þli > > N > > ; :1 i¼1 2E½mð1Þ 2E½m2 ð1Þli To simplify the result of (27), we neglected the terms involved with E½m2 ð1Þ that are very smaller compared to other terms in the event of small values of misadjustment. After the proceedings, we have J ex ð1Þ J min trðRÞ
Cðn þ 1Þ ¼ CðnÞ E½mðnÞCðnÞK E½mðnÞKCðnÞ
(23)
Let q ¼ diag½1 r1 ; 1 r2 ; . . . ; 1 rN be a diagonal matrix with ri given in (21). We proceed (23) as follows:
where lmax and lmin are the largest and smallest eigenvalues of R, respectively. From (16) we can easily find the following bound on the mean step-size: 2
(22)
E½m2 ð1Þ Ns2r þ . 2E½mð1Þ 2E½mð1Þ
(28)
Next, we shall find the forms for E½mð1Þ and E½m2 ð1Þ, respectively. Taking expectations on both sides of (7) and assuming that n approaches to infinity yields lim E½mðnÞ ¼ a lim E½mðn 1Þ þ g lim E½yðnÞ.
n!1
n!1
n!1
(29)
From (29), we can approximate the steady-state mean step-size as follows: E½mð1Þ
g limn!1 E½yðnÞ . 1a
(30)
The term limn!1 E½yðnÞ in (30) is given by AðnÞ . lim E½yðnÞ ¼ lim E n!1 n!1 BðnÞ
(31)
Note that AðnÞ and BðnÞ are with recursive forms as shown in (8). Assuming that n approaches to infinity and
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
limn!1 E½e2 ðnÞ is slowly varying, we have the following approximations: lim E½AðnÞ
1 lim E½e2 ðnÞ, 1 a n!1
(32)
lim E½BðnÞ
1 lim E½e2 ðnÞ. 1 b n!1
(33)
n!1
n!1
AðnÞ E½AðnÞ lim . lim E n!1 n!1 E½BðnÞ BðnÞ
(34)
This cannot really hold for the algorithm, but approximately true since that limn!1 AðnÞ and limn!1 BðnÞ will vary slowly. Using the results of (32) and (33), we can approximate (31) as
n!1
(35)
gð1 bÞ . ð1 aÞð1 aÞ
(36)
Here, we would like to drop a point that comparing to the steady-state mean VSS step-size shown in (3), Eq. (36) shows that the steady-state mean MRVSS step-size is now free from the steady-state MSE, i.e., J min þ J ex ð1Þ. Since the assumption in (34) is approximately true, we may conclude that the MRVSS algorithm should be much less sensitive to the measurement noise power when compared to the VSS algorithm. This will be demonstrated in the simulation section. To proceed the steady-state mean-square MRVSS stepsize, taking the expectations of the squared (6) and assuming that n approaches to infinity yields n!1
2
þ g2 E½y ðnÞg.
(37)
For small g, the last term in (37) involving g2 is negligible compared with the other terms. So after eliminating this small term, we can approximate as lim E½m2 ðnÞ
n!1
2agE½mð1ÞE½yð1Þ . 1 a2
(38)
Substituting the results of (35) and (36) into (38) yields E½m2 ð1Þ
2ag2 ð1 bÞ2 ð1 a2 Þð1 aÞð1 aÞ2
.
(39)
Eq. (39) gives the steady-state mean-square MRVSS stepsize. Substituting the result of mð1Þ (36) and m2 ð1Þ (39) into (28) and using the definition of M ¼ J ex ð1Þ=J min yields the misadjustment of the MRVSS algorithm:
agð1 bÞ ð1 aÞð1 aÞN s2r trðRÞ þ . M 2gð1 bÞJ min ð1 a2 Þð1 aÞ
agð1 bÞ trðRÞ. ð1 a2 Þð1 aÞ
(43)
We can observe from (43) that the steady-state misadjustment of the MRVSS algorithm is proportional to the ratio of ð1 bÞ=ð1 aÞ, which is consistent with the result of the steady-state mean MRVSS step-size shown in (36). For the case that a is very close to one, we can approximate (43) as M 12 E½mð1Þ trðRÞ.
(44)
The result of (44) is then consistent with the result for the fixed LMS algorithm as shown in [1–4]. 5. Mean behavior of the proposed VSS In this section, we investigate the mean behavior of the MRVSS step-size in a nonstationary environment where the plant varies following the model of (10). Taking the expectations of (6) we obtain E½mðn þ 1Þ ¼ aE½mðnÞ þ gE½yðnÞ.
lim E½m2 ðnÞ ¼ lim fa2 E½m2 ðn 1Þ þ 2agE½mðnÞE½yðnÞ
n!1
We can observe from (42) that smaller a should provide better tracking performance. We will demonstrate this point in the simulation section. In stationary environments, i.e., s2r ¼ 0, the misadjustment is then given by M
1b . 1a
Substituting the result of (35) into (30) yields E½mð1Þ
optimize the choice for g given a; a and b to achieve the minimum misadjustment. When both terms are equal, the optimal choice of g is given by sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð1 aÞð1 a2 Þð1 aÞ2 Ns2r g ¼ . (41) 2að1 bÞ2 J min trðRÞ And the result of minimum misadjustment follows: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2aN s2r trðRÞ . (42) M min ¼ ð1 þ aÞJmin
To proceed (31), we assume that
lim E½yðnÞ ¼ E½yð1Þ
71
(45)
Next, we need to evaluate the mean value of yðnÞ, which is given by AðnÞ . (46) E½yðnÞ ¼ E BðnÞ To proceed further, we make the following assumption: AðnÞ E½AðnÞ . (47) E BðnÞ E½BðnÞ This is approximately true for the decaying factors of a and b close to one. Using (47) in (46), we then only need to evaluate E½AðnÞ and E½BðnÞ, which are given by E½AðnÞ ¼ aE½Aðn 1Þ þ E½e2 ðnÞ,
(48)
E½BðnÞ ¼ bE½Bðn 1Þ þ E½e2 ðnÞ.
(49)
Using the form of the output error eðnÞ ¼ eo ðnÞ þ x^ T ðnÞvðnÞ, we proceed E½e2 ðnÞ as E½e2 ðnÞ ¼ E½ðeo ðnÞ þ x^ T ðnÞvðnÞÞ2
(40)
Since the first term in (40) is directly proportional to g and the second term is inversely proportional to g, we can
^ ¼ E½e2o ðnÞ þ 2E½eo ðnÞx^ T ðnÞvðnÞvT ðnÞxðnÞ ^ þ E½x^ T ðnÞvðnÞvT ðnÞxðnÞ ðJ min þ kT cðnÞÞ,
(50)
ARTICLE IN PRESS 72
S. Zhao et al. / Signal Processing 89 (2009) 67–76
where cðnÞ is given in (19) and k is the column vector of the eigenvalues of R. Note that during the derivation of (50) we have used the independent assumption and the assumptions that the input signal xðnÞ and desired response dðnÞ are jointly Gaussian and the measurement noise eo ðnÞ is zero-mean white Gaussian and statistically independent of xðnÞ, which justifies the Gaussian moment factoring theorem and the simplification. Using (46)–(50), we have E½yðnÞ
aE½Aðn 1Þ þ ðJ min þ kT cðnÞÞ bE½Bðn 1Þ þ ðJ min þ kT cðnÞÞ
.
(51)
Since cðnÞ requires the value of E½m2 ðn 1Þ, we need to establish the transient behavior for E½m2 ðnÞ. Taking the expectations of the squared (6) we obtain E½m2 ðnÞ a2 E½m2 ðn 1Þ þ 2agE½mðn 1ÞE½yðn 1Þ 2
þ g2 E½y ðn 1Þ.
(52)
Note that we have used E½mðnÞyðnÞ E½mðnÞE½yðnÞ based on the independence assumption. Neglecting the very small term involving with g2 , we can approximate (52) as follows: E½m2 ðnÞ a2 E½m2 ðn 1Þ þ 2agE½mðn 1ÞE½yðn 1Þ.
(53)
Now the mean behavior of the MRVSS step-size is completed described by (45), (48), (49), (51), (53) and (19)–(21). 6. Design of constant parameters In this section, we derive some design guidelines for the constant parameters of the new VSS scheme. We recommend that the choices of a and g are a ¼ aVSS and g ¼ gVSS =N to obtain the optimal transient behavior, where aVSS and gVSS should be designed follows the guidelines in [20]. To show the effectiveness of the choices of the parameters a and b on the steady-state performance of MRVSS algorithm, we make a ratio of the misadjustments of MRVSS algorithm and VSS algorithm as follows: M agð1 bÞ trðRÞ ¼ M VSS ð1 a2 Þð1 aÞ 1 a2VSS . gVSS ð3 aVSS ÞðJmin þ Jex ð1ÞÞ trðRÞ
MRVSS algorithm is more significant. The choices of a and b do not affect the transient behavior of the mean MRVSS step-size much, but do affect the steady-state mean MRVSS step-size, which we will demonstrate in the simulation section. For nonstationary environments, a should be chosen small for good tracking performance according to (42) and g are given from (41). However, in the simulation we found g =3 rather gives a better tracking performance, this may be due to the assumptions and approximations we have made in the derivation of g . The decaying factors a and b are chosen similarly as made for stationary case.
7. Simulation results This section presents four examples to corroborate the analytic results derived in this paper and to illustrate the properties of the MRVSS algorithm. In the examples, the performances of VSS algorithm [8], NRVSS algorithm [18,19] and MRVSS algorithm are compared to demonstrate that MRVSS can lead to a better steady-state performance for the same initial transient behavior in stationary environments, and in nonstationary environments we demonstrate in Example 3 that MRVSS can provide a faster tracking property. The mean MRVSS stepsize behavior via the choices of a and b, and via the measurement noise power are demonstrated in Example 1. The sensitivities of VSS, NRVSS and MRVSS to noise power are compared in Example 2. Examples 1–3 are based on the problem of system identification. Example 4 presents a comparison between NRVSS and MRVSS algorithms when applied to noise cancelation. To begin with, let us describe the experimental setup and parameter settings. Unless stated otherwise, in all examples the input signal xðnÞ is with unit power and the additive measurement noise eo ðnÞ is zero-mean, white Gaussian and statistically independent of xðnÞ. The unknown plant and adaptive filters have the same number of 10 taps (N ¼ 10), where the plant impulse response is designed using a Hanning window with unit norm (wT0 ð0Þw0 ð0Þ ¼ 1). The parameters are set according to
(54)
[19], where aVSS ¼ 0:9997; gVSS ¼ 2 105 ; aNRVSS ¼ aVSS ; gNRVSS ¼ NgVSS =2; a ¼ aVSS ; g ¼ gVSS =N; a ¼ 0:9; b ¼
For the settings a ¼ aVSS and g ¼ gVSS =N, Eq. (54) can be simplified as
1 105 . For all the algorithms, we used mð0Þ ¼ mmax ¼ 0:01 and mmin ¼ 0. It can easily be checked that the mean stability condition (17) is satisfied with such parameter
M aVSS ð1 bÞ . ¼ M VSS Nð3 aVSS Þð1 aÞðJ min þ Jex ð1ÞÞ
(55)
Considering that aVSS is close to one and Jex ð1Þ5J min , we can approximate (55) as M ð1 bÞ . M VSS 2NJmin ð1 aÞ
(56)
From (56), we can see that the MRVSS algorithm can be designed to achieve smaller steady-state misadjustment by choosing ð1 bÞ5ð1 aÞ for any level of measurement noise power and filter length N. It is not hard to observe that for larger measurement noise power (larger Jmin ), the improvement of the steady-state misadjustment with
Table 1 Steady-state misadjustments for VSS, NRVSS and MRVSS algorithms under different SNR conditions and comparisons between theoretical models and Monte Carlo simulations for MRVSS algorithm (all results are in dB) SNR
20 10 0 10
Analytical values
Simulated values
MRVSS (Eq. (43))
VSS
NRVSS
MRVSS
54.78 54.78 54.78 54.78
24.56 14.45 12.64 12.64
32.82 23.77 15.27 12.72
53.01 52.23 52.20 52.18
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
setting. The results were obtained by averaging 500 ensemble trials in Monte Carlo simulations. The SNR is calculated by SNR ¼ 10 log10 E½x2 ðnÞ=E½e2o ðnÞ.
0 −5
EMSE (dB)
−10 −15 −20
Example 1. The plant impulse response is time invariant (w0 ðnÞ ¼ w0 ðn 1Þ) and the input is white Gaussian. We used SNR ¼ 60, 20, 10, 0 and 10 dB. Table 1 presents the steady-state misadjustments for VSS, NRVSS and MRVSS for SNR ranging from 20 to 10 dB. The theoretical values are also obtained based on Eq. (43), which are compared with the simulation results. The main causes of mismatch between the analytical and simulation results seem to be the following: (a) the assumption of (34); (b) the neglected terms involving g2 or E½m2 ð1Þ in (38) and (28), respectively; and (c) the independent assumption. Figs. 1 and 2 show the EMSE learning curves and mean step-size behaviors, respectively, of VSS, NRVSS and MRVSS algorithms up to the iteration 40,000 at SNR ¼ 10 dB. All algorithms present approximately the
(a)
−25 −30
(b)
−35
(c)
−40 −45
0
0.5
1
1.5
2 2.5 iterations
3
3.5
4
x 104
Fig. 1. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 1: the white Gaussian input and stationary plant with SNR ¼ 10 dB.
10−2
(a)
10−3
10−2 10−3 Mean step size
Mean step size
(b)
10−4
10−5
(c)
0
0.5
1
1.5
2 2.5 iterations
10−4 10−5 (a)=(b)=(c)=(d) 10−6
(d) 10−6
73
3
3.5
4
10−7
x 104
Fig. 2. Comparisons of mean behavior of the step-size (mðnÞ) between (a) VSS, (b) NRVSS, (c) MRVSS simulations, and (d) MRVSS model for Example 1: the white Gaussian input and stationary plant with SNR ¼ 10 dB.
0
0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 iterations x 105
2
Fig. 4. Comparisons of mean behavior of the MRVSS step-size (mðnÞ) for Example 1: the white Gaussian input and stationary plant with SNR: (a) 10 dB, (b) 0 dB, (c) 10 dB and (d) 20 dB.
10−2
Mean step size
10−3 10−4 10−5 10
(e)
−6
(c) (b)
10−7
(a)
10−8 10−9
(d) 0
1
2
3
4 iterations
5
6
7
8 x 104
Fig. 3. Comparisons of mean behavior of the MRVSS step-size (mðnÞ) for Example 1: the white Gaussian input and stationary plant with (a) a ¼ 0:9, b ¼ 1 105 , (b) a ¼ 0:99, b ¼ 1 105 , (c) a ¼ 0:999, b ¼ 1 105 , (d) a ¼ 0:9, b ¼ 1 106 and (e) a ¼ 0:9, b ¼ 1 104 .
ARTICLE IN PRESS 74
S. Zhao et al. / Signal Processing 89 (2009) 67–76
same initial transient behavior but MRVSS achieves lower steady-state EMSE. The predicted and simulation mean step-size behaviors agree well with each other. Fig. 3 shows the evolution of the mean MRVSS step-size for different settings of the decaying factors a and b at SNR ¼ 60 dB. It can be seen that the transient behavior of MRVSS algorithm is not sensitive to the choices of a and b, but the steady-state EMSE or misadjustment is determined by a and b. The smaller a or larger b gives the lower steady-state EMSE. However, if small b or large a is used, poor steady-state EMSE may be produced, which is consistent with our analytic result. Fig. 4 shows the evolution of the MRVSS step-size for SNR ¼ 20; 10; 0 and 10 dB. It can be seen that the mean MRVSS step-size has low sensitivity to the SNR changes, so does the steady-state misadjustment of MRVSS algorithm as demonstrated in Example 2.
Example 2. The plant impulse response is time invariant, and both white Gaussian input and correlated first-order Markov input ðxðnÞ ¼ a1 xðn 1Þ þ uðnÞÞ with a1 ¼ 0:9 were tested. The measurement noise has an abrupt power variation at iteration 50,000 such that SNR changes from 60 to 20 dB. Figs. 5 and 7 show the EMSE behaviors of VSS, NRVSS and MRVSS algorithms, and Figs. 6 and 8 show the evolution of the mean step-sizes of VSS, NRVSS and MRVSS algorithms. It can be seen that all algorithms perform equally under high SNR conditions, whereas under low SNR conditions, the MRVSS algorithm shows a better performance in steady-state conditions (Figs. 5–8). Example 3. The plant impulse response is time variant modeled as a first-order random walk process as in Eq. (10) with s2r ¼ 1010 , and the input is a correlated first-order Markov process (xðnÞ ¼ a1 xðn 1Þ þ uðnÞ) with a1 ¼ 0:7 [12]. We tested at SNR ¼ 20 dB. The parameters
0 0
−10
−10
−20 EMSE (dB)
−20 EMSE (dB)
−30 −40
(a)
−50
(b)
−60
(c)
(a)=(b)=(c)
−90
−50
0
1
2
3
4 5 6 iterations
−80 7
8
(b) (c)
(a)=(b)=(c) 0
1
2
3
9 10 x 104
Fig. 5. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the white Gaussian input and stationary plant with SNR change from 60 to 20 dB.
4
5 6 iterations
7
8
9 x
10 104
Fig. 7. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the correlated first-order Markov input and stationary plant with SNR change from 60 to 20 dB.
10−2
10−2 10−3
(a)
10−4
(b)
10−5
10−3 Mean step size
Mean step size
(a)
−70
−80
(c)
10−6 (a) (c)
10−7 10−8 10−9
−40
−60
−70
−100
−30
1
2
3
4
5 6 iterations
(b)
10−4 10−5
(c)
10−6 (a)
10−7
(b) 0
(a)
10−8 7
8
9
10
x 104
Fig. 6. Comparisons of mean behavior of the step-size (mðnÞ) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the white Gaussian input and stationary plant with SNR change from 60 to 20 dB.
(c) (b) 0
1
2
3
4
5 6 iterations
7
8
9
10
x 104
Fig. 8. Comparisons of mean behavior of the step-size (mðnÞ) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the correlated first-order Markov input and stationary plant with SNR change from 60 to 20 dB.
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
75
0 2
EMSE (dB)
−5 −10
0
−15
−2
−25
2
−30
0
−35
−2
−40
(a)
(b)
−45 −50
5000
10000
15000
0
5000
10000
15000
0
5000
10000
15000
2
(c) 0
0.5
1
1.5 iterations
2
2.5
0
3
x 104
−2
Fig. 9. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 3: the correlated first-order Markov input and nonstationary plant with SNR ¼ 20 dB.
0.01
iterations Fig. 11. Comparisons between NRVSS and MRVSS algorithms applied to noise cancelation application. (a) The desired square wave plus a pure sinusoidal noise; (b) NRVSS processed square wave; and (c) MRVSS processed square wave.
0.009
0.01 Step−size instantaneous value
0.008 Mean step size
0
−20
0.007 0.006 0.005 0.004 0.003 0.002 (b)
0.001 0
(a)
(c) 0
0.5
1
1.5 iterations
2
2.5
3
x 104
Fig. 10. Comparisons of mean behavior of the step-size ðmðnÞÞ between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 3: the correlated first-order Markov input and nonstationary plant with SNR ¼ 20 dB.
of MRVSS algorithm were reset as a ¼ 0:5 and g ¼ 0:2 according to Eqs. (42) and (41). All the other parameters were unchanged. Figs. 9 and 10 show the tracking behaviors and the evolution of mean step-sizes of VSS, NRVSS and MRVSS algorithms. It can be seen that the MRVSS algorithm achieves a faster tracking behavior and a slightly lower misadjustment. We need to point out that the accumulative nature of MRVSS may lower its recovery ability for abrupt changes of the plant impulse response when small g is used. This is because the MRVSS has the memory of past power of output errors. In such cases, the suggestion in [12] could be used. That is, the stepsizeshould switch automatically to mmax at the abrupt change, where the system is monitored and the abrupt changes are detected employing the power of output error with suitable threshold (chosen according to the application at hand). The power of output error is recursively updated as EðnÞ ¼ bEðn 1Þ þ ð1 bÞe2 ðnÞ. The additional
0.009 0.008 0.007 0.006 0.005 0.004 0.003
(a)
0.002
(b)
0.001 0
0
5000
10000
15000
iterations Fig. 12. Instantaneous value of NRVSS (a) and MRVSS (b) step-sizes for Example 4: the application of noise cancelation.
cost is two extra multiplications and one extra addition. For MRVSS algorithm, the values of AðnÞ and BðnÞ are reinitialized consequently. Although such scheme is applicable in the applications, NRVSS and VSS are more recommended in such situations for their better recovery abilities. Example 4. In this example, we study the performance of MRVSS algorithm for the application of noise cancelation where the output error eðnÞ is expected to converge to the desired noiseless signal, which is correlated in time. The task in this example is to remove an additive pure sinusoidal noise generated by ðuðnÞ ¼ 0:8 sinðon þ 0:5pÞ with o ¼ 0:1p from a square wave generated by ðsðnÞ ¼ 2 ððmodðn; 1000Þo1000=2Þ 0:5ÞÞ where modðn; 1000Þ computes the modulus of n over 1000. The summation of sðnÞ and uðnÞ severs as the reference response of the adaptive filters. The input to the adaptive filters is also a
ARTICLE IN PRESS 76
S. Zhao et al. / Signal Processing 89 (2009) 67–76
pffiffiffi sinusoidal signal (xðnÞ ¼ 2 sinðonÞ) with o ¼ 0:1p, which is in unit power. The SNR is therefore 0 dB. The parameter settings were the same as used in Examples 1 and 2. The length of the adaptive filters is 10. Since the VSS algorithm diverged in the simulation, only the performances of NRVSS algorithm and MRVSS algorithm are compared. Fig. 11 shows the reference response and the processed results by the NRVSS algorithm and MRVSS algorithm. It can be seen that both algorithms are capable of suppressing the interference. Fig. 12 shows the instantaneous stepsize behavior of both algorithms. The step-size values in steady-state are approximately equal while MRVSS achieves slightly faster initial adaptation velocity. 8. Conclusion This work presented a quotient form for the VSS adjustment of LMS algorithm. The resulting MRVSS algorithm was demonstrated to be less sensitive to the power of the measurement noise when compared to VSS and NRVSS. The additive computational cost is negligible and real-time implementation of MRVSS is realizable. The analytical results in the paper can be served as design guidelines for the setting of two new added decaying factors. Monte Carlo simulations corroborated the analytical results and illustrated superior properties of MRVSS. References [1] B. Widrow, S.D. Steam, Adaptive Signal Processing, Prentice-Hall, Englewood Cliffs, NJ, 1985. [2] S. Haykin, Adaptive Filter Theory, fourth ed., Prentice-Hall, Englewood Cliffs, NJ, 2002. [3] A.H. Sayed, Fundamentals of Adaptive Filtering, Wiley, New York, 2003. [4] B. Widrow, J.M. McCool, Stationary and nonstationary learning characteristics of the LMS filter, Proc. IEEE 64 (8) (August 1976) 1151–1162.
[5] A. Feuer, E. Weinstein, Convergence analysis of LMS filters with uncorrelated Gaussian data, IEEE Trans. Acoust. Speech Signal Process. ASSP-33 (1) (February 1985) 222–230. [6] R. Harris, D. Chabries, F.A. Bishop, A variable step (VS) adaptive filter algorithm, IEEE Trans. Acoust. Speech Signal Process. ASSP-34 (2) (April 1986) 309–316. [7] V.J. Mathews, Z. Xie, A stochastic gradient adaptive filter with gradient adaptive step size, IEEE Trans. Signal Process. 41 (6) (June 1993) 2075–2087. [8] R. Kwong, E.W. Johnston, A variable step size LMS algorithm, IEEE Trans. Signal Process. 40 (7) (July 1992) 1633–1642. [9] T. Aboulnasr, K. Mayyas, A robust variable step-size lms-type algorithm: analysis and simulations, IEEE Trans. Signal Process. 45 (3) (March 1997) 631–639. [10] D.I. Pazaitis, A.G. Constantinides, A novel kurtosis driven variable step-size adaptive algorithm, IEEE Trans. Signal Process. 47 (3) (March 1999) 864–872. [11] W.P. Ang, B. Farhang-Boroujeny, A new class of gradient adaptive step-size LMS algorithms, IEEE Trans. Signal Process. 49 (4) (April 2001) 805–810. [12] T.I. Haweel, A simple variable step size LMS adaptive algorithm, Internat. J. Circ. Theor. Appl. 32 (November 2004) 523–536. [13] A.I. Sulyman, A. Zerguine, Convergence and steady-state analysis of a variable step-size NLMS algorithm, Signal Processing 83 (June 2003) 1255–1273. [14] A. Mader, H. Puder, G.U. Schmidt, Step-size control for acoustic echo cancellation filters—an overview, Signal Processing 80 (September 2000) 1697–1719. [15] H.C. Shin, A.H. Sayed, W.J. Song, Variable step-size NLMS and affine projection algorithms, IEEE Signal Process. Lett. 11 (2) (February 2004) 132–135. [16] S. Koike, A class of adaptive step-size control algorithms for adaptive filters, IEEE Trans. Signal Process. 50 (6) (June 2002) 1315–1326. [17] J. Okello, et al., A new modified variable step-size for the LMS algorithm, in: Proceedings of the International Symposium on Circuits and Systems, 1998, pp. 170–173. [18] M.H. Costa, J.C.M. Bermudez, A robust variable step-size algorithm for LMS adaptive filters, in: Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2006, pp. 93–96. [19] M.H. Costa, J.C.M. Bermudez, A noise resilient variable step-size LMS algorithm, Signal Processing 88 (March 2008) 733–748. [20] C.G. Lopes, J.C.M. Bermudez, Evaluation and design of variable stepsize adaptive algorithms, in: Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2001, pp. 3845–3848.
Contents lists available at ScienceDirect
Signal Processing journal homepage: www.elsevier.com/locate/sigpro
Variable step-size LMS algorithm with a quotient form Shengkui Zhao a,, Zhihong Man b, Suiyang Khoo a, Hong Ren Wu c a
School of Computer Engineering, Nanyang Technological University (NTU), Nanyang Avenue, Singapore 639798, Singapore School of Engineering, Monash University (Malaysia), Jalan Lagoon Selatan, Bandar Sunway, Petaling Jaya 46150, Malaysia c School of Electrical and Computer Engineering, RMIT University, VIC 3001, Australia b
a r t i c l e in fo
abstract
Article history: Received 21 January 2008 Received in revised form 14 July 2008 Accepted 14 July 2008 Available online 25 July 2008
An improved robust variable step-size least mean square (LMS) algorithm is developed in this paper. Unlike many existing approaches, we adjust the variable step-size using a quotient form of filtered versions of the quadratic error. The filtered estimates of the error are based on exponential windows, applying different decaying factors for the estimations in the numerator and denominator. The new algorithm, called more robust variable step-size (MRVSS), is able to reduce the sensitivity to the power of the measurement noise, and improve the steady-state performance for comparable transient behavior, with negligible increase in the computational cost. The mean convergence, the steady-state performance and the mean step-size behavior of the MRVSS algorithm are studied under a slow time-varying system model, which can be served as guidelines for the design of MRVSS algorithm in practical applications. Simulation results are demonstrated to corroborate the analytic results, and to compare MRVSS with the existing representative approaches. Superior properties of the MRVSS algorithm are indicated. & 2008 Elsevier B.V. All rights reserved.
Keywords: Least mean square (LMS) Variable step-size System identification
1. Introduction The least mean square (LMS) [1] adaptive algorithm is simple, robust, and has been used in numerous applications in the area of signal processing. To implement LMS in practice, the parameter of step-size that governs the convergence speed and steady-state excess mean-square error (EMSE) must be properly selected. It is shown that the rate of convergence is proportional to the step-size, while the steady-state EMSE is inversely proportional to the step-size [2,3]. When the fixed step-size LMS algorithm is used, the tradeoff between the rate of convergence and the steady-state EMSE must be considered. The problem of the fixed step-size LMS algorithm has motivated many researchers to approach to a time Corresponding author.
E-mail addresses: [email protected] (S. Zhao), [email protected] (Z. Man), [email protected] (S. Khoo), [email protected] (H.R. Wu). 0165-1684/$ - see front matter & 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.sigpro.2008.07.013
varying step-size. The main task of using a variable stepsize (VSS) approach is to speed up the rate of convergence using large step-size at the initial stages, and reduce the estimation error with small step-size in steady-state. Numerous VSS adjustment schemes have been proposed [6–19] to meet the conflicting requirements of fast convergence speed and low steady-state EMSE or misadjustment with increase in computational cost. All these schemes are shown to be based on a specific criterion. Different criteria usually lead to distinct performance improvements and limitations. For instance, the schemes based on the squared instantaneous error [8], on the timeaveraged estimate of the error autocorrelation at adjacent time [9], on the sign of the instantaneous error [12], and on the estimate of gradient power [7] have low-computational cost, but their steady-state performances are highly dependent on the measurement noise power level. As a result, these algorithms suffer from the applications where low signal-to-noise ratios (SNR) present. The schemes introduced in [16,17] were shown to be largely
ARTICLE IN PRESS 68
S. Zhao et al. / Signal Processing 89 (2009) 67–76
insensitive to the power of measurement noise, but the increases in computational cost are significant compared to the fixed step-size LMS algorithm for real-time applications. Recently, it was demonstrated in [20] that the VSS LMS algorithm developed by Kwong and Johnston in [8] is probably the best low-complexity VSS LMS algorithm available in the literature if well designed, except for its limited robustness to the measurement noise power. More recently, a modified version of the VSS algorithm called noise resilient variable step-size (NRVSS) LMS algorithm was presented in [18,19]. Compared with the VSS algorithm, the NRVSS algorithm was shown to achieve the same transient behavior and less sensitivity to the measurement noise for small increase in the computational cost. Since the optimal design of the NRVSS algorithm requires the reference signal power, it is specially indicated for the applications where estimation of the input signal power is possible. In this paper, we propose a more general VSS LMS algorithm where the VSS adjustment is based on the quotient of filtered quadratic error. We shall demonstrate that the newly proposed algorithm called more robust variable step-size (MRVSS) LMS algorithm can be designed to achieve lower steady-state EMSE or misadjustment, faster tracking property in nonstationary environment, and equal performance for the application of noise cancelation where the desired noiseless signal is correlated in time, when compared to NRVSS. Since the parameter setting of the MRVSS algorithm is free from estimation of the reference signal power, more applications should be found. This paper is organized as follows. In Section 2, the techniques of the VSS algorithm [8] and NRVSS algorithm [18,19] are introduced first, and then the proposed MRVSS algorithm is formulated. Next, we provide an analysis on the mean convergence of the MRVSS algorithm in Section 3, and discuss the steady-state performance of the MRVSS algorithm in Section 4. In Section 5, the mean stepsize behavior of the MRVSS algorithm is investigated. Section 6 discusses the design guidelines for new added constant parameters, and Section 7 presents extensive computer simulations. Finally, Section 8 concludes this paper.
2. Algorithm formulation Consider a general adaptive filtering problem of system identification, where xðnÞ is input signal, w0 is the vector consisting of the coefficients of the unknown system, wðnÞ is the vector consisting of the coefficients of the adaptive filter that models the unknown system, eo ðnÞ denotes the measurement noise, which is assumed to be zero mean and statistically independent of xðnÞ, and dðnÞ represents the desired response consisting of system output plus the measurement noise. A VSS LMS algorithm uses the following weight vector update equation: wðn þ 1Þ ¼ wðnÞ þ mðnÞeðnÞxðnÞ,
(1)
where mðnÞ is a scalar VSS, eðnÞ is the instantaneous estimation error given by eðnÞ ¼ dðnÞ wT ðnÞxðnÞ, the
superscript T denotes matrix transposition, and xðnÞ is the tap-delayed input vector. The VSS algorithm [8] uses the following VSS adjustment scheme: VSS:
mðn þ 1Þ ¼ aVSS mðnÞ þ gVSS e2 ðnÞ,
(2)
where aVSS and gVSS are the constant parameters, which are chosen as 0oaVSS o1 and gVSS 40. To ensure the stability and desired steady-state performance of the algorithm, the VSS is usually bounded as mmin pmðnÞp mmax . The extensive studies in [8,18–20] show that the VSS algorithm can achieve the mean step-size behavior that is the closest to the optimal step-size behavior for stationary environments. The main complaint about the VSS algorithm goes to its intrinsic large sensitivity to the measurement noise power. This high sensitivity could be explained by the analytic results of mean step-size and steady-state misadjustment derived in [8] where n is assumed to equal infinity, which are listed as follows: VSS:
E½mð1Þ ¼
M VSS
gVSS ðJmin þ Jex ð1ÞÞ , 1 aVSS
gVSS ð3 aVSS ÞðJmin þ Jex ð1ÞÞ trðRÞ, 1 a2VSS
(3)
(4)
where Jmin denotes the minimum MSE, which is equal to the power of measurement noise, Jex ð1Þ denotes the steady-state EMSE, which is given by J ex ð1Þ ¼ trfRKVSS ð1Þg with KVSS ðnÞ ¼ E½eðnÞeT ðnÞ, eðnÞ ¼ w0 wðnÞ the weight error vector, and trðRÞ the trace of the input correlation matrix R9E½xðnÞxT ðnÞ. We would like to write R as R ¼ Q KQ T with Q the unitary eigenvector matrix and K the diagonal eigenvalue matrix, which will be used in our latter analysis. It is seen from (3) and (4) that the mean step-size and steady-state misadjustment of VSS algorithm are proportional to the power of measurement noise. It indicates that the VSS algorithm will produce large steady-state EMSE or misadjustment in the applications where low SNR presents. When the power of measurement noise increases, the mean stepsize of the VSS algorithm also increases, which may diverge the algorithm for a large increase in the noise power. The work presented in [18,19] addressed this problem with the following step-size adjustment scheme: NRVSS: mðn þ 1Þ ¼ aNRVSS mðnÞ þ gNRVSS ½kxT ðnÞxðnÞ 1e2 ðnÞ, (5) where the constant parameter settings, aNRVSS ¼ aVSS and gNRVSS ¼ NgVSS =2, are provided in [18,19], and the newly added parameter k is optimally designed as k ¼ 1=ðr x NÞ with r x the input power and N the filter length. The absolute value of mðnÞ in (5) is used for the weight vector update. It was shown in [18,19] that the NRVSS algorithm can achieve the same initial transient behavior, whereas less sensitivity to the noise power when compared to VSS algorithm. In this paper, we would like to further address the problem of the VSS algorithm by proposing a new approach to adjust the VSS based on a quotient of filtered
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
quadratic error. The proposed scheme is listed as follows:
mðn þ 1Þ ¼ amðnÞ þ gyðnÞ,
MRVSS: Pn
yðnÞ ¼ Pni¼0
ai e2 ðn iÞ
j 2 j¼0 b e ðn
jÞ
,
(6)
(7)
where a and b are decaying factors used for the exponential windows in numerator and denominator of (7) and bounded as 0oaobo1. The constant parameter settings, a ¼ aVSS and g ¼ gVSS =N, should be recommended such that the MRVSS algorithm achieves the almost optimal mean step-size behavior as the VSS algorithm for stationary environments according to the design guidelines of [20]. For nonstationary environments, we shall demonstrate that the parameter g in (6) could be optimally designed and small a should be chosen for good tracking performance. The use of a quotient form in (6) and (7) serves as two objectives. First, the quotient is expected for a smoothing decrease of the step-size, where the transient behavior of the proposed VSS in stationary environment may be described by a reformulation of (7) as follows: AðnÞ aAðn 1Þ þ e2 ðnÞ ¼ yðnÞ ¼ BðnÞ bBðn 1Þ þ e2 ðnÞ a e2 ðnÞ . yðn 1Þ þ b bBðn 1Þ
(8)
Note that in derivation of (8), we have neglected the value of e2 ðnÞ in the denominator, since compared to e2 ðnÞ the error cumulant bBðn 1Þ becomes much larger during adaptation because the decaying factor b is very close to one, which will be discussed in Section 6. Assuming that the initial step-size mð0Þ is set to be mmax for initial fast transient behavior, from (8) we may observe that since the ratio of e2 ðnÞ and bBðn 1Þ decreases following the decrease of the power of output error, the VSS mðnÞ should also decrease from its maximum value following the decrease of yðnÞ. Second, as shown in the analysis given in the latter sections of the paper, in steady-state, the EMSE of the algorithm should be much smaller compared to the power of measurement noise. This implies that the measurement noise dominates the numerator and denominator of (7). In statistic sense, the power of measurement noise in the numerator and denominator could be canceled out, leaving the steady-state mean stepsize determined only by the constant parameters. Therefore, the decaying factors a and b could be designed beforehand for a desired steady-state mean step-size level. Comparing the computational complexities of the NRVSS and MRVSS algorithms, we find that the MRVSS algorithm requires the same number of multiplications/ divisions and additions per iteration by using the recursive forms of AðnÞ and BðnÞ in (8). Note that there is a division computation in the MRVSS algorithm, which usually requires some more processing time compared to additions or multiplications. Both of the algorithms are considered to be low complexity and real-time algorithms. Comparing the parameter design simplicity of NRVSS and MRVSS added on the VSS algorithm, the NRVSS algorithm
69
adds one new design parameter k, which could be optimally designed with the reference signal power known, and the MRVSS algorithm adds two new decaying factors a and b, which could provide a flexible design as discussed in latter sections and demonstrated in simulations. In the following sections, we shall study the convergence and performance of the MRVSS algorithm in both stationary and nonstationary environments, and the design guidelines of the decaying factors a and b. 3. Convergence in mean sense For the problem of system identification being considered in this paper, we assume that the desired response of the system arises from the model dðnÞ ¼ wT0 xðnÞ þ eo ðnÞ.
(9)
We would like to consider an unknown plant whose weight vector follows a random walk process [2]: w0 ðn þ 1Þ ¼ w0 ðnÞ þ rðnÞ,
(10)
where rðnÞ represents a driving vector of zero-mean white sequence with a diagonal autocorrelation matrix of s2r I, and I is an identity matrix. It is known that for s2r ¼ 0, the unknown plant is considered as time invariant and the adaptive filters do not need to perform a tracking task. In this paper, we consider a general case of s2r . Using (1) and (10), the transient behavior of the weight vector error, eðnÞ, can be modeled as
eðn þ 1Þ ¼ eðnÞ mðnÞeðnÞxðnÞ þ rðnÞ.
(11)
Consider that the output error can be reformed as eðnÞ ¼ eo ðnÞ þ xT ðnÞeðnÞ.
(12)
Substituting (12) into (11) and applying the unitary transformation Q T on both sides of (11) yields T
^ x^ ðnÞvðnÞ mðnÞeo ðnÞxðnÞ ^ þ r^ ðnÞ, vðn þ 1Þ ¼ ½I mðnÞxðnÞ (13) ^ where r^ ðnÞ; xðnÞ, and vðnÞ are the transformed versions of rðnÞ; xðnÞ, and eðnÞ by Q T, respectively. To proceed further, we introduce the independence assumption. That is, the input data vectors xð1Þ; xð2Þ; . . . ; xðnÞ are mutually independent, and xðnÞ is statistically independent of the previous samples of the system response dðnÞ. We also require the following averaging principle commonly used in [4,5,8,9]: T
T
^ x^ ðnÞ E½mðnÞE½xðnÞ ^ x^ ðnÞ. E½mðnÞxðnÞ
(14)
This is approximately true since the step-size mðnÞ varies slowly around its mean value when compared to eðnÞ and xðnÞ for small g. Therefore, this justifies the independence assumption of mðnÞ and m2 ðnÞ with eðnÞ; xðnÞ and wðnÞ. Based on above assumptions, Eq. (13) is stable in the mean sense if and only if the eigenvalues of the matrix
ARTICLE IN PRESS 70
S. Zhao et al. / Signal Processing 89 (2009) 67–76 T
^ x^ ðnÞ are all within the unit circle. DðnÞ9E½I mðnÞxðnÞ This may be solved as follows:
In the steady-state, we can assume that limn!1 cðn þ 1Þ ¼ limn!1 cðnÞ. From (19), we then obtain
det½DðnÞ ZI ¼ detfI E½mðnÞK ZIg
cð1Þ ¼ ½I Lð1Þ1 fE½m2 ð1ÞJ min k þ s2r 1g,
¼
N Y
ð1 Z E½mðnÞli Þ ¼ 0,
(15)
i¼1
where Z denotes the eigenvalues of DðnÞ; li is the ith eigenvalue of the input correlation matrix R. To ensure all the solutions of (15) within the unit circle, we only need to find the conditions to make sure both the smallest and the largest solutions to be within the unit circle. That is 1o1 E½mðnÞlmax o1 E½mðnÞlmin o1,
where n is assumed to equal infinity. Using (22) into the steady-state excess MSE given by Jex ð1Þ ¼ tr½RKð1Þ ¼ kT cð1Þ, we obtain J ex ð1Þ ¼ kT ½I Lð1Þ1 fE½m2 ð1ÞJmin k þ s2r 1g.
0oE½mðnÞo
lmax
.
J ex ð1Þ ¼ kT fq kE½m2 ð1ÞkT g1 fE½m2 ð1ÞJmin k þ s2r 1g ¼ kT fq1 þ q1 kðE½m2 ð1Þ1 kT q1 kÞ1 kT q1 g
(16)
(17)
The condition in (17) guarantees the stability of the MRVSS algorithm in the mean sense for nonstationary environments. For the special case of s2r ¼ 0, the algorithm is guaranteed to be convergent in the mean sense. Alternative derivations of condition (17) could be found in [8,9]. For the case of a constant step-size mðnÞ, condition (17) is the same as the condition for mean convergence of the fixed step-size LMS algorithm as shown in [4,5]. 4. Analysis of steady-state performance In this section, we shall study the steady-state performance of MRVSS algorithm in terms of the misadjustment. To proceed further, besides the independent assumption we need to further assume that the input signal xðnÞ and the desired response dðnÞ are jointly Gaussian. Postmultiplying both sides of (13) by vT ðn þ 1Þ and then taking the expectations yields
fE½m2 ð1ÞJmin k þ s2r 1g
2
2
þ 2E½m ðnÞKCðnÞK þ E½m ðnÞK trðKCðnÞÞ þ E½m2 ðnÞJmin K þ s2r I,
¼ f1 E½m2 ð1ÞkT q1 kg1 fE½m2 ð1ÞJmin kT q1 k þ s2r kT q1 1g.
(18)
where CðnÞ ¼ E½vðnÞvT ðnÞ denotes the transformed weight error covariance of eðnÞ. Note that in the derivation and simplification of (18), the Gaussian moment factoring theorem [2] was used. Eq. (18) gives the transient behavior of the transformed weight error covariance. Now, let cðnÞ be the vector with the entries of the diagonal elements of CðnÞ, let k be the column vector of eigenvalues of R, and let 1 be a column vector of 1’s with the same length as k. Following the analyzed results in [5,8], from (18) we have cðn þ 1Þ ¼ LðnÞcðnÞ þ E½m2 ðnÞJ min k þ s2r 1,
(19)
LðnÞ ¼ diag½r1 ; r2 ; . . . ; rN þ E½m2 ðnÞkkT ,
(20)
ri ¼ 1 2E½mðnÞli þ 2E½m2 ðnÞl2i .
(21)
(24)
It is easy to obtain the following equalities:
kT q1 k ¼
kT q1 1 ¼
N X
li
i¼1
2E½mð1Þ 2E½m2 ð1Þli
N X i¼1
,
(25)
1 . 2E½mð1Þ 2E½m2 ð1Þli
(26)
Substituting the equalities of (25) and (26) into (24) yields 9 8 P > > E½m2 ð1Þli > > > > J min N = < i¼1 2E½mð1Þ 2E½m2 ð1Þli J ex ð1Þ ¼ 2 > > PN E½m ð1Þli > > > > ; :1 i¼1 2E½mð1Þ 2E½m2 ð1Þli 9 8 2 PN > > sr > > > > = < i¼1 2E½mð1Þ 2E½m2 ð1Þli . (27) þ 2 > > P E½m ð1Þli > > N > > ; :1 i¼1 2E½mð1Þ 2E½m2 ð1Þli To simplify the result of (27), we neglected the terms involved with E½m2 ð1Þ that are very smaller compared to other terms in the event of small values of misadjustment. After the proceedings, we have J ex ð1Þ J min trðRÞ
Cðn þ 1Þ ¼ CðnÞ E½mðnÞCðnÞK E½mðnÞKCðnÞ
(23)
Let q ¼ diag½1 r1 ; 1 r2 ; . . . ; 1 rN be a diagonal matrix with ri given in (21). We proceed (23) as follows:
where lmax and lmin are the largest and smallest eigenvalues of R, respectively. From (16) we can easily find the following bound on the mean step-size: 2
(22)
E½m2 ð1Þ Ns2r þ . 2E½mð1Þ 2E½mð1Þ
(28)
Next, we shall find the forms for E½mð1Þ and E½m2 ð1Þ, respectively. Taking expectations on both sides of (7) and assuming that n approaches to infinity yields lim E½mðnÞ ¼ a lim E½mðn 1Þ þ g lim E½yðnÞ.
n!1
n!1
n!1
(29)
From (29), we can approximate the steady-state mean step-size as follows: E½mð1Þ
g limn!1 E½yðnÞ . 1a
(30)
The term limn!1 E½yðnÞ in (30) is given by AðnÞ . lim E½yðnÞ ¼ lim E n!1 n!1 BðnÞ
(31)
Note that AðnÞ and BðnÞ are with recursive forms as shown in (8). Assuming that n approaches to infinity and
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
limn!1 E½e2 ðnÞ is slowly varying, we have the following approximations: lim E½AðnÞ
1 lim E½e2 ðnÞ, 1 a n!1
(32)
lim E½BðnÞ
1 lim E½e2 ðnÞ. 1 b n!1
(33)
n!1
n!1
AðnÞ E½AðnÞ lim . lim E n!1 n!1 E½BðnÞ BðnÞ
(34)
This cannot really hold for the algorithm, but approximately true since that limn!1 AðnÞ and limn!1 BðnÞ will vary slowly. Using the results of (32) and (33), we can approximate (31) as
n!1
(35)
gð1 bÞ . ð1 aÞð1 aÞ
(36)
Here, we would like to drop a point that comparing to the steady-state mean VSS step-size shown in (3), Eq. (36) shows that the steady-state mean MRVSS step-size is now free from the steady-state MSE, i.e., J min þ J ex ð1Þ. Since the assumption in (34) is approximately true, we may conclude that the MRVSS algorithm should be much less sensitive to the measurement noise power when compared to the VSS algorithm. This will be demonstrated in the simulation section. To proceed the steady-state mean-square MRVSS stepsize, taking the expectations of the squared (6) and assuming that n approaches to infinity yields n!1
2
þ g2 E½y ðnÞg.
(37)
For small g, the last term in (37) involving g2 is negligible compared with the other terms. So after eliminating this small term, we can approximate as lim E½m2 ðnÞ
n!1
2agE½mð1ÞE½yð1Þ . 1 a2
(38)
Substituting the results of (35) and (36) into (38) yields E½m2 ð1Þ
2ag2 ð1 bÞ2 ð1 a2 Þð1 aÞð1 aÞ2
.
(39)
Eq. (39) gives the steady-state mean-square MRVSS stepsize. Substituting the result of mð1Þ (36) and m2 ð1Þ (39) into (28) and using the definition of M ¼ J ex ð1Þ=J min yields the misadjustment of the MRVSS algorithm:
agð1 bÞ ð1 aÞð1 aÞN s2r trðRÞ þ . M 2gð1 bÞJ min ð1 a2 Þð1 aÞ
agð1 bÞ trðRÞ. ð1 a2 Þð1 aÞ
(43)
We can observe from (43) that the steady-state misadjustment of the MRVSS algorithm is proportional to the ratio of ð1 bÞ=ð1 aÞ, which is consistent with the result of the steady-state mean MRVSS step-size shown in (36). For the case that a is very close to one, we can approximate (43) as M 12 E½mð1Þ trðRÞ.
(44)
The result of (44) is then consistent with the result for the fixed LMS algorithm as shown in [1–4]. 5. Mean behavior of the proposed VSS In this section, we investigate the mean behavior of the MRVSS step-size in a nonstationary environment where the plant varies following the model of (10). Taking the expectations of (6) we obtain E½mðn þ 1Þ ¼ aE½mðnÞ þ gE½yðnÞ.
lim E½m2 ðnÞ ¼ lim fa2 E½m2 ðn 1Þ þ 2agE½mðnÞE½yðnÞ
n!1
We can observe from (42) that smaller a should provide better tracking performance. We will demonstrate this point in the simulation section. In stationary environments, i.e., s2r ¼ 0, the misadjustment is then given by M
1b . 1a
Substituting the result of (35) into (30) yields E½mð1Þ
optimize the choice for g given a; a and b to achieve the minimum misadjustment. When both terms are equal, the optimal choice of g is given by sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð1 aÞð1 a2 Þð1 aÞ2 Ns2r g ¼ . (41) 2að1 bÞ2 J min trðRÞ And the result of minimum misadjustment follows: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2aN s2r trðRÞ . (42) M min ¼ ð1 þ aÞJmin
To proceed (31), we assume that
lim E½yðnÞ ¼ E½yð1Þ
71
(45)
Next, we need to evaluate the mean value of yðnÞ, which is given by AðnÞ . (46) E½yðnÞ ¼ E BðnÞ To proceed further, we make the following assumption: AðnÞ E½AðnÞ . (47) E BðnÞ E½BðnÞ This is approximately true for the decaying factors of a and b close to one. Using (47) in (46), we then only need to evaluate E½AðnÞ and E½BðnÞ, which are given by E½AðnÞ ¼ aE½Aðn 1Þ þ E½e2 ðnÞ,
(48)
E½BðnÞ ¼ bE½Bðn 1Þ þ E½e2 ðnÞ.
(49)
Using the form of the output error eðnÞ ¼ eo ðnÞ þ x^ T ðnÞvðnÞ, we proceed E½e2 ðnÞ as E½e2 ðnÞ ¼ E½ðeo ðnÞ þ x^ T ðnÞvðnÞÞ2
(40)
Since the first term in (40) is directly proportional to g and the second term is inversely proportional to g, we can
^ ¼ E½e2o ðnÞ þ 2E½eo ðnÞx^ T ðnÞvðnÞvT ðnÞxðnÞ ^ þ E½x^ T ðnÞvðnÞvT ðnÞxðnÞ ðJ min þ kT cðnÞÞ,
(50)
ARTICLE IN PRESS 72
S. Zhao et al. / Signal Processing 89 (2009) 67–76
where cðnÞ is given in (19) and k is the column vector of the eigenvalues of R. Note that during the derivation of (50) we have used the independent assumption and the assumptions that the input signal xðnÞ and desired response dðnÞ are jointly Gaussian and the measurement noise eo ðnÞ is zero-mean white Gaussian and statistically independent of xðnÞ, which justifies the Gaussian moment factoring theorem and the simplification. Using (46)–(50), we have E½yðnÞ
aE½Aðn 1Þ þ ðJ min þ kT cðnÞÞ bE½Bðn 1Þ þ ðJ min þ kT cðnÞÞ
.
(51)
Since cðnÞ requires the value of E½m2 ðn 1Þ, we need to establish the transient behavior for E½m2 ðnÞ. Taking the expectations of the squared (6) we obtain E½m2 ðnÞ a2 E½m2 ðn 1Þ þ 2agE½mðn 1ÞE½yðn 1Þ 2
þ g2 E½y ðn 1Þ.
(52)
Note that we have used E½mðnÞyðnÞ E½mðnÞE½yðnÞ based on the independence assumption. Neglecting the very small term involving with g2 , we can approximate (52) as follows: E½m2 ðnÞ a2 E½m2 ðn 1Þ þ 2agE½mðn 1ÞE½yðn 1Þ.
(53)
Now the mean behavior of the MRVSS step-size is completed described by (45), (48), (49), (51), (53) and (19)–(21). 6. Design of constant parameters In this section, we derive some design guidelines for the constant parameters of the new VSS scheme. We recommend that the choices of a and g are a ¼ aVSS and g ¼ gVSS =N to obtain the optimal transient behavior, where aVSS and gVSS should be designed follows the guidelines in [20]. To show the effectiveness of the choices of the parameters a and b on the steady-state performance of MRVSS algorithm, we make a ratio of the misadjustments of MRVSS algorithm and VSS algorithm as follows: M agð1 bÞ trðRÞ ¼ M VSS ð1 a2 Þð1 aÞ 1 a2VSS . gVSS ð3 aVSS ÞðJmin þ Jex ð1ÞÞ trðRÞ
MRVSS algorithm is more significant. The choices of a and b do not affect the transient behavior of the mean MRVSS step-size much, but do affect the steady-state mean MRVSS step-size, which we will demonstrate in the simulation section. For nonstationary environments, a should be chosen small for good tracking performance according to (42) and g are given from (41). However, in the simulation we found g =3 rather gives a better tracking performance, this may be due to the assumptions and approximations we have made in the derivation of g . The decaying factors a and b are chosen similarly as made for stationary case.
7. Simulation results This section presents four examples to corroborate the analytic results derived in this paper and to illustrate the properties of the MRVSS algorithm. In the examples, the performances of VSS algorithm [8], NRVSS algorithm [18,19] and MRVSS algorithm are compared to demonstrate that MRVSS can lead to a better steady-state performance for the same initial transient behavior in stationary environments, and in nonstationary environments we demonstrate in Example 3 that MRVSS can provide a faster tracking property. The mean MRVSS stepsize behavior via the choices of a and b, and via the measurement noise power are demonstrated in Example 1. The sensitivities of VSS, NRVSS and MRVSS to noise power are compared in Example 2. Examples 1–3 are based on the problem of system identification. Example 4 presents a comparison between NRVSS and MRVSS algorithms when applied to noise cancelation. To begin with, let us describe the experimental setup and parameter settings. Unless stated otherwise, in all examples the input signal xðnÞ is with unit power and the additive measurement noise eo ðnÞ is zero-mean, white Gaussian and statistically independent of xðnÞ. The unknown plant and adaptive filters have the same number of 10 taps (N ¼ 10), where the plant impulse response is designed using a Hanning window with unit norm (wT0 ð0Þw0 ð0Þ ¼ 1). The parameters are set according to
(54)
[19], where aVSS ¼ 0:9997; gVSS ¼ 2 105 ; aNRVSS ¼ aVSS ; gNRVSS ¼ NgVSS =2; a ¼ aVSS ; g ¼ gVSS =N; a ¼ 0:9; b ¼
For the settings a ¼ aVSS and g ¼ gVSS =N, Eq. (54) can be simplified as
1 105 . For all the algorithms, we used mð0Þ ¼ mmax ¼ 0:01 and mmin ¼ 0. It can easily be checked that the mean stability condition (17) is satisfied with such parameter
M aVSS ð1 bÞ . ¼ M VSS Nð3 aVSS Þð1 aÞðJ min þ Jex ð1ÞÞ
(55)
Considering that aVSS is close to one and Jex ð1Þ5J min , we can approximate (55) as M ð1 bÞ . M VSS 2NJmin ð1 aÞ
(56)
From (56), we can see that the MRVSS algorithm can be designed to achieve smaller steady-state misadjustment by choosing ð1 bÞ5ð1 aÞ for any level of measurement noise power and filter length N. It is not hard to observe that for larger measurement noise power (larger Jmin ), the improvement of the steady-state misadjustment with
Table 1 Steady-state misadjustments for VSS, NRVSS and MRVSS algorithms under different SNR conditions and comparisons between theoretical models and Monte Carlo simulations for MRVSS algorithm (all results are in dB) SNR
20 10 0 10
Analytical values
Simulated values
MRVSS (Eq. (43))
VSS
NRVSS
MRVSS
54.78 54.78 54.78 54.78
24.56 14.45 12.64 12.64
32.82 23.77 15.27 12.72
53.01 52.23 52.20 52.18
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
setting. The results were obtained by averaging 500 ensemble trials in Monte Carlo simulations. The SNR is calculated by SNR ¼ 10 log10 E½x2 ðnÞ=E½e2o ðnÞ.
0 −5
EMSE (dB)
−10 −15 −20
Example 1. The plant impulse response is time invariant (w0 ðnÞ ¼ w0 ðn 1Þ) and the input is white Gaussian. We used SNR ¼ 60, 20, 10, 0 and 10 dB. Table 1 presents the steady-state misadjustments for VSS, NRVSS and MRVSS for SNR ranging from 20 to 10 dB. The theoretical values are also obtained based on Eq. (43), which are compared with the simulation results. The main causes of mismatch between the analytical and simulation results seem to be the following: (a) the assumption of (34); (b) the neglected terms involving g2 or E½m2 ð1Þ in (38) and (28), respectively; and (c) the independent assumption. Figs. 1 and 2 show the EMSE learning curves and mean step-size behaviors, respectively, of VSS, NRVSS and MRVSS algorithms up to the iteration 40,000 at SNR ¼ 10 dB. All algorithms present approximately the
(a)
−25 −30
(b)
−35
(c)
−40 −45
0
0.5
1
1.5
2 2.5 iterations
3
3.5
4
x 104
Fig. 1. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 1: the white Gaussian input and stationary plant with SNR ¼ 10 dB.
10−2
(a)
10−3
10−2 10−3 Mean step size
Mean step size
(b)
10−4
10−5
(c)
0
0.5
1
1.5
2 2.5 iterations
10−4 10−5 (a)=(b)=(c)=(d) 10−6
(d) 10−6
73
3
3.5
4
10−7
x 104
Fig. 2. Comparisons of mean behavior of the step-size (mðnÞ) between (a) VSS, (b) NRVSS, (c) MRVSS simulations, and (d) MRVSS model for Example 1: the white Gaussian input and stationary plant with SNR ¼ 10 dB.
0
0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 iterations x 105
2
Fig. 4. Comparisons of mean behavior of the MRVSS step-size (mðnÞ) for Example 1: the white Gaussian input and stationary plant with SNR: (a) 10 dB, (b) 0 dB, (c) 10 dB and (d) 20 dB.
10−2
Mean step size
10−3 10−4 10−5 10
(e)
−6
(c) (b)
10−7
(a)
10−8 10−9
(d) 0
1
2
3
4 iterations
5
6
7
8 x 104
Fig. 3. Comparisons of mean behavior of the MRVSS step-size (mðnÞ) for Example 1: the white Gaussian input and stationary plant with (a) a ¼ 0:9, b ¼ 1 105 , (b) a ¼ 0:99, b ¼ 1 105 , (c) a ¼ 0:999, b ¼ 1 105 , (d) a ¼ 0:9, b ¼ 1 106 and (e) a ¼ 0:9, b ¼ 1 104 .
ARTICLE IN PRESS 74
S. Zhao et al. / Signal Processing 89 (2009) 67–76
same initial transient behavior but MRVSS achieves lower steady-state EMSE. The predicted and simulation mean step-size behaviors agree well with each other. Fig. 3 shows the evolution of the mean MRVSS step-size for different settings of the decaying factors a and b at SNR ¼ 60 dB. It can be seen that the transient behavior of MRVSS algorithm is not sensitive to the choices of a and b, but the steady-state EMSE or misadjustment is determined by a and b. The smaller a or larger b gives the lower steady-state EMSE. However, if small b or large a is used, poor steady-state EMSE may be produced, which is consistent with our analytic result. Fig. 4 shows the evolution of the MRVSS step-size for SNR ¼ 20; 10; 0 and 10 dB. It can be seen that the mean MRVSS step-size has low sensitivity to the SNR changes, so does the steady-state misadjustment of MRVSS algorithm as demonstrated in Example 2.
Example 2. The plant impulse response is time invariant, and both white Gaussian input and correlated first-order Markov input ðxðnÞ ¼ a1 xðn 1Þ þ uðnÞÞ with a1 ¼ 0:9 were tested. The measurement noise has an abrupt power variation at iteration 50,000 such that SNR changes from 60 to 20 dB. Figs. 5 and 7 show the EMSE behaviors of VSS, NRVSS and MRVSS algorithms, and Figs. 6 and 8 show the evolution of the mean step-sizes of VSS, NRVSS and MRVSS algorithms. It can be seen that all algorithms perform equally under high SNR conditions, whereas under low SNR conditions, the MRVSS algorithm shows a better performance in steady-state conditions (Figs. 5–8). Example 3. The plant impulse response is time variant modeled as a first-order random walk process as in Eq. (10) with s2r ¼ 1010 , and the input is a correlated first-order Markov process (xðnÞ ¼ a1 xðn 1Þ þ uðnÞ) with a1 ¼ 0:7 [12]. We tested at SNR ¼ 20 dB. The parameters
0 0
−10
−10
−20 EMSE (dB)
−20 EMSE (dB)
−30 −40
(a)
−50
(b)
−60
(c)
(a)=(b)=(c)
−90
−50
0
1
2
3
4 5 6 iterations
−80 7
8
(b) (c)
(a)=(b)=(c) 0
1
2
3
9 10 x 104
Fig. 5. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the white Gaussian input and stationary plant with SNR change from 60 to 20 dB.
4
5 6 iterations
7
8
9 x
10 104
Fig. 7. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the correlated first-order Markov input and stationary plant with SNR change from 60 to 20 dB.
10−2
10−2 10−3
(a)
10−4
(b)
10−5
10−3 Mean step size
Mean step size
(a)
−70
−80
(c)
10−6 (a) (c)
10−7 10−8 10−9
−40
−60
−70
−100
−30
1
2
3
4
5 6 iterations
(b)
10−4 10−5
(c)
10−6 (a)
10−7
(b) 0
(a)
10−8 7
8
9
10
x 104
Fig. 6. Comparisons of mean behavior of the step-size (mðnÞ) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the white Gaussian input and stationary plant with SNR change from 60 to 20 dB.
(c) (b) 0
1
2
3
4
5 6 iterations
7
8
9
10
x 104
Fig. 8. Comparisons of mean behavior of the step-size (mðnÞ) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 2: the correlated first-order Markov input and stationary plant with SNR change from 60 to 20 dB.
ARTICLE IN PRESS S. Zhao et al. / Signal Processing 89 (2009) 67–76
75
0 2
EMSE (dB)
−5 −10
0
−15
−2
−25
2
−30
0
−35
−2
−40
(a)
(b)
−45 −50
5000
10000
15000
0
5000
10000
15000
0
5000
10000
15000
2
(c) 0
0.5
1
1.5 iterations
2
2.5
0
3
x 104
−2
Fig. 9. Comparisons of excess mean-square error (EMSE) between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 3: the correlated first-order Markov input and nonstationary plant with SNR ¼ 20 dB.
0.01
iterations Fig. 11. Comparisons between NRVSS and MRVSS algorithms applied to noise cancelation application. (a) The desired square wave plus a pure sinusoidal noise; (b) NRVSS processed square wave; and (c) MRVSS processed square wave.
0.009
0.01 Step−size instantaneous value
0.008 Mean step size
0
−20
0.007 0.006 0.005 0.004 0.003 0.002 (b)
0.001 0
(a)
(c) 0
0.5
1
1.5 iterations
2
2.5
3
x 104
Fig. 10. Comparisons of mean behavior of the step-size ðmðnÞÞ between (a) VSS, (b) NRVSS and (c) MRVSS simulations for Example 3: the correlated first-order Markov input and nonstationary plant with SNR ¼ 20 dB.
of MRVSS algorithm were reset as a ¼ 0:5 and g ¼ 0:2 according to Eqs. (42) and (41). All the other parameters were unchanged. Figs. 9 and 10 show the tracking behaviors and the evolution of mean step-sizes of VSS, NRVSS and MRVSS algorithms. It can be seen that the MRVSS algorithm achieves a faster tracking behavior and a slightly lower misadjustment. We need to point out that the accumulative nature of MRVSS may lower its recovery ability for abrupt changes of the plant impulse response when small g is used. This is because the MRVSS has the memory of past power of output errors. In such cases, the suggestion in [12] could be used. That is, the stepsizeshould switch automatically to mmax at the abrupt change, where the system is monitored and the abrupt changes are detected employing the power of output error with suitable threshold (chosen according to the application at hand). The power of output error is recursively updated as EðnÞ ¼ bEðn 1Þ þ ð1 bÞe2 ðnÞ. The additional
0.009 0.008 0.007 0.006 0.005 0.004 0.003
(a)
0.002
(b)
0.001 0
0
5000
10000
15000
iterations Fig. 12. Instantaneous value of NRVSS (a) and MRVSS (b) step-sizes for Example 4: the application of noise cancelation.
cost is two extra multiplications and one extra addition. For MRVSS algorithm, the values of AðnÞ and BðnÞ are reinitialized consequently. Although such scheme is applicable in the applications, NRVSS and VSS are more recommended in such situations for their better recovery abilities. Example 4. In this example, we study the performance of MRVSS algorithm for the application of noise cancelation where the output error eðnÞ is expected to converge to the desired noiseless signal, which is correlated in time. The task in this example is to remove an additive pure sinusoidal noise generated by ðuðnÞ ¼ 0:8 sinðon þ 0:5pÞ with o ¼ 0:1p from a square wave generated by ðsðnÞ ¼ 2 ððmodðn; 1000Þo1000=2Þ 0:5ÞÞ where modðn; 1000Þ computes the modulus of n over 1000. The summation of sðnÞ and uðnÞ severs as the reference response of the adaptive filters. The input to the adaptive filters is also a
ARTICLE IN PRESS 76
S. Zhao et al. / Signal Processing 89 (2009) 67–76
pffiffiffi sinusoidal signal (xðnÞ ¼ 2 sinðonÞ) with o ¼ 0:1p, which is in unit power. The SNR is therefore 0 dB. The parameter settings were the same as used in Examples 1 and 2. The length of the adaptive filters is 10. Since the VSS algorithm diverged in the simulation, only the performances of NRVSS algorithm and MRVSS algorithm are compared. Fig. 11 shows the reference response and the processed results by the NRVSS algorithm and MRVSS algorithm. It can be seen that both algorithms are capable of suppressing the interference. Fig. 12 shows the instantaneous stepsize behavior of both algorithms. The step-size values in steady-state are approximately equal while MRVSS achieves slightly faster initial adaptation velocity. 8. Conclusion This work presented a quotient form for the VSS adjustment of LMS algorithm. The resulting MRVSS algorithm was demonstrated to be less sensitive to the power of the measurement noise when compared to VSS and NRVSS. The additive computational cost is negligible and real-time implementation of MRVSS is realizable. The analytical results in the paper can be served as design guidelines for the setting of two new added decaying factors. Monte Carlo simulations corroborated the analytical results and illustrated superior properties of MRVSS. References [1] B. Widrow, S.D. Steam, Adaptive Signal Processing, Prentice-Hall, Englewood Cliffs, NJ, 1985. [2] S. Haykin, Adaptive Filter Theory, fourth ed., Prentice-Hall, Englewood Cliffs, NJ, 2002. [3] A.H. Sayed, Fundamentals of Adaptive Filtering, Wiley, New York, 2003. [4] B. Widrow, J.M. McCool, Stationary and nonstationary learning characteristics of the LMS filter, Proc. IEEE 64 (8) (August 1976) 1151–1162.
[5] A. Feuer, E. Weinstein, Convergence analysis of LMS filters with uncorrelated Gaussian data, IEEE Trans. Acoust. Speech Signal Process. ASSP-33 (1) (February 1985) 222–230. [6] R. Harris, D. Chabries, F.A. Bishop, A variable step (VS) adaptive filter algorithm, IEEE Trans. Acoust. Speech Signal Process. ASSP-34 (2) (April 1986) 309–316. [7] V.J. Mathews, Z. Xie, A stochastic gradient adaptive filter with gradient adaptive step size, IEEE Trans. Signal Process. 41 (6) (June 1993) 2075–2087. [8] R. Kwong, E.W. Johnston, A variable step size LMS algorithm, IEEE Trans. Signal Process. 40 (7) (July 1992) 1633–1642. [9] T. Aboulnasr, K. Mayyas, A robust variable step-size lms-type algorithm: analysis and simulations, IEEE Trans. Signal Process. 45 (3) (March 1997) 631–639. [10] D.I. Pazaitis, A.G. Constantinides, A novel kurtosis driven variable step-size adaptive algorithm, IEEE Trans. Signal Process. 47 (3) (March 1999) 864–872. [11] W.P. Ang, B. Farhang-Boroujeny, A new class of gradient adaptive step-size LMS algorithms, IEEE Trans. Signal Process. 49 (4) (April 2001) 805–810. [12] T.I. Haweel, A simple variable step size LMS adaptive algorithm, Internat. J. Circ. Theor. Appl. 32 (November 2004) 523–536. [13] A.I. Sulyman, A. Zerguine, Convergence and steady-state analysis of a variable step-size NLMS algorithm, Signal Processing 83 (June 2003) 1255–1273. [14] A. Mader, H. Puder, G.U. Schmidt, Step-size control for acoustic echo cancellation filters—an overview, Signal Processing 80 (September 2000) 1697–1719. [15] H.C. Shin, A.H. Sayed, W.J. Song, Variable step-size NLMS and affine projection algorithms, IEEE Signal Process. Lett. 11 (2) (February 2004) 132–135. [16] S. Koike, A class of adaptive step-size control algorithms for adaptive filters, IEEE Trans. Signal Process. 50 (6) (June 2002) 1315–1326. [17] J. Okello, et al., A new modified variable step-size for the LMS algorithm, in: Proceedings of the International Symposium on Circuits and Systems, 1998, pp. 170–173. [18] M.H. Costa, J.C.M. Bermudez, A robust variable step-size algorithm for LMS adaptive filters, in: Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2006, pp. 93–96. [19] M.H. Costa, J.C.M. Bermudez, A noise resilient variable step-size LMS algorithm, Signal Processing 88 (March 2008) 733–748. [20] C.G. Lopes, J.C.M. Bermudez, Evaluation and design of variable stepsize adaptive algorithms, in: Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2001, pp. 3845–3848.
