Performance Comparison of Two Implementations of the Leaky LMS Adaptive Filter Scott C. Douglas Department of Electrical Engineering University of Utah Salt Lake City, Utah 84112
Abstract{ The leaky LMS adaptive lter can be implemented either directly or by adding
random white noise to the input signal of the LMS adaptive lter. In this paper, we analyze and compare the mean-square performances of these two adaptive lter implementations for system identi cation tasks with zero mean i.i.d. input signals. Our results indicate that the performance of the direct implementation is superior to that of the random noise implementation in all respects. However, for small leakage factors, these performance di erences are negligible. Simulations verify the results of the analysis and the conclusions drawn.
submitted to IEEE TRANSACTIONS ON SIGNAL PROCESSING
EDICS Category No. SP 2.6.4
Please address correspondence to: Scott C. Douglas, Department of Electrical Engineering, University of Utah, Salt Lake City, UT 84112. (801) 581-4445. FAX: (801) 581-5281. Electronic mail address:
[email protected]. World Wide Web: http://www.elen.utah.edu/douglas. 0 Permission of the IEEE to publish this abstract separately is granted. 0 This research was supported in part by NSF Grant No. MIP-9501680.
1 Introduction The leaky least-mean-square (LMS) adaptive lter is a useful variant of the LMS adaptive lter for several communications and signal processing tasks [1, 2, 3, 4, 5]. The leaky LMS coecient update is given by
Wk+1 = (1 ? )Wk + ek Xk ek = dk ? WkT Xk ;
(1) (2)
where Wk = [w0;k w1;k wL?1;k ]T is the L-dimensional coecient vector, Xk = [xk xk?L+1 ]T is the input signal vector, dk is the desired response signal, ek is the error signal, is the step size, and is the leakage parameter. For = 0, equations (1){(2) describe the coecient updates for the standard LMS adaptive lter. An approximate analysis of the updates in (1){(2) for > 0 shows that the coecients slowly decay towards zero values if the desired response signal is uncorrelated with the input signal vector [2]. Recently, a more accurate analysis of the leaky LMS algorithm for jointly Gaussian input and desired response signals has enabled useful comparisons of the behaviors of the leaky LMS and LMS adaptive lters to be made [6]. It is well-known that a stochastic variant of the leaky LMS adaptive lter can be implemented by adding zero-mean random white noise to the input signal prior to the application of the LMS adaptive lter [7]. The resulting coecient updates are
Wk+1 = Wk + ek Xk ek = dk ? WkT Xk ;
(3) (4)
where the noisy input signal vector Xk is de ned as
Xk = Xk + Mk ;
(5)
and Mk = [mk mk?L+1 ]T is a noise vector with zero-mean uncorrelated elements. If the noise power m2 = E [m2k ] is chosen as
m2 = ;
(6)
then it can be shown that the mean behaviors of the algorithms in (1){(2) and (3){(4) are approximately the same. If the noise signal mk can be easily generated (using a maximum-length shift register in VLSI hardware, for example), then this stochastic implementation of the leaky LMS adaptive lter avoids the L multiplies used to compute the scaled coecient vector (1 ? )Wk in (1). However, to obtain an error signal that is free of the noise introduced within the adaptation process, the error e(n) in (2) must be computed, and thus the two algorithms in (1){(2) and 1
(2){(4) are of similar complexity. Note that the algorithm in (2){(4) is also obtained in situations where the input signal is dithered prior to sampling, and thus this alternative implementation is of practical interest in its own right. Even though the mean behaviors of the two algorithms are similar, it is not clear how the mean-square performances of the two algorithms di er in any particular situation. The mean-square behavior of an adaptive lter is a more accurate measure of its overall performance and stability characteristics than its mean behavior; thus, it is not clear which algorithm is to be preferred in any particular situation. In this paper, we compare the mean-square performances of the two algorithms in (1){(2) and (2){(4), respectively, assuming a system identi cation desired response signal model of the form T X +n ; dk = Wopt k k
(7)
where Wopt is the optimal coecient vector, nk is an uncorrelated noise signal, and xk and mk are assumed to be independent, identically-distributed (i.i.d.) signals with even-symmetric probability density functions pX (x) and pM (m), respectively. In addition, we shall also assume that vectors within the sequences fXk g and fMk g are independent of each other and of the noise sequence nk . Such assumptions are similar to the independence assumptions often used in analyses of this sort [7, 8]. While never true for an FIR lter con guration, these assumptions lead to reasonably accurate descriptions of the adaptation behaviors of the algorithms, and they allow meaningful comparisons of di erent algorithms to be made. Through our analyses, we show the following: For any particular value of , the range of stable step sizes for the LMS adaptive lter with noisy inputs is smaller than that for the leaky LMS adaptive lter.
For any stable values of and , the LMS adaptive lter with noisy inputs converges no faster than the leaky LMS adaptive lter.
For any stable values of and , the LMS adaptive lter with noisy inputs has a higher steady-state excess mean-square error (MSE) than that for the leaky LMS adaptive lter.
Thus, from a performance standpoint, the leaky LMS adaptive lter is to be preferred in this situation. Simulations verify the analytical results and the above conclusions.
2 Analysis
2.1 Leaky LMS Adaptive Filter
For our analyses, we de ne the coecient error vector as
Vk = Wk ? Wopt: 2
(8)
With this de nition, we can write the leaky LMS adaptive lter update in (1){(2) as
Vk+1 = ((1 ? )I ? Xk XTk )Vk + nk Xk ? Wopt:
(9)
Taking expectations of both sides of (9) and employing our assumptions, we nd that
E [Vk+1] = (1 ? (x2 + ))E [Vk] ? Wopt;
(10)
where x2 = E [x2k ] for our input signal model. To determine a description for the mean-square behavior of (1){(2), we post-multiply both sides of (9) by their respective transposes and take expectations of both sides of the resulting equation. This operation results in
V V
E [ k+1 kT+1]
?
T = E [((1 ? )I ? Xk XTk )Vk VkT ((1 ? )I ? Xk XTk )] + 2 n2 x2 I + 2 Wopt Wopt ? T + Wopt E [VT ((1 ? )I ? Xk XT )] ; (11) ? E [((1 ? )I ? Xk XTk )Vk ]Wopt k k
where n2 = E [n2k ]. Employing the assumptions described above and relying on results already available for the LMS adaptive lter with i.i.d. inputs [8], we can take the trace of both sides of (11). After simplifying the result, we nd the update given by trE [Vk+1VkT+1 ] =
1 ? 2(x2 + ) + 2 ((L ? 1 + x )x4 + 2x2 + 2) trE [Vk VkT ] T E [V ]; + 2 Lx2 n2 + 2 2jjWoptjj2 ? 2 (1 ? (x2 + ))Wopt (12) k
where x = E [x4k ]=x4 . Note that this update depends on the mean coecient error vector E [Vk]. We can de ne the (L + 1)-dimensional vectors Yk and B and matrix A as "
#
k VkT ] ; Yk = trEE[V [Vk ]
"
#
2 2 2 2jjWoptjj2) ; B = (Lx?n +W opt
"
(13) #
2 2 4 2 2 2 T A = 1 ? 2(x + ) + ((L ?0 1 + x )x + 2x + ) ?2 (1(1??((2x++
))))IWopt ; x
(14)
respectively. With these de nitions, we can represent the updates in (10) and (12) as
Yk+1 = AYk + B:
(15)
Note that the excess MSE at time k is given by
MSE;k = E [(VkT Xk )2] = x2 trE [Vk VkT ];
(16)
where trE [Vk VkT ] is the rst entry of Yk . Thus, the form of (15) allows us to determine both the transient and steady-state mean-square behaviors of the leaky LMS adaptive lter for a known initial state vector Y0. 3
For stability, we can determine the values of and that guarantee that all of the eigenvalues of A are less than one in magnitude. Because of the triangular form of A in (14), this transition matrix has L eigenvalues equal to 1 ? (x2 + ) and one eigenvalue equal to the rst entry in the matrix. It can be easily shown that a necessary and sucient condition to guarantee all of the eigenvalues of A to be less than one is
j1 ? 2(x2 + ) + 2 ((L ? 1 + x)x4 + 2x2 + 2)j < 1;
(17)
from which we determine the mean-square stability conditions on to be x2 + ) (18) 0 < < (L ? 1 + 2( x )x4 + 2x2 + 2 : For step sizes that satisfy (18), we can solve for the steady-state excess MSE by rst calculating = (I ? A)?1 B;
lim Y k!1 k
(19)
from which the excess MSE is simply the rst entry of the steady-state value of Yk scaled by x2 . Using the form of A and B in (13){(14), we nd after some algebra that 2
lim Y = k!1 k
6 6 4
?
3
2 2 4 x n + (L ? 2 + x )x + 2(2 + ) L ? ((L ? 1 + x)x4 + 2x2 + 2) 775 ; x ? 2 + Wopt x
(20)
where we have de ned 2
= (2 + )2 jjWoptjj2: x
(21)
From this result, we nd that the steady-state excess MSE for the leaky LMS adaptive lter is 4 ?L 2 + (L ? 2 + x ) 2 x 2 (22) MSE;ss = x + 2(2 + ) ? ((nL ? 1 + ) 4 + 2x 2 + 2) : x x x x The rst term on the right-hand-side of (22) is the excess MSE due to the bias in the lter coecients in steady-state. The second term is the excess MSE due to coecient uctuations caused by a nonzero adaptation speed.
2.2 LMS Adaptive Filter with Noisy Input Signals We now analyze the behavior of the LMS adaptive lter with noisy input signals. Similar to the leaky LMS adaptive lter, we write the coecient updates in (3){(4) in terms of the coecient error vector as
Vk+1 = (I ? Xk XTk )Vk + nk Xk ? Xk MTk Wopt; 4
(23)
where Xk is as de ned in (5). Taking expectations of both sides and using our assumptions, we determine that the mean coecient error vector evolves according to
E [Vk+1] = (1 ? (x2 + m2 ))E [Vk] ? m2 Wopt:
(24)
Thus, if m2 = , the evolution equation in (24) is the same as that in (10). Therefore, we set m2 = in what follows. To determine a mean-square description of the adaptation behavior of this algorithm, we postmultiply both sides of (23) by their respective transposes and take expectations of boths sides. The resulting relation is
E [Vk+1VkT+1 ] = E [(I ? Xk XTk )Vk VkT (I ? Xk XTk )] + 2 (x2 + )n2 I T M XT ] + E [X MT W VT (I ? X XT )] ? E [(I ? Xk XTk )Vk Wopt k k opt k k k k k T T Mk X ]: (25) + 2 E [Xk MTk WoptWopt k Similar to the rst analysis, we take the trace of both sides of (25) and employ our analysis assumptions to simplify the resulting equation, as given by trE [Vk+1VkT+1 ] =
?
1 ? 2(x2 + ) + 2 ((L ? 1 + x )x4 + 2(L + 2)x2 + (L ? 1 + m ) 2 ) trE [Vk VkT ] ? T E [Vk ] ? 2 1 ? ((L + 2)x2 + (L ? 1 + m ) ) Wopt + 2 L(x2 + )n2 + 2 (Lx2 + (L ? 1 + m ) )jjWopt jj2; (26)
where we have de ned m = E [m4k ]= 2. As in the previous case, we can describe the mean-square evolution equation for the LMS adaptive lter with noisy input signals as
Yk+1 = AYk + B;
(27)
where A and B are given by
A
"
2 2 T = A + 2 2(L + 1)x + (L ? 2 + m ) 2((L + 1)x + (L ? 2 + m ) )Wopt "
0
#
2 2 2 B = B + 2 nL + (Lx + (L ?02 + m ) )jjWoptjj ;
0
#
(28) (29)
respectively. As in the case of the rst analysis, the stability of the LMS adaptive lter with noisy input signals is guaranteed if the rst element of A is less than one in magnitude. This criterion results in step size bounds of 2 + ) 0 < < (L ? 1 + ) 4 + 2(2(L+x 2) (30) x2 + (L ? 1 + m ) 2 : x x 5
In addition, we can solve for the stationary point of (27) for stable step sizes in a similar manner as before; the resulting excess MSE for the LMS adaptive lter with noisy inputs is 2 2 2 4 ?1 2 2 x (L(x + )n + (Lx + (2L ? 6 + x + m )x + L )x ) : (31) MSE;ss = x2 + 2(x2 + ) ? ((L ? 1 + x )x4 + 2(L + 2)x2 + (L ? 1 + m ) 2) Using these results, we can compare the mean-square behaviors of the two adaptive lters.
3 Performance Comparison Examining the upper step size bounds in (18) and (30), we note that the denominator of the upper bound in (30) is always greater than that of (18). Thus, the range of stable step sizes for the LMS adaptive lter with noisy inputs is in general smaller than that for the leaky LMS adaptive lter, and the di erence between these two ranges increases if either L, , or m increases. As for the convergence rates of the two systems, we note that the transition matrices A and A share L eigenvalues, and thus we can compare the eigenvalues represented by the rst entries of both matrices directly. From (28), we see that the rst entry of A is always larger than the corresponding entry of A for positive leakage factors. Since both of these entries are always positive, the convergence speed of the LMS adaptive lter is always slower than the convergence speed of the leaky LMS adaptive lter with the same step size and leakage factor. The di erence in convergence speeds becomes negligible as the step size and leakage factors are decreased, however, and thus we can conclude that the two adaptive systems converge at nearly the same rate if both and are suitably small-valued. We now compare the steady-state excess MSE of the two adaptive systems. We can express MSE;ss in (22) as
MSE;ss = x2 + cc1 ; 2
(32)
where c1 and c2 are the numerator and denominator, respectively, of the second term on the righthand-side of (22). Comparing this equation with (31), we see that 2 ?L 2 + (L 4 ?1 + (L ? 4 + m ) 2 + L ) 2 c + 1 x n x x x : 2 MSE;ss ? x = (33) c2 ? (2(L + 1)x2 + (L ? 2 + m ) ) For any value of L and any distribution of mk , the numerator and denominator of the right-handside of (33) will be larger than c1 and smaller than c2 , respectively. Thus, the steady-state excess MSE of the LMS adaptive lter with noisy inputs is always greater than that of the leaky LMS adaptive lter with the same values of and . Combining the above results, we see that the noisy LMS adaptive lter always performs more poorly than the leaky LMS adaptive lter for i.i.d. input signals. The di erence in performance will in general increase if either , m , and/or L are increased. 6
4 Simulation Results We now verify the above conclusions via simulation. An L = 50 coecient system identi cation task was chosen for these simulations, where Wopt = [1 1 1]T . The input and observation noise signals were both chosen to be zero-mean white Gaussian with x2 = 1 and n2 = 0:00001, respectively. For the noisy LMS adaptive lter, the noise signal mk was chosen to be zero-mean white Gaussian-distributed. One hundred simulation runs were averaged in each case. Figure 1 shows the convergence of the total coecient error power trE [VkVkT ] for the leaky LMS, noisy LMS, and LMS adaptive lters in this situation, where = 0:01 and = 10?5 . As can be seen, the leaky LMS adaptive lter's convergence rate is nearly the same as that for the LMS adaptive lter with noisy inputs; however, the steady-state coecient error power is about 50 times larger for the noisy LMS adaptive lter. Also plotted are the convergence behaviors of the systems as obtained from the analyses, showing that the theory accurately predicts convergence behavior. Figure 2 shows the total coecient error powers in steady state for the three algorithms as a function of for = 10?5 . As can be seen, the LMS adaptive lter with noisy inputs has a larger error in steady-state as compared to that of the leaky LMS and LMS adaptive lters. Simulated behavior closely matches the theoretical predictions of performance. Since both of the leaky LMS adaptive lters converge at nearly the same rate for small step sizes, a useful quantity for comparison purposes is the fraction of the additional steady-state excess MSE due to coecient uctuations for the noisy LMS adaptive lter with respect to that of the leaky LMS adaptive lter, given by
=
MSE;ss ? MSE;ss MSE;ss ? x2 :
(34)
Figure 3 plots as a function of for the case described above, with = 0:01. Clearly, the behavior of follows two di erent trends for small and large leakage factors, respectively. It can be shown for large signal-to-noise ratios x2 jjWoptjj2=n2 1 and for x2 that 2 2 xjjWoptjj 2 : x n2 + 2jjWoptjj2 Moreover, the maximum value of occurs at = x n =jjWoptjj and is approximately
(35)
1 pSNR ; max (36) d
2 where SNRd = x2 jjWoptjj2=n2 is the signal-to-noise ratio of dk . We can see that the analysis is accurate in predicting simulated behavior, as indicated by the simulation results. It should be noted from (35) that the value of tends to zero as the leakage factor is reduced. If one considers = 0:1 to be an acceptable fraction of excess MSE that can be tolerated in the 7
adaptation process, then for values of satisfying
x2 > 10 SNRd ;
(37)
both the leaky LMS and noisy LMS adaptive lters have similar behaviors. In other words, if the input-signal-power-to-leakage ratio x2 = is large relative to the signal-to-noise ratio of dk , either implementation gives satisfactory performance.
5 Conclusions In this paper, we have provided an analysis of two competing implementations of the leaky LMS adaptive lter. We have shown through both theory and simulation that by adding noise to the input signal of the the LMS adaptive lter, one obtains a system whose mean behavior is similar to that of the leaky LMS adaptive lter. However, for every mean-square performance criterion studied, the mean-square behavior of this adaptive lter is worse than that of the leaky LMS adaptive lter, and this performance di erence is particularly large for large signal-to-noise ratios and moderate values of the leakage factor. We have also identi ed the range of leakage factors for which both implementations perform satisfactorily. Simulations verify the analyses and indicate the magnitude of the performance di erences.
References [1] D.L. Cohn and J.L. Melson, \The residual encoder: An improved ADPCM system for speech digitization," IEEE Trans. Comm., vol. COMM-23, no. 9, pp. 935-941, September 1975. [2] R.D. Gitlin, H.C. Meadors, and S.B. Weinstein, \The tap-leakage algorithm: An algorithm for the stable operation of a digitally-implemented fractionally-spaced adaptive equalizer," Bell Sys. Tech. J., vol. 61, no. 10, pp. 1817-1840, October 1982. [3] J.R Treichler, C.R. Johnson, Jr., and M. G. Larimore, Theory and Design of Adaptive Filters (New York: Wiley-Interscience, 1987). [4] J.R. Gonzal, R.R. Bitmead, and C.R. Johnson, Jr., \The dynamics of bursting in simple adaptive feedback systems with leakage," IEEE Trans. Circuits Systems, vol. CAS-38, no. 5, pp. 476-488, May 1991. [5] J. Cio and Y.-S. Byun, \Adaptive lters," in Handbook of Digital Signal Processing, S.K. Mitra and J.F. Kaiser, eds. (New York: Wiley, 1993), pp. 1102-1104. [6] K.A. Mayyas and T. Aboulnasr, \The leaky LMS algorithm: MSE analysis for Gaussian data," accepted for publication in IEEE Trans. Signal Processing; to appear. [7] S.S. Haykin, Adaptive Filter Theory, 3rd. Ed., (Englewood Cli s, NJ: Prentice-Hall, 1995). [8] W.A. Gardner, \Learning characteristics of stochastic-gradient-descent algorithms: A general study, analysis, and critique," Signal Processing, vol. 6, no. 2, pp. 113-133, April 1984. 8
List of Figures Figure 1: Convergence of total coecient error power, theory and simulation: leaky LMS, noisy LMS, and LMS adaptive lters, white Gaussian input signals, = 0:01, = 0:00001. Figure 2: Steady-state total coecient error power as a function of , theory and simulation: leaky LMS, noisy LMS, and LMS adaptive lters, white Gaussian input signals, = 0:00001. Figure 3: Penalty factor as a function of the leakage factor , leaky and noisy LMS adaptive lters, = 0:01.
9
2
10
1
10
Theory Leaky LMS (Sim.) Noisy LMS (Sim.)
0
Total Coefficient Error Power
10
LMS (Sim.) -1
10
-2
10
-3
10
-4
10
-5
10
-6
10
0
200
400
600
800 1000 1200 number of iterations
1400
1600
1800
2000
Figure 1: Convergence of total coecient error power, theory and simulation: leaky LMS, noisy LMS, and LMS adaptive lters, white Gaussian input signals, = 0:01, = 0:00001.
10
-2
10
-3
10
Theory
Steady-State Coefficient Error Power
Leaky LMS (Sim.) Noisy LMS (Sim.) -4
10
LMS (Sim.)
-5
10
-6
10
-7
10
-8
10
-9
10 -5 10
-4
10
-3
10 mu
-2
10
-1
10
Figure 2: Steady-state total coecient error power as a function of , theory and simulation: leaky LMS, noisy LMS, and LMS adaptive lters, white Gaussian input signals, = 0:00001.
11
4
10
Theory Simulation Eq. (35) 3
alpha
10
2
10
1
10
0
10 -6 10
-5
10
-4
10 gamma
-3
10
-2
10
Figure 3: Penalty factor as a function of the leakage factor , leaky and noisy LMS adaptive lters, = 0:01.
12