Combined Analog and Digital Error-Correcting Codes For ... - CiteSeerX
Combined Analog and Digital Error-Correcting Codes For Analog Information Sources Isaac Rosenhouse and Anthony J. Weiss School of Electrical Engineering, Tel Aviv University, Tel Aviv, 69978, Israel Email: ,
Abstract
This contribution focuses on error-correcting codes for analog signals. Generally, error-correcting codes exhibit a threshold effect, which degrades performance at the low signal to noise ratio (SNR) range. We propose two error correction alternatives based on chaotic dynamical systems, which improve threshold performance relative to previously proposed methods. The first one, which is based on principles of diversity, exhibits a very simple decoder. The second alternative, which is based on a low-density-parity-check (LDPC) code, attains a significantly lower threshold. Theoretical analysis and simulations are provided.
Index Terms – analog signals, error correction, chaotic systems, diversity, LDPC, threshold extension.
Isaac Rosenhouse is with Ceragon Networks, Israel. This research was supported by the Israel Science Foundation (grant No. 1232/04)
/06/2007
I. Introduction In the last five decades, numerous error-correcting codes for digital information sources have been developed. However, when the information source is analog (i.e., continuous-valued) quantization is applied prior to coding. This approach is optimal when the channel signal to noise ratio (SNR) is known at the transmitter. The quantization and coding are done according to the channel’s capacity. However, when the SNR is not known the optimal number of quantization levels cannot be determined. Cases in which the transmitter does not know the SNR include broadcast channels, where there are multiple receivers each with different SNR, as well as systems operating in timeselective fading, where feedback from the receiver is not available. An alternative approach to the simple quantization and coding is error correction in the analog signals domain. In this case no quantization takes place. Chen & Wornell [1] proposed an analog error correcting code based on chaotic dynamics. Their algorithm encodes an analog scalar value into a vector of analog values by generating a discrete time chaotic trajectory originating in the scalar value. Several papers, including [2],[3],[4], examined the estimation of the chaotic trajectory’s initial value. Indeed, their scheme achieves coding gain, but a threshold effect limits the gain, especially for low-rate codes. The threshold effect is a consequence of errors in the estimated signs (itinerary) of the chaotic trajectory. The authors of [1] discuss the threshold, and its effect on the sign error probability and on the resulting mean square error of the estimated analog values. However, we are not aware of any attempt to reduce the threshold effect, so far. In this work we explore efficient ways for coding the signs embedded in the chaotic trajectory. We employ longer chaotic trajectories with redundancy embedded in their itineraries. The error correction scheme exploits the redundancy and yields better-estimated itinerary. We refer to the first type of redundancy as diversity. The chaotic sequence is obtained by iterating a scalar value using a symmetric tent map [2]. In addition, iterating the scalar value /06/2007
backwards with an equal number of iterations doubles the sequence length. One can choose the signs of the backward iterated values arbitrarily. Here we choose them to be the signs of the forward iterations multiplied by a pseudo random signs sequence. This arrangement provides the desired diversity. Although the performance of this method is not optimal, it provides an additional coding gain at the low SNR range while maintaining a very simple decoder. The second type of redundancy we explore provides excellent coding gain but requires a more sophisticated decoder. It is based on a low-density-parity-check (LDPC) code. The LDPC code was originally introduced by Gallager [5] in 1962 and was rediscovered by MacKay et al. [6] in 1996. It exhibits performance extremely close to the best possible as determined by the Shannon capacity expression [7]. Since good coding requires long code words we shall encode a vector of analog samples rather than a single sample. We perform forward and backwards iterations on each of the vector entries. The signs of the backwards-iterated samples are determined according to the parity bits of a systematic LDPC code. The information bits for the LDPC encoder are the signs of all forward iterated samples. We show by means of analysis and simulation that this arrangement provides excellent coding gain at the low SNR range. The paper is organized as follows. In Section II we formulate the system and review some previous results. We demonstrate the threshold effect and motivate coding of the itineraries. Section III presents a capacity bound on the threshold SNR. Section IV presents the diversity method for reducing the threshold effect. Sections V - VII present the LDPC code based approach for threshold effect reduction. In Section VIII we discuss the elements responsible for the gap between the theoretical bounds and the achieved performance. Section IX includes our concluding remarks.
II. Formulation of the System and Review of Previous Results We consider transmission of N1 samples of a random signal distributed uniformly in the open interval (-1,1) over an additive white Gaussian noise (AWGN) channel. The signal samples are /06/2007
denoted by {zn,0} where n = 0,1,…N1-1. Throughout the paper we refer to them as the source samples. Each of the N1 source samples is iterated (N2-1) times using a tent map
z defined as
follows:
z
1 2z .
(1)
The ensemble of trajectories generated by this function is white and also distributed uniformly in the range (-1,1) [8]. The trajectories are denoted by {zn,m}, where m = 0,1,…N2-1 and n =
0,1,…N1-1. The mathematical relation between the entries of the trajectories is given by z n ,m
z n ,m
1
m 1,2 ,...N 2 1 . n 0 ,1,...N 1 1
(2)
In order to define an inverse function for
z , we must specify a sign for the obtained value. We
use the subscript s to denote it. The sign of zn,m is denoted by s[n,m] . Therefore, the function 1 s [ n ,m ]
z n ,m
z n ,m
1
1 z n ,m 1 s n ,m , 2 1 z n ,m 0 sign( z n ,m ) . 1 z n ,m 0
1 s [ n ,m ]
s [ n ,m ]
is defined as
z n ,m
1
(3)
In addition to the forward iterations (2), we perform N3 backwards-iterations on each of the N1 source samples according to a given signs array s[n,m] :
z n ,m
1 s [ n ,m ]
z n ,m
1
m
N 3 , N 3 1,... 1 . n 0 ,1,...N 1 1
(4)
Finally, we transmit the whole array {zn,m}, m=-N3,…N2-1, n=0,1,…N1-1 through an AWGN channel with noise variance of
yn,m
z n,m
2 v
. The received signal is denoted by {yn,m} and given by
vn,m ,
(5)
where vn,m stands for the noise samples. The source vector is estimated in the receiver using the received array. The samples obtained by the forward and backwards iterations may be considered as “parity samples” in a systematic /06/2007
analog error-correcting code. For the scalar case (N1 = 1) with no backwards iterations (N3=0) performance analysis has been carried out in [1] and [2]. We review the main results for this case with some adaptations to the formulation presented above. Maximum likelihood (ML) estimation of the source value is done in two steps. At the first step the signs and magnitudes of the transmitted trajectory are estimated via a filtering process. For simplicity, we shall omit the index n for scalar cases, i.e. zm
zn,m and ym
yn,m. Let ˆz k | j
denote the estimated value of zk based on the j+1 samples {ym}, m = 0,1,…j. The filtering process may be done recursively using the following expressions [2]
ˆz [ 0 | 0 ]
y0 ,
ˆz [ m | m 1 ] ˆz m | m
(6a)
ˆz m 1 | m 1 .
3 2 2 m ym
(6b)
2 2 m 1 ˆz m | m 1 4 2 2m 1
m
1
3 ym 4
1 ˆz m | m 1 . 4
(6c)
Notice that since the values of z are limited to the interval (-1,1), the estimates ˆz m | m should also be hard limited to this interval during the filtering process. The second step involves a smoothing process. It performs backwards iterations using the signs of the estimated samples obtained in the first step:
ˆz m | N 2 1
1 sˆ [ m ]
ˆz m 1 | N 2 1 ,
(7)
where ˆs m denotes the sign of ˆz m | m in equation (6). The Cramer-Rao lower bound (CRLB) on the variance of the estimates has been presented in [2]: var ˆz m | m var ˆz 0 | N 2 1
2 v
3 4 0.25 m 2 v
m
3 4
N2
1
1
3 4
2 v
,
(8)
.
(9)
As seen from equations (8) and (9), the filtering process reduces the noise effect by 1.25 dB, asymptotically, and the smoothing process reduces the noise further by an exponential factor of
N2. Therefore, the smoothing is the dominant element in the noise effect reduction. However, the /06/2007
smoothing process approaches the Cramer-Rao bound only asymptotically when there are practically no errors in the estimated signs ˆs m . We show in the sequel that the errors in the estimated signs are the dominant factor of the performance degradation of the proposed error correction scheme. The sign error probability is approximately [1]:
PS
1 2 2
SNR
,
(10)
where the SNR is defined as: 2 z 2 v
SNR and
2 z
1 3
2 v
,
(11)
is the variance of the transmitted signal.
An estimated value in the smoothing process gets a wrong sign if a sign error event (e) occurs. In this case the variance of its estimation error is approximately [1]: 2 ˆz|e
2 / 3 SNR
(12)
Let w denote the source value estimation error, i.e. w
ˆz0
z0 .
The variance of the estimation error is given by [1]: 2 w
var ˆz 0 | N 2 1
2 v
3 4 N2
1
1 PS
N2 1
2 ˆz|e
1
N
1 PS / 4 2 1 1 PS / 4
1
PS .
(13)
Observe that as the sign error probability decreases to zero the estimation error variance in (13) converges to the CRLB in (9). However, the sign error probability decays very slowly as the SNR increases as indicated by (10), and therefore, practically, the second term in (13) dominates the variance over a large range of SNR values. The coding gain (in dB) is defined by:
CG 10 log 10
2 v 2 w
10 log 10 N 2
N3 .
(14)
/06/2007
Even though output distortion or SNR may be regarded by some as a more fundamental performance measure of analog signals, we believe that coding gain is the natural approach here. It concisely presents the ratio of the output SNR to the channel SNR normalized by the bandwidth expansion (N2+N3). Figure 1 shows the analytical coding gain according to (14) as a function of SNR for different numbers of forward iterations. Observe that a threshold effect exists as predicted by equations (13) and (14). Even for a small number of forward iterations such as N2=4, the coding gain approaches its asymptotic limit at an SNR of approximately 50 dB. The additional N3 backwards-iterations should provide the necessary redundancy for coding the itinerary, and hence reduce considerably the threshold effect. Notice that the fact that the coding gain saturates at some point implies that the ratio between the output SNR and channel SNR converges to a constant, hence, as the channel SNR increases, the output SNR increases proportionally without limit. This behavior is typical for all coding schemes discussed in this contribution, and is in fact the whole point of using an analog scheme in the first place.
III. A Capacity Bound on the Threshold SNR A simple upper bound on the achievable coding gain is obtained by using a combination of the rate-distortion and channel capacity theorems. A bound on the distortion (
2 w
) has been
derived in [1]. With some slight adaptations and using the coding gain definition (14) we obtain 2 w
2 w ,Bound
CG CGBound
2 1 1 e 3 v2 10 log 10
N2 N3
,
(15a)
2 v
N2
N3
2 w ,Bound
.
(15b)
Notice that in the high SNR range (SNR>>1), the bound is approximated by:
/06/2007
CGBound
N2
N 3 1 10 log 10 SNR
10 log 10
6 N2 e
N3
,
(15c)
which implies that the coding gain need not saturate as a function of SNR as in our coding schemes. This phenomenon is discussed later on after we present our coding schemes. Achieving the coding gain bound requires optimal coding and in the case of N2+N3 > 2 there is no single coding scheme, which achieves it for all SNR values [15]. In the previous section we have motivated coding the itineraries of the chaotic trajectories by demonstrating the threshold effect caused by the sign errors. In this paper we confine ourselves to the case in which the required coding redundancy is embedded in itineraries of longer chaotic trajectories. Since maximum-likelihood decoding is exponentially complex, we resort to sub-optimal estimation of the transmitted signal. In our proposed method the estimation process begins with filtering as shown in the previous section and an itinerary decoder follows it. There are two main causes for the sub-optimality: The filtering processes disregards the code word embedded in the itinerary, and the itinerary decoder regards the magnitudes as uncorrelated random variables rather than a series of values evolving in a deterministic fashion. Under these simplifications we wish to obtain a bound on the threshold SNR for coded itineraries. We model the coded itinerary as a binary source with values of 1. We model the channel through which the itinerary is transmitted as a composition of two elements: a gain element and a noise element. The gain element multiplies the source by a random gain distributed uniformly in the range (0,1). The noise element is an AWGN element with variance 2 u
(Figure 2). Notice that the chaotic trajectory has a deterministic relation between the
magnitudes of its samples. This relation is exploited in the filtering process (6) in order to reduce the AWGN. Therefore, the noise variance in the model,
2 u
, represents the noise after filtering.
For capacity calculations we use the asymptotic relation in (8) for determining 2 u
3 4
2 v
, which implies a filtering gain of 1.25 dB. /06/2007
2 u
, i.e.
The amount of information which can be transmitted through a channel for a given source distribution is determined by the mutual information between its input (x) and its output (y). We refer to this quantity as the capacity per symbol and denote it by C [9]. It is given by
C
2
I X ;Y
!
p x xk
k 1
f y|x y xk
log 2 f y|x y xk
2
log 2
p x xn
f y|x y xn
dy .
(16)
n 1
Y
Here px(x) denotes the probability distribution of the source x, and fy|x(y|x) denotes the conditional probability density of the channel output (y) given the channel input (x). Throughout the analysis we distinguish between two cases: 1. The transmitted magnitudes are provided to the itinerary decoder. 2. The transmitted magnitudes are uniformly distributed in (0,1) and no other a priori information regarding them is provided to the itinerary decoder. In the first case the conditional probability is given by:
f y|x y | x
y "x
1
e
2 u
2
2
2
2 u
,
(17a)
where " designates the transmitted magnitude and the capacity is obtained by averaging it over #
!
the distribution of " , i.e. C
1
f " " C" d"
#
! C" d" , where C" is the capacity for a given " . 0
In the second case we have
f y|x y | x
1 2
1
y *x
!e
2
2 u
2 u 0
2
d*
1) 'Q x(
y
Q
u
x y & $, u %
(17b)
where Q(x) is the complementary error function defined in (31). For a symmetric source distribution i.e. px(1)=px(-1)=0.5, equation (16) simplifies to f y|x y | x 1
#
C
!f #
y|x
y | x 1 log 2
1 2
f y|x y | x 1
f y|x y | x
1
dy .
(18)
Equation (18) has been evaluated numerically for both cases and the results are shown in Fig. 3, which plots the capacity per symbol vs. SNR. Notice that the SNR in the figure is defined according to (11), that is, the ratio between the signal power after the gain multiplication and the /06/2007
AWGN power before filtering. Inspection of the figure reveals that using an error correction code with rate ½ the obtained threshold SNR values are approximately 0.65 dB and 2.05 dB for the known and unknown magnitudes respectively. Codes with rate ½ will be used in examples later on. In the introduction we stated that we deal with cases in which channel SNR is unknown to the transmitter. This contradicts the need to choose a code rate according to a given SNR. Strictly speaking, we still need to know that channel SNR is above some threshold, however, as long as this condition is fulfilled, the distortion of the estimated signal will decrease with any increase of the channel SNR.
IV. Diversity Based Coding of Itineraries A simple method for coding the itinerary of a single trajectory is based on generating another chaotic trajectory whose itinerary is related to the source itinerary by a pseudo random sequence known at both ends of the system. We refer to the second trajectory as the parity trajectory. By selecting a pseudo random itinerary for the parity trajectory, its magnitudes become statistically independent of the source trajectory. As we shall see this condition decreases the sign error probability by a factor of a square root of the SNR. Notice that we have here a sort of rate ½ repetition code, which will be replaced by an LDPC code in the following section. (Simple repetition codes do not have any coding gain, while this code has). Practically, the parity trajectory may be generated using N3=N2 backwards iterations starting from the source value. We denote the sign of a corresponding sample at the parity trajectory by
sign z
m 1
+ m sign z m ,
m = 0,1,…N2-1.
(19)
where + m denotes a pseudo-random sign. Estimation of the source value is done in three steps. First the received chaotic trajectory is filtered according to (6). The estimation error in this stage is designated by u: /06/2007
um
ˆz m | m
zm ,
ˆz m | m
where
(20)
denotes the estimated value of zm based on the received samples
y-N3,…ym. The variance of the estimation error generally depends on m, but for m >> -N3 it converges to a constant value as indicated by (8). In the following derivation and analysis we assume that it is independent of m and equals to 2 u
2 v
3
/4.
(21)
In the second step the itinerary of the transmitted sequence is estimated using,
ˆs m
ˆs m 1 + m
sign ˆz m | m
ˆz
m 1 | m 1 + m , m = 0,1,…N2-1.
(22)
Finally, smoothing is performed according to (7). The sign error probability is bounded by the following expression [appendix A]:
PS ,Diversity
8 SNR
1
.
(23)
Comparing (23) with (10), we see that the sign error probability for diversity-coded itineraries is reduced by a factor proportional to the square root of the SNR. As a consequence, the threshold SNR is reduced by a factor of approximately 2 on a logarithmic scale. Figure 4 shows the coding gain vs. SNR for different number of forward and backward iterations. The coding gain in this figure is defined in equation (14) where equation (13) is used for
2 w
and the expressions for PS ,Diversity and
2 ˆz|e ,Diversity
are provided in Appendix A in equations
(48) and (52) respectively. The effect of threshold reduction is evident by comparison with Figure 1. It is worth noting that the price paid for the threshold shift is a reduction in the asymptotic coding gain.
V. LDPC Coding of the Itineraries In order to approach the threshold bound presented in section III a highly efficient coding scheme should be employed. It is well known that in order to approach the Shannon limit, long information sequences should be coded. Therefore, in this section we discuss coding of a vector /06/2007
of analog samples as opposed to coding of a scalar analog sample that was considered in previous sections. Each of the N1 elements in the source vector is iterated forward N2-1 times using the tent map function (1). As a result we get an array of analog samples with N1 rows and N2 columns (2). We wish to encode the signs of the elements in the array. Negative values are mapped to one and non-negative values are mapped to zero. This information is digital, and therefore, standard digital encoding techniques may be employed here. The digital information source to be coded is denoted by a row vector x and its elements are defined as follows:
xn N 2
1 2
m
1 sign z n ,m ,
m 0 ,1,...N 2 1,
n 0 ,1,...N 1 1.
(24)
The vector x has K=N1 N2 elements. An LDPC code is a linear block code with a generator matrix GKxN and a parity check matrix H(N-K)xN. (The subscripts denote the size of the matrices). The code word c is generated by
c 1 xN
x1 xK G KxN .
(25)
The code rate is denoted by R=K/N. The generator matrix G should be expressed in a systematic form and the number of parity bits, denoted by M, should be an integer multiple of N1:
G
IK
M
N
PKxM , K
(26)
N1 N 3 .
(27)
Each of the N1 source samples is now iterated backwards N3 times using signs matching the parity bits generated by the encoder. We obtain an array, denoted by Z, composed of N1 rows and
(N2+N3) columns. Each row consists of a valid discrete time chaotic trajectory. In addition, the signs of the samples in Z consist of a valid LDPC code word. Figure 5 illustrates this structure. The code word array is transmitted through an AWGN channel (5). A few steps are employed in the estimation of the source vector. As seen in the previous cases, the first step is filtering according to (6). The filtering is done on a row-by-row basis. The second step involves /06/2007
error correction of the estimated signs. In this stage we can either assume that the magnitudes of the transmitted samples are unknown and uniformly distributed in the range (0,1) or perform smoothing as in [1] (7), and rely on the estimated magnitudes for decoding the LDPC code. In the second strategy the decoder is provided with additional information, however, due to itinerary estimation errors, it may be inaccurate at this stage. The preferred strategy is yet an open issue, however, our simulations suggest that for a rate ½ LDPC code with N2=N3=8 the second is definitely superior (See Figures 6,7). Anyhow, expressions for both decoding strategies are provided. The third step consists of a smoothing according to (7) using the decoded signs. Additional decoding-smoothing iterations may be applied while providing the decoder updated magnitude estimations from previous iterations. The process halts when a valid itinerary is obtained or when a maximal number of decoding-smoothing iterations have been performed. An itinerary may be validated by calculating the syndrome:
s
cˆ H T ,
(28)
where cˆ denotes the decoded codeword. For a valid codeword (itinerary) the syndrome is zero. As originally suggested by Tanner [10], LDPC codes are well represented by bipartite graphs in which one set of nodes, the variable nodes, corresponds to elements of the codeword and the other set of nodes, the check nodes, correspond to the set of parity-check constraints, which define the code. The first step of LDPC decoders based on message-passing is a calculation of the loglikelihood-ratio (LLR) of variable nodes. The LLR of bit i, which is denoted by Li, is defined as Li
ln
Pr ci
0| y
Pr ci
1| y
.
Here ci denotes the ith bit of the codeword and y denotes the vector of received samples. It is possible to derive a closed form expression for this LLR. However, this is beyond the scope of this paper. The derivation and performance will be shown in a future paper. In the derivation
/06/2007
provided here we shall use an approximation based on the filtered samples. Filtered sample i shall be denoted by ˆzi|i . The approximate LLR we propose is given by Li
ln
Pr ci
0 | ˆz i|i
Pr ci
1 | ˆzi|i
.
(29)
Let " i denote the magnitude of the transmitted sample corresponding to xi. Assuming the magnitudes are uniformly distributed unknowns at the receiver the LLR is given by 1 LUnknownMagnitude
ln
Pr zi
0 | ˆzi|i
Pr zi
0 | ˆzi|i
ln
2
e 2 !
2
2
2 u
dt
u 0
ˆzi|i t
0
1 2
ˆzi|i t
1
2 u
!
e
2
2 u
Q
dt
Q
1 ˆzi|i
u
ln
2
ˆz i|i
Q
1 ˆz i|i u
u
Q
ˆzi|i
, (30)
u
1
where the Q function is given by:
1 2
Qx
#
!e
t2 2
dt .
(31)
x
Equation (30) is easily derived using Bayes law assuming equal probabilities of zeros and ones in the codeword. Similarly, assuming the magnitudes of the transmitted samples, "i, are known at the receiver, we arrive at the following formula for the LLR: LKnownMagnitudes
2" i ˆz i|i 2 u
.
(32)
The following steps of the iterative decoder are identical to the standard algorithm. Let qij denote the message passed from variable node i to check node j. Let rji denote the message passed from check node j to variable node i. The message qij represents the log-likelihood ratio of variable node i, given the LLRs of the check nodes connected to it, rj’i, except check node j and given the channel LLR, L. It is given by qij
Li
(33)
r j'i j', j
The message rji represents the LLR of check node j, given the LLRs of all variable nodes connected to it except variable node i. It is given by /06/2007
r ji
2 tanh
1
- tanh i', i
qi'j 2
.
(34)
Equations (33) and (34) constitute one decoding iteration and each variable node message is initialized with the channel LLR, (30) or (32). Iterations are repeated until hard decisions on the variable nodes LLRs yield a valid code word, or until a maximal number of decoding iterations have been performed.
VI. Performance Evaluation of the LDPC Coded Itineraries Algorithm An elegant analysis method of LDPC codes has been proposed by Richardson & Urbanke in [11]. The method, known as density evolution, predicts a threshold phenomenon: as the codeword length tends to infinity, an arbitrary small error probability can be achieved if the noise level is smaller than a certain threshold. Conversely, for a noise level above this threshold, the error probability is larger than a positive constant. They proved a general concentration result, which shows that performance of an LDPC decoder on random graphs converges to its expected value as the length of the code increases. Therefore, analysis is done for an ensemble of infinitely long (i.e. cycle free) codes determined by a given length and degree distributions. Degree distribution refers to the degrees of the variable and check nodes. Specifically, let
.i denote the fraction of messages emerging from variable nodes with degree i, and let / i denote the fraction of messages emerging from check nodes with degree i. The vectors . and
/ determine the degree distribution of the ensemble. The method, which we use here, is based on approximating the message densities as Gaussian mixtures, and expressing them as a function of their mean as shown in [14]. While the Gaussian approximation holds for the check node messages, the variable node messages are no longer Gaussian in our case. Therefore, slight modifications are performed in order to adjust the
/06/2007
Gaussian approximation (GA) algorithm to our needs. The threshold is approximated and shows good agreement with simulation results. Before we proceed with the GA derivation we wish to emphasize that even though we treat the channel as memoriless it is by no means so. The chaotic trajectory evolves deterministically, and hence, so does the noise variance. If this memory had been taken into account in the decoding process and analysis, performance would have probably been improved. The analysis here is done under the assumption that the magnitudes of the transmitted symbols are distributed uniformly in (0,1) and are known at the receiver. This assumption is justified since the iterative analog error correction, as explained in the previous section, involves detection of the signs of the transmitted symbols as well as estimation of their magnitudes. Let q and r denote random variables corresponding to the probability density of the variable nodes messages (33) and the check nodes messages (34), respectively. Let mrh denote the mean of the check nodes messages after h iterations. Let Peh denote the bit error probability after h iterations. The Gaussian Approximation method provides an expression for the bit error probability at iteration h+1 as a nonlinear function of the bit error probability after h iterations. An LDPC code is a linear block code and therefore the performance depends only on the additive noise. Throughout the analysis it is assumed that the sent message is the zero vector, i.e. all the transmitted symbols have non-negative signs. Given Peh , it can be shown that the following check node messages have a symmetric Gaussian distribution fm(r) with mean m
fm r
mrh
1
1 e 4 m 2Q
1
1 2
Wc
r m 4m
mrh
1
[14]. The distribution is given by
2
/ j 1 1 2 Peh
(35)
j 1
(36)
j 2
Let the distribution of the initial variable nodes messages (32) be denoted by f L
. Consider
a variable node with degree i. From equation (33) it is evident that the distribution of its /06/2007
and a symmetric Gaussian with mean i 1 mrh 1 . For
messages is a convolution between f L
short we denote this mean value by * . The resulting distribution is denoted by f Q q | * . The initial error probability is given by (see (10) and [1]) Pe0
0
!f
1
d
L
2 2
#
,
SNR
(37)
and the error probability at iteration h+1 is given by h 1 e
P
Wv
Pr q 0
0
!f
.i
i 2
Q
q|*
i 1 mrh
1
dq ,
(38)
#
where Wv is the maximal variable node degree. Equations (37), (36) and (38) constitute a set of nonlinear iterative functions which enable calculation of the messages densities and the corresponding error probabilities. Explicit expressions for the quantities involved in these equations were derived and they are presented below. The log-likelihood function of the received samples is given by 2" i ˆz i|i
LKnownMgnitudes 2 u
Recall that
2" i " i xi
2 u
2 u
ui
2 x
1
2 u
" i2 " i u i .
(39)
denotes the noise variance after the filtering process (6). It is related to the channel 2 v
noise variance
through equation (21) and to the channel SNR through equation (11).
The distribution of these values is given by [Appendix B]:
fL
1 2
1
!e
"
1 u 2 2"
2 u
u
2"
0
d"
(40)
The distribution of the variable nodes messages conditioned by * is given by [Appendix C]
1
fQ q | *
!2 0
1
u 2 u
* 2"
2
) 0 ' 1 0 exp ' 0 ' 4* 0 '(
4*
1 2
2 2 u 2
*
"
q2
q 2
2"
2
*
4
2 u
2 u
&3 $ 00 $ 2d" . $0 $% 0 1
Finally, the associated bit error rate is given by: /06/2007
(41)
0
!
1
f Q q | * dq
#
!4 0
g 1 ( * ," ,
g 2 ( * ," ,
2 u
2 u
)
)
g 2 ( * ," , g 1 ( * ," ,
1 4*
)
2 u
4* 4
2 u
2 u
2 u
* 2" 2
)
1 2
exp
1 16 g 1 ( * ," ,
2 u
)
1 g 2 ( * ," ,
2 u
3 1 2 Q )1 2 g 1 ( * ," ,
2 u
)
d" ,
,
2 2 u 2
*
(42a)
(42b)
" .
(42c)
These expressions may be evaluated numerically. The integrals are defined over a finite range, and the integrands are fairly smooth.
VII. Simulation Results In our first example we used the following parameters: The size of the analog source vector
N1=125; the number of forward iterations per source sample N2-1=7; the number of backwardsiterations per source sample N3=8. Hence, the length of the LDPC codeword is
N1 (N2+N3)=2000, and the LDPC code rate is 0.5. Notice that in terms of the analog signal, the coding rate is 1/(N2+N3)=1/16 and the bandwidth expansion factor is (N2+N3)=16. The LDPC code has been designed according to algorithms presented in [12] and [13]. The optimal degree distributions for rate-½ codes with a maximum variable nodes and check nodes weight of 7 using GA over a simple AWGN channel [14] are:
/ x
0.4 x 5 0.6 x 6 and . x
0.3078 x 0.27287 x 2 0.41933 x 6 .
Even though our channel is not a simple AWGN, the code used here is considered good also for Rayleigh fading channels [16], which resemble our channel to some extent, and therefore, the same distribution has been used. The maximal number of iterations of the LDPC decoder was set to 50. Figure 6 presents the sign error probability of the LPDC coded itineraries vs. SNR for several cases. Notice that we use the SNR as defined in equation (11) in order to maintain a unified basis for comparison between the various results. The dashed line is related to the case in which the /06/2007
transmitted magnitudes are considered a-priori unknowns. The lack of magnitudes information is probably the reason that the LDPC decoder yields the worst sign-error probability. If smoothing is applied according to [1], some information regarding the transmitted magnitudes is obtained. Providing this information to the LDPC decoder boosts the performance by 1.5dB, as seen in the diamonds curve. A second smoothing process, now with the corrected itinerary provides some more information regarding the transmitted magnitudes. If this information is exploited, performance is improved additional 0.1 dB, as seen in the asterisks curve. How far is the obtained performance from the case in which full magnitudes information is available? According to the simulation (squares curve), approximately 0.3 dB. Throughout the simulations and analysis we assumed that the filtering process provides a gain of 1.25 dB. However, this value is asymptotic, and the first samples in each row don’t enjoy this gain. By providing an artificial gain of 1.25 dB to all samples, instead of running the filtering process, the circles curve, with additional performance improvement of 0.3dB was obtained. Notice that even though this curve is artificial, we can still compare it with the Gaussian approximation bound (solid curve) since both assume the full 1.25dB filtering gain. The last two curves have close threshold values. Figure 7 presents the corresponding coding gains of the LDPC coded itinerary method vs. SNR. The asterisks curve represents the obtained performance using two LDPC decoding passes. The first one uses magnitudes obtained by smoothing as in [1] and the second uses magnitudes obtained by smoothing with the corrected itinerary. We obtain a threshold of 4.2 dB, which is a tremendous improvement relative to the previous methods shown in Figures 1 and 4. Next we analyzed the effect of changing the LDPC code rate while keeping the bandwidth expansion factor, N2+N3, constantly at 8 and the number of source samples at N1=250. Figure 8 presents the results. The solid line is the coding gain bound (15). The asterisks graph represents no coding of the itinerary and is identical to the method presented previously in [1]. The /06/2007
remaining graphs show the obtained coding gain when the LPDC rate is gradually increased. While there is still a substantial distance of the obtained results from the theoretical bound, they significantly outperform previously published methods.
VIII. Discussion By comparing the performance of our analog coding scheme with the theoretical ratedistortion bound as shown for example in Figure 8, we see that the gap is quite significant. This is disturbing especially due to the fact that a powerful LDPC code has been employed. In addition, the coding gain of our proposed coding scheme saturates while the bound does not. Regarding the first problem, while there are several elements, which contribute to the gap, we believe that the main element is the information embedded in the magnitudes, which is hardly being exploited. The value of the last sample in each row is backward iterated according to signs extracted from the received array. All other magnitudes provide information for filtering the received sequence, which eventually yields a coding gain bounded by only 1.25 dB.
We
strongly believe that filtering gain may be significantly larger for fading channels, a topic which is not covered in this contribution. Regarding the second problem, we have to recall the roles of the N2-1 forward iterations and the N3 backward iterations applied to each of the source samples. The forward iterations are responsible for the asymptotic coding gain. The backward iterations are responsible for removing the erratic local minima in the estimation problem, by coding the itinerary. The amount of redundancy needed (N3) is a function of the channel SNR. As SNR improves, the required number of backward iterations decreases. Therefore, for a given bandwidth expansion (N2 + N3), more and more backward iterations may be traded by forward iterations as SNR increases. This tradeoff is the key to increasing coding gain, as a function of SNR. Since we don’t assume any knowledge at the transmitter regarding the channel SNR, our coding gain saturates.
/06/2007
IX. Concluding Remarks We have presented a combination of analog and digital error-correcting codes. The combination enjoys the benefits of both analog and digital worlds. The digital error correction field is a mature topic, which has enjoyed massive research through the years, but on the other hand, analog error-correcting codes have failed to attract much interest. Transmission of analog signals doesn’t involve quantization. Therefore, as long as the channel SNR is above some threshold level, the distortion of the received signal monotonically decreases with the increase of the SNR. Even though this paper considered channels with a constant signal to noise ratio, it is speculated that the combination of joint analog and digital error correction is advantageous especially in varying SNR conditions. Further research in this field is likely to be fruitful. The proposed combination may prove to be efficient in error concealment applications. While digital errors tend to generate pronounced effects, analog errors usually generate a much softer effect. Finally, the field of joint source and channel coding may benefit from the combined approach. Traditionally, source compression is done after quantization. Once the information is represented in a digital format it is usually best to transmit it using standard digital modulations. However, lossy compression techniques such as discrete cosine transform (DCT) and linear prediction coding (LPC) represent the information via coefficients. These coefficients may be treated as analog signals and thus suitable for joint analog-digital channel coding.
Appendix A. Performance analysis for Diversity Coded Itineraries We start by deriving the upper bound on the sign error probability in equation (22). In order to analyze the performance of this scheme we assume, without loss of generality, that the two corresponding values in the source and parity trajectories are positive, i.e.:
zm , z
m 1
0.
(43)
The sum of the estimated values as appears in (22) is given by /06/2007
ˆz m | m
ˆz
m 1| m 1
zm
z
m 1
um u
m 1
,
(44)
where um and u-m-1 denote the estimation errors (20). We shall use the following notation for short:
~z u~
zm
z
um u
m 1
,
(45)
1.
m
The distribution of ~z is triangular and is given by
~z
0 ~ z 1 0 ~ ~ 2 z 1 z 2 . 0 0 otherwise
f Z~ ~z
(46)
The distribution of ~ u is a zero mean Gaussian with variance 2 u~
2 u
2
.
(47)
A sign error event will occur if the sum (44) is negative. Therefore the sign error probability is given by 1
PS ,Diversity
t
!t Q
2
!
dt
u~
0
2 t Q
t
2
dt
u~
1
!t
Q
t
dt
u~
0
(48)
2 2 u~
#
u~
!t
Q t dt
2 u~
0
!t
Q t dt
0
1 4
2 u~
1 2
2 u
2 v
3 8
1 . 8 SNR
which completes the derivation. The bound in (48) provides clear insight about the performance improvement, and it is apparently quite tight for medium SNR and up. The performance evaluation of the diversity coded itineraries method requires also evaluation of a sign error event effect on the variance of the associated sample,
2 ˆz|e ,Diversity
. Since the effect of a sign error event is to provide a fairly good
estimate of –z rather than z, the variance may be expressed as follows: 2 ˆz|e ,Diversity
1
! 2z
2
f z|e z | e dz ,
(49)
1
f z|e z | e
f e|z e | z
fz z
PS ,Diversity
,
(50) /06/2007
1
!Q
f e|z e | z
z t 2
0
) 1 z '1 z Q ' 2 u2 (
dt
2 u
z Q
& $ $ %
z 2
2 u
) 'e ' (
2 u
2 2
z2 2 u
4
1 z 4
e
2
2 u
& $ . (51) $ %
For medium and high SNR conditions, the limits of integral (49) may be pushed to infinity, and after a few manipulations we obtain the following tight bound: 2 ˆz|e ,Diversity
1 . SNR
(52)
Application of equation (13) and (14) with the quantities obtained in (48) and (52) provides the required coding gain.
Appendix B. Distribution of the Received Signals LLR The log-likelihood ratio of the received signals is given by (39) L
2
" 2 "u
2 u
g" u ,
(53) 2 u
where u has a zero mean Gaussian distribution with variance
, and " has a uniform
distribution in the interval (0,1) denoted by f " " . The CDF of the LLR is given by 1
FL
! f" "
Pr L
d" ,
Pr g" u
(54)
0
Pr g" u
Pr
2
" 2 "u
2 u
3 2 1
Pr u
2 u
2"
3
u
1
2"
"2 1 Q
"
.
(55)
u
Combining the last two equations: 1
FL
!1
Q
0
u
"
2"
d" .
(56)
u
And the PDF is obtained by differentiating the CDF with respect to : 1
fL
4 1 Q u ! 4 0 2"
" u
1
d"
4 u !0 4 Q 2"
" u
1
d"
! 0
1 e 2
1 u 2 2"
" u
2 u
2"
d" .
/06/2007
(57)
Appendix C. Distribution of The Variable Nodes Messages The variable nodes messages distribution is a convolution of the received samples LLR distribution (39) and a Gaussian distribution with mean * and variance of 2 * : )1 1 !#'' !0 2 e '(
#
!f
fQ q | *
|* d
fR q
L
#
1
! 0
1 2
1 u 2 2"
#
)# 1 u ' !e 4 * 2" ' # '(
"
1 u 2 2"
2
u
q
e
* 4*
2
"
2
& 1 d" $ e $ 2" 4 * $% u
u
*
q
4*
2
d
(58)
& d $ d" . $
%$The internal integral is evaluated as follows: #
!e
1 u 2 2"
2
"
q
e
u
* 4*
2
0 )1 u ' exp !# 0 '( 2 2"
#
d
#
2 0 ) u !#exp 0 '' 8" 2 (
#
1 4u
2
#
! exp
a
2
b
c d
exp
c
#
* 2 & 30
q
q * 4*
2 u
b2 3 2 4a 1
a
$ 2d $% 01
4*
u
"2
q 2*
2
2
"
2
& 30 $ 2d %$ 01
5
(59)
.
Finally: 1
fQ q | *
!2
1
u 2 u
0
1
!2 0
1
u 2 u
* 2"
2
* 2" 2
2 q * 0 ) 2 *" exp ' 4 * u2 0 '(
) 0 ' 1 0 exp ' 0 ' 4* 0 '(
4*
1 2
2 2 u 2
*
q2
2
q 2
"
2 u
1 q 2 * 2"
2
*
4
2 u
2
*" 2 & 30 $ 2d" 2 2" 2 $% 0 u* 1
2 u
&3 $ 00 $ 2d" . $0 $% 0 1
(60)
Acknowledgement The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. One of the authors (I.R.) wishes to thank Gil Shamir for helpful discussions on information theoretic aspects of the subject. References [1] B.Chen, G.W. Wornell, “Analog Error-Correction Codes Based on Chaotic Dynamical Systems”, IEEE Trans. on Communications, Vol. 46, No. 7, pp. 881-890, July 1998. /06/2007
[2] H.C.Papadopoulus, G.W. Wornell, “Maximum likelihood Estimation of a Class of Chaotic Signals”, IEEE Trans. on Information Theory, Vol. 41, No. 1, pp. 312-317, January 1995. [3] C.Pantaleon, D. Lunego, I. Santamaria, “Optimal Estimation of Chaotic Signals Generated by Piecewise-Linear Maps”, IEEE Sig. Processing Letters, Vol. 7, No. 8, pp. 235-237, August 2000. [4] S.Kay, V. Nagesha, “Methods for Chaotic Signal Estimation”, IEEE Trans. on Signal Processing, Vol. 43, No. 8, pp. 2013-2016, August 1995. [5] R.G.Gallager, “Low Density Parity Check Codes”, IRE Trans. on Information Theory, Vol. IT-8, pp. 21-28, January 1962. [6] D.J.C. MacKay, R.M. Neal, “Near Shannon Limit Performance of Low Density Parity Check Codes”, Electronic Letters, Vol. 32, pp. 1645-1646, 1996. [7] T.J.Richardson, M.A.Shokrollahi, R.L.Urbanke, “Design of Capacity-Approaching Irregular Low-Density Parity-Check Codes”, IEEE Trans. on Information Theory, Vol. 47, No. 2, pp. 619637, February. 2001. [8] W. Schwarz, M. Götz, K. Kelber, A.Abel, T. Falk, F. Dachselt , “Statistical Analysis and Design of Chaotic Systems”, in M.P.Kennedy, R.Rovatti, G. Setti, “Chaotic Electronics in
Telecommunications”, CRC press, 2000, chapter 9. [9] T.M.Cover, J.A.Thomas, “Elements of Information Theory”, New York, Wiley 1991. [10] M. Tanner, “A recursive approach to low complexity codes”, IEEE Trans. on Information
Theory, Vol. IT-27, pp. 533-547, September 1981. [11] T.J.Richardson, R.L.Urbanke, “The Capacity of Low-Density-Parity-Check Codes Under Message-Passing Decoding”, IEEE. Trans. on Information Theory, Vol. 47, No. 2, pp. 599-618, February. 2001. [12] M. Yang, W.E.Ryan, Y.Li, “Design of Efficiently Encodable Moderate-Length High-Rate Irregular LDPC”, IEEE Trans. on Communications, Vol. 52, No. 4, pp. 564-571, April 2004. [13] J.Campello, D.S.Modha, “Extended Bit-Filling and LDPC Code Design”, in Proc.
GlobeCom, San Antonio, TX, 2001. [14] S.Y. Chung, T.J. Richardson, R.L. Urbanke, “Analysis of Sum-Product Decoding of LowDensity Parity-Check Codes Using a Gaussian Approximation”, IEEE Trans. on. Information
Theory, Vol. 47, No. 2, pp. 657-670, February 2001. [15] J. Ziv, “The Behavior of Analog Communication Systems”, IEEE Trans. on Information
Theory, Vol. 16, No. 5, pp. 587-594, September 1970. [16] F. Peng, W.E. Ryan, R.D. Wesel, “Surrogate-Channel Design of Universal LDPC Codes”,
IEEE Communications Letters, Vol. 10, No. 6, pp. 480-482, June 2006.
/06/2007
Figure 1. Analytical coding gains vs. SNR of the analog error-correcting code for a scalar information source (N1=1) and no backwards iterations (N3=0).
/06/2007
Source
Channel Gain
1 x
AWGN
+
y
-1
Figure 2. A model of the itinerary as a binary information source and of the channel through which it is transmitted.
/06/2007
Figure 3. The capacity per symbol for the channel depicted in figure 2, taking into account noise reduction of 1.25 dB due to filtering gain.
/06/2007
Figure 4. Analytical coding gains of the analog error-correcting code for a scalar information source (N1=1) with diversity coded itineraries. The circles denote simulation results for N2=8 iterations.
/06/2007
N1 source samples
N3 backwards iterations with signs according to the parity bits for each source value
N2-1 forward iterations for each source value
Figure 5. An illustration of the analog code word array Z.
/06/2007
Figure 6. Sign error probability of an estimated chaotic itinerary vs. SNR. The solid line presents the Gaussian approximation assuming a filtering gain of 1.25 dB and known magnitudes. All other curves present simulation results corresponding to different cases as described in the legend of the figure and in the text.
/06/2007
Figure 7. Coding gain vs. SNR of the analog error-correcting code based on LDPC coding of the itineraries of the chaotic trajectories (from simulations.)
/06/2007
Figure 8. Coding gain vs. SNR of the analog error-correcting code based on LDPC coding of the itineraries of the chaotic trajectories for various LDPC code rates and a constant bandwidth expansion factor of 8.
/06/2007
Abstract
This contribution focuses on error-correcting codes for analog signals. Generally, error-correcting codes exhibit a threshold effect, which degrades performance at the low signal to noise ratio (SNR) range. We propose two error correction alternatives based on chaotic dynamical systems, which improve threshold performance relative to previously proposed methods. The first one, which is based on principles of diversity, exhibits a very simple decoder. The second alternative, which is based on a low-density-parity-check (LDPC) code, attains a significantly lower threshold. Theoretical analysis and simulations are provided.
Index Terms – analog signals, error correction, chaotic systems, diversity, LDPC, threshold extension.
Isaac Rosenhouse is with Ceragon Networks, Israel. This research was supported by the Israel Science Foundation (grant No. 1232/04)
/06/2007
I. Introduction In the last five decades, numerous error-correcting codes for digital information sources have been developed. However, when the information source is analog (i.e., continuous-valued) quantization is applied prior to coding. This approach is optimal when the channel signal to noise ratio (SNR) is known at the transmitter. The quantization and coding are done according to the channel’s capacity. However, when the SNR is not known the optimal number of quantization levels cannot be determined. Cases in which the transmitter does not know the SNR include broadcast channels, where there are multiple receivers each with different SNR, as well as systems operating in timeselective fading, where feedback from the receiver is not available. An alternative approach to the simple quantization and coding is error correction in the analog signals domain. In this case no quantization takes place. Chen & Wornell [1] proposed an analog error correcting code based on chaotic dynamics. Their algorithm encodes an analog scalar value into a vector of analog values by generating a discrete time chaotic trajectory originating in the scalar value. Several papers, including [2],[3],[4], examined the estimation of the chaotic trajectory’s initial value. Indeed, their scheme achieves coding gain, but a threshold effect limits the gain, especially for low-rate codes. The threshold effect is a consequence of errors in the estimated signs (itinerary) of the chaotic trajectory. The authors of [1] discuss the threshold, and its effect on the sign error probability and on the resulting mean square error of the estimated analog values. However, we are not aware of any attempt to reduce the threshold effect, so far. In this work we explore efficient ways for coding the signs embedded in the chaotic trajectory. We employ longer chaotic trajectories with redundancy embedded in their itineraries. The error correction scheme exploits the redundancy and yields better-estimated itinerary. We refer to the first type of redundancy as diversity. The chaotic sequence is obtained by iterating a scalar value using a symmetric tent map [2]. In addition, iterating the scalar value /06/2007
backwards with an equal number of iterations doubles the sequence length. One can choose the signs of the backward iterated values arbitrarily. Here we choose them to be the signs of the forward iterations multiplied by a pseudo random signs sequence. This arrangement provides the desired diversity. Although the performance of this method is not optimal, it provides an additional coding gain at the low SNR range while maintaining a very simple decoder. The second type of redundancy we explore provides excellent coding gain but requires a more sophisticated decoder. It is based on a low-density-parity-check (LDPC) code. The LDPC code was originally introduced by Gallager [5] in 1962 and was rediscovered by MacKay et al. [6] in 1996. It exhibits performance extremely close to the best possible as determined by the Shannon capacity expression [7]. Since good coding requires long code words we shall encode a vector of analog samples rather than a single sample. We perform forward and backwards iterations on each of the vector entries. The signs of the backwards-iterated samples are determined according to the parity bits of a systematic LDPC code. The information bits for the LDPC encoder are the signs of all forward iterated samples. We show by means of analysis and simulation that this arrangement provides excellent coding gain at the low SNR range. The paper is organized as follows. In Section II we formulate the system and review some previous results. We demonstrate the threshold effect and motivate coding of the itineraries. Section III presents a capacity bound on the threshold SNR. Section IV presents the diversity method for reducing the threshold effect. Sections V - VII present the LDPC code based approach for threshold effect reduction. In Section VIII we discuss the elements responsible for the gap between the theoretical bounds and the achieved performance. Section IX includes our concluding remarks.
II. Formulation of the System and Review of Previous Results We consider transmission of N1 samples of a random signal distributed uniformly in the open interval (-1,1) over an additive white Gaussian noise (AWGN) channel. The signal samples are /06/2007
denoted by {zn,0} where n = 0,1,…N1-1. Throughout the paper we refer to them as the source samples. Each of the N1 source samples is iterated (N2-1) times using a tent map
z defined as
follows:
z
1 2z .
(1)
The ensemble of trajectories generated by this function is white and also distributed uniformly in the range (-1,1) [8]. The trajectories are denoted by {zn,m}, where m = 0,1,…N2-1 and n =
0,1,…N1-1. The mathematical relation between the entries of the trajectories is given by z n ,m
z n ,m
1
m 1,2 ,...N 2 1 . n 0 ,1,...N 1 1
(2)
In order to define an inverse function for
z , we must specify a sign for the obtained value. We
use the subscript s to denote it. The sign of zn,m is denoted by s[n,m] . Therefore, the function 1 s [ n ,m ]
z n ,m
z n ,m
1
1 z n ,m 1 s n ,m , 2 1 z n ,m 0 sign( z n ,m ) . 1 z n ,m 0
1 s [ n ,m ]
s [ n ,m ]
is defined as
z n ,m
1
(3)
In addition to the forward iterations (2), we perform N3 backwards-iterations on each of the N1 source samples according to a given signs array s[n,m] :
z n ,m
1 s [ n ,m ]
z n ,m
1
m
N 3 , N 3 1,... 1 . n 0 ,1,...N 1 1
(4)
Finally, we transmit the whole array {zn,m}, m=-N3,…N2-1, n=0,1,…N1-1 through an AWGN channel with noise variance of
yn,m
z n,m
2 v
. The received signal is denoted by {yn,m} and given by
vn,m ,
(5)
where vn,m stands for the noise samples. The source vector is estimated in the receiver using the received array. The samples obtained by the forward and backwards iterations may be considered as “parity samples” in a systematic /06/2007
analog error-correcting code. For the scalar case (N1 = 1) with no backwards iterations (N3=0) performance analysis has been carried out in [1] and [2]. We review the main results for this case with some adaptations to the formulation presented above. Maximum likelihood (ML) estimation of the source value is done in two steps. At the first step the signs and magnitudes of the transmitted trajectory are estimated via a filtering process. For simplicity, we shall omit the index n for scalar cases, i.e. zm
zn,m and ym
yn,m. Let ˆz k | j
denote the estimated value of zk based on the j+1 samples {ym}, m = 0,1,…j. The filtering process may be done recursively using the following expressions [2]
ˆz [ 0 | 0 ]
y0 ,
ˆz [ m | m 1 ] ˆz m | m
(6a)
ˆz m 1 | m 1 .
3 2 2 m ym
(6b)
2 2 m 1 ˆz m | m 1 4 2 2m 1
m
1
3 ym 4
1 ˆz m | m 1 . 4
(6c)
Notice that since the values of z are limited to the interval (-1,1), the estimates ˆz m | m should also be hard limited to this interval during the filtering process. The second step involves a smoothing process. It performs backwards iterations using the signs of the estimated samples obtained in the first step:
ˆz m | N 2 1
1 sˆ [ m ]
ˆz m 1 | N 2 1 ,
(7)
where ˆs m denotes the sign of ˆz m | m in equation (6). The Cramer-Rao lower bound (CRLB) on the variance of the estimates has been presented in [2]: var ˆz m | m var ˆz 0 | N 2 1
2 v
3 4 0.25 m 2 v
m
3 4
N2
1
1
3 4
2 v
,
(8)
.
(9)
As seen from equations (8) and (9), the filtering process reduces the noise effect by 1.25 dB, asymptotically, and the smoothing process reduces the noise further by an exponential factor of
N2. Therefore, the smoothing is the dominant element in the noise effect reduction. However, the /06/2007
smoothing process approaches the Cramer-Rao bound only asymptotically when there are practically no errors in the estimated signs ˆs m . We show in the sequel that the errors in the estimated signs are the dominant factor of the performance degradation of the proposed error correction scheme. The sign error probability is approximately [1]:
PS
1 2 2
SNR
,
(10)
where the SNR is defined as: 2 z 2 v
SNR and
2 z
1 3
2 v
,
(11)
is the variance of the transmitted signal.
An estimated value in the smoothing process gets a wrong sign if a sign error event (e) occurs. In this case the variance of its estimation error is approximately [1]: 2 ˆz|e
2 / 3 SNR
(12)
Let w denote the source value estimation error, i.e. w
ˆz0
z0 .
The variance of the estimation error is given by [1]: 2 w
var ˆz 0 | N 2 1
2 v
3 4 N2
1
1 PS
N2 1
2 ˆz|e
1
N
1 PS / 4 2 1 1 PS / 4
1
PS .
(13)
Observe that as the sign error probability decreases to zero the estimation error variance in (13) converges to the CRLB in (9). However, the sign error probability decays very slowly as the SNR increases as indicated by (10), and therefore, practically, the second term in (13) dominates the variance over a large range of SNR values. The coding gain (in dB) is defined by:
CG 10 log 10
2 v 2 w
10 log 10 N 2
N3 .
(14)
/06/2007
Even though output distortion or SNR may be regarded by some as a more fundamental performance measure of analog signals, we believe that coding gain is the natural approach here. It concisely presents the ratio of the output SNR to the channel SNR normalized by the bandwidth expansion (N2+N3). Figure 1 shows the analytical coding gain according to (14) as a function of SNR for different numbers of forward iterations. Observe that a threshold effect exists as predicted by equations (13) and (14). Even for a small number of forward iterations such as N2=4, the coding gain approaches its asymptotic limit at an SNR of approximately 50 dB. The additional N3 backwards-iterations should provide the necessary redundancy for coding the itinerary, and hence reduce considerably the threshold effect. Notice that the fact that the coding gain saturates at some point implies that the ratio between the output SNR and channel SNR converges to a constant, hence, as the channel SNR increases, the output SNR increases proportionally without limit. This behavior is typical for all coding schemes discussed in this contribution, and is in fact the whole point of using an analog scheme in the first place.
III. A Capacity Bound on the Threshold SNR A simple upper bound on the achievable coding gain is obtained by using a combination of the rate-distortion and channel capacity theorems. A bound on the distortion (
2 w
) has been
derived in [1]. With some slight adaptations and using the coding gain definition (14) we obtain 2 w
2 w ,Bound
CG CGBound
2 1 1 e 3 v2 10 log 10
N2 N3
,
(15a)
2 v
N2
N3
2 w ,Bound
.
(15b)
Notice that in the high SNR range (SNR>>1), the bound is approximated by:
/06/2007
CGBound
N2
N 3 1 10 log 10 SNR
10 log 10
6 N2 e
N3
,
(15c)
which implies that the coding gain need not saturate as a function of SNR as in our coding schemes. This phenomenon is discussed later on after we present our coding schemes. Achieving the coding gain bound requires optimal coding and in the case of N2+N3 > 2 there is no single coding scheme, which achieves it for all SNR values [15]. In the previous section we have motivated coding the itineraries of the chaotic trajectories by demonstrating the threshold effect caused by the sign errors. In this paper we confine ourselves to the case in which the required coding redundancy is embedded in itineraries of longer chaotic trajectories. Since maximum-likelihood decoding is exponentially complex, we resort to sub-optimal estimation of the transmitted signal. In our proposed method the estimation process begins with filtering as shown in the previous section and an itinerary decoder follows it. There are two main causes for the sub-optimality: The filtering processes disregards the code word embedded in the itinerary, and the itinerary decoder regards the magnitudes as uncorrelated random variables rather than a series of values evolving in a deterministic fashion. Under these simplifications we wish to obtain a bound on the threshold SNR for coded itineraries. We model the coded itinerary as a binary source with values of 1. We model the channel through which the itinerary is transmitted as a composition of two elements: a gain element and a noise element. The gain element multiplies the source by a random gain distributed uniformly in the range (0,1). The noise element is an AWGN element with variance 2 u
(Figure 2). Notice that the chaotic trajectory has a deterministic relation between the
magnitudes of its samples. This relation is exploited in the filtering process (6) in order to reduce the AWGN. Therefore, the noise variance in the model,
2 u
, represents the noise after filtering.
For capacity calculations we use the asymptotic relation in (8) for determining 2 u
3 4
2 v
, which implies a filtering gain of 1.25 dB. /06/2007
2 u
, i.e.
The amount of information which can be transmitted through a channel for a given source distribution is determined by the mutual information between its input (x) and its output (y). We refer to this quantity as the capacity per symbol and denote it by C [9]. It is given by
C
2
I X ;Y
!
p x xk
k 1
f y|x y xk
log 2 f y|x y xk
2
log 2
p x xn
f y|x y xn
dy .
(16)
n 1
Y
Here px(x) denotes the probability distribution of the source x, and fy|x(y|x) denotes the conditional probability density of the channel output (y) given the channel input (x). Throughout the analysis we distinguish between two cases: 1. The transmitted magnitudes are provided to the itinerary decoder. 2. The transmitted magnitudes are uniformly distributed in (0,1) and no other a priori information regarding them is provided to the itinerary decoder. In the first case the conditional probability is given by:
f y|x y | x
y "x
1
e
2 u
2
2
2
2 u
,
(17a)
where " designates the transmitted magnitude and the capacity is obtained by averaging it over #
!
the distribution of " , i.e. C
1
f " " C" d"
#
! C" d" , where C" is the capacity for a given " . 0
In the second case we have
f y|x y | x
1 2
1
y *x
!e
2
2 u
2 u 0
2
d*
1) 'Q x(
y
Q
u
x y & $, u %
(17b)
where Q(x) is the complementary error function defined in (31). For a symmetric source distribution i.e. px(1)=px(-1)=0.5, equation (16) simplifies to f y|x y | x 1
#
C
!f #
y|x
y | x 1 log 2
1 2
f y|x y | x 1
f y|x y | x
1
dy .
(18)
Equation (18) has been evaluated numerically for both cases and the results are shown in Fig. 3, which plots the capacity per symbol vs. SNR. Notice that the SNR in the figure is defined according to (11), that is, the ratio between the signal power after the gain multiplication and the /06/2007
AWGN power before filtering. Inspection of the figure reveals that using an error correction code with rate ½ the obtained threshold SNR values are approximately 0.65 dB and 2.05 dB for the known and unknown magnitudes respectively. Codes with rate ½ will be used in examples later on. In the introduction we stated that we deal with cases in which channel SNR is unknown to the transmitter. This contradicts the need to choose a code rate according to a given SNR. Strictly speaking, we still need to know that channel SNR is above some threshold, however, as long as this condition is fulfilled, the distortion of the estimated signal will decrease with any increase of the channel SNR.
IV. Diversity Based Coding of Itineraries A simple method for coding the itinerary of a single trajectory is based on generating another chaotic trajectory whose itinerary is related to the source itinerary by a pseudo random sequence known at both ends of the system. We refer to the second trajectory as the parity trajectory. By selecting a pseudo random itinerary for the parity trajectory, its magnitudes become statistically independent of the source trajectory. As we shall see this condition decreases the sign error probability by a factor of a square root of the SNR. Notice that we have here a sort of rate ½ repetition code, which will be replaced by an LDPC code in the following section. (Simple repetition codes do not have any coding gain, while this code has). Practically, the parity trajectory may be generated using N3=N2 backwards iterations starting from the source value. We denote the sign of a corresponding sample at the parity trajectory by
sign z
m 1
+ m sign z m ,
m = 0,1,…N2-1.
(19)
where + m denotes a pseudo-random sign. Estimation of the source value is done in three steps. First the received chaotic trajectory is filtered according to (6). The estimation error in this stage is designated by u: /06/2007
um
ˆz m | m
zm ,
ˆz m | m
where
(20)
denotes the estimated value of zm based on the received samples
y-N3,…ym. The variance of the estimation error generally depends on m, but for m >> -N3 it converges to a constant value as indicated by (8). In the following derivation and analysis we assume that it is independent of m and equals to 2 u
2 v
3
/4.
(21)
In the second step the itinerary of the transmitted sequence is estimated using,
ˆs m
ˆs m 1 + m
sign ˆz m | m
ˆz
m 1 | m 1 + m , m = 0,1,…N2-1.
(22)
Finally, smoothing is performed according to (7). The sign error probability is bounded by the following expression [appendix A]:
PS ,Diversity
8 SNR
1
.
(23)
Comparing (23) with (10), we see that the sign error probability for diversity-coded itineraries is reduced by a factor proportional to the square root of the SNR. As a consequence, the threshold SNR is reduced by a factor of approximately 2 on a logarithmic scale. Figure 4 shows the coding gain vs. SNR for different number of forward and backward iterations. The coding gain in this figure is defined in equation (14) where equation (13) is used for
2 w
and the expressions for PS ,Diversity and
2 ˆz|e ,Diversity
are provided in Appendix A in equations
(48) and (52) respectively. The effect of threshold reduction is evident by comparison with Figure 1. It is worth noting that the price paid for the threshold shift is a reduction in the asymptotic coding gain.
V. LDPC Coding of the Itineraries In order to approach the threshold bound presented in section III a highly efficient coding scheme should be employed. It is well known that in order to approach the Shannon limit, long information sequences should be coded. Therefore, in this section we discuss coding of a vector /06/2007
of analog samples as opposed to coding of a scalar analog sample that was considered in previous sections. Each of the N1 elements in the source vector is iterated forward N2-1 times using the tent map function (1). As a result we get an array of analog samples with N1 rows and N2 columns (2). We wish to encode the signs of the elements in the array. Negative values are mapped to one and non-negative values are mapped to zero. This information is digital, and therefore, standard digital encoding techniques may be employed here. The digital information source to be coded is denoted by a row vector x and its elements are defined as follows:
xn N 2
1 2
m
1 sign z n ,m ,
m 0 ,1,...N 2 1,
n 0 ,1,...N 1 1.
(24)
The vector x has K=N1 N2 elements. An LDPC code is a linear block code with a generator matrix GKxN and a parity check matrix H(N-K)xN. (The subscripts denote the size of the matrices). The code word c is generated by
c 1 xN
x1 xK G KxN .
(25)
The code rate is denoted by R=K/N. The generator matrix G should be expressed in a systematic form and the number of parity bits, denoted by M, should be an integer multiple of N1:
G
IK
M
N
PKxM , K
(26)
N1 N 3 .
(27)
Each of the N1 source samples is now iterated backwards N3 times using signs matching the parity bits generated by the encoder. We obtain an array, denoted by Z, composed of N1 rows and
(N2+N3) columns. Each row consists of a valid discrete time chaotic trajectory. In addition, the signs of the samples in Z consist of a valid LDPC code word. Figure 5 illustrates this structure. The code word array is transmitted through an AWGN channel (5). A few steps are employed in the estimation of the source vector. As seen in the previous cases, the first step is filtering according to (6). The filtering is done on a row-by-row basis. The second step involves /06/2007
error correction of the estimated signs. In this stage we can either assume that the magnitudes of the transmitted samples are unknown and uniformly distributed in the range (0,1) or perform smoothing as in [1] (7), and rely on the estimated magnitudes for decoding the LDPC code. In the second strategy the decoder is provided with additional information, however, due to itinerary estimation errors, it may be inaccurate at this stage. The preferred strategy is yet an open issue, however, our simulations suggest that for a rate ½ LDPC code with N2=N3=8 the second is definitely superior (See Figures 6,7). Anyhow, expressions for both decoding strategies are provided. The third step consists of a smoothing according to (7) using the decoded signs. Additional decoding-smoothing iterations may be applied while providing the decoder updated magnitude estimations from previous iterations. The process halts when a valid itinerary is obtained or when a maximal number of decoding-smoothing iterations have been performed. An itinerary may be validated by calculating the syndrome:
s
cˆ H T ,
(28)
where cˆ denotes the decoded codeword. For a valid codeword (itinerary) the syndrome is zero. As originally suggested by Tanner [10], LDPC codes are well represented by bipartite graphs in which one set of nodes, the variable nodes, corresponds to elements of the codeword and the other set of nodes, the check nodes, correspond to the set of parity-check constraints, which define the code. The first step of LDPC decoders based on message-passing is a calculation of the loglikelihood-ratio (LLR) of variable nodes. The LLR of bit i, which is denoted by Li, is defined as Li
ln
Pr ci
0| y
Pr ci
1| y
.
Here ci denotes the ith bit of the codeword and y denotes the vector of received samples. It is possible to derive a closed form expression for this LLR. However, this is beyond the scope of this paper. The derivation and performance will be shown in a future paper. In the derivation
/06/2007
provided here we shall use an approximation based on the filtered samples. Filtered sample i shall be denoted by ˆzi|i . The approximate LLR we propose is given by Li
ln
Pr ci
0 | ˆz i|i
Pr ci
1 | ˆzi|i
.
(29)
Let " i denote the magnitude of the transmitted sample corresponding to xi. Assuming the magnitudes are uniformly distributed unknowns at the receiver the LLR is given by 1 LUnknownMagnitude
ln
Pr zi
0 | ˆzi|i
Pr zi
0 | ˆzi|i
ln
2
e 2 !
2
2
2 u
dt
u 0
ˆzi|i t
0
1 2
ˆzi|i t
1
2 u
!
e
2
2 u
Q
dt
Q
1 ˆzi|i
u
ln
2
ˆz i|i
Q
1 ˆz i|i u
u
Q
ˆzi|i
, (30)
u
1
where the Q function is given by:
1 2
Qx
#
!e
t2 2
dt .
(31)
x
Equation (30) is easily derived using Bayes law assuming equal probabilities of zeros and ones in the codeword. Similarly, assuming the magnitudes of the transmitted samples, "i, are known at the receiver, we arrive at the following formula for the LLR: LKnownMagnitudes
2" i ˆz i|i 2 u
.
(32)
The following steps of the iterative decoder are identical to the standard algorithm. Let qij denote the message passed from variable node i to check node j. Let rji denote the message passed from check node j to variable node i. The message qij represents the log-likelihood ratio of variable node i, given the LLRs of the check nodes connected to it, rj’i, except check node j and given the channel LLR, L. It is given by qij
Li
(33)
r j'i j', j
The message rji represents the LLR of check node j, given the LLRs of all variable nodes connected to it except variable node i. It is given by /06/2007
r ji
2 tanh
1
- tanh i', i
qi'j 2
.
(34)
Equations (33) and (34) constitute one decoding iteration and each variable node message is initialized with the channel LLR, (30) or (32). Iterations are repeated until hard decisions on the variable nodes LLRs yield a valid code word, or until a maximal number of decoding iterations have been performed.
VI. Performance Evaluation of the LDPC Coded Itineraries Algorithm An elegant analysis method of LDPC codes has been proposed by Richardson & Urbanke in [11]. The method, known as density evolution, predicts a threshold phenomenon: as the codeword length tends to infinity, an arbitrary small error probability can be achieved if the noise level is smaller than a certain threshold. Conversely, for a noise level above this threshold, the error probability is larger than a positive constant. They proved a general concentration result, which shows that performance of an LDPC decoder on random graphs converges to its expected value as the length of the code increases. Therefore, analysis is done for an ensemble of infinitely long (i.e. cycle free) codes determined by a given length and degree distributions. Degree distribution refers to the degrees of the variable and check nodes. Specifically, let
.i denote the fraction of messages emerging from variable nodes with degree i, and let / i denote the fraction of messages emerging from check nodes with degree i. The vectors . and
/ determine the degree distribution of the ensemble. The method, which we use here, is based on approximating the message densities as Gaussian mixtures, and expressing them as a function of their mean as shown in [14]. While the Gaussian approximation holds for the check node messages, the variable node messages are no longer Gaussian in our case. Therefore, slight modifications are performed in order to adjust the
/06/2007
Gaussian approximation (GA) algorithm to our needs. The threshold is approximated and shows good agreement with simulation results. Before we proceed with the GA derivation we wish to emphasize that even though we treat the channel as memoriless it is by no means so. The chaotic trajectory evolves deterministically, and hence, so does the noise variance. If this memory had been taken into account in the decoding process and analysis, performance would have probably been improved. The analysis here is done under the assumption that the magnitudes of the transmitted symbols are distributed uniformly in (0,1) and are known at the receiver. This assumption is justified since the iterative analog error correction, as explained in the previous section, involves detection of the signs of the transmitted symbols as well as estimation of their magnitudes. Let q and r denote random variables corresponding to the probability density of the variable nodes messages (33) and the check nodes messages (34), respectively. Let mrh denote the mean of the check nodes messages after h iterations. Let Peh denote the bit error probability after h iterations. The Gaussian Approximation method provides an expression for the bit error probability at iteration h+1 as a nonlinear function of the bit error probability after h iterations. An LDPC code is a linear block code and therefore the performance depends only on the additive noise. Throughout the analysis it is assumed that the sent message is the zero vector, i.e. all the transmitted symbols have non-negative signs. Given Peh , it can be shown that the following check node messages have a symmetric Gaussian distribution fm(r) with mean m
fm r
mrh
1
1 e 4 m 2Q
1
1 2
Wc
r m 4m
mrh
1
[14]. The distribution is given by
2
/ j 1 1 2 Peh
(35)
j 1
(36)
j 2
Let the distribution of the initial variable nodes messages (32) be denoted by f L
. Consider
a variable node with degree i. From equation (33) it is evident that the distribution of its /06/2007
and a symmetric Gaussian with mean i 1 mrh 1 . For
messages is a convolution between f L
short we denote this mean value by * . The resulting distribution is denoted by f Q q | * . The initial error probability is given by (see (10) and [1]) Pe0
0
!f
1
d
L
2 2
#
,
SNR
(37)
and the error probability at iteration h+1 is given by h 1 e
P
Wv
Pr q 0
0
!f
.i
i 2
Q
q|*
i 1 mrh
1
dq ,
(38)
#
where Wv is the maximal variable node degree. Equations (37), (36) and (38) constitute a set of nonlinear iterative functions which enable calculation of the messages densities and the corresponding error probabilities. Explicit expressions for the quantities involved in these equations were derived and they are presented below. The log-likelihood function of the received samples is given by 2" i ˆz i|i
LKnownMgnitudes 2 u
Recall that
2" i " i xi
2 u
2 u
ui
2 x
1
2 u
" i2 " i u i .
(39)
denotes the noise variance after the filtering process (6). It is related to the channel 2 v
noise variance
through equation (21) and to the channel SNR through equation (11).
The distribution of these values is given by [Appendix B]:
fL
1 2
1
!e
"
1 u 2 2"
2 u
u
2"
0
d"
(40)
The distribution of the variable nodes messages conditioned by * is given by [Appendix C]
1
fQ q | *
!2 0
1
u 2 u
* 2"
2
) 0 ' 1 0 exp ' 0 ' 4* 0 '(
4*
1 2
2 2 u 2
*
"
q2
q 2
2"
2
*
4
2 u
2 u
&3 $ 00 $ 2d" . $0 $% 0 1
Finally, the associated bit error rate is given by: /06/2007
(41)
0
!
1
f Q q | * dq
#
!4 0
g 1 ( * ," ,
g 2 ( * ," ,
2 u
2 u
)
)
g 2 ( * ," , g 1 ( * ," ,
1 4*
)
2 u
4* 4
2 u
2 u
2 u
* 2" 2
)
1 2
exp
1 16 g 1 ( * ," ,
2 u
)
1 g 2 ( * ," ,
2 u
3 1 2 Q )1 2 g 1 ( * ," ,
2 u
)
d" ,
,
2 2 u 2
*
(42a)
(42b)
" .
(42c)
These expressions may be evaluated numerically. The integrals are defined over a finite range, and the integrands are fairly smooth.
VII. Simulation Results In our first example we used the following parameters: The size of the analog source vector
N1=125; the number of forward iterations per source sample N2-1=7; the number of backwardsiterations per source sample N3=8. Hence, the length of the LDPC codeword is
N1 (N2+N3)=2000, and the LDPC code rate is 0.5. Notice that in terms of the analog signal, the coding rate is 1/(N2+N3)=1/16 and the bandwidth expansion factor is (N2+N3)=16. The LDPC code has been designed according to algorithms presented in [12] and [13]. The optimal degree distributions for rate-½ codes with a maximum variable nodes and check nodes weight of 7 using GA over a simple AWGN channel [14] are:
/ x
0.4 x 5 0.6 x 6 and . x
0.3078 x 0.27287 x 2 0.41933 x 6 .
Even though our channel is not a simple AWGN, the code used here is considered good also for Rayleigh fading channels [16], which resemble our channel to some extent, and therefore, the same distribution has been used. The maximal number of iterations of the LDPC decoder was set to 50. Figure 6 presents the sign error probability of the LPDC coded itineraries vs. SNR for several cases. Notice that we use the SNR as defined in equation (11) in order to maintain a unified basis for comparison between the various results. The dashed line is related to the case in which the /06/2007
transmitted magnitudes are considered a-priori unknowns. The lack of magnitudes information is probably the reason that the LDPC decoder yields the worst sign-error probability. If smoothing is applied according to [1], some information regarding the transmitted magnitudes is obtained. Providing this information to the LDPC decoder boosts the performance by 1.5dB, as seen in the diamonds curve. A second smoothing process, now with the corrected itinerary provides some more information regarding the transmitted magnitudes. If this information is exploited, performance is improved additional 0.1 dB, as seen in the asterisks curve. How far is the obtained performance from the case in which full magnitudes information is available? According to the simulation (squares curve), approximately 0.3 dB. Throughout the simulations and analysis we assumed that the filtering process provides a gain of 1.25 dB. However, this value is asymptotic, and the first samples in each row don’t enjoy this gain. By providing an artificial gain of 1.25 dB to all samples, instead of running the filtering process, the circles curve, with additional performance improvement of 0.3dB was obtained. Notice that even though this curve is artificial, we can still compare it with the Gaussian approximation bound (solid curve) since both assume the full 1.25dB filtering gain. The last two curves have close threshold values. Figure 7 presents the corresponding coding gains of the LDPC coded itinerary method vs. SNR. The asterisks curve represents the obtained performance using two LDPC decoding passes. The first one uses magnitudes obtained by smoothing as in [1] and the second uses magnitudes obtained by smoothing with the corrected itinerary. We obtain a threshold of 4.2 dB, which is a tremendous improvement relative to the previous methods shown in Figures 1 and 4. Next we analyzed the effect of changing the LDPC code rate while keeping the bandwidth expansion factor, N2+N3, constantly at 8 and the number of source samples at N1=250. Figure 8 presents the results. The solid line is the coding gain bound (15). The asterisks graph represents no coding of the itinerary and is identical to the method presented previously in [1]. The /06/2007
remaining graphs show the obtained coding gain when the LPDC rate is gradually increased. While there is still a substantial distance of the obtained results from the theoretical bound, they significantly outperform previously published methods.
VIII. Discussion By comparing the performance of our analog coding scheme with the theoretical ratedistortion bound as shown for example in Figure 8, we see that the gap is quite significant. This is disturbing especially due to the fact that a powerful LDPC code has been employed. In addition, the coding gain of our proposed coding scheme saturates while the bound does not. Regarding the first problem, while there are several elements, which contribute to the gap, we believe that the main element is the information embedded in the magnitudes, which is hardly being exploited. The value of the last sample in each row is backward iterated according to signs extracted from the received array. All other magnitudes provide information for filtering the received sequence, which eventually yields a coding gain bounded by only 1.25 dB.
We
strongly believe that filtering gain may be significantly larger for fading channels, a topic which is not covered in this contribution. Regarding the second problem, we have to recall the roles of the N2-1 forward iterations and the N3 backward iterations applied to each of the source samples. The forward iterations are responsible for the asymptotic coding gain. The backward iterations are responsible for removing the erratic local minima in the estimation problem, by coding the itinerary. The amount of redundancy needed (N3) is a function of the channel SNR. As SNR improves, the required number of backward iterations decreases. Therefore, for a given bandwidth expansion (N2 + N3), more and more backward iterations may be traded by forward iterations as SNR increases. This tradeoff is the key to increasing coding gain, as a function of SNR. Since we don’t assume any knowledge at the transmitter regarding the channel SNR, our coding gain saturates.
/06/2007
IX. Concluding Remarks We have presented a combination of analog and digital error-correcting codes. The combination enjoys the benefits of both analog and digital worlds. The digital error correction field is a mature topic, which has enjoyed massive research through the years, but on the other hand, analog error-correcting codes have failed to attract much interest. Transmission of analog signals doesn’t involve quantization. Therefore, as long as the channel SNR is above some threshold level, the distortion of the received signal monotonically decreases with the increase of the SNR. Even though this paper considered channels with a constant signal to noise ratio, it is speculated that the combination of joint analog and digital error correction is advantageous especially in varying SNR conditions. Further research in this field is likely to be fruitful. The proposed combination may prove to be efficient in error concealment applications. While digital errors tend to generate pronounced effects, analog errors usually generate a much softer effect. Finally, the field of joint source and channel coding may benefit from the combined approach. Traditionally, source compression is done after quantization. Once the information is represented in a digital format it is usually best to transmit it using standard digital modulations. However, lossy compression techniques such as discrete cosine transform (DCT) and linear prediction coding (LPC) represent the information via coefficients. These coefficients may be treated as analog signals and thus suitable for joint analog-digital channel coding.
Appendix A. Performance analysis for Diversity Coded Itineraries We start by deriving the upper bound on the sign error probability in equation (22). In order to analyze the performance of this scheme we assume, without loss of generality, that the two corresponding values in the source and parity trajectories are positive, i.e.:
zm , z
m 1
0.
(43)
The sum of the estimated values as appears in (22) is given by /06/2007
ˆz m | m
ˆz
m 1| m 1
zm
z
m 1
um u
m 1
,
(44)
where um and u-m-1 denote the estimation errors (20). We shall use the following notation for short:
~z u~
zm
z
um u
m 1
,
(45)
1.
m
The distribution of ~z is triangular and is given by
~z
0 ~ z 1 0 ~ ~ 2 z 1 z 2 . 0 0 otherwise
f Z~ ~z
(46)
The distribution of ~ u is a zero mean Gaussian with variance 2 u~
2 u
2
.
(47)
A sign error event will occur if the sum (44) is negative. Therefore the sign error probability is given by 1
PS ,Diversity
t
!t Q
2
!
dt
u~
0
2 t Q
t
2
dt
u~
1
!t
Q
t
dt
u~
0
(48)
2 2 u~
#
u~
!t
Q t dt
2 u~
0
!t
Q t dt
0
1 4
2 u~
1 2
2 u
2 v
3 8
1 . 8 SNR
which completes the derivation. The bound in (48) provides clear insight about the performance improvement, and it is apparently quite tight for medium SNR and up. The performance evaluation of the diversity coded itineraries method requires also evaluation of a sign error event effect on the variance of the associated sample,
2 ˆz|e ,Diversity
. Since the effect of a sign error event is to provide a fairly good
estimate of –z rather than z, the variance may be expressed as follows: 2 ˆz|e ,Diversity
1
! 2z
2
f z|e z | e dz ,
(49)
1
f z|e z | e
f e|z e | z
fz z
PS ,Diversity
,
(50) /06/2007
1
!Q
f e|z e | z
z t 2
0
) 1 z '1 z Q ' 2 u2 (
dt
2 u
z Q
& $ $ %
z 2
2 u
) 'e ' (
2 u
2 2
z2 2 u
4
1 z 4
e
2
2 u
& $ . (51) $ %
For medium and high SNR conditions, the limits of integral (49) may be pushed to infinity, and after a few manipulations we obtain the following tight bound: 2 ˆz|e ,Diversity
1 . SNR
(52)
Application of equation (13) and (14) with the quantities obtained in (48) and (52) provides the required coding gain.
Appendix B. Distribution of the Received Signals LLR The log-likelihood ratio of the received signals is given by (39) L
2
" 2 "u
2 u
g" u ,
(53) 2 u
where u has a zero mean Gaussian distribution with variance
, and " has a uniform
distribution in the interval (0,1) denoted by f " " . The CDF of the LLR is given by 1
FL
! f" "
Pr L
d" ,
Pr g" u
(54)
0
Pr g" u
Pr
2
" 2 "u
2 u
3 2 1
Pr u
2 u
2"
3
u
1
2"
"2 1 Q
"
.
(55)
u
Combining the last two equations: 1
FL
!1
Q
0
u
"
2"
d" .
(56)
u
And the PDF is obtained by differentiating the CDF with respect to : 1
fL
4 1 Q u ! 4 0 2"
" u
1
d"
4 u !0 4 Q 2"
" u
1
d"
! 0
1 e 2
1 u 2 2"
" u
2 u
2"
d" .
/06/2007
(57)
Appendix C. Distribution of The Variable Nodes Messages The variable nodes messages distribution is a convolution of the received samples LLR distribution (39) and a Gaussian distribution with mean * and variance of 2 * : )1 1 !#'' !0 2 e '(
#
!f
fQ q | *
|* d
fR q
L
#
1
! 0
1 2
1 u 2 2"
#
)# 1 u ' !e 4 * 2" ' # '(
"
1 u 2 2"
2
u
q
e
* 4*
2
"
2
& 1 d" $ e $ 2" 4 * $% u
u
*
q
4*
2
d
(58)
& d $ d" . $
%$The internal integral is evaluated as follows: #
!e
1 u 2 2"
2
"
q
e
u
* 4*
2
0 )1 u ' exp !# 0 '( 2 2"
#
d
#
2 0 ) u !#exp 0 '' 8" 2 (
#
1 4u
2
#
! exp
a
2
b
c d
exp
c
#
* 2 & 30
q
q * 4*
2 u
b2 3 2 4a 1
a
$ 2d $% 01
4*
u
"2
q 2*
2
2
"
2
& 30 $ 2d %$ 01
5
(59)
.
Finally: 1
fQ q | *
!2
1
u 2 u
0
1
!2 0
1
u 2 u
* 2"
2
* 2" 2
2 q * 0 ) 2 *" exp ' 4 * u2 0 '(
) 0 ' 1 0 exp ' 0 ' 4* 0 '(
4*
1 2
2 2 u 2
*
q2
2
q 2
"
2 u
1 q 2 * 2"
2
*
4
2 u
2
*" 2 & 30 $ 2d" 2 2" 2 $% 0 u* 1
2 u
&3 $ 00 $ 2d" . $0 $% 0 1
(60)
Acknowledgement The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. One of the authors (I.R.) wishes to thank Gil Shamir for helpful discussions on information theoretic aspects of the subject. References [1] B.Chen, G.W. Wornell, “Analog Error-Correction Codes Based on Chaotic Dynamical Systems”, IEEE Trans. on Communications, Vol. 46, No. 7, pp. 881-890, July 1998. /06/2007
[2] H.C.Papadopoulus, G.W. Wornell, “Maximum likelihood Estimation of a Class of Chaotic Signals”, IEEE Trans. on Information Theory, Vol. 41, No. 1, pp. 312-317, January 1995. [3] C.Pantaleon, D. Lunego, I. Santamaria, “Optimal Estimation of Chaotic Signals Generated by Piecewise-Linear Maps”, IEEE Sig. Processing Letters, Vol. 7, No. 8, pp. 235-237, August 2000. [4] S.Kay, V. Nagesha, “Methods for Chaotic Signal Estimation”, IEEE Trans. on Signal Processing, Vol. 43, No. 8, pp. 2013-2016, August 1995. [5] R.G.Gallager, “Low Density Parity Check Codes”, IRE Trans. on Information Theory, Vol. IT-8, pp. 21-28, January 1962. [6] D.J.C. MacKay, R.M. Neal, “Near Shannon Limit Performance of Low Density Parity Check Codes”, Electronic Letters, Vol. 32, pp. 1645-1646, 1996. [7] T.J.Richardson, M.A.Shokrollahi, R.L.Urbanke, “Design of Capacity-Approaching Irregular Low-Density Parity-Check Codes”, IEEE Trans. on Information Theory, Vol. 47, No. 2, pp. 619637, February. 2001. [8] W. Schwarz, M. Götz, K. Kelber, A.Abel, T. Falk, F. Dachselt , “Statistical Analysis and Design of Chaotic Systems”, in M.P.Kennedy, R.Rovatti, G. Setti, “Chaotic Electronics in
Telecommunications”, CRC press, 2000, chapter 9. [9] T.M.Cover, J.A.Thomas, “Elements of Information Theory”, New York, Wiley 1991. [10] M. Tanner, “A recursive approach to low complexity codes”, IEEE Trans. on Information
Theory, Vol. IT-27, pp. 533-547, September 1981. [11] T.J.Richardson, R.L.Urbanke, “The Capacity of Low-Density-Parity-Check Codes Under Message-Passing Decoding”, IEEE. Trans. on Information Theory, Vol. 47, No. 2, pp. 599-618, February. 2001. [12] M. Yang, W.E.Ryan, Y.Li, “Design of Efficiently Encodable Moderate-Length High-Rate Irregular LDPC”, IEEE Trans. on Communications, Vol. 52, No. 4, pp. 564-571, April 2004. [13] J.Campello, D.S.Modha, “Extended Bit-Filling and LDPC Code Design”, in Proc.
GlobeCom, San Antonio, TX, 2001. [14] S.Y. Chung, T.J. Richardson, R.L. Urbanke, “Analysis of Sum-Product Decoding of LowDensity Parity-Check Codes Using a Gaussian Approximation”, IEEE Trans. on. Information
Theory, Vol. 47, No. 2, pp. 657-670, February 2001. [15] J. Ziv, “The Behavior of Analog Communication Systems”, IEEE Trans. on Information
Theory, Vol. 16, No. 5, pp. 587-594, September 1970. [16] F. Peng, W.E. Ryan, R.D. Wesel, “Surrogate-Channel Design of Universal LDPC Codes”,
IEEE Communications Letters, Vol. 10, No. 6, pp. 480-482, June 2006.
/06/2007
Figure 1. Analytical coding gains vs. SNR of the analog error-correcting code for a scalar information source (N1=1) and no backwards iterations (N3=0).
/06/2007
Source
Channel Gain
1 x
AWGN
+
y
-1
Figure 2. A model of the itinerary as a binary information source and of the channel through which it is transmitted.
/06/2007
Figure 3. The capacity per symbol for the channel depicted in figure 2, taking into account noise reduction of 1.25 dB due to filtering gain.
/06/2007
Figure 4. Analytical coding gains of the analog error-correcting code for a scalar information source (N1=1) with diversity coded itineraries. The circles denote simulation results for N2=8 iterations.
/06/2007
N1 source samples
N3 backwards iterations with signs according to the parity bits for each source value
N2-1 forward iterations for each source value
Figure 5. An illustration of the analog code word array Z.
/06/2007
Figure 6. Sign error probability of an estimated chaotic itinerary vs. SNR. The solid line presents the Gaussian approximation assuming a filtering gain of 1.25 dB and known magnitudes. All other curves present simulation results corresponding to different cases as described in the legend of the figure and in the text.
/06/2007
Figure 7. Coding gain vs. SNR of the analog error-correcting code based on LDPC coding of the itineraries of the chaotic trajectories (from simulations.)
/06/2007
Figure 8. Coding gain vs. SNR of the analog error-correcting code based on LDPC coding of the itineraries of the chaotic trajectories for various LDPC code rates and a constant bandwidth expansion factor of 8.
/06/2007
