On the symmetric information rate of two ... - Semantic Scholar
ITW2003,Paris, France, March 31 - April 4, 2003
On the Symmetric Information Rate of Two-Dimensional Finite State IS1 Channels Jiangxin Chen and Paul H. Siege1 Department of Electrical and Computer Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093, USA [email protected],[email protected] Abstract - We derive upper and lower bounds on the there is no direct counterpart of the BCJR algorithm that can symmetric information rate of a two-dimensional finite- simplify the APP calculations for a large sample output m a y state intersymbol-interference(ISI) channel model. in the two-dimensional setting. To overcome this difficulty, we derive upper and lower bounds on the entropy rate of a I. INTRODUCTION large output array based upon conditional entropies of smaller One approach to achieving higher information storage den- output arrays. By adapting the one-dimensional Monte Carlo sity is the use of page-oriented data recording technologies, technique to the calculation of these conditional entropies, we such as holographic memory. Instead of recording the data in are able to compute fairly tight upper and lower bounds on one-dimensional tracks, these technologies store the data on the SIR for two-dimensional IS1 channels with small impulse two dimensional surfaces. A commonly used model for such response support. The paper is organized as follows. In Section 11,we derive a two-dimensional recording channel is the two-dimensional finite-state IS1 channel with additive white Gaussian noise the upper and lower bounds on the SIR of two-dimensionalfinite state IS1 channels. Section I11 describes the Monte Carlo (AWGN), as described by approach to computing these bounds. Section IV provides num n merical results for a two-dimensional IS1 channel whose impulse response is supported on a 2 x 2 region. Section V concludes the paper. where z[i,j] is the channel input with finite alphabet, y[i,j] IS1 11. BOUNDSON THE SIR OF TWO-DIMENSIONAL is the channel output, and n[i,j] is i.i.d, zero-mean Gaussian CHANNELS noise with variance g2 = No/2. As in the case of one-dimensional channels, the capacity A. Conditional Entropies of Two-Dimensional Arrays of this two-dimensional channel is defined as the maximum The input array { X [ i ,j ] } is a two-dimensional discrete ranmutual information rate I ( X ,y ) over all input distributions dom process consisting of i.i.d random variables. The output where X = { z [ i , j ] } and y = { g [ i , j ] } . When the input array Y,, = { Y [ l I], , . . . ,Y [ l , n ]Y;[ 2 , 1 3 , .. . ,Y [ m ,n ] } is a is i.i.d., equiprobable, the mutual information rate is called two-dimensional continuous random process. It is easy to see the symmetric information rate (SIR). The capacity and the that both are stationary random processes. The mutual inforSIR provide useful measures of the storage density that can, mation rate between X and Y is in principle, be achieved with page-oriented technologies, and also serve as performance benchmarks for channel coding and I ( X ; Y ) = H ( Y ) - H ( Y l X ) = H ( Y ) - H(N), (2) detection methods. Various bounds on the capacity and the SIR have been de- where H ( Y ) = limm,n+oo &H(Ymn) is the entropy rate veloped for certain one-dimensional IS1 channels; see, e.g. of the output process and H ( N ) = log(aeN0) is the noise [11-[4]. Recently, several authors independently proposed a entropy rate. new Monte-Carlo approach to calculating a convergent seFor a stationary one-dimensional random process Z = The {Z[i]}, quence of lower bounds on these informationrates [5]-[7]. the limit H ( Z ) = limn+w ;H(Z[I], . . .,~ [ n ex]) method requires the simulation of an a posteriori probability ists. It is not difficult to show that limm,n+w &H(Ymn) also (APP) detector, matched to the information source and chan- exists when { Y [i,j]} is stationary. nel, on a long sample realization of the channel output. The Although we could estimate H ( y ) by calculating the samforward recursion of the sum-product (BCJR) algorithm [8], ple entropy rate of a very large array, similar to the approach applied to the combined source-channeltrellis, can be used to used in [SI-[7],the huge computational complexity makes it reduce the overall computational complexity of the APP cal- impractical. Instead, we will use conditional entropies based culations. The method has been further extended to evaluate upon a smaller array to derive upper and lower bounds on the information rates of multi-track recording channels [9]. H ( y ) , and subsequently, the SIR. Conditional entropies have In this paper, we investigate the extension of this approach been used to bound the entropy rate of one-dimensionalhidden to two-dimensional IS1 channels. The main problem is that Markov processes (see, e.g. [lo]).We will adapt the approach 'This work was supported by the Center for Magnetic Recording Research to accommodate the nature of the two-dimensional processes under consideration. at the University of California, San Diego.
0-7803-7799-0/03/$17.00
02003 IEEE
320
Authorized licensed use limited to: Univ of Calif San Diego. Downloaded on March 13, 2009 at 13:18 from IEEE Xplore. Restrictions apply.
(1) iej
(2) i>=j (2) i>=j
Figure 1: The region Pust{Y[i,j]}. Figure 2: Before developing the bounds on the SIR, we first define an ordering -i of the elements in an m x n array Y . We start from Y [ 1 , 1 ]then , follow with Y[1,2],Y [ 2 , 1 ] YI2,2], , YI1,3],and so on. In general, when a k x k block has been ordered, the next elements are those in the (k + 1)th column: Y [ 1 ,k + 1],Y[2,k + 13,. ... Y [ k ,k + 11, and when a k x (IC + 1)block has been ordered, the next elements are those in the ( k 1)th row: Y[k+ 1 , 1 ] , Y [ k+ 1,2],... , Y [ k 1, k 11. If m > n and we have ordered an n x n block, we only order the rows R. + 1 through m without increasing the number of columns. A similar approach is used when m < n. We denote by Past{Y[i,j]} the subset of elements that precede Y [ i , j ]according to the ordering + . Fig. 1 depicts Past{Y[i,j ] } for the two scenarios where i < j and i >_ j . Theset Past{Y[i,j]}\ I& is obtainedfromPast{Y[i,j]} by omitting its first k columns, the first TJ rows, and the last T rows. Having defined the ordering 4,we can apply the chain rule to the entropy H(Y,,), resulting in the expression
+
+
+
C-shape S t r z p e { Y [ i , j ] } , Out(St{Y[i, j]}) and
ww%f}).
Similarly, we can prove that, fori
>j ,
H ( Y [ i jl , IPast{Y[i,A ) ) IH(Y[j, jllPast{Yli7 A}). In general, then, we have the following upper bound on the conditional entropy
H(Y[.L', jlIPast{Y[4jl})
5 H ( Y [min{i , j } min{ i, j }] \Past{Y [mi.{ i ,j } ,min{ i, j }]}) (6)
= a[min{i, j } ] .
This bound will be useful in the derivation below of Upper Bound I on the SIR. For a large enough array, we define a "C-shape" stripe Stripe{Y[i,j]}in P a s t { Y [ i , j ] }for certain elements Y [ i , j ] where i # j . For i < j , Stripe{Y[i,j]}is defined as the (i - k)th row, the (i + Z)th row, and the ( j - p)th column of P a s t { Y [ i , j ] }where , 0 < IC < i, 0 < p < j . More specifiH(Y[i,jllPast{Y[i,jl>). (3) cally,
H(Ymn) = l < i S m
Stripe{Y [i,j ] } = {Y[i- IC, j - p ] ,.... Y [ i- k , j ] } u ( Y [ i + 2 , j - p ] , ..., Y [ i + l , j - l ] } U {Y[i - IC + 1 , j - p ] ,.... Y [ i z - 1,j -PI}, (7)
lsjsn Without loss of generality, in the rest of the paper we will consider an m x n array where m n . We next define the sequence 4 2 1 , where
>
+
as shown in the left side of Fig. 2. Note that k , I , p can a[i] = H ( Y [ i , i ] ( P a s t { Y ( i , i ) ) )for 1 5 i j . Combined with the chain rule (3), Lemma 11.1 leads to the H(Y)I H ( Y [ m fnfllPast{Y[mf, , nrl}), following lower bound on the entropy rate H ( y ) . where H ( Y [ m fnf]lPast{Y[mf, , n f ] } )is the conditional Theorem 11.3 A lower bound on the entropy rate H ( y ) is entropy of Y [ m fnf3 , on an mf x nf array. is
It is proved by combining (3) and (6), and using a technique similar to that in [lo, pp. 64-651.
Theorem 11.2 An upper bound on the entropy rate H ( y ) is
H(Y) >min { H (Y[il,jl]lIn ( S ~ { W I , ~, X~ (S~{W1,j111)) I}) ,
H (Y[i2,jzllIn(St{Y[i2,j21)) ,x(St{Y[i2,jzl}))} , where il iZ<j2,
+ +15
> j 1 , r < il, s < j 1 , s t IC j ~ r, < 21, s < j 1 , s t + 1 5 2 1 , and i 2 < . j Z , IC < iz, p < j,, IC + 1 + 1 I j 2 - 1.
+
It is proved by applying (3), (8) and the fact that the number of elements that satisfy (8) grows at the same rate as the array size m x n. Combining the theorems with the trivial upper bound I ( X ;Y ) I 1 bitlchannel use, we get the following two upper bounds on the SIR. Upper Bound I:
111. COMPUTING THE SIR BOUNDS The bounds derived in Section I1 convert the problem of estimating the entropy of a large two-dimensional arT ( X ;Y ) 5 min (1, H ( Y b f ,nfllPast{Y[mf,nfl}) ray into the problem of estimating conditional entropies 1 over a smaller array. As suggested in [5], the conditional -- log(reN0)). 2 entropy h(dlB) = -E[logP{a(b}]can be estimated by Erzl logP{a(k))Ib(k)},where a ( k ) ,b(k) are the lcth reUpper Bound 11: alizations of A and B , respectively. By the law of large numI ( X ; Y ) 5 min{l,max bers, the estimate converges to h(dlf?) with probability 1 as { H (Y[ii,jl]Pn ( S t { Y [ i l , j ~U ] }Stripe{Y[il,jl])) ) , N -+ W. Therefore, we simulate the channel N times, each , the corretime with i.i.d channel inputs { z [ i , j ] }generating H (Y[iz,jz](In (St(Y[iz,jz]}) U Stripe{Y[iz,j2]})} sponding channel outputs { y [ i , j ] } on an m x n array. For each 1 -- log(reN0)). of these two-dimensional realizations, we calculate the con2 ditional probabilities needed to estimate the conditional entropies above. The calculation of the conditional probability C. Lower Bound on the SIR P{AIB} is, in turn, converted to the calculation of the joint Using an approach similar to one described in [lo], we now derive a lower bound on H ( y ) given certain state information probability P { A , B } and the probability P { B } . For Upper Bound I, A corresponds to Y [m, n] while B corresponds to XF,j l . Past{Y [m,n]}. Similar correspondence can be established j ] } ) for the We define the state information X (St{Y[i, for the other bounds. stripe Stripe{Y[i,j]} as the set of { X [ i , j ] }that are related In order to compute the joint probability of the twovia the transfer function h[i,j]. to Stripe{Y[i,j]} dimensional arrays, we adapt the technique proposed in First, consider the case i < j . It is easy to see that, given X ( S t ( Y [ i , f } )Y[z,jJ , is independent of the channel outputs [SI-[7] for one-dimensional sequences, treating each row vector as a variable and calculating the joint probaOut (St{Y[i,j]}). The information that Out (St{Y[i,j]}) bility P { y l ,y;, . . .,y,} row-by-row using the forwardprovides regarding Y [ i j, ] is all contained in X (St{Y[i, j]}), recursion of the BCJR algorithm. due to the Markovianity of the channel input process. This observation leads to a lower bound on H(Y[i,j]lPast{Y[i,j]}) Since each row vector is considered as a single variable, the number of states for each variable increases exponentially as fori < j . the number of columns in the array increases. Therefore, this Lemma 11.1 Given Y [i,j ] , and any IC > 0, p > 0, I > 0 numerical scheme can only be used to calculate the probability of two-dimensional arrays with relatively few columns. This such that k < i, p < j , IC 1 1 5 j - 1, we have limitation, in turn, requires that the two-dimensional channel impulse response have a small region of support. H ( y [ i , j l l m(St{Y[i,jI)) (St{Y[i,jl))) 5 (Y[it,jt]lPast{Y[it,jtl}) We note that if a new impulse response { h I [ z , j ] }is the transpose of the original impulse response h[i,j],i.e., for any 0 < it 5 j t . hl[i,j]= hb,i],the new channel outputs {yl[i,j]} are also
-&
+ + ,x
0-7803-7799-0/03/$17.00 @ 2003 IEEE
322
Authorized licensed use limited to: Univ of Calif San Diego. Downloaded on March 13, 2009 at 13:18 from IEEE Xplore. Restrictions apply.
the transpose of the original channel outputs. This property enables us to use the same block size (m x n) to calculate the conditional entropy for the case i > j. In the event that the impulse response is symmetric, i.e., h[i,j]= hlj, 21, it is sufficient to calculate only one of the two conditional entropies in Upper Bound I1 and in the Lower Bound.
ACKNOWLEDGMENTS The authors would like to thank Henry D. Pfister for very helpful discussions that suggested Upper Bound II. 0 89 -0
IV. NUMERICALRESULTS In this section, we present numerical results for the upper and lower bounds on the SIR of the two-dimensional IS1 channel with impulse response
h= The
07-
[ E: “0:; ] .
coefficients
have been normalized such that = 1. The channel inputs are i.i.d, equiprobable Bernoulli samples over the input alphabet (1, -l}; thus, each channel input symbol has energy ES = 1. The noise variance is N0/2. The numerical bounds are based upon a 15 x 4 array of output samples. For Upper Bound I, we choose the conditional entropy corresponding to the output Y [15,4] on the lower right comer. For Upper Bound I1 and the Lower Bound, the output of interest is Y[8,4], which is in the middle of the last column. For each EslNo point, the corresponding conditional entropies are computed from 100,000 realizations of such an output array. Fig. 3 shows the numerical results. It can be seen that, even though only a relatively small sample array is used, the upper and lower bounds are still very close, particularly at low signal-to-noise ratios. In general, Upper Bound I1 is slightly tighter than Upper Bound I. The maximum differencebetween Upper Bound I1 and the Lower Bound is approximately 0.05. The discrepancy at higher values of EslNo reflects the larger impact of the IS1 on the conditional entropies and, therefore, on the resulting tightness of the bounds.
CL=, E,’=,h2[k,I ]
V. CONCLUDING REMARKS We have developed upper and lower bounds on the symmetric information rate (SIR) of two-dimensional finite-state IS1 channels. These bounds are expressed in terms of conditional entropies of channel output arrays. We also describe Monte Carlo methods that can be used to compute these conditional entropies when the array size is not too large. We then present numerical results for a channel with 2 x 2 impulse response support, demonstratingthat we can obtain fairly tight bounds on the SIR using conditional entropies on an array which is only a few times larger than the support of the impulse response. The computational complexity of the proposed methods currently limits their applicability to two-dimensional IS1 channels with small impulse response support. Further research is required to develop computable,tight bounds on the SIR for general two-dimensional IS1 channels; see, for example, [1 I] for some proposed approaches.
0-7803-7799-0/03/$17.00 @ 2003 IEEE
0‘
-io
0
-5
5
EdNo (dB)
Figure 3: Upper and lower bounds on the SIR of the twodimensional IS1 channel.
REFERENCES [l] W. Hirt, “Capacity and information rates of discrete-time channels with
memory,” Doctoral dissertation (Diss. ETH no. 8671), Swiss Federal Inst. of Tech. (ETH), Zurich, Switzerland, 1988. [2] W. Hirt and J. L. Massey, “Capacity of the discrete-time Gaussian channel with intersymbol interference,” IEEE Trans. Inform. Theory, vol. 34, no. 3, pp. 380-388, May 1988. [3] S . Shamai (Shitz), L. H. Ozarow, and A. D. Wyner, “Information rates for a discrete-time Gaussian channel with intersymbol interference and stationary inputs,” ZEEE Trans. Inform. Theory, vol. 37, no. 6, pp.15271539, November 1991. [4] S . Shamai (Shitz) and R. Laroia, “The intersymbol interference channel: lower bounds on capacity and channel precoding loss,” IEEE Trans. Inform. Theory, vol. 42, no. 5, pp.1388-1404, September 1996. [5] D. Arnold and H. Loeliger, “On the information rate of binary-input channels with memory”, in Pmc. ZEEE Int. ConJ Commun., Helsinki, Finland, June 2001, vol. 9, pp.2692-2695. [6] V. Sharma and S. K. Singh, “Entropy and channel capacity in the regenerative setup with applications to Markov channels,” in Pmc. IEEE Znt. Symp. Inform. Theory, Washington, DC, USA, June 2001, pp. 283. [7] H. D. Pfister, J . B. Soriaga, and P. H. Siegel, “On the achievable information rate of finite state IS1 channels,” in Pmc. GLOBECOM2001, San Antonio, TX, November 2001, vol. 5, pp. 2992-2996. [8] L. R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” ZEEE Trans. Inform. Theory, vol. 20, pp. 284-287, March 1974. [9] 2. Zhang, T. M. Duman, and E. Kurtas, “On information rates of single track and multi-track magnetic recording channels in intertrack interference,” in Pmc. IEEE Int. Symp. Inform. Theory, Lausanne, Switzerland, June, 2002, pp. 163. [lo] T. M. Cover and J . A . Thomas, Elements of Information Theory, John Wiley &Sons, 1991. [ll] J. Chen and P. H. Siegel, “Information rates of two-dimensional finite state IS1 channels,” submitted to 2003 ZEEE Znt. Symp. Inform. Theory.
323
Authorized licensed use limited to: Univ of Calif San Diego. Downloaded on March 13, 2009 at 13:18 from IEEE Xplore. Restrictions apply.
On the Symmetric Information Rate of Two-Dimensional Finite State IS1 Channels Jiangxin Chen and Paul H. Siege1 Department of Electrical and Computer Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093, USA [email protected],[email protected] Abstract - We derive upper and lower bounds on the there is no direct counterpart of the BCJR algorithm that can symmetric information rate of a two-dimensional finite- simplify the APP calculations for a large sample output m a y state intersymbol-interference(ISI) channel model. in the two-dimensional setting. To overcome this difficulty, we derive upper and lower bounds on the entropy rate of a I. INTRODUCTION large output array based upon conditional entropies of smaller One approach to achieving higher information storage den- output arrays. By adapting the one-dimensional Monte Carlo sity is the use of page-oriented data recording technologies, technique to the calculation of these conditional entropies, we such as holographic memory. Instead of recording the data in are able to compute fairly tight upper and lower bounds on one-dimensional tracks, these technologies store the data on the SIR for two-dimensional IS1 channels with small impulse two dimensional surfaces. A commonly used model for such response support. The paper is organized as follows. In Section 11,we derive a two-dimensional recording channel is the two-dimensional finite-state IS1 channel with additive white Gaussian noise the upper and lower bounds on the SIR of two-dimensionalfinite state IS1 channels. Section I11 describes the Monte Carlo (AWGN), as described by approach to computing these bounds. Section IV provides num n merical results for a two-dimensional IS1 channel whose impulse response is supported on a 2 x 2 region. Section V concludes the paper. where z[i,j] is the channel input with finite alphabet, y[i,j] IS1 11. BOUNDSON THE SIR OF TWO-DIMENSIONAL is the channel output, and n[i,j] is i.i.d, zero-mean Gaussian CHANNELS noise with variance g2 = No/2. As in the case of one-dimensional channels, the capacity A. Conditional Entropies of Two-Dimensional Arrays of this two-dimensional channel is defined as the maximum The input array { X [ i ,j ] } is a two-dimensional discrete ranmutual information rate I ( X ,y ) over all input distributions dom process consisting of i.i.d random variables. The output where X = { z [ i , j ] } and y = { g [ i , j ] } . When the input array Y,, = { Y [ l I], , . . . ,Y [ l , n ]Y;[ 2 , 1 3 , .. . ,Y [ m ,n ] } is a is i.i.d., equiprobable, the mutual information rate is called two-dimensional continuous random process. It is easy to see the symmetric information rate (SIR). The capacity and the that both are stationary random processes. The mutual inforSIR provide useful measures of the storage density that can, mation rate between X and Y is in principle, be achieved with page-oriented technologies, and also serve as performance benchmarks for channel coding and I ( X ; Y ) = H ( Y ) - H ( Y l X ) = H ( Y ) - H(N), (2) detection methods. Various bounds on the capacity and the SIR have been de- where H ( Y ) = limm,n+oo &H(Ymn) is the entropy rate veloped for certain one-dimensional IS1 channels; see, e.g. of the output process and H ( N ) = log(aeN0) is the noise [11-[4]. Recently, several authors independently proposed a entropy rate. new Monte-Carlo approach to calculating a convergent seFor a stationary one-dimensional random process Z = The {Z[i]}, quence of lower bounds on these informationrates [5]-[7]. the limit H ( Z ) = limn+w ;H(Z[I], . . .,~ [ n ex]) method requires the simulation of an a posteriori probability ists. It is not difficult to show that limm,n+w &H(Ymn) also (APP) detector, matched to the information source and chan- exists when { Y [i,j]} is stationary. nel, on a long sample realization of the channel output. The Although we could estimate H ( y ) by calculating the samforward recursion of the sum-product (BCJR) algorithm [8], ple entropy rate of a very large array, similar to the approach applied to the combined source-channeltrellis, can be used to used in [SI-[7],the huge computational complexity makes it reduce the overall computational complexity of the APP cal- impractical. Instead, we will use conditional entropies based culations. The method has been further extended to evaluate upon a smaller array to derive upper and lower bounds on the information rates of multi-track recording channels [9]. H ( y ) , and subsequently, the SIR. Conditional entropies have In this paper, we investigate the extension of this approach been used to bound the entropy rate of one-dimensionalhidden to two-dimensional IS1 channels. The main problem is that Markov processes (see, e.g. [lo]).We will adapt the approach 'This work was supported by the Center for Magnetic Recording Research to accommodate the nature of the two-dimensional processes under consideration. at the University of California, San Diego.
0-7803-7799-0/03/$17.00
02003 IEEE
320
Authorized licensed use limited to: Univ of Calif San Diego. Downloaded on March 13, 2009 at 13:18 from IEEE Xplore. Restrictions apply.
(1) iej
(2) i>=j (2) i>=j
Figure 1: The region Pust{Y[i,j]}. Figure 2: Before developing the bounds on the SIR, we first define an ordering -i of the elements in an m x n array Y . We start from Y [ 1 , 1 ]then , follow with Y[1,2],Y [ 2 , 1 ] YI2,2], , YI1,3],and so on. In general, when a k x k block has been ordered, the next elements are those in the (k + 1)th column: Y [ 1 ,k + 1],Y[2,k + 13,. ... Y [ k ,k + 11, and when a k x (IC + 1)block has been ordered, the next elements are those in the ( k 1)th row: Y[k+ 1 , 1 ] , Y [ k+ 1,2],... , Y [ k 1, k 11. If m > n and we have ordered an n x n block, we only order the rows R. + 1 through m without increasing the number of columns. A similar approach is used when m < n. We denote by Past{Y[i,j]} the subset of elements that precede Y [ i , j ]according to the ordering + . Fig. 1 depicts Past{Y[i,j ] } for the two scenarios where i < j and i >_ j . Theset Past{Y[i,j]}\ I& is obtainedfromPast{Y[i,j]} by omitting its first k columns, the first TJ rows, and the last T rows. Having defined the ordering 4,we can apply the chain rule to the entropy H(Y,,), resulting in the expression
+
+
+
C-shape S t r z p e { Y [ i , j ] } , Out(St{Y[i, j]}) and
ww%f}).
Similarly, we can prove that, fori
>j ,
H ( Y [ i jl , IPast{Y[i,A ) ) IH(Y[j, jllPast{Yli7 A}). In general, then, we have the following upper bound on the conditional entropy
H(Y[.L', jlIPast{Y[4jl})
5 H ( Y [min{i , j } min{ i, j }] \Past{Y [mi.{ i ,j } ,min{ i, j }]}) (6)
= a[min{i, j } ] .
This bound will be useful in the derivation below of Upper Bound I on the SIR. For a large enough array, we define a "C-shape" stripe Stripe{Y[i,j]}in P a s t { Y [ i , j ] }for certain elements Y [ i , j ] where i # j . For i < j , Stripe{Y[i,j]}is defined as the (i - k)th row, the (i + Z)th row, and the ( j - p)th column of P a s t { Y [ i , j ] }where , 0 < IC < i, 0 < p < j . More specifiH(Y[i,jllPast{Y[i,jl>). (3) cally,
H(Ymn) = l < i S m
Stripe{Y [i,j ] } = {Y[i- IC, j - p ] ,.... Y [ i- k , j ] } u ( Y [ i + 2 , j - p ] , ..., Y [ i + l , j - l ] } U {Y[i - IC + 1 , j - p ] ,.... Y [ i z - 1,j -PI}, (7)
lsjsn Without loss of generality, in the rest of the paper we will consider an m x n array where m n . We next define the sequence 4 2 1 , where
>
+
as shown in the left side of Fig. 2. Note that k , I , p can a[i] = H ( Y [ i , i ] ( P a s t { Y ( i , i ) ) )for 1 5 i j . Combined with the chain rule (3), Lemma 11.1 leads to the H(Y)I H ( Y [ m fnfllPast{Y[mf, , nrl}), following lower bound on the entropy rate H ( y ) . where H ( Y [ m fnf]lPast{Y[mf, , n f ] } )is the conditional Theorem 11.3 A lower bound on the entropy rate H ( y ) is entropy of Y [ m fnf3 , on an mf x nf array. is
It is proved by combining (3) and (6), and using a technique similar to that in [lo, pp. 64-651.
Theorem 11.2 An upper bound on the entropy rate H ( y ) is
H(Y) >min { H (Y[il,jl]lIn ( S ~ { W I , ~, X~ (S~{W1,j111)) I}) ,
H (Y[i2,jzllIn(St{Y[i2,j21)) ,x(St{Y[i2,jzl}))} , where il iZ<j2,
+ +15
> j 1 , r < il, s < j 1 , s t IC j ~ r, < 21, s < j 1 , s t + 1 5 2 1 , and i 2 < . j Z , IC < iz, p < j,, IC + 1 + 1 I j 2 - 1.
+
It is proved by applying (3), (8) and the fact that the number of elements that satisfy (8) grows at the same rate as the array size m x n. Combining the theorems with the trivial upper bound I ( X ;Y ) I 1 bitlchannel use, we get the following two upper bounds on the SIR. Upper Bound I:
111. COMPUTING THE SIR BOUNDS The bounds derived in Section I1 convert the problem of estimating the entropy of a large two-dimensional arT ( X ;Y ) 5 min (1, H ( Y b f ,nfllPast{Y[mf,nfl}) ray into the problem of estimating conditional entropies 1 over a smaller array. As suggested in [5], the conditional -- log(reN0)). 2 entropy h(dlB) = -E[logP{a(b}]can be estimated by Erzl logP{a(k))Ib(k)},where a ( k ) ,b(k) are the lcth reUpper Bound 11: alizations of A and B , respectively. By the law of large numI ( X ; Y ) 5 min{l,max bers, the estimate converges to h(dlf?) with probability 1 as { H (Y[ii,jl]Pn ( S t { Y [ i l , j ~U ] }Stripe{Y[il,jl])) ) , N -+ W. Therefore, we simulate the channel N times, each , the corretime with i.i.d channel inputs { z [ i , j ] }generating H (Y[iz,jz](In (St(Y[iz,jz]}) U Stripe{Y[iz,j2]})} sponding channel outputs { y [ i , j ] } on an m x n array. For each 1 -- log(reN0)). of these two-dimensional realizations, we calculate the con2 ditional probabilities needed to estimate the conditional entropies above. The calculation of the conditional probability C. Lower Bound on the SIR P{AIB} is, in turn, converted to the calculation of the joint Using an approach similar to one described in [lo], we now derive a lower bound on H ( y ) given certain state information probability P { A , B } and the probability P { B } . For Upper Bound I, A corresponds to Y [m, n] while B corresponds to XF,j l . Past{Y [m,n]}. Similar correspondence can be established j ] } ) for the We define the state information X (St{Y[i, for the other bounds. stripe Stripe{Y[i,j]} as the set of { X [ i , j ] }that are related In order to compute the joint probability of the twovia the transfer function h[i,j]. to Stripe{Y[i,j]} dimensional arrays, we adapt the technique proposed in First, consider the case i < j . It is easy to see that, given X ( S t ( Y [ i , f } )Y[z,jJ , is independent of the channel outputs [SI-[7] for one-dimensional sequences, treating each row vector as a variable and calculating the joint probaOut (St{Y[i,j]}). The information that Out (St{Y[i,j]}) bility P { y l ,y;, . . .,y,} row-by-row using the forwardprovides regarding Y [ i j, ] is all contained in X (St{Y[i, j]}), recursion of the BCJR algorithm. due to the Markovianity of the channel input process. This observation leads to a lower bound on H(Y[i,j]lPast{Y[i,j]}) Since each row vector is considered as a single variable, the number of states for each variable increases exponentially as fori < j . the number of columns in the array increases. Therefore, this Lemma 11.1 Given Y [i,j ] , and any IC > 0, p > 0, I > 0 numerical scheme can only be used to calculate the probability of two-dimensional arrays with relatively few columns. This such that k < i, p < j , IC 1 1 5 j - 1, we have limitation, in turn, requires that the two-dimensional channel impulse response have a small region of support. H ( y [ i , j l l m(St{Y[i,jI)) (St{Y[i,jl))) 5 (Y[it,jt]lPast{Y[it,jtl}) We note that if a new impulse response { h I [ z , j ] }is the transpose of the original impulse response h[i,j],i.e., for any 0 < it 5 j t . hl[i,j]= hb,i],the new channel outputs {yl[i,j]} are also
-&
+ + ,x
0-7803-7799-0/03/$17.00 @ 2003 IEEE
322
Authorized licensed use limited to: Univ of Calif San Diego. Downloaded on March 13, 2009 at 13:18 from IEEE Xplore. Restrictions apply.
the transpose of the original channel outputs. This property enables us to use the same block size (m x n) to calculate the conditional entropy for the case i > j. In the event that the impulse response is symmetric, i.e., h[i,j]= hlj, 21, it is sufficient to calculate only one of the two conditional entropies in Upper Bound I1 and in the Lower Bound.
ACKNOWLEDGMENTS The authors would like to thank Henry D. Pfister for very helpful discussions that suggested Upper Bound II. 0 89 -0
IV. NUMERICALRESULTS In this section, we present numerical results for the upper and lower bounds on the SIR of the two-dimensional IS1 channel with impulse response
h= The
07-
[ E: “0:; ] .
coefficients
have been normalized such that = 1. The channel inputs are i.i.d, equiprobable Bernoulli samples over the input alphabet (1, -l}; thus, each channel input symbol has energy ES = 1. The noise variance is N0/2. The numerical bounds are based upon a 15 x 4 array of output samples. For Upper Bound I, we choose the conditional entropy corresponding to the output Y [15,4] on the lower right comer. For Upper Bound I1 and the Lower Bound, the output of interest is Y[8,4], which is in the middle of the last column. For each EslNo point, the corresponding conditional entropies are computed from 100,000 realizations of such an output array. Fig. 3 shows the numerical results. It can be seen that, even though only a relatively small sample array is used, the upper and lower bounds are still very close, particularly at low signal-to-noise ratios. In general, Upper Bound I1 is slightly tighter than Upper Bound I. The maximum differencebetween Upper Bound I1 and the Lower Bound is approximately 0.05. The discrepancy at higher values of EslNo reflects the larger impact of the IS1 on the conditional entropies and, therefore, on the resulting tightness of the bounds.
CL=, E,’=,h2[k,I ]
V. CONCLUDING REMARKS We have developed upper and lower bounds on the symmetric information rate (SIR) of two-dimensional finite-state IS1 channels. These bounds are expressed in terms of conditional entropies of channel output arrays. We also describe Monte Carlo methods that can be used to compute these conditional entropies when the array size is not too large. We then present numerical results for a channel with 2 x 2 impulse response support, demonstratingthat we can obtain fairly tight bounds on the SIR using conditional entropies on an array which is only a few times larger than the support of the impulse response. The computational complexity of the proposed methods currently limits their applicability to two-dimensional IS1 channels with small impulse response support. Further research is required to develop computable,tight bounds on the SIR for general two-dimensional IS1 channels; see, for example, [1 I] for some proposed approaches.
0-7803-7799-0/03/$17.00 @ 2003 IEEE
0‘
-io
0
-5
5
EdNo (dB)
Figure 3: Upper and lower bounds on the SIR of the twodimensional IS1 channel.
REFERENCES [l] W. Hirt, “Capacity and information rates of discrete-time channels with
memory,” Doctoral dissertation (Diss. ETH no. 8671), Swiss Federal Inst. of Tech. (ETH), Zurich, Switzerland, 1988. [2] W. Hirt and J. L. Massey, “Capacity of the discrete-time Gaussian channel with intersymbol interference,” IEEE Trans. Inform. Theory, vol. 34, no. 3, pp. 380-388, May 1988. [3] S . Shamai (Shitz), L. H. Ozarow, and A. D. Wyner, “Information rates for a discrete-time Gaussian channel with intersymbol interference and stationary inputs,” ZEEE Trans. Inform. Theory, vol. 37, no. 6, pp.15271539, November 1991. [4] S . Shamai (Shitz) and R. Laroia, “The intersymbol interference channel: lower bounds on capacity and channel precoding loss,” IEEE Trans. Inform. Theory, vol. 42, no. 5, pp.1388-1404, September 1996. [5] D. Arnold and H. Loeliger, “On the information rate of binary-input channels with memory”, in Pmc. ZEEE Int. ConJ Commun., Helsinki, Finland, June 2001, vol. 9, pp.2692-2695. [6] V. Sharma and S. K. Singh, “Entropy and channel capacity in the regenerative setup with applications to Markov channels,” in Pmc. IEEE Znt. Symp. Inform. Theory, Washington, DC, USA, June 2001, pp. 283. [7] H. D. Pfister, J . B. Soriaga, and P. H. Siegel, “On the achievable information rate of finite state IS1 channels,” in Pmc. GLOBECOM2001, San Antonio, TX, November 2001, vol. 5, pp. 2992-2996. [8] L. R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” ZEEE Trans. Inform. Theory, vol. 20, pp. 284-287, March 1974. [9] 2. Zhang, T. M. Duman, and E. Kurtas, “On information rates of single track and multi-track magnetic recording channels in intertrack interference,” in Pmc. IEEE Int. Symp. Inform. Theory, Lausanne, Switzerland, June, 2002, pp. 163. [lo] T. M. Cover and J . A . Thomas, Elements of Information Theory, John Wiley &Sons, 1991. [ll] J. Chen and P. H. Siegel, “Information rates of two-dimensional finite state IS1 channels,” submitted to 2003 ZEEE Znt. Symp. Inform. Theory.
323
Authorized licensed use limited to: Univ of Calif San Diego. Downloaded on March 13, 2009 at 13:18 from IEEE Xplore. Restrictions apply.
