Coding for Discrete Source - Cal Poly Pomona
EGR 544 Communication Theory
3. Coding for Discrete Sources
Z. Aliyazicioglu Electrical and Computer Engineering Department Cal Poly Pomona
Coding for Discrete Source Coding • Represent source data effectively in digital form for transmission or storage • A measure of the efficiency of a source-encoding method can be obtained by comparing the average number of binary digits per output letter from the source to the entropy H(X). • Two types of source coding – Lossless (Huffman coding algorithm, Lembel-Ziv Algorithm..) – Lossy (rate-distortion, quantization, waveform coding..) X
bits
Source encoding
Cal Poly Pomona
Channel transmission
bits
_
Source decoding
Electrical & Computer Engineering Dept.
X
_
X = X _
X ≤ X EGR 544-2 2
1
Coding for Discrete Memoryless Source • DMS source produce an output letter every τs second. • Source has finite alphabet of symbol xi , i=1,2,…L with probabilities P (xi ) • The entropy of the DMS in bits per source symbol is L
H ( X ) = − ∑ P( xi ) log 2 P( xi ) ≤ log 2 L i =1
• If symbols have same probability L 1 1 H ( X ) = − ∑ log 2 = log 2 L L L i =1
• The source rate in bits/s is
Cal Poly Pomona
H ( X ) /τ s
Electrical & Computer Engineering Dept.
EGR 544-2 3
Fixed-length code words • Let’s assign a unique set of R binary digits to each symbols • Since there is L possible symbols, R will gives us code rate in bits per symbols as R = log 2 L
• When L is not a power of 2, it is R = log 2 L + 1
denotes the largest integer less than log 2 L
log 2 L
Since
H ( X ) ≤ log 2 L
H(X ) R Cal Poly Pomona
R ≥ H(X )
Ratio shows the efficiency of the encoding for DMS
Electrical & Computer Engineering Dept.
EGR 544-2 4
2
When L is power of 2 and source letters are equally probable, Fixed length code of R bits per symbol attains 100 percent efficiency
R = H(X ) When L is not power of 2 and source letters are equally probable, R will be different than H(X) at most 1 bit. Shannon coding Theorem: Based on the sequences, the lossless coding exists as long as R≥H(X). Lossless code does not exits for any R
3. Coding for Discrete Sources
Z. Aliyazicioglu Electrical and Computer Engineering Department Cal Poly Pomona
Coding for Discrete Source Coding • Represent source data effectively in digital form for transmission or storage • A measure of the efficiency of a source-encoding method can be obtained by comparing the average number of binary digits per output letter from the source to the entropy H(X). • Two types of source coding – Lossless (Huffman coding algorithm, Lembel-Ziv Algorithm..) – Lossy (rate-distortion, quantization, waveform coding..) X
bits
Source encoding
Cal Poly Pomona
Channel transmission
bits
_
Source decoding
Electrical & Computer Engineering Dept.
X
_
X = X _
X ≤ X EGR 544-2 2
1
Coding for Discrete Memoryless Source • DMS source produce an output letter every τs second. • Source has finite alphabet of symbol xi , i=1,2,…L with probabilities P (xi ) • The entropy of the DMS in bits per source symbol is L
H ( X ) = − ∑ P( xi ) log 2 P( xi ) ≤ log 2 L i =1
• If symbols have same probability L 1 1 H ( X ) = − ∑ log 2 = log 2 L L L i =1
• The source rate in bits/s is
Cal Poly Pomona
H ( X ) /τ s
Electrical & Computer Engineering Dept.
EGR 544-2 3
Fixed-length code words • Let’s assign a unique set of R binary digits to each symbols • Since there is L possible symbols, R will gives us code rate in bits per symbols as R = log 2 L
• When L is not a power of 2, it is R = log 2 L + 1
denotes the largest integer less than log 2 L
log 2 L
Since
H ( X ) ≤ log 2 L
H(X ) R Cal Poly Pomona
R ≥ H(X )
Ratio shows the efficiency of the encoding for DMS
Electrical & Computer Engineering Dept.
EGR 544-2 4
2
When L is power of 2 and source letters are equally probable, Fixed length code of R bits per symbol attains 100 percent efficiency
R = H(X ) When L is not power of 2 and source letters are equally probable, R will be different than H(X) at most 1 bit. Shannon coding Theorem: Based on the sequences, the lossless coding exists as long as R≥H(X). Lossless code does not exits for any R
