Download Report

Digital Communications III (ECE 154C)
Introduction to Coding and Information Theory
Tara Javidi
These lecture notes were originally developed by late Prof. J. K. Wolf.
UC San Diego
Spring 2014
1 / 12
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
Noiseless Source Coding
Fundamental Limits
2 / 12
Shannon Entropy
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
• Let an I.I.D. source S , have M source letters that occur with
PM
probabilities p1 , p2 , ..., pM ,
i=1 pi = 1
• The entropy of the source S is denoted H(S) and is defined as
Ha (S) =
M
X
i=1
M
X
1
pi loga pi
pi loga
=−
pi
i=1
1
Ha (S) = E loga
pi
• The base of the logarithms is usually taken to be equal to 2. In
that case, H2 (S) is simply written as H(S) and is measured in
units of “bits”.
3 / 12
Shannon Entropy
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
• Other bases can be used and easily transformed to. Note:
loga x = (logb x)(loga b) =
(logb x)
(logb a)
Hence,
Hb (S)
Ha (S) = Hb (S) · loga b =
logb a
Theorem
• Source Coding
Theorem
A Useful Theorem
Let p1 , p2 , ..., pM be one set of probabilities and let p′1 , p′2 , ..., p′M
be another (note
PM
i=1 pi = 1,
M
X
i=1
PM
′ = 1). Then
p
i=1 i
M
X
1
1
pi log ≤
pi log ′ ,
pi
pi
i=1
with equality iff pi = p′i for i = 1, 2, ..., M
4 / 12
Shannon Entropy
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Proof.
First note that ln x ≤ x − 1 with equality iff x = 1.
Theorem
• Source Coding
Theorem
Placeholder Figure
M
X
i=1
pi log
A
p′i
pi
=
M
X
i=1
pi ln
p′i
pi
!
log e
5 / 12
Shannon Entropy
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
proof continued.
M
X
pi ln
i=1
p′i
pi
!
log e ≤ (log e)
= (log e)
Theorem
• Source Coding
Theorem
M
X
pi
i=1
M
X
pi
p′i −
i=1
=0
p′i
−1
M
X
i=1
pi
!
In other words,
M
X
i=1
M
X
1
1
pi log ′
pi log ≤
pi
pi
i=1
6 / 12
Shannon Entropy
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
1.
2.
For an equally likely i.i.d. source with M source letters
H2 (S) = log2 M
(aHa (S) = loga M, any a)
For any i.i.d. source with M source letters
0 ≤ H2 (S) ≤ log2 M,
3.
for alla
1
This follows from previous theorem with p′i = M
all i.
Consider an iid source with source alphabet S . If we consider
encoding M source letters at a time, this is an iid source with
alphabet S M for source letters. Call this the M tuple extension
of the source and denote it is by S M . Then
H2 (S M ) = mH2 (S),
(and similarly for all a, Ha (S M ) = mHa (S)).
The proofs are omitted but are easy.
7 / 12
Computation of Entropy (base 2)
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
Example 1: Consider a source with M = 2 and
(p1 , p2 ) = (0.9, 0.1).
1
1
+ .1 log2
= 0.469bits
.9
1.1
H2 (S) = .9 log2
From before we gave Huffman Codes for this source and extensions
of this source for which
L1 = 1
L 2 = 0.645
2
L 3 = 0.533
3
L 4 = 0.49
4
In other words, we note that L m ≥ H2 (s). Furthermore, as m gets
m
m
is getting closer to
larger Im
H2 (S).
8 / 12
Performance of Huffman Codes
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
−→ One can prove that, in general, for a binary Huffman Code,
1
m < H2 (S) +
H2 (S) ≤ L m
m
Example 2: Consider a source with M = 3 and
(p1 , p2 , p3 ) = (.5, .35, .15).
1
1
1
+ .15 log2
= 1.44 bits
H2 (S) = .5 log2 + .35 log2
.5
.35
.15
Again we have already given codes for this source such that
1 symbol at a time L1 = 1.5
2 symbols at a time L 2 = 1.46
2
9 / 12
Performance of Huffman Codes
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
Example 3: Consider M = 4 and (p1 , p2 , p3 , p4 ) = ( 12 , 14 , 18 , 81 ).
H2 (S) =
1
1
1
1
1
1
1
1
log2
+ log2
+ log2
+ log2
2
1/2 4
1/4 8
1/8 8
1/8
= 1.75bits
But from before we gave the code
Source Symbols Codewords
A
0
B
10
C
110
D
111
for which L1 = H2 (S).
This means that one cannot improve on the efficiency of this
Huffman code by encoding several source symbols at a time.
10 / 12
Fundamental Source Coding Limit
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
Theorem [Lossless Source Coding Theorem]
For any U.D. Binary Code corresponding to the N th extension of the
I.I.D. Source S , for every N = 1, 2, ...
LN
≥ H(S)
N
• For a binary Huffman Code corresponding to the N th extension
of the I.I.D. Source S
LN
1
H2 (S) +
>
≥ H2 (S)
m
N
• But this implies that Huffman coding is asymptotically optimal, i.e.
LN
−→ H2 (S)
lim
N →∞ N
11 / 12
NON-BINARY CODE WORDS
Source Coding
• Entropy
• A Useful Theorem
• A Useful Theorem
• A Useful Theorem
• More on Entropy
• More on Entropy
• Fundamental Limits
• More on Entropy
• Source Coding
Theorem
• Source Coding
Theorem
The code symbols that make up the codewords can be from a higher
order alphabet than 2.
Example 4: I.I.D. Source {A,B,C,D,E} with U.D. codes (where each
concatenation of code words can be decoded in only one way):
SYMBOLS
A
B
C
D
E
TERNARY
QUATERNARY
0
1
20
21
22
0
1
2
30
31
5-ary
0
1
2
3
4
• A lower bound to the average code length of any U.D. n-ary code
(with n-letters) is Hn (S)
• For example, for a ternary code, the average length (per source
L
M
, is no less than H3 (S).
letter), M
12 / 12