Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing)

Preface to the Second Edition.

Preface to the First Edition.

Acknowledgments for the Second Edition.

Acknowledgments for the First Edition.

1. Introduction and Preview.

1.1 Preview of the Book.

2. Entropy, Relative Entropy, and Mutual Information.

2.1 Entropy.

2.2 Joint Entropy and Conditional Entropy.

2.3 Relative Entropy and Mutual Information.

2.4 Relationship Between Entropy and Mutual Information.

2.5 Chain Rules for Entropy, Relative Entropy, and Mutual Information.

2.6 Jensen’s Inequality and Its Consequences.

2.7 Log Sum Inequality and Its Applications.

2.8 Data-Processing Inequality.

2.9 Sufficient Statistics.

2.10 Fano’s Inequality.

Summary.

Problems.

Historical Notes.

3. Asymptotic Equipartition Property.

3.1 Asymptotic Equipartition Property Theorem.

3.2 Consequences of the AEP: Data Compression.

3.3 High-Probability Sets and the Typical Set.

Summary.

Problems.

Historical Notes.

4. Entropy Rates of a Stochastic Process.

4.1 Markov Chains.

4.2 Entropy Rate.

4.3 Example: Entropy Rate of a Random Walk on a Weighted Graph.

4.4 Second Law of Thermodynamics.

4.5 Functions of Markov Chains.

Summary.

Problems.

Historical Notes.

5. Data Compression.

5.1 Examples of Codes.

5.2 Kraft Inequality.

5.3 Optimal Codes.

5.4 Bounds on the Optimal Code Length.

5.5 Kraft Inequality for Uniquely Decodable Codes.

5.6 Huffman Codes.

5.7 Some Comments on Huffman Codes.

5.8 Optimality of Huffman Codes.

5.9 Shannon–Fano–Elias Coding.

5.10 Competitive Optimality of the Shannon Code.

5.11 Generation of Discrete Distributions from Fair Coins.

Summary.

Problems.

Historical Notes.

6. Gambling and Data Compression.

6.1 The Horse Race.

6.2 Gambling and Side Information.

6.3 Dependent Horse Races and Entropy Rate.

6.4 The Entropy of English.

6.5 Data Compression and Gambling.

6.6 Gambling Estimate of the Entropy of English.

Summary.

Problems.

Historical Notes.

7. Channel Capacity.

7.1 Examples of Channel Capacity.

7.2 Symmetric Channels.

7.3 Properties of Channel Capacity.

7.4 Preview of the Channel Coding Theorem.

7.5 Definitions.

7.6 Jointly Typical Sequences.

7.7 Channel Coding Theorem.

7.8 Zero-Error Codes.

7.9 Fano’s Inequality and the Converse to the Coding Theorem.

7.10 Equality in the Converse to the Channel Coding Theorem.

7.11 Hamming Codes.

7.12 Feedback Capacity.

7.13 Source–Channel Separation Theorem.

Summary.

Problems.

Historical Notes.

8. Differential Entropy.

8.1 Definitions.

8.2 AEP for Continuous Random Variables.

8.3 Relation of Differential Entropy to Discrete Entropy.

8.4 Joint and Conditional Differential Entropy.

8.5 Relative Entropy and Mutual Information.

8.6 Properties of Differential Entropy, Relative Entropy, and Mutual Information.

Summary.

Problems.

Historical Notes.

9. Gaussian Channel.

9.1 G..