ECE587: Information Theory

Yao Xie, Duke University, Fall semester 2012
Yao XieClaude Shannon

This is the website for ECE 587, Duke University, Fall semester 2012.

These lecture notes is a tribute to the beloved Thomas M. Cover, from whom I learned information theory.

Information theory is the science of operations on data. The central themes of information theory include compression, storage, and communication. The birth of information theory was in 1948, marked by Claude E. Shannon's paper entitled "A Mathematical Theory of Communication". Surprisingly, this paper paved way for many later revolutionary ideas such as digital communication, storage and even internet. Our course will explore the basic concepts of Information theory. It is highly recommended for students planning to delve into the fields of communications, data compression, and statistical signal processing. Also encouraged to enroll are students who are interested in seeing the interplay between information theory and probability theory.

This class provides an introduction to information theory. Information measures: entropy, mutual information, relative entropy and differential entropy. These topics are connected to practical problems in communications, compression, and inference, including lossless data compression, Huffman coding, asymptotic equipartition property, channel capacity, Gaussian channels, rate distortion theory, and Fisher information. The class is appropriate for beginning graduate students in electrical engineering, computer science, and mathematics who have a background in probability. Prerequisite: ECE255 or equivalent, or consent of instructor.


Welcome to ECE587!
  • Second midterm is at Tues., Nov. 6, 1:25pm - 2:40pm, close book, in class.
  • TA midterm 2 review session, by Miao, Hudson 208, Mon., Nov. 5, 5:30pm - 6:30pm.
  • TA midterm review session, by Miao, has been scheduled to be at Hudson 208, Mon., Nov.5, 5:30pm - 6:30pm.
  • First midterm is at Thurs., Oct. 4, 1:25pm - 2:40pm, close book, in class.
  • TA midterm review session, by Miao, has been scheduled to be at Hudson 208, on Wed., Oct. 3, 5:30pm - 6:30pm.
  • Room for TA office hour has been changed to: CIEMAS 3472A.
  • Good reference on machine learning and information theory: Information Theory, Inference and Learning Algorithms, by D. J. C. MacKay
  • A good reference for basic probability is MIT Open Courseware
  • Homeworks and solutions will be posted on Sakai.
  • Want to know more about Claude Shannon and the early information theory? Claude Shannon - Father of the Information Age
  • An interesting US News article: Quantitative Approach to Information Processing