Lecture Notes

David Tse(dntse@stanford.edu) Winter 2016-2017

Each student has to scribe notes for a single lecture. Each scribe for a given lecture will prepare a LaTeX document written in full prose with figures, understandable to a student who may have missed class. Please use this LaTeX template file and style file. All the relevant files should be submitted to Vivek (vbagaria@stanford.edu) no later than 72 hours after the lecture.

Please sign up to scribe a specific lecture through this spreadsheet.

Part I: Information measures

  • Lecture 1: Historical context, introduction to entropy.

  • Lecture 2: Entropy, mutual information and the chain rule.

  • Lecture 3: Relative entropy and its nonnegativity.

Part II: Compression

Part III: Communication

  • Lecture 8: Introduction to communication over noisy channels and channel capacity.

  • Lecture 9: Channel capacity and the converse to the noisy channel coding theorem.

  • Lecture 10: Sphere packing view of the coding theorem.

  • Lecture 11: Achieving capacity via random coding.

  • Lecture 12: Achieving capacity efficiently: polar codes I (Slides for polar codes).

  • Lecture 13: Polar codes II

  • Lecture 14: Gaussian channel and information measures for continuous variables I.

  • Lecture 15: Information measures for continuous variables II and entropy maximization.

Part IV: Machine Learning and statistics

  • Lecture 16: Maximum entropy principle, exponential families and Gaussian graphical models.

  • Lecture 17: The Ising model, maximum conditional entropy approach to supervised learning.

  • Lecture 18: Supervised learning II, decision theoretic interpretation of MAXENT.

  • Lecture 19: Information theory, machine learning and statistics: three views of the same coin.