Measuring uncertainty

How much do we know—and how can we measure it? Build an intuition for the mathematics behind every prediction.

YOUR STUDY DESK

01 ✧

Read the lesson

Follow the ideas, mathematics, and worked examples.

Open lesson 02 ✧

Open your notebook

Test your understanding with hands-on experiments.

Start experimenting 03 ✧

Watch the conversation

Learn alongside the recorded session.

View recordings

Readings

Easy

Information Theory & Entropy

• Visual Information Theory by Chris Olah — Beautiful visual explanation of entropy, mutual information, and KL divergence

Compression Algorithms

• Huffman Coding — How variable-length codes achieve near-optimal compression using entropy
• Arithmetic Coding — A more sophisticated approach that can achieve compression arbitrarily close to the entropy limit

N-grams & Language Models

• Chapter 3: N-gram Language Models from Speech and Language Processing by Jurafsky & Martin — The classic textbook chapter on n-grams and perplexity

Advanced

Information Theory Foundations

• A Mathematical Theory of Communication by Claude Shannon (1948) — The original paper that started it all (Sections I-II are most relevant)