Hey,

This week I went back to 1948. I finally sat down with Claude Shannon's A Mathematical Theory of Communication and read the actual paper instead of the Wikipedia summary. It kicks off a new series on the blog: From Shannon to Transformers, where we walk through the original papers behind modern AI and unpack the maths without the hand-waving.

What I learned this week

Shannon built a language model with a book and random numbers. His "approximations to English" pick each letter based on the previous one or two. I rebuilt it in about 60 lines of Python on Moby Dick: at order 4 it starts writing "WHALE" and "MY LIFE" without ever being told what a word is.
The entropy formula isn't a convention. Shannon lists three common-sense properties a measure of "choice" should have and proves only one function satisfies them: H = −Σ p log p.
The trap: measure entropy on a small text and English looks far more predictable than it is. On Alice in Wonderland my 8th-order estimate fell to 0.61 bits/letter, because most long contexts appear only once. That's overfitting, 1948 edition, and it's why we test language models on held-out data.

👉 Read Part 1: Shannon (1948) — Entropy, N-grams, and Guessing the Next Letter

Worth your time

A Mathematical Theory of Communication (Bell Labs) — Read Section 3 first; it's short, and it's where the "predict the next letter" idea starts.
Prediction and Entropy of Printed English (1951) — Shannon turns people into the language model with a guessing game. This is Part 2, so read ahead if you like.
karpathy/makemore — A character-level repo that starts at a bigram lookup table and climbs to a Transformer. It's basically this series in code form.

That's it for this week. Hit reply and tell me which paper you want dissected next; I read every reply.

— Unique

Want the next issue?