Lecture 01 · Part 1

Overview, Tokenization

Percy opens the course with its reason for existing — researchers have drifted from prompting back up the stack to understanding, and full understanding requires building — and its central framing, that accuracy equals efficiency times resources. He tours the five units, then starts the first one. A tokenizer converts strings to integer tokens; character, byte, and word tokenizers each fail in their own way; byte-pair encoding builds a data-driven vocabulary by repeatedly merging the most frequent adjacent pair.

▶ Lecture video Executable lecture (.py) ↗

Why build language models from scratch

Percy’s opening diagnosis: researchers are becoming disconnected from the underlying technology. Around 2016, AI researchers implemented and trained their own models; by 2018 they downloaded pretrained models like BERT and fine-tuned them; today most people prompt an API model. There’s nothing wrong with prompting — moving up levels of abstraction boosts productivity — but these abstractions are leaky, and “by simply prompting a model, you’re vastly constraining the set of options, the design space you’re looking at.” The claim underwriting the whole course: full understanding of this technology is necessary for fundamental research, and the way to understanding is building.

But there’s one small problem: the industrialization of language models. GPT-4 supposedly cost $100M to train back in 2023, and costs today are speculated to be on the order of $1B; xAI built a 230K-GPU cluster for Grok. And nobody publishes details — the GPT-4 technical report says outright that due to the competitive landscape and safety implications, it will share nothing about how the model was built. Frontier models are out of reach.

We could build small models instead (under 1B parameters), but small models are not necessarily representative of large ones, and Percy gives two examples. First, where compute goes shifts with scale: the fraction of FLOPs spent in MLP layers is around 44% at small scale but 80% at 175B — so an attention optimization tuned at small scale may not matter at large scale. Second, emergence: zero-shot and few-shot abilities look flat, like nothing is working, until a critical scale where they suddenly improve.

So what can a class like this teach that transfers? Three types of knowledge: mechanics (how things work — what a Transformer is, how model parallelism works), mindset (squeeze the most out of the hardware, take scaling seriously), and intuitions (which data and modeling decisions yield good accuracy). Mechanics and mindset transfer and are what this course teaches; intuitions only partially transfer across scales — for those you need to run experiments somewhere with real scale.

Efficiency is the whole game

Percy pauses on the bitter lesson, because it gets misread. The wrong interpretation is that scale is all that matters and algorithms don’t. The right interpretation is that algorithms that scale are what matter.

The framing for the entire course follows: what is the best model one can build given a fixed compute and data budget? For pretraining we mostly assume more data than compute, so the question becomes: maximize efficiency. Nearly every topic on the syllabus is this one idea wearing different clothes — systems is obviously about compute efficiency; tokenization exists because raw bytes are compute-inefficient with today’s architectures; many architecture changes are motivated by reducing memory or FLOPs; data filtering avoids wasting gradient updates on bad data; scaling laws are hyperparameter tuning done cheaply on small models.

From Shannon to agents

Language models are old. Shannon used them in the 1950s to measure the entropy of English, and n-gram models were long a component of machine translation and speech recognition systems. The modern lineage comes from neural ingredients accumulated through the 2010s: LSTMs, Bengio’s first neural language model in 2003 (a feedforward network over a small context, not an LSTM), seq2seq’s bold claim that a whole sentence can be compressed into a vector, the Adam optimizer, the attention mechanism and then the Transformer (both developed for machine translation), mixture of experts, and model parallelism.

By the late 2010s, ELMo and BERT established the pretrain-then-fine-tune recipe. Then OpenAI opened the floodgates by embracing scaling: GPT-2, then scaling laws, then GPT-3 at 175B — more than 10x the largest model of its time — showing emergent in-context learning. Google responded with the massive but undertrained PaLM, while DeepMind’s Chinchilla worked out compute-optimal scaling laws. Early open replications (EleutherAI’s Pile and GPT-J, Meta’s OPT-175B with its many hardware issues, BigScience’s BLOOM) were not very strong; the last three years changed that, with Llama, Mistral, and a wave of Chinese models — DeepSeek, Qwen, Kimi, GLM, and more — producing open-weight models that genuinely approach the closed ones. A further line — AI2’s Olmo, NVIDIA’s Nemotron, and the Marin project Percy works on — releases weights plus paper, code, and data. Percy is explicit that this course would not be possible without the open ecosystem: published papers about big MoEs and RL systems let us triangulate how frontier models are built, even if key details like data mixtures stay hidden.

What a “language model” is has shifted each era: 2018, something you fine-tune; 2020, something you prompt; 2022, something you talk to; 2026, something that acts autonomously as an agent. Yet the fundamentals haven’t changed much — still GPUs and kernels, still stochastic-gradient-like optimization, still the Transformer and attention. The specs are different: longer contexts, and inference efficiency matters even more.

Five units, five assignments

The course has five parts mirroring the five assignments. Basics (Assignment 1): tokenize the data, define the architecture, implement the optimizer and training loop — you implement a BPE tokenizer, the Transformer, cross-entropy loss, AdamW, do resource accounting, train on TinyStories and OpenWebText, and race a leaderboard to minimize OpenWebText perplexity given 45 minutes on a B200. The high-level principle: everything balances expressivity (represent complex dependencies), stability (keep parameter and gradient norms in the goldilocks zone so runs neither blow up nor vanish), and efficiency (run fast on hardware).

Systems (Assignment 2): get the most out of the hardware. Resource accounting comes first — the $6ND$ formula for training FLOPs, and the cartoon picture that your memory is not where your compute is, so parameters and activations must move between the two (a B200 does 2.25 PFLOP/s in bf16 against 8 TB/s of memory bandwidth). Then kernels — a kernel is a function that runs on the GPU, and the principle is organizing computation to minimize data movement, e.g. fusing two operations so data crosses the HBM boundary once instead of twice — then parallelism across thousands of GPUs via collective operations and sharding, then inference with its compute-bound prefill and memory-bound one-token-at-a-time decode.

Scaling laws (Assignment 3): if you had 1e25 FLOPs, what would you train? You can’t hyperparameter-tune at that scale, so think in terms of a scaling recipe — a mapping from FLOP budget to hyperparameters — fit scaling laws at small scale, and extrapolate. “Scaling laws are not laws of nature… you have to will them into existence,” and predictability is at least as important as optimality. The classic Chinchilla result gives the rule of thumb $D = 20N$ : a 70B model should see roughly 1.4T tokens. The assignment simulates the high-stakes version: submit training jobs to a cached training API, fit scaling laws, and get judged on where your extrapolated config lands.

Data (Assignment 4): “data quality basically specifies how good your model is going to be.” Evaluation splits into internal metrics (perplexity — smoothness across scales, relative performance) and external metrics (ecological validity), and they shouldn’t be conflated. Data doesn’t fall from the sky: it’s crawled, licensed, or contested; it arrives as HTML and PDF, and must be transformed, filtered, deduplicated, and mixed. You start from a raw web crawl and do the dirty work yourself.

Alignment (Assignment 5): improve a pretrained model with weak supervision, because it’s often easier to critique than to generate. The template — generate responses, score them with a human, verifier, or LM judge, update the model to prefer better ones — is instantiated by PPO, GRPO, or (for preference data, more simply) DPO.

Tokenization: strings to integers

Now the first unit, which Percy notes was inspired by Andrej Karpathy’s tokenization video. Raw text is Unicode strings; a language model places a probability distribution over sequences of tokens represented as integer indices. So we need to convert between the two.

Playing with real tokenizers reveals their quirks: a word and its preceding space form one token (” world”), so “hello” at the start of a line and ” hello” mid-sentence are completely unrelated indices; numbers get chopped every few digits, sometimes predictably, sometimes not. These oddities are why people keep hoping tokenizers will die.

Three baselines that fail

Character tokenizer. A Unicode string is a sequence of characters, each convertible to a code point via ord (97 for “a”, 127757 for 🌍) and back via chr. This round-trips fine, but there are roughly 150K Unicode characters, so the vocabulary is huge — and many characters are rare, which wastes vocabulary on things the model barely sees. Large vocabulary and mediocre compression: “the worst of both worlds.”

Byte tokenizer. Encode the string as UTF-8 bytes: “a” is one byte, 🌍 is four. Now the vocabulary is a tidy 256 — but the compression ratio is exactly 1, so sequences are as long as they can possibly be. With quadratic attention and limited context, “this is not looking great.”

Word tokenizer. The classic NLP approach: split on a regular expression like regex.findall(r"\w+|.", string), which keeps alphanumeric runs together. Each token is meaningful — humans invented words — and compression is good, but the vocabulary is the number of distinct chunks in the training data: effectively unbounded, full of rare words the model learns little about, and any unseen word at test time needs a special UNK token, “which is ugly and can mess up your perplexity calculations.”

Byte-pair encoding

BPE was introduced by Philip Gage in 1994 for data compression — long before language models were on the scene — adapted to NLP for neural machine translation by Sennrich et al., and first used for an LLM in GPT-2. The basic idea: train the tokenizer on raw text, constructing a vocabulary tailored to the data. Common byte sequences become single tokens; rare sequences split into many tokens, so everything is tokenizable and there is never an UNK.

The algorithm: start with each byte as a token, then repeatedly merge the most frequent pair of adjacent tokens into a new token.

Code walkthrough — Training BPE on 'the cat in the hat'

Training starts from the raw bytes and iterates: count adjacent pairs, take the most common, mint a new token index for it, and rewrite the sequence.

def train_bpe(string: str, num_merges: int) -> BPETokenizerParams:
    indices = list(map(int, string.encode("utf-8")))
    merges: dict[tuple[int, int], int] = {}          # (index1, index2) -> merged index
    vocab: dict[int, bytes] = {x: bytes([x]) for x in range(256)}

    for i in range(num_merges):
        counts = count_adjacent_pairs(indices)       # e.g. (116, 104) appears twice
        pair = max(counts, key=counts.get)           # most common pair
        new_index = 256 + i
        merges[pair] = new_index
        vocab[new_index] = vocab[pair[0]] + vocab[pair[1]]
        indices = merge(indices, pair, new_index)    # replace every occurrence

    return BPETokenizerParams(vocab=vocab, merges=merges)

On "the cat in the hat" with three merges: the pair (116, 104) — “t”, “h” — occurs most often, so token 256 now means “th” and both occurrences are replaced. Next round, (256, 101) wins and token 257 means “the”; a third merge yields 258. The sequence shrinks while the vocabulary grows, and the compression ratio on this toy string ends at 1.5.

Encoding new text like "the quick brown fox" conceptually just replays the recorded merges, in order, against the new byte sequence — and decoding concatenates each token’s bytes back into a string:

def encode(self, string: str) -> list[int]:
    indices = list(map(int, string.encode("utf-8")))
    for pair, new_index in self.params.merges.items():   # very slow: loops over ALL merges
        indices = merge(indices, pair, new_index)
    return indices

This is a complete, working BPE implementation — and extremely slow. Assignment 1 asks you to go beyond it: loop only over the merges that matter (which requires building indices), detect and preserve special tokens like <|endoftext|>, pre-tokenize with a regex (as GPT-2 does) so the tokenizer runs per-chunk rather than over one giant string, and generally make it as fast as possible — at some point you may conclude Python is the bottleneck, and Rust or C is fair game.

Back of the chalk tray — what to remember

Understanding via building: prompting constrains your design space, and fundamental research requires tearing up the whole stack — but since frontier models are industrial secrets, we learn mechanics and mindset (which transfer across scale) and accept that intuitions only partially do.
accuracy = efficiency × resources, and efficiency matters more at scale, not less; almost every course topic is efficiency in disguise.
A tokenizer converts strings to integer tokens and back, and must round-trip; its quality is measured by compression ratio (bytes per token) against vocabulary size.
Character (150K sparse vocabulary), byte (compression ratio 1), and word (unbounded vocabulary, UNK) tokenizers are each suboptimal in their own way.
BPE trains on raw text: start from bytes, repeatedly merge the most frequent adjacent pair; common sequences become single tokens and rare ones split, so nothing is ever out-of-vocabulary.
Tokenization buys compression (shorter sequences under quadratic attention) and, more subtly, adaptive computation — properties any future byte-level replacement must also deliver.

Check yourself

Exercises in the lecture’s spirit — a tokenizer trace and some napkin reasoning, doable on paper.

Problem 1.1 compression ratio

A tokenizer encodes the string “Olá, 世界!” into 5 tokens. Compute the compression ratio, and explain in one sentence why a higher ratio matters for a Transformer.

Show solution

Count UTF-8 bytes: “O”, “l”, ”,”, ” ”, ”!” are 1 byte each (5), “á” is 2 bytes, and “世” and “界” are 3 bytes each (6) — total $5 + 2 + 6 = 13$ bytes. Compression ratio $= 13 / 5 = 2.6$ bytes per token. It matters because attention is quadratic in sequence length, so fewer tokens for the same text means less compute (and effectively longer usable context).

Problem 1.2 BPE by hand

Run the lecture’s BPE training algorithm on the string “aaabab” for two merges (bytes: “a” = 97, “b” = 98; break count ties by whichever pair was counted first, scanning left to right). Give the merge table, the final token sequence, and the compression ratio.

Show solution

Initial indices: [97, 97, 97, 98, 97, 98]. Adjacent pair counts: (97, 97) appears 2 times, (97, 98) 2 times, (98, 97) 1 time. Tie between (97, 97) and (97, 98); (97, 97) was counted first, so merge it as token 256 (“aa”). Rewriting left to right: [256, 97, 98, 97, 98]. Second round: (256, 97), (98, 97) each appear once, (97, 98) appears twice — merge (97, 98) as token 257 (“ab”). Final sequence: [256, 257, 257]. Merge table: (97, 97) → 256, (97, 98) → 257. Compression ratio: 6 bytes / 3 tokens = 2.0.

Problem 1.3 baseline trade-offs

For the string “你好世界” (four Chinese characters, 3 UTF-8 bytes each), compute the compression ratio under a character tokenizer and under a byte tokenizer, and state the vocabulary problem each one has. Why does BPE escape both?

Show solution

The string is $4 \times 3 = 12$ bytes. Character tokenizer: 4 tokens, ratio $12/4 = 3.0$ — good compression, but the vocabulary must cover roughly 150K Unicode characters, most of them rare, which wastes capacity. Byte tokenizer: 12 tokens, ratio $12/12 = 1.0$ — a tidy 256-entry vocabulary, but sequences are maximally long under quadratic attention. BPE starts from the 256 bytes (so everything is representable and the vocabulary size is a chosen budget, not an accident of the data) and spends its merges only on byte sequences that are actually frequent — frequent chunks compress to one token, rare ones fall back to more tokens.

Problem 1.4 scaling rule of thumb

Using the compute-optimal rule of thumb from the lecture, how many tokens should a 3B-parameter model be trained on? Name one reason a lab might deliberately train on far more than that number.

Show solution

The Chinchilla rule of thumb is $D = 20N$ : $20 \times 3\times10^9 = 60\times10^9$ , so about 60B tokens. Labs overshoot it deliberately for inference reasons: the rule optimizes training compute only, but a smaller model trained on many more tokens is cheaper to serve, so models intended for heavy inference are trained well past compute-optimal. (The lecture also cautions the constant varies with dataset and architecture.)