Architectures, Hyperparameters

Everything you didn’t want to know

Architecture “has always been pretty inscrutable” to Tatsu, so his approach is to tell you everything: track every dense model release — over 19 in one year — into one comparison table, and learn from the experience of others, since the best teacher (training your own models and varying things) doesn’t fit in a lecture. With that much data, patterns emerge: some choices are fixed across every effective architecture; others vary freely without hurting anything.

The historical arc: experimentation through GPT-3, then Llama 2 lands and “everyone’s like, wow, Llama 2 is great, I want my own Llama 2” — a wave of Llama-alikes, then stability modifications, then long-context variations. It’s why Assignment 1 asks for prenorm, RoPE, SwiGLU, and no biases: “we’ve copied a lot of this over from Llama, so did everyone else.”

Where the norm goes (and what kind)

The one thing everyone agrees on: the original transformer put the LayerNorm in the wrong place — inside the residual path, after each block (postnorm). Every modern model moves it outside the residual stream, usually to the front of each block (prenorm); the lone exception is OPT-350M, and “I don’t know why.”

The original motivation was removing learning-rate warmup, but the reason prenorm stuck is “keep your residual stream clean”: the input $x$ propagates untouched to the top of the network, gradients flow straight through the backward pass, and gradient sizes stay constant across layers at initialization where postnorm distorts them (Xiong 2020); prenorm also shrinks gradient spikes (Salazar and Nguyen). Once outside the stream, the norm needn’t come before the computation: Grok and Gemma 2 also norm after each block (“double norm”), OLMo 2 only after.

Activations: the gate is what matters

There’s a zoo — ReLU, GeLU, Swish, GeGLU, SeLU, SwiGLU, LiGLU — and Tatsu once made it “a point of pride to never know what a SwiGLU is.” The compression: plain activations work (Chinchilla on ReLU, GPT-3 on GeLU), but nearly every credible modern model uses a gated linear unit.

The evidence is unusually clean: Shazeer 2020 trained parameter-matched replicates with error bars and the GLU variants win almost every time; Narang et al. corroborate (SwiGLU final loss 1.789 vs 1.838 vanilla). Gating isn’t necessary — Nemotron-340B used a squared ReLU, “a crazy choice, but that works too” — but it’s a nearly free gain. One idea that fell out of favor: parallel layers (GPT-J, PaLM), computing attention and MLP simultaneously for a claimed 15% speedup; serial won, the systems gain no longer worth what is effectively “you’ve lost half of your depth.”

RoPE: position as rotation

Attention is position-independent — shuffle the tokens and the inner products don’t change — so position must be injected. Sines and cosines (original transformer), learned absolute embeddings (GPT-1/2/3), and relative offsets added to the attention matrix (T5, Chinchilla) all had runs; what won, in essentially every model past 2024, is RoPE — remarkably, a GPT-J-era innovation from a little-known blog post.

The trick rests on one fact: inner products are invariant to rotation. So rotate each word’s vector by an angle proportional to its position. In “we know that,” we stays put and know rotates one step; in “of course we know,” we rotates two steps and know three — the relative angle is still one step, so the score is unchanged. In $d$ dimensions, do the simplest possible thing: cut the vector into $d/2$ pairs and rotate each pair in 2D, with frequencies from fast (adjacency) to slow (long-range structure). The sines and cosines multiply the queries and keys inside every attention operation — never added — so nothing leaks absolute position.

Hyperparameters nobody varies (much)

The search space looks daunting until you notice the space people actually use:

Hyperparameter	Consensus	Notable exceptions
$d_{ff} / d_{model}$	4 (non-gated), $8/3 \approx 2.67$ (GLU)	Llama 2 / Mistral at 3.5; T5 at 64
heads $\times$ head-dim vs $d_{model}$	ratio $\approx 1$	T5 (16), LaMDA (2)
Aspect ratio $d_{model} / n_{layer}$	$\approx$ 100–200	T5-11B (33)
Vocabulary	30–50k monolingual, 100–250k multilingual	—
Regularization	weight decay 0.1, no dropout	Qwen keeps dropout

The feedforward ratio has empirical cover: Kaplan et al. 2020 found a flat basin from about 1 to 10, then sharp degradation. T5’s bold $d_{ff} = 65{,}536$ against $d_{model} = 1024$ — a 64$\times$ multiplier, justified by keeping TPU matmuls fat — still trained a fine model, but T5 v1.1, the improved version, quietly went back to a standard 2.5 multiplier.

Aspect ratio is the interesting one: depth buys expressiveness but is miserable to parallelize — cutting up layers means pipeline parallelism, “something that most people really, really do not want to deal with” — while width splits easily via tensor parallelism. The trade-off lands near 100 (GPT-3 and Mistral at 128, Llama at 102), inside a wide flat optimum at every scale. Regularization is the counterintuitive one: single-pass SGD over trillions of tokens essentially cannot overfit, yet modern models keep weight decay 0.1 — because it interacts with learning-rate decay, converging to a better minimum under a cosine schedule (Andriushchenko et al. 2023). It’s an optimization intervention, not a regularizer.

Stability: beware of softmaxes

If the loss spikes and blows up mid-run, you may have spent millions on an untrainable model. The usual suspects are the softmaxes — an exponential divided by a normalizer that can hit zero — and a language model has exactly two: the output softmax and the attention softmax.

For the output, note $\log P(x) = U_r(x) - \log Z(x)$: the logit $U_r$ is the well-behaved residual-stream output, but the log-normalizer can drift. Since softmax is shift-invariant, penalize the drift without changing the model: add $\alpha \log^2 Z(x)$ to the loss — the z-loss (Devlin 2014), used by PaLM with $\alpha = 10^{-4}$ and since by Baichuan 2, DCLM, and OLMo — pinning $\log Z$ near 0 where the computation is numerically stable.

For attention, the fix is QK norm: RMSNorm the queries and keys right before their dot product, so the softmax inputs arrive at scale roughly one. It came from the multimodal world (Idefics, Chameleon) and is now standard — Gemma, Qwen 3, OLMo — with a slight quality gain in NVIDIA’s comparison (perplexity 10.84 vs 11.19 baseline), because it lets you raise the learning rate. The blunter logit soft-capping, $\text{logits} \leftarrow c \cdot \tanh(\text{logits}/c)$ (Gemma 2/3/4), is safe but capping confidence costs quality alone (11.24 vs 11.19).

Attention variants: paying for inference

Training keeps GPUs happy: over a whole sequence, attention does about $bnd^2$ arithmetic against $bnd + bhn^2 + d^2$ memory accesses, so intensity is high. Generation is different — autoregressive decoding emits one token at a time, reusing past keys and values from the KV cache, and that changes the accounting.

MQA costs real expressiveness, though. Grouped-query attention (GQA) interpolates: keep all query heads, reduce keys and values to a few groups. Ainslie 2023 shows the trade is remarkably favorable — GQA matches multi-head quality at close to MQA’s speed — which is why almost every modern model adopts it.

The other standard trick is sliding-window attention interleaved with full attention: GPT-3 already alternated full and banded patterns, and Cohere Command A revived the idea with one full-attention layer per four, the rest attending only within a local window — local layers aggregate nearby information, periodic global layers integrate it across the context. Llama 4, Gemma 4, and OLMo 3 all do a version, some using NoPE (no position embedding) on the long-range layers; Qwen 3.5 swaps the cheap layers for a state-space model, which is where the next lecture picks up.

Check yourself

Napkin-scale exercises in the lecture’s survey spirit — each uses a rule of thumb or derivation from above.

Problem 3.1 GLU parameter matching

A model has $d_{model} = 4096$ and a non-gated FFN with the standard $d_{ff} = 4\, d_{model}$. You switch it to SwiGLU. What $d_{ff}$ keeps the FFN parameter count unchanged, and what is the new ratio $d_{ff}/d_{model}$?

Show solution

Non-gated parameters: $W_1$ and $W_2$ give $2 \cdot d_{model} \cdot d_{ff} = 2 \cdot 4096 \cdot 16384 \approx 134\text{M}$. SwiGLU adds the gate matrix $V$, so parameters are $3 \cdot d_{model} \cdot d_{ff}'$. Setting $3\,d_{ff}' = 2 \cdot 16384$ gives $d_{ff}' = \frac{2}{3} \cdot 16384 \approx 10{,}923$ (rounded in practice to a hardware-friendly 10,752 or 11,008). The ratio is $\frac{8}{3} \approx 2.67$ — exactly the 2.6-ish cluster in the model survey, and why Shazeer’s GLU comparisons were fair fights.

Problem 3.2 RoPE relativity check

In 2D, RoPE embeds word $x$ at position $i$ as $f(x, i) = R_{i\theta}\,W x$, where $R_{\alpha}$ is rotation by angle $\alpha$. Show that $\langle f(x,i), f(y,j) \rangle$ depends only on $i - j$, and explain in one sentence why additive sine embeddings fail the same test.

Show solution

Rotations preserve inner products, so $\langle R_{i\theta} u,\; R_{j\theta} v \rangle = \langle u,\; R_{i\theta}^\top R_{j\theta} v \rangle = \langle u,\; R_{(j-i)\theta} v \rangle$ with $u = Wx$, $v = Wy$. The absolute positions cancel; only the offset survives, so shifting the whole sentence leaves every attention score unchanged. Additive embeddings compute $\langle v_x + PE_i,\; v_y + PE_j \rangle$, which expands into cross terms like $\langle PE_i, v_y \rangle$ that carry absolute position — the inner product is not a function of $i - j$ alone.

Problem 3.3 KV cache accounting

A 32-layer model has $d_{model} = 4096$ with 32 attention heads of dimension 128, storing its KV cache in bf16. (a) How many bytes of KV cache per token with full multi-head attention? (b) Serving batch 16 at context length 8192, does the cache fit alongside the weights on one 80 GB GPU? (c) What does GQA with 8 KV heads change?

Show solution

(a) Per token per layer: keys and values, each $d_{model}$ elements at 2 bytes, so $2 \cdot 4096 \cdot 2 = 16{,}384$ bytes; across 32 layers, $\approx 0.5$ MB per token. (b) Cache: $0.5\,\text{MB} \cdot 8192 \cdot 16 \approx 69$ GB — before the weights (a ~7B model in bf16 is another ~14 GB), so no. (c) GQA with 8 of 32 KV heads shrinks keys and values by $4\times$: about 17 GB of cache, which fits comfortably. This is exactly the memory-movement term that makes incremental decoding memory-bound, and why nearly every modern model ships with GQA.

Problem 3.4 sizing by aspect ratio

Size a roughly 7B-parameter model using the lecture’s rules of thumb: aspect ratio $d_{model}/n_{layer} = 128$, SwiGLU FFN at $\frac{8}{3} d_{model}$, and heads matching $d_{model}$. Estimate $n_{layer}$ and $d_{model}$, ignoring embeddings.

Show solution

Per layer: attention $Q, K, V, O$ cost $4\,d^2$; the SwiGLU FFN costs $3 \cdot d \cdot \frac{8}{3} d = 8\,d^2$; total $\approx 12\,d^2$, so $N \approx 12\,L\,d^2$. Substituting $d = 128L$: $12 \cdot L \cdot (128L)^2 = 196{,}608\,L^3 = 7\times10^9$, giving $L^3 \approx 35{,}600$, so $L \approx 33$ and $d \approx 4224$. Rounding to friendly numbers: 32 layers, $d_{model} = 4096$, 32 heads of dimension 128 — almost exactly the published Llama-7B shape. The rules of thumb reproduce real models because that’s where they came from.