Scaling Laws I

The 10,000-B200 problem

Your very wealthy friend has given you ten thousand B200s for a month and asked for a good open-source language model. The infra team is assembled (assignment 2), the pretraining dataset exists (assignment 4) — now you have to pick the run. Architecture, optimizer, depth, batch size: every choice from the earlier lectures reappears, except a wrong one now costs millions of dollars. For many choices you’ll copy well-adopted best practices out of the literature, and that’s fine — but at the frontier “you can’t just copy the choices of others and get something that’s better than the state of the art.”

Scaling laws are the tool. Tatsu calls them a paradigm — for people doing scaling work in big labs, “almost kind of a way of life.” The naive approach tunes hyperparameters on the big runs themselves, which is hopelessly wasteful; the scaling-law approach does all the optimization at small scale and extrapolates up with a simple rule.

A line on a log-log plot says two things: the error decays polynomially, and you are far from your asymptote — near the irreducible-error floor the curve tapers off instead. That’s why this lecture keeps models comfortably bigger than the data (roughly 10x is plenty): it keeps every measurement inside the power-law regime.

Older than the transformer

Theorists have thought about “scaling” for a long time: a generalization bound is exactly an upper bound on loss as a function of training-set size — though a bound, not a realized value. The first genuine data scaling law paper is arguably Cortes, Vapnik, and colleagues at Bell Labs in 1993: training classifiers on large databases is expensive, so fit a curve to how error decays on small samples and extrapolate the learning curve out. Banko and Brill (2001) drew the NLP moral — maybe spend the money on data rather than algorithms, since every learner they tested kept improving predictably with corpus size. Kolachina et al. (2012) explicitly raced functional forms against each other for machine translation BLEU and landed on the same Pow3/Pow4 power laws we use today.

The paper Tatsu insists on citing in every scaling lecture is Hestness et al. (2017): power-law data scaling across speech, translation, and language modeling, years before OpenAI’s versions — plus early forms of emergence (accuracy is discontinuous even when loss is smooth), scaling by compute, and the observation that once you scale by compute, speed and systems optimization turn into accuracy. “A lot of these phenomena were known far before the modern era.”

Why power laws? A three-slide statistics class

Why should log-log plots be linear at all? Because estimation error naturally decays polynomially.

Here is the intriguing mystery: fitted neural exponents are nothing like $-1$. Hestness finds about $-0.13$ for translation and $-0.3$ for speech; Kaplan’s language-modeling data law is $L = (D / 5.4\times10^{13})^{-0.095}$. Still polynomial, but much, much slower. Where do slow rates come from? Nonparametric estimation. To estimate an arbitrary smooth function on the 2D unit box, cut space into boxes of side $n^{-1/4}$: about $\sqrt n$ boxes, each getting $\sqrt n$ samples, for error roughly $1/\sqrt n$. In $d$ dimensions this becomes $n^{-1/d}$ — a log-log slope of $-1/d$. So one mental model is that these networks learn like nonparametric regressors in about ten dimensions. Bahri et al. argue the stronger version — that the exponent literally measures the data’s intrinsic dimensionality — which Tatsu finds interesting but doesn’t fully buy, since intrinsic-dimension estimators are sketchy.

Data scaling you can engineer with

A univariate data law tells you how fast your model learns — nice for forecasting, not much else. The engineering questions are about composition. For mixtures (how much news vs Wikipedia?), the classical view says dataset composition moves the offset of the scaling law while the model class sets the slope — and the intercept as a function of mixture proportion is interestingly shaped, rewarding diversity. So you can fit mixture-dependent scaling laws on small models and extrapolate the optimum (the Data Mixing Laws idea). In practice reality is noisier: the large-scale DataDecide study found that just picking the best mix at small scale and scaling it up works well — which is exactly what you’d predict if slopes don’t change.

For repetition, the data-constrained scaling study showed that up to about 4 epochs, repeated data is nearly as good as fresh data; past that, the realized curve falls badly below the fresh-data projection (at 40 epochs, repeating is worthless), and a modified functional form with an “effective data” term predicts the decay. Pushing to the extreme — Tatsu’s recent infinite-compute work with Percy — epoching and parameter count both saturate, and you end up reaching for ensembles to squeeze more from fixed data. And filtering is scale-dependent: with little compute you filter aggressively down to the highest-quality data; with lots of compute the optimal filter loosens, because repeating pristine data loses to admitting fresh, lower-quality data.

Designing the model without training it

The scaling-law design procedure: train a few smaller models across a range of compute, fit lines on log-log plots, and if the fit is clean, trust that the gap persists and deploy the winner on the big run. Are transformers better than LSTMs? The brute-force answer trains an LSTM GPT-3 for tens of millions; the scaling-law answer (in Kaplan) shows LSTMs with a worse intercept and arguably a worse slope — and a worse slope is disqualifying, because it means your model loses harder as you scale. Every architecture paper now carries this plot. The Tay et al. T5-scale study is the striking validation: at compute scales far below today’s, its scaling trends already picked out the interventions frontier models actually adopted — gated linear units and mixture-of-experts scale well, efficient-attention variants like Performer don’t.

The same analysis settles shape questions. One layer is catastrophic; beyond that, depth gives diminishing returns. The trick is to find scale-invariant quantities: raw layer count grows with scale, but the aspect ratio (width over depth) has its minimum near 100 at every model size Kaplan tested, shifting only slightly — which is what licenses “fix the aspect ratio and scale up.” But not all parameters are created equal: Kaplan’s with-embedding curves looked so funky they dropped embedding parameters from the count entirely — a choice with real consequences later. In the mixture-of-experts era the question sharpens, and the Apple/MIT IsoFLOP study shows optimal sparsity rising with total parameters — inactive experts still reduce loss.

There’s one more caution before you trust any of this: scaling laws live on the perplexity side. In the Yi Tay architecture study, perplexity scales beautifully with parameters, yet the best-perplexity model (NL12) loses badly on SuperGLUE to a perplexity-mediocre one (NL32-XL). Upstream-downstream transfer is much less certain than the clean lines suggest — don’t be the pretraining person who hands over a great perplexity and someone else’s problem.

The two knobs you must retune: batch size and learning rate

Most choices you’ll copy; batch size and learning rate you must re-derive, and they’re coupled. The systems lectures want batch size as large as possible (data parallelism feeds on it), so the real question is how large before you suffer.

The reason this lives in the scaling lecture: $B_{\text{crit}}$ itself follows a power law in the target loss — McCandlish et al. fit $B_{\text{crit}} \approx 2.1\times10^8 \cdot L^{-4.8}$ tokens — so the better the model you’re aiming for, the larger the batch you can productively use. Variance reduction matters most when you’re grinding out tiny differences near the minimum.

Learning rate is the messier half, saved mostly for the advanced lecture. Naively, the optimal learning rate shifts as you widen a network — the rule of thumb scales it like one over width. Two philosophies coexist: fit how the loss-vs-learning-rate minimum moves and extrapolate it (its shift is quite predictable), or reparameterize initialization and per-layer step sizes so the minimum stays put across scales — the muP approach. Both have run successfully at scale; anecdotally the extrapolation camp is winning.

More data or a bigger model? The Chinchilla saga

Someone hands you FLOPs, and FLOPs are roughly parameters times data. A tiny model fed tons of tokens flatlines — pure waste. Rosenfeld and Kaplan almost simultaneously proposed joint functional forms; Rosenfeld’s is just two scaling laws added together, $\text{Error} = n^{-\alpha} + m^{-\beta} + C$. Take limits to sanity-check it: infinite data leaves a pure model-size law, infinite model a pure data law. These fits extrapolate remarkably well from a corner of small, cheap runs to the high-compute regime — so minimize the fitted surface subject to your FLOPs constraint. Kaplan’s solution said $N_{\text{opt}} \propto C^{0.73}$, $D_{\text{opt}} \propto C^{0.27}$: tokens per parameter fall with compute, so train ever-bigger models. That prescription drove the GPT-3-era stampede toward hundred-billion-parameter dense models.

Then Hoffmann et al. (2022) — Chinchilla — said the fits were badly off. They estimated the trade-off three independent ways: the lower envelope of training curves, IsoFLOP profiles (fix a FLOP budget, sweep the parameter-data split, fit the valley — Tatsu’s favorite, and the method that has endured everywhere from diffusion models to MoE sparsity), and a parametric joint fit. Methods 1 and 2 agree almost exactly.

Why did two reasonable papers disagree so badly? The autopsy is the real lesson. One replication showed a chain of seemingly minor choices walks Kaplan’s exponent exactly to Chinchilla’s: count the last-layer (softmax) parameters that Kaplan had excluded along with embeddings, fix warmup schedules that were too long for the smallest models, and tune batch size per scale instead of fixing one big value. Pearce and Song, without training anything, simulated Kaplan’s setup from Chinchilla’s fitted curves and reproduced the 0.73 from just the non-embedding parameter convention plus Kaplan’s lower compute range. And the fun addendum: Chinchilla’s method 3 never quite agreed with 1 and 2 — until Epoch AI extracted the data points from the paper’s own plots, refit the surface, and found the original had underfit. The refit lands almost exactly on tokens-per-parameter of 20. The authors were more right than they knew.

One closing twist: you probably don’t want the Chinchilla ratio. It optimizes training compute, but most frontier-lab compute goes to R&D and serving, and serving wants small, capable models. Hence deliberate “overtraining”: GPT-3 sat at 2 tokens per parameter, Chinchilla at 20, Llama 2 70B at 29, Mistral 7B at 110, Llama 3 70B at 215. The more usage you expect, the more it pays to keep training past optimal.

Check yourself

Exercises in the lecture’s spirit — fit, extrapolate, and take limits on paper.

Problem 9.1 exponents and effective dimension

A data scaling law fits your model’s excess loss as $L = 3.9 \cdot n^{-0.125}$ for $n$ training tokens. What is the slope on a log-log plot? Under the nonparametric mental model, what effective dimension is the model learning in? And how much more data do you need to halve the excess loss — compared against a classical estimator with $1/n$ scaling?

Show solution

The log-log slope is the exponent, $-0.125$. The nonparametric rate is $n^{-1/d}$, so $1/d = 0.125$ gives an effective dimension of $d = 8$. Halving excess loss needs a data multiplier $m$ with $m^{-0.125} = 1/2$, i.e. $m = 2^{1/0.125} = 2^8 = 256$ times the data. A classical $1/n$ estimator (slope $-1$) needs only $2\times$. That gap is the whole economics of scale: slow exponents make each halving of loss brutally expensive, which is why the intercept-moving interventions of this lecture matter so much.

Problem 9.2 compute-optimal allocation

You are handed $C = 10^{24}$ FLOPs. Using $C \approx 6ND$ and the Chinchilla prescription of $D/N \approx 20$, solve for the model size $N$ and token count $D$. Then explain qualitatively how a Kaplan-style rule ($N_{\text{opt}} \propto C^{0.73}$) and a serving-heavy deployment would each pull your answer.

Show solution

Substitute $D = 20N$ into $6ND = C$: $120 N^2 = 10^{24}$, so $N^2 = 8.3\times10^{21}$ and $N \approx 9.1\times10^{10}$ — about a 91B-parameter model on $D = 20N \approx 1.8$T tokens. Kaplan’s $C^{0.73}$ rule puts far more of each marginal FLOP into parameters, so it would prescribe a much larger model trained on fewer tokens — the GPT-3-era pattern that Chinchilla showed was undertrained. A serving-heavy deployment pulls the opposite way: shrink $N$ below 91B and push $D$ well past 20 tokens per parameter (“overtraining”), accepting worse training-compute efficiency to cut inference cost.

Problem 9.3 critical batch size

A batch-size sweep at a fixed target loss fits $S_{\min} = 5\times10^4$ steps and $E_{\min} = 1.5\times10^9$ examples. Compute the critical batch size, then the steps and examples used if you train exactly at $B_{\text{crit}}$, and verify they satisfy $S/S_{\min} - 1 = (E/E_{\min} - 1)^{-1}$. Which regime are you in on either side of $B_{\text{crit}}$, and why does $B_{\text{crit}}$ grow as your target loss shrinks?

Show solution

$B_{\text{crit}} = E_{\min}/S_{\min} = 1.5\times10^9 / 5\times10^4 = 3\times10^4$ examples per batch. At the balance point each side costs twice its minimum: $S = 2S_{\min} = 10^5$ steps and $E = S \cdot B_{\text{crit}} = 3\times10^9 = 2E_{\min}$ examples. Check: $S/S_{\min} - 1 = 1$ and $(E/E_{\min} - 1)^{-1} = 1$. Below $B_{\text{crit}}$ you are noise-limited — added examples reduce gradient variance, so doubling the batch roughly halves the steps (perfect scaling). Above it you are bias-limited: the local gradient direction disagrees with the path to the optimum no matter how noiseless it is. $B_{\text{crit}}$ grows as target loss falls (the fitted $L^{-4.8}$ power law) because near the minimum you are resolving finer differences, so variance reduction is worth more.

Problem 9.4 slopes versus intercepts

Three training recipes fit as $L_A = 5.2 \cdot N^{-0.09}$, $L_B = 5.6 \cdot N^{-0.09}$, and $L_C = 4.4 \cdot N^{-0.075}$. Which comparisons can you extrapolate confidently? Find where recipe B overtakes recipe C, and state the lecture’s verdict on interventions like C.

Show solution

A vs B share a slope, so their ratio $5.2/5.6 \approx 0.93$ holds at every scale — A wins everywhere, and this is the trustworthy kind of extrapolation (parallel lines, the common case). C has a better intercept but a worse slope, so it wins small and loses big. Crossover with B: $4.4 N^{-0.075} = 5.6 N^{-0.09}$ gives $N^{0.015} = 5.6/4.4 \approx 1.273$, so $N = \exp(\ln 1.273 / 0.015) \approx e^{16.1} \approx 10^7$ parameters — tiny by modern standards. C looks great in small experiments and is exactly the intervention to reject: “you certainly don’t want something where your slopes are worse, because that means as you scale up larger and larger, your models will eventually do worse.”