Scaling Laws II

Two ways to tame scale-sensitive hyperparameters

The classical scaling canon — Kaplan, Hestness, Chinchilla — leaves you current to about 2022. Since then, the detailed scaling papers have mostly come from the Chinese open-source community, and Tatsu’s goal for this lecture is to curate them: what would you need to “speedrun your way from Chinchilla all the way to Kimi K2”? The organizing problem is that some hyperparameters are genuinely scale sensitive. Optimizer behavior, initialization, learning rate, and batch size all shift as models grow, and you cannot brute-force a big run hundreds of times to find them.

Two papers anchor the lecture because they take opposite approaches to this problem. MiniCPM (2024, a high-performance 1–2.5B model from a Tsinghua-affiliated group) tries to stabilize the hyperparameters — reparameterize the network so the optimal learning rate stops moving with scale. DeepSeek LLM (2024, “one of the more nicely executed scaling analyses in the open world”) instead fits scaling laws to the hyperparameters themselves and extrapolates. Everything else in the lecture hangs off this dichotomy.

MiniCPM: muP and the scaling ladder

MiniCPM’s plan is a scaling ladder: train models from 9M to 0.5B parameters, nail the sensitive hyperparameters on the ladder, then jump the roughly 5x gap to the release model with the hyperparameters already right the first time. The enabling trick is a special initialization scheme, muP, whose whole point is that the optimal learning rate should be the same as you scale up and down. Concretely (with scale_emb = 12, scale_depth = 1.4, init_std = 0.1, lr = 0.01), MiniCPM multiplies the embedding output by 12, scales each residual-connection increment by $1.4/\sqrt{\text{num\_layers}}$, sets the initialization std of every matrix-shaped tensor to $0.1/\sqrt{d_m/d_{\text{base}}}$, scales the learning rate of each matrix tensor by $1/(d_m/d_{\text{base}})$ — per-parameter learning rates, which is exotic if you haven’t seen them — and divides the LM-head logits by the same ratio.

Does it work? In their experiments, very cleanly: sweeping learning rate across model sizes from 0.04B to 2.1B, the loss minimum sits essentially at $10^{-2}$ for every size. That removes learning rate from the tuning budget. Batch size, however, still moves — it depends on both data and model size. MiniCPM runs many fixed-batch training runs, fits quadratics to equal-loss contours, and recovers exactly the Kaplan-style critical batch size result: optimal batch size is a power law in the target loss,

$\log(\text{BS}) = -6.24 \cdot \log(L) + 20.91,$

so lower target loss means bigger batch, and given a loss target from your scaling law you can set batch size precisely.

WSD: the trapezoid that makes Chinchilla cheap

MiniCPM’s other lasting contribution fixes an annoying cost in Chinchilla-style analysis. A cosine schedule needs to know the total training horizon before you start — so a run to 8M sequences cannot be produced by continuing a 4M-sequence run, and every point on the data axis means retraining from scratch. Fitting a scaling law becomes quadratic in cost.

Because the stable phase carries no horizon information, you can rewind any run to its last stable checkpoint, continue at the constant rate, and re-decay whenever you want a measurement. Each extra data point costs one decay (roughly 10% of a run) instead of a full retrain — the scaling-law sweep along the data axis becomes linear. Performance-wise, WSD curves “look a little bit funky”: they underperform through the stable phase, then reclaim everything during the decay, matching and sometimes exceeding a well-tuned cosine. Anecdotally cosine is slightly better in many cases, but WSD is basically as good and far more versatile.

Equipped with WSD, MiniCPM replicates Chinchilla methods 1 and 3 — which Tatsu calls “the least reliable of the Chinchilla methods” — and gets reasonable-looking fits, but with pretty different exponents from Chinchilla and a startlingly high data-to-model ratio ($D_{\text{opt}}/N_{\text{opt}} \approx 96$ at $10^{21}$ FLOPs from the joint fit). Whether that reflects reality or a strange fit is, in his words, highly unclear.

DeepSeek: fit the hyperparameters a scaling law

DeepSeek uses no muP at all. Instead they run extensive grid searches over learning rate and batch size at several compute scales, mark the loss minimizer at each scale, and fit power laws in compute to the minimizers:

$\eta_{\text{opt}} = 0.3118 \cdot C^{-0.1250}, \qquad B_{\text{opt}} = 0.2920 \cdot C^{0.3271}.$

Higher compute means bigger batches and lower learning rates, and the stars — their actual 7B and 67B production runs — sit on the extrapolated lines. The batch-size fit genuinely looks like a line; the learning-rate fit is, per Tatsu, “not quite… the best linear fits that I’ve seen in my life.” That is the weakness of the grid approach: unless your grid lands in the right place, quantization error can leave you fitting an arbitrary-looking line. For the Chinchilla replication they use a WSD-style schedule with a quirky two-step decay (down to 31.6% of max after 80% of tokens, 10% after 90%), run clean IsoFLOP sweeps — method 2, which gives them much nicer curves than MiniCPM’s choices — and accurately predict the final loss of both production models from small-scale fits. The takeaway: Chinchilla’s analysis has been independently replicated at large-ish compute, and it is generally sound.

The newer releases compress all of this to a paragraph because it is now assumed knowledge. Qwen 2.5 fits hyperparameter scaling laws exactly like DeepSeek; Qwen 3 says it does what Qwen 2.5 did. What is new is MoE scaling: companies switching to mixture-of-experts redo the analysis over sparsity.

Hunyuan does the same and lands at 96 tokens per active parameter; LLaMA 3 runs IsoFLOPs (about a 39:1 token-to-parameter ratio) and adds a two-stage compute-to-downstream prediction — a power law from compute to negative log-likelihood, then a sigmoid from log-likelihood to benchmark accuracy, with visible systematic deviations but a useful coupling between loss and downstream numbers. MiniMax-01 does architecture selection by scaling law: softmax attention, lightning (linear) attention, and a hybrid all scale comparably, which is the evidence they use to ship the hybrid.

The StepFun landscape: what actually governs LR and batch size

If you collect the published prescriptions — OpenAI, Microsoft, DeepSeek, Porian, MiniCPM, MeiTuan — they do not even agree on the inputs: Kaplan scales batch size with terminal loss, DeepSeek with compute, StepFun with data. The StepFun “Predictable Scale” study is the largest attempt to settle this: burn a ton of compute gridding learning rate and batch size across model and data sizes, and map the whole loss landscape. Two observations survive. First, the landscape is convex and smooth along each slice — the minimum is well defined and gridding is a sane procedure. Second, the scaling trends separate cleanly:

Under Chinchilla scaling $N$ and $D$ are both driven by compute, the two dependencies partially cancel, and you recover something DeepSeek-shaped: learning rate falls with compute, batch size rises, albeit with different exponents. The fitted laws transfer reasonably to MoEs when you control for active parameters, but the optima do shift when the training data changes — the constants are contingent on the data recipe. Asked whether students should just use these laws instead of running their own grids, Tatsu’s answer is: near StepFun’s compute regime, these are probably better defaults than anything you would pull out of a hat, but any serious pre-training run has enough minor differences that people redo the analysis.

Optimizers at scale, and the Muon story

Optimizers are the sharpest case study in scale-dependent conclusions. On the NanoGPT speedrun (the benchmark that inspired assignment 1), Muon posted big, cheap gains over Adam. Then scaling studies — the “Fantastic Pretraining Optimizers” paper by Kaiyue Wen, David Hall, Tengyu Ma, and Percy — complicated the picture. Two traps first: tuning. Badly tuned AdamW looks terrible next to everything (tuning its learning rate alone is a 2x speedup: $8\times10^{-3}$ versus $6\times10^{-4}$ at 130M), and hyperparameters do not transfer across optimizers — Lion’s optimal weight decay is around 0.6, wildly far from Adam’s. Then the scaling result: measured properly, matrix-based optimizers (Muon, Soap, Kron) consistently beat scalar-based ones, but the speedup over AdamW shrinks with scale, from about 1.4x at 130M to about 1.1x at 1.2B.

Two axes must always be checked in this kind of study: compute, and the Chinchilla ratio of tokens to parameters. Some algorithms shine in the overparameterized regime and fall apart data-rich, or vice versa — a big confounder that even careful papers skip. And establishing scaling at all is nontrivial: the Marin project (Percy’s open-source training effort, with Will Held) published a run where a beautiful Chinchilla-style trend held for orders of magnitude past $10^{20}$ FLOPs, then drifted, worsened, and finally diverged outright — fixed only by more careful muP-style parameterization and optimizer changes.

Muon itself is worth understanding. Take a standard momentum step $B_t = \mu B_{t-1} + G_t$, but before applying the update, orthogonalize it: write $B_t = U S V^\top$ and use $U V^\top$ — all singular values pushed to 1 — computed not by an SVD (slow on GPUs) but by NewtonSchulz5, a matmul-only iteration that approximates the orthogonalization. The intuition mirrors Adam: Adam normalizes per coordinate so every coordinate moves at unit size; Muon normalizes in the spectral norm so every direction moves at unit size. That only makes sense for matrix-valued parameters, so Muon handles the attention and MLP matrices while AdamW keeps the vector-shaped ones (RMSNorm gains and the like). The story’s ending: after scaling studies suggested the gains fade, Kimi K2 trained its entire flagship model with Muon — with bells and whistles to prevent blowups — and it is an outstandingly good model. No ablation exists at that scale, so “is Muon better than AdamW at scale” remains open; but Muon demonstrably works at scale, and the small-scale-to-large-scale pipeline, speedrun included, is still how the field does science.

Deriving muP: physicist’s math for hyperparameters

Finally, what muP actually is. The goal picture: under standard parameterization, widening the network shifts the optimal learning rate; under muP it stays put. The knobs are per-layer initializations, per-parameter learning rates, and occasional residual scalings. The derivation (following the “spectral condition for feature learning” paper by Yang, Simon, and Bernstein — the accessible one, next to the “somewhat inscrutable” tensor-programs series) rests on two asserted invariants as width $n_l$ grows: A1, activations at initialization stay $\Theta(1)$; A2, the change in activations after one gradient step stays $\Theta(1)$. A2 is feature learning — in the neural-tangent-kernel limit activation changes vanish with width, and you don’t want that. If entries are $\Theta(1)$, the activation vector’s norm is $\Theta(\sqrt{n_l})$.

For a deep linear network $h_l = W_l h_{l-1}$ with Gaussian init, matrix concentration gives $\lVert W_l \rVert_* \to \sigma(\sqrt{n_{l-1}} + \sqrt{n_l})$, and propagating $\lVert h_l \rVert \approx \lVert W_l \rVert_* \lVert h_{l-1} \rVert$ inductively pins down the init scale. For A2, the SGD update is a rank-1 outer product $\Delta W_l = -\eta \nabla_{h_l}\ell \, h_{l-1}^\top$; expanding $\Delta h_l = W_l \Delta h_{l-1} + \Delta W_l (h_{l-1} + \Delta h_{l-1})$, demanding every term be $\Theta(\sqrt{n_l})$, and adding the least palatable assumption — the loss decreases by $\Theta(1)$ per step regardless of scale — lets you solve for the learning rate. This is order-of-magnitude bookkeeping, not rigor: “It’s physicists math. No offense to physicists.”

Empirically the program holds up well. Cerebras-GPT found muP models tracked their projected scaling law almost exactly while standard-parameterization models fluctuated wildly. An independent stress-test replication found learning-rate transfer across 4x width jumps works as advertised, and that many modern components the theory doesn’t cover — SwiGLU, initialization variations — mostly don’t break it. Three things do: learnable RMSNorm gains (removable with little performance loss), sign-based optimizers like Lion, and — most concerning — strong decoupled weight decay, the one significant failure.

Check yourself

Exercises in the lecture’s spirit — scaling-law arithmetic and order-of-magnitude reasoning, paper and calculator only.

Problem 11.1 WSD compute accounting

You want Chinchilla-style loss measurements for one model size at 20N, 40N, 60N, and 80N tokens. Compare total training compute using (a) cosine schedules, which require a fresh run per horizon, and (b) a WSD schedule where each measurement costs a decay phase equal to 10% of that horizon on top of one shared stable run to 80N.

Show solution

Cosine: each horizon retrains from scratch, so $20 + 40 + 60 + 80 = 200N$ tokens of training. WSD: one stable run to $80N$, plus decays of $0.1 \times (20 + 40 + 60 + 80) = 20N$, totaling $100N$ — half the cost with just four points. The gap grows with the number of measurements: cosine cost is quadratic in the number of horizons (you re-pay the full prefix each time) while WSD pays the longest run once plus a 10% surcharge per point, which is why the lecture calls WSD the enabler of cheap Chinchilla analysis.

Problem 11.2 DeepSeek law extrapolation

Using DeepSeek’s fits $\eta_{\text{opt}} = 0.3118\, C^{-0.125}$ and $B_{\text{opt}} = 0.2920\, C^{0.3271}$: if compute grows by a factor of $10^4$ (say $10^{20} \to 10^{24}$ FLOPs), by what factors do the optimal learning rate and batch size change? Which of the two fits did the lecture flag as less trustworthy, and why?

Show solution

Learning rate scales by $(10^4)^{-0.125} = 10^{-0.5} \approx 0.32$ — it drops about 3.2x. Batch size scales by $(10^4)^{0.3271} = 10^{1.31} \approx 20$x. So four orders of magnitude of compute turn into a roughly 3x smaller learning rate and 20x bigger batch. The learning-rate fit is the shakier one: the gridded optima scatter widely at fixed compute (different model shapes at the same FLOPs), and grid quantization error means the “line” through them may be partly an artifact — whereas the batch-size points genuinely fall on a line.

Problem 11.3 muP per-layer learning rates

A transformer MLP has an up-projection with fan-in $n$ and fan-out $4n$, and a down-projection with fan-in $4n$ and fan-out $n$. Under muP, what learning-rate ratios do the two layers get (relative to a square $n \times n$ layer) with SGD, and with Adam? If you double the width $n$ everywhere, what happens to each Adam learning rate?

Show solution

SGD muP sets $\eta_l = \Theta(n_l/n_{l-1})$: the up-projection gets $4n/n = 4$x the square layer’s rate, the down-projection $n/4n = 1/4$x. Adam muP sets $\eta_l = \Theta(1/n_{l-1})$: up-projection $1/n$, down-projection $1/(4n)$ — the wide-fan-in layer gets a 4x smaller rate. Doubling the width doubles every fan-in, so every Adam per-layer rate halves; that is exactly the mechanism by which the base learning rate can stay fixed while the effective rates absorb the scale change — the invariance MiniCPM verified empirically.

Problem 11.4 critical batch size

MiniCPM’s fit is $\log(\text{BS}) = -6.24 \log(L) + 20.91$ (natural log). Your scaling law says a planned run will reach loss 3.0 where a previous run reached 4.0. By what factor should the batch size grow, and what is the general principle this reflects from the Kaplan lineage?

Show solution

Only the loss-dependent term matters for the ratio: $\text{BS}_{3.0}/\text{BS}_{4.0} = (3.0/4.0)^{-6.24} = (4/3)^{6.24}$. Since $\ln(4/3) \approx 0.288$, this is $e^{6.24 \times 0.288} \approx e^{1.80} \approx 6$ — the batch should grow about 6x. The principle is critical batch size scaling: lower target loss supports (and needs) larger batches, so batch size is set from the loss target you back out of the scaling law, not from the hardware. This is also why, under compute scaling where loss keeps falling, optimal batch size keeps rising — the DeepSeek and StepFun batch laws are the same phenomenon in different variables.