Mid/Post-Training: SFT, RLHF

From GPT-3 to ChatGPT: the two-part recipe

Everything in the course so far gets you a souped-up GPT-3 — a very big, strong base model whose utility is “ultimately a little bit limited.” The only things people really did with GPT-3 were copywriting and fun tasks where you don’t need reliability. This lecture covers the process that closes most of that gap — post-training — and Tatsu is upfront that it is “really artisanal”: messy data work rather than clean architecture. Next lecture goes from ChatGPT to o1-style thinking models.

Pre-training remains the basis — “if we ignore pre-training and we just try to train our way to victory, we will get none of the things that we want. Absolutely none of it.” But once you have the “primordial soup” of pre-training, you have to extract the behaviors you want with explicit data collection and steering. The recipe, straight from the RLHF papers, has two parts: collect demonstration data and supervised-fine-tune on it, then do reinforcement learning against human judgments of which responses are good.

One caveat governs the whole lecture: frontier post-training information is sparse. The detailed public sources — Stiennon 2020’s learning-to-summarize paper with its annotation guidelines, Anthropic’s HH paper from 2022 — predate the ChatGPT-era competition. Since then the data has become trade secret, vividly illustrated by leaked Scale AI documents in which annotators were instructed to reverse-engineer why GPT-4 beat Bard and “write responses better than GPT-4.” Algorithms are well understood; the leverage is in the data.

SFT is a data problem

How to SFT is “basically exactly the same as pre-training” — the whole method slide is a standard training loop played almost as a joke:

model.train()
for epoch in range(num_epochs):
    for batch in train_dataloader:
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

So the lecture walks the data chronologically. FLAN (used for T5) was the visionary first move: NLP already has supervised datasets, so convert them all into input–output pairs and train on everything. But the data is strange — an Enron email with “write a subject line for this email” bolted on, summarization targets that are short and often hallucinated — and the unnaturalness and deficiencies of the source benchmarks carry straight into the model. FLAN also bet on scale, importing the pre-training intuition that post-training needs tons of data; later work showed that with a strong base model, few high-quality examples suffice. Self-instruct was forward-looking in the other direction: let models generate the data. Alpaca (by Tatsu’s students) distilled ChatGPT traces — natural inputs, long chatty outputs — and reliably induced ChatGPT-like behavior on Llama, the point at which people realized chat-style data made ChatGPT-style systems “not that hard.” Open Assistant was the Wikipedia-style crowdsourced attempt at the highest-quality human version, producing on the order of 10,000 examples with long, expert responses before stalling. WizardLM and Tulu 3 pushed increasingly elaborate synthetic generation, and the newest datasets — NVIDIA’s Nemotron SFT — are agentic: assistant turns with parallel tool calls and to-do lists supervised directly into the model.

Three axes vary across all of these: chattiness (people don’t want to talk to an NLP benchmark), detail and references, and tool use. Length and style deserve special suspicion: in preference evaluations, both humans and GPT-based judges reliably favor bullet-pointed lists and longer, more detailed responses. Training on different instruction sets swings preference scores like AlpacaEval dramatically while leaving standard benchmarks (MMLU, GSM) mostly unchanged — your model isn’t smarter, your engagement signal just moved. “You want to think about style control separately from capabilities control.”

Where SFT bites back: knowledge, safety, and how little you need

This is the folklore, and it has empirical support: fine-tuning a model on facts it does not know makes it hallucinate, while training only on known facts does not. John Schulman’s argument goes one step further — teaching a model what it knows has to be policy-dependent, which is a reason RLHF exists at all: “you can’t have an external person shoving knowledge down your throat if you want the model to be calibrated about what it knows and doesn’t.” The folk mechanism: if the model has an internal “I know this” direction, RL can reward references in the know-direction and punish them in the don’t-know direction, extracting calibration into the output policy. So, counterintuitively, you may not want to fine-tune on tail knowledge even when that knowledge is factually correct.

Safety is the post-training team’s burden — “you are the last line of defense” against misuse — and the framework is a Pareto trade-off between the violation rate (bad queries that get through) and the false refusal rate (refusing “how do I kill a Python process”). Public detail is even sparser than for capabilities: Llama 2 used a few thousand safety examples without saying how many; Tulu 3, the most detailed open pipeline, has roughly 50,000 safety examples, largely mined from WildChat — free chat access traded for logs, from which unsafe attempts and jailbreaks were filtered and paired with refusals.

Midtraining: instruction tuning dissolves into pre-training

The decay phase gets the good data for two reasons: it’s closest to deployment, and the learning rate is lowest. And because midtraining is much shorter than pre-training, you can run ten decay-phase data ablations for the price of one full run — in practice labs ablate mixtures on the decay phase and reflect the estimates back into the pre-training mix. (You can’t just make all of pre-training high quality; you’d run out of tokens.)

RLHF: stop fitting a distribution, start maximizing a reward

Pre-training and SFT are generative modeling: fit $\hat{p}(y|x) \approx p^*(y|x)$ for some reference distribution. RLHF changes the game — find a policy $\hat{p}(y|x)$ maximizing $\mathbb{E}_p[R(y, x)]$ for some reward. The conceptual difference matters: a reward maximizer is free to collapse its distribution onto a single answer per prompt, and that would be fine as long as the reward is good. Language models become policies, not models of a distribution.

Why rate outputs instead of collecting more demonstrations? Because there’s a gap between what people generate and what they prefer.

The data is pairwise preference ratings over sampled model outputs (temperature 1, since post-SFT models are still diverse). InstructGPT’s leaked-into-the-appendix guidelines balance helpful, truthful, and harmless; Google Bard’s leaked instructions look similar but use Likert scales. The annotator workforce has shifted upward: on one Scale AI platform roughly 70 percent hold bachelor’s or master’s degrees, median wages exceed $50/hour, and bespoke experts — doctors, lawyers — earn over $100/hour. The hard problems are verifying annotators aren’t using ChatGPT themselves, and getting real correctness checks under time pressure (Bard annotators protested having under a minute per long response). Annotators also imprint on the model: post-trained models’ opinions drifted toward Buddhist, Hindu, and atheist poll respondents — matching the Southeast Asian and West Coast annotator demographics in InstructGPT’s appendix — and non-expert annotators over-reward formatting while under-checking factuality and consistency. Subtler still, “emergent misalignment” results show innocuous-looking model-generated data can transmit hidden preferences.

Meanwhile, model-based annotation won the open-source race: GPT-4’s rankings correlate near-perfectly with human system rankings at a tenth the cost, with agreement near human inter-annotator levels. Hugging Face’s Zephyr team tried hard to collect human feedback through the standard vendors, found it slow, costly, and no better, and shipped with AI feedback; UltraChat, UltraFeedback, and the whole Tulu 3 pipeline are model-annotated. If you’re catching up to the frontier, distillation and AI feedback are the answer; only pushing the frontier (or extracting expert tacit knowledge) still demands humans.

PPO, and the search for something simpler: DPO

The objective, from equation 2 of InstructGPT, is reward with a KL leash:

$\max_{\pi_\theta}\; \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)}\big[r_\phi(x, y)\big] \;-\; \beta\, \mathbb{D}_{\mathrm{KL}}\big[\pi_\theta(y \mid x)\,\|\,\pi_{\mathrm{ref}}(y \mid x)\big]$

where the reward model is a binary classifier trained on pairs (the Stiennon loss), $\mathcal{L}(r_\theta) = -\mathbb{E}\left[\log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big)\right]$. The baby description of PPO: start from the policy gradient identity, $\nabla_\theta \mathbb{E}_{p_\theta}[R(z)] = \mathbb{E}_{p_\theta}[R(z)\, \nabla_\theta \log p_\theta(z)]$ — which “really just looks like SFT with weighted examples.” But sampling per step is expensive, so you go off-policy and reuse rollouts; TRPO adds an importance-weighting correction with a trust-region constraint so local reward estimates don’t blow up; PPO replaces the awkward constraint with a heuristic ratio clip. (Full treatment next lecture — you’ll need it for the assignment.)

Can we get rid of PPO? Many reasonable ideas fail: SFT with [GOOD]/[BAD] control tokens prepended to pairs, training only on the preferred outputs, reward-model-filtered SFT (somewhat works). The one that stuck is DPO, built on the most naive intuition possible: take gradient steps toward the log-probability of good responses and negative gradient steps away from bad ones, weighted appropriately.

The gradient is the most intuitive form: increase the likelihood of the winner, decrease the likelihood of the loser, with step size scaled by $\sigma(\hat{r}_\theta(x, y_l) - \hat{r}_\theta(x, y_w))$ — how wrong the implied reward model currently is. If the model already strongly prefers the winner, tiny step; if it rated them equal, big step. Is DPO worse than PPO? The results are highly contingent on experimental setup — Ai2 published both “PPO beats DPO” and “DPO done right beats PPO.” “It’s good enough for Llama”: Llama’s core RLHF primitive was DPO inside an outer loop of rejection sampling and re-SFT. Variants (SimPO drops the reference normalizer, length-normalized DPO divides by response length to blunt length hacking) don’t seem to matter much.

Over-optimization and mode collapse

Can you RLHF your way to superintelligence by collecting enough thumbs-up? No — push the optimization hard and you overfit the learned reward model: proxy reward keeps climbing while true win-rate peaks and falls, across PPO, best-of-$n$, and expert iteration alike. The KL regularizer is critical for keeping a strong optimizer honest. And because an RLHF’d model is a policy rather than a distribution, it can and does collapse: diversity drops, and calibration breaks — GPT-4’s post-RLHF calibration plot was one of the few open problems OpenAI published, and nobody has really solved it. Entropy and exploration become critical in the next lecture, where the question is whether there are rewards — verifiable ones — that you can pour compute into without over-optimizing. That is RLVR, and it’s why reasoning models work.

Check yourself

Exercises in the lecture’s spirit — a couple of small calculations on the DPO machinery and two judgment calls about data.

Problem 15.1 DPO gradient weighting

With $\beta = 0.1$, consider two preference pairs. Pair A: the model’s log-ratios $\log \frac{\pi_\theta}{\pi_{\mathrm{ref}}}$ are 1.0 for the winner and 1.5 for the loser. Pair B: 3.0 for the winner and $-2.0$ for the loser. Compute the DPO gradient weight $\sigma(\hat{r}_\theta(x, y_l) - \hat{r}_\theta(x, y_w))$ for each pair. Which pair drives the larger update, and why does that make sense?

Show solution

The implied rewards are $\hat{r}_\theta = \beta \log \frac{\pi_\theta}{\pi_{\mathrm{ref}}}$ (the $\beta \log Z(x)$ term cancels in the difference). Pair A: winner 0.10, loser 0.15, weight $\sigma(0.15 - 0.10) = \sigma(0.05) \approx 0.512$. Pair B: winner 0.30, loser $-0.20$, weight $\sigma(-0.5) \approx 0.378$. Pair A gets the larger step: the implied reward model there is wrong — it rates the loser above the winner — while in Pair B it already strongly prefers the winner, so DPO barely touches it. Step size scales with the prediction error of the implied reward model.

Problem 15.2 exponential tilt

A reference policy puts probabilities $(0.5, 0.3, 0.2)$ on three responses with rewards $(1, 0, -1)$. With $\beta = 1$, compute the optimal tilted policy $\pi_r(y) \propto \pi_{\mathrm{ref}}(y)\, e^{r(y)/\beta}$. What happens as $\beta \to 0$ and as $\beta \to \infty$, and which RLHF failure mode does the $\beta \to 0$ limit correspond to?

Show solution

Unnormalized weights: $0.5 e^{1} \approx 1.359$, $0.3 e^{0} = 0.3$, $0.2 e^{-1} \approx 0.074$. Normalizer $Z \approx 1.733$, so $\pi_r \approx (0.784, 0.173, 0.043)$. As $\beta \to 0$ the tilt sharpens until all mass sits on the argmax response — the policy collapses to a point, which is exactly the mode collapse / diversity loss RLHF’d models exhibit (and why calibration breaks: the model is no longer a fitted distribution). As $\beta \to \infty$ the reward is ignored and $\pi_r \to \pi_{\mathrm{ref}}$ — the KL leash dominates.

Problem 15.3 tail knowledge judgment

Your annotation vendor delivers 20,000 expert-written SFT responses, dense with accurate citations to niche journal articles, for a 7B model. A colleague argues this is ideal: the data is factually flawless. Using the lecture’s two-mechanisms argument, explain what is likely to go wrong, and name two mitigations the lecture suggests.

Show solution

Each example teaches two things at once: the specific knowledge (the citation contents) and the format (good answers emit references). A 7B model almost certainly does not know most niche citations, so it generalizes the format without the knowledge and learns to forcibly emit references it doesn’t have — hallucination, exactly the Gekhman-style finding that training on unknown facts hurts while training on known facts doesn’t. Mitigations: filter the SFT set toward knowledge the model already has (train on known facts, drop tail knowledge even when correct), and rely on RL-style policy-dependent feedback, which can reward reference-emission only when the model’s internal “I know this” state supports it.

Problem 15.4 reading a preference eval

After swapping in a new instruction-tuning set, your AlpacaEval-style win rate jumps from 50 percent to 65 percent, while MMLU and GSM scores are unchanged. The new set’s responses average 2.1x the length of the old ones and heavily use bulleted lists. What can you conclude, what can’t you, and what does the lecture propose you control for?

Show solution

You can conclude the model’s outputs became more preferred by the judges — but both human and GPT-based judges systematically favor longer, list-formatted responses, and it has even been shown you can RLHF on length alone and climb these benchmarks. Since capability benchmarks didn’t move, the parsimonious reading is a style shift, not a capability gain: engagement signals moved while the model got no smarter. You cannot conclude the new data taught anything beyond tone. The lecture’s prescription is to treat style control and capabilities control as separate axes — measure them separately (e.g., length-controlled or style-controlled evaluation) before crediting the data with an improvement.