Post-Training: RLVR

From RLHF’s ceiling to verifiable rewards

Last lecture ended on a downer: RLHF is fundamentally annotation-bottlenecked. You collect preferences, fit a reward model, and RL against it — but you can’t keep putting compute into the same reward model, because eventually you overfit it. No amount of regularization saves you from overoptimization.

Now compare that to the domains where RL believers point — AlphaGo. The difference, Tatsu argues, is that AlphaGo optimizes exactly what we want: the win–loss condition of Go has no sloppiness in its definition, so you can put in as much compute as you like and trust that improvement is real. These are closer to search problems, where RLHF is a learning problem. Domains like formal mathematics, natural-language math, and coding have this same flavor — verifiable, and therefore far more amenable to reinforcement learning. That is RLVR: reinforcement learning from verifiable rewards, the engine behind O1-style thinking models with their very long chains of thought.

The lecture runs in two parts, mirroring the course’s usual structure: first the core algorithms (PPO, then GRPO — you will implement this in the assignment), then a tour of open-source releases showing how those algorithms surface in real technical reports. The algorithms won’t differ much from last lecture; where they land will differ a lot.

PPO: easy pseudocode, 37 implementation details

Everything in RL for language models comes back to one equation — the REINFORCE policy gradient:

$\nabla_\theta \, \mathbb{E}_{p_\theta}[R(z)] = \mathbb{E}_{p_\theta}\!\left[R(z)\, \nabla_\theta \log p_\theta(z)\right]$

Gradient descent on rewards, taken as weighted SFT updates where the weights can be positive or negative. “This is in some sense the core from which everything else is derived.” PPO builds on it to reuse rollouts across several gradient steps, clipping the policy ratio so you don’t stray too far from the policy that generated the data. The spinning-up pseudocode looks harmless: sample trajectories, estimate advantages, clip, update, fit a value function.

Then you see a blog post titled “The 37 implementation details of PPO,” and — in Tatsu’s words — “this should strike fear into your heart.” PPO for language models means generalized advantage estimation, an experience buffer, a KL term that operates token by token, and a value model being trained alongside the policy. Tatsu walks through his own lab’s AlpacaFarm implementation: the outer loop is fine, the loss computation follows the paper, and then you hit the hacks — a KL penalty clipped at zero (which “totally ruins the point of a KL divergence” but blows up immediately if removed), and a generalized advantage estimator run at gamma = lambda = 1, a degenerate setting that quietly turns the whole thing back into a bandit problem. The point isn’t that PPO is impossible — labs run it turnkey at scale — but that it is finicky and sensitive for anyone implementing from scratch. It also demands a value model as big as the policy itself, eating memory you’d rather spend on models or inference servers.

Why not DPO, then? DPO is a specific solution to a specific problem — pairwise Bradley–Terry feedback — and math problems don’t arrive as pairwise comparisons. “PPO is the more general hammer that you can hit everything with.” What the community wanted was that generality without the pain.

GRPO: delete the value function

GRPO, from the DeepSeekMath paper, starts with PPO and removes its most complicated, annoying part. The value network exists to provide a baseline that reduces gradient variance — but it’s a whole neural network that destabilizes training. GRPO replaces it with statistics you can compute for free: sample a group of $G$ rollouts from the same prompt, and score each one against its siblings.

The full objective keeps PPO’s min-clipped ratio plus a KL term to the reference policy — but in the online case, where you roll out and immediately update, the ratio $\pi_\theta / \pi_{\theta_{\text{old}}}$ is 1, the clipping never fires, and the whole thing collapses to policy gradient with group-normalized rewards minus a KL penalty. That simplicity is why nearly all open-source RLVR work builds on it. The reference implementation Tatsu shows (from McGill’s nano-aha-moment) fits the entire advantage computation in half a slide:

# for one prompt: G rollouts, one scalar reward each
rewards = np.array(rewards)
advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-4)

The recipe you’ll implement: compute a reward for each of your $k$ rollouts, normalize within the group, compute the KL term, and take gradient updates on the combination — with one stop-grad placed carefully so autodiff does the right thing. The 1e-4 is the only departure from the paper: it keeps the division alive when every rollout in a group gets the identical reward, which genuinely happens when the model fails a math problem every single time. In DeepSeekMath’s results, GRPO beats rejection fine-tuning (training only on your correct answers), with some further gains from process supervision — a detail that becomes interesting when R1 abandons it.

What the z-score smuggles in

Is GRPO’s advantage a valid advantage? Sutton and Barto’s REINFORCE-with-baseline theorem says you may subtract any state-dependent baseline $b(s)$ from your rewards — in the bandit setting, any prompt-dependent constant — and still descend the true reward objective. Subtracting the group mean qualifies. Dividing by the standard deviation does not, and neither does GRPO’s other quiet move: normalizing per token by the sequence length. Derive the update from first principles and you get neither term — which is exactly what the Dr. GRPO paper (Liu et al. 2025) did, ending up close to REINFORCE with leave-one-out.

DeepSeek R1: how simple the recipe can be

R1 kicked off the open-source RLVR wave because it matched OpenAI O1’s behavior — long CoT, clearly RL, strong on hard math — with a recipe anyone could run. The cleanest result is R1-Zero: take the DeepSeek-V3 base model and run GRPO directly on it with just two rewards — accuracy rewards for correct final answers, and format rewards for enclosing the CoT in thinking tags (so it can be stripped out later). No SFT, no RLHF, no production mess. That alone lands only a little worse than O1: 71.0 versus 74.4 pass@1 on AIME 2024.

Two celebrated phenomena deserve skepticism. CoTs growing steadily longer during training is arguably a side effect of GRPO’s length normalization, not evidence of deeper thought. And the viral “aha moment” — the model flagging its own realization mid-proof — appears in the base model too, so it can’t be a product of the RL algorithm; RL merely extracts a behavior pre-training already installed.

The production R1 stacks the pieces the course has studied in isolation: DeepSeek-V3, then reasoning SFT, then RL with GRPO, then SFT/RLHF at the end as the user-facing polish. The SFT stage is “a small amount of long CoT data” — read the tech report between the lines about where that came from — plus verification filtering. The RL stage adds a language-consistency reward, purely because R1-Zero-style training would switch languages mid-CoT, which was uninterpretable and slightly disturbing. Two further findings held up well. Distillation: feed R1’s CoT traces (800k of them) into Qwen 2.5 and even Llama models, and you get much of the thinking-model juice from pure SFT.

And failed attempts: DeepSeek openly reports that process reward models — the star of DeepSeekMath — didn’t do much, and that MCTS, the AlphaGo-style speculation everyone had about O1, couldn’t be made to work at scale. Outcome rewards scale better, and that’s where the action is.

Kimi K1.5: a different derivation, the same place

Kimi K1.5 came out simultaneously with R1 and also beat O1, yet gets a fraction of the attention. It matters here because it does several things differently, and the fact that both work tells you which components are load-bearing.

On data, Kimi is far more explicit: broad coverage across STEM domains, multiple-choice excluded, and — most importantly — difficulty filtering. If the model succeeds on a problem within best-of-8 samples, the problem teaches it little; if a problem is hopeless, there is no reward signal at all (“if you get no rewards, you have no signal”). Filtering to the medium band keeps RL progressing at a steady pace, and once a problem is mastered it leaves the pool. Essentially everyone doing RL now does this.

On the algorithm, Kimi starts from the same KL-regularized reward maximization and runs a DPO-style derivation — solve for the reward at the optimum, then put a squared loss on the resulting identity as a surrogate. Optimization purists would be horrified, but take the gradient and out falls a policy gradient with a group-mean baseline plus a KL-like regularizer: the group-normalized baseline reinvented through entirely different means, with no standard-deviation division and no length normalizer.

Kimi’s view of length is the opposite of celebrating growing CoTs: long CoTs are wasteful, because thinking time is inference cost you subsidize for the user. So they add an explicit length reward — correct answers are pushed to be short, while incorrect answers are only penalized when longer than the middle of the group’s range. The asymmetry is deliberate: if you crush wrong answers to zero length, a model that is bad at geometry can never think its way back to a correct geometry answer, and it’s stuck forever. Two more details close the loop. Verification is itself a rabbit hole — for math they train a chain-of-thought reward model for answer-equivalence checking (98.5% accurate versus 84.4% for a classic RM), because models box answers wrong in endless ways. And RL infra is hard because it welds training to inference: one Riemann-hypothesis rollout can stall a whole batch, weights must ship from trainer to inference workers, and reusing rollouts for utilization buys you off-policy instability.

Qwen 3 and the agentic turn

Qwen 3 is the tried-and-tested playbook executed well: long-CoT cold start, reasoning RL, thinking-mode fusion, general RLHF, then strong-to-weak distillation for the small models. The data work compounds — difficulty filtering by best-of-n, dropping problems solvable without CoT, decontaminating against validation sets, manual CoT quality checks — and the striking number is that the reasoning-RL stage uses just 3,995 examples. Its distinctive contribution is thinking-mode fusion: thinking and non-thinking behavior live in one model, switched by a prompt tag, with a special early-exit string that forces an immediate answer mid-thought. Performance degrades gracefully as the thinking budget shrinks — and the stage-by-stage ablation shows general RL lifting chat benchmarks while math and coding dip slightly, a trade-off Qwen later found unacceptable enough to un-fuse the modes in newer releases.

Qwen3-Coder-Next extends the recipe to agents, and the lesson is the course’s oldest one: there is no new agent-training algorithm, “data is the important thing.” An extensive mid-training phase injects the capabilities — 600B tokens of repository-level long-context data, pull requests with RAG-constructed context, synthetic coding QA, traces from running public coding agents. Then, unusually, four expert models (web dev, UX, QA, software engineering) are trained separately and distilled back into one model. The SWE expert does RL on 800k automatically constructed SWE-bench-style environments, landing 70.6% on SWE-bench Verified with only 3B active parameters.

Check yourself

Exercises in the lecture’s spirit — group statistics, bias hunting, and reward design, all doable on paper.

Problem 16.1 GRPO advantage computation

One prompt, $G = 8$ rollouts, binary rewards: 3 correct, 5 incorrect. Compute the GRPO advantage for a correct and an incorrect rollout. Then explain what happens to the computation when all 8 rollouts fail, and how implementations handle it.

Show solution

Mean reward: $3/8 = 0.375$. The rewards are Bernoulli, so the (population) standard deviation is $\sqrt{0.375 \cdot 0.625} \approx 0.484$. Correct rollout: $A = (1 - 0.375)/0.484 \approx +1.29$. Incorrect: $A = (0 - 0.375)/0.484 \approx -0.78$. If all 8 rollouts fail, mean $= 0$ and std $= 0$: the numerator is 0 but so is the denominator, and the division blows up. This is why the reference implementation adds a small stability constant, computing $(r_i - \bar r)/(\text{std} + 10^{-4})$ — with identical rewards the advantage becomes $0/10^{-4} = 0$, correctly contributing no gradient. All-zero groups are common in RLVR: they’re exactly the “too hard” problems that difficulty filtering tries to remove.

Problem 16.2 std-normalization bias

Two problems, 8 rollouts each with binary rewards. Problem E is easy: the model gets 7 of 8 correct. Problem M is medium: 4 of 8 correct. Compare the magnitude of the GRPO advantage assigned to an incorrect rollout on each problem, and explain what this says about which problems the std normalizer emphasizes.

Show solution

Problem E: mean $= 0.875$, std $= \sqrt{0.875 \cdot 0.125} \approx 0.331$. The one incorrect rollout gets $A = (0 - 0.875)/0.331 \approx -2.65$. Problem M: mean $= 0.5$, std $= 0.5$, so an incorrect rollout gets $A = -0.5/0.5 = -1$. The easy problem’s failure receives a gradient about 2.6x larger than the medium problem’s, purely because its group variance is small. Symmetrically, a lone success on a nearly-impossible problem gets a huge positive advantage. Dividing by std therefore upweights the two extremes — problems the model almost always solves or almost never solves — which is backwards if you want learning concentrated in the model’s solvability range. That is the bias Dr. GRPO removes by dropping the denominator.

Problem 16.3 length-bias mechanics

GRPO normalizes each rollout’s contribution by its own token length. Using a rollout with reward $-1$, argue why this encourages the model to generate long outputs specifically when it is wrong, and why the same term cannot shrink correct answers below a floor. What happens to average CoT length during training once the normalizer is removed?

Show solution

With per-sequence length normalization, a rollout’s advantage is spread as roughly $A_i / |o_i|$ per token. For a wrong answer ($A_i$ negative), a longer $|o_i|$ makes the per-token penalty smaller — in the limit, an infinitely long wrong answer divides its $-1$ by infinity and escapes punishment entirely. So once the model senses it cannot solve a problem, blabbing on is gradient-optimal. For correct answers the same term pushes toward brevity (larger per-token credit for shorter outputs), but there is a hard floor: a correct solution needs some minimum number of tokens to actually solve the problem, so it can’t shrink indefinitely. The net effect is length growth driven by the incorrect rollouts — exactly what the Dr. GRPO controlled comparisons show, where the incorrect-output length grows steadily under GRPO but stays flat under the debiased objective, and total CoT length caps at a constant instead of growing forever.

Problem 16.4 Kimi length reward

Kimi’s length reward for rollout $i$ in a batch is $\lambda$ if correct and $\min(0, \lambda)$ if incorrect, where $\lambda = 0.5 - \frac{\text{len}(i) - \text{min\_len}}{\text{max\_len} - \text{min\_len}}$. For a group with min_len $= 200$ and max_len $= 1000$, compute the length reward for (a) a correct answer of 400 tokens, (b) a correct answer of 900 tokens, (c) an incorrect answer of 400 tokens, (d) an incorrect answer of 900 tokens. Explain the design intent behind case (c).

Show solution

Compute $\lambda$ in each case. (a) $\lambda = 0.5 - 200/800 = +0.25$; correct, so reward $= +0.25$. (b) $\lambda = 0.5 - 700/800 = -0.375$; correct, so reward $= -0.375$ — even correct answers are penalized for rambling. (c) $\lambda = +0.25$, but the answer is incorrect, so reward $= \min(0, 0.25) = 0$: a short wrong answer is not rewarded for its shortness, just left unpenalized. (d) $\lambda = -0.375$, incorrect, so reward $= -0.375$. The clamp in case (c) is the recovery safeguard from the lecture’s geometry example: if short wrong answers earned positive reward, a model bad at some topic would collapse its CoTs toward zero length and lose any chance of ever reasoning to a correct answer. Incorrect answers are only nudged to be shorter than the middle of the group’s range, not crushed.