Evaluation

What does “good” even mean?

The course has now covered everything needed to train a language model — architecture, optimizer, training loop, kernels, parallelism, scaling laws, inference. The missing piece is data, and data shapes model behavior: train on code and you get a coder, train only on DNA sequences and it probably can’t speak English. But before deciding what to train on, you need to decide what behavior you want, and that is evaluation: given a model, how good is it?

Evaluation looks mechanical — define prompts, collect responses, compute accuracy — so why a whole lecture? Because “evaluation is what shapes the development of AI.” It sets North Stars; every developer, open or closed, reads benchmarks as progress. Percy offers four superficial answers to what good means, none of them the correct one: a model is good if it scores well on benchmarks (Artificial Analysis’s intelligence index), if it scores well and is cheap to run (intelligence plotted against inference cost — the two correlate, but not perfectly), if people prefer its responses (Arena AI, formerly Chatbot Arena), or if people simply pay to use it (OpenRouter’s usage rankings, the economic lens).

Perplexity: the believer’s metric

A language model is a distribution $p(x)$ over token sequences, and the most natural way to evaluate a distribution is to ask how much probability mass it puts on held-out data.

This was language modeling research through the 2010s: in-distribution evaluation on Penn Treebank, WikiText-103, or the One Billion Word Benchmark, train split to test split. Real progress was measured this way — a famous 2016 paper drove 1BW perplexity from 51.3 to 30.0 with pure CNNs and LSTMs, the first definitive result that neural beats n-gram hybrids. GPT-2 (2019) broke the paradigm: trained on WebText (40 GB of pages linked from Reddit), evaluated zero-shot on everyone else’s datasets — out-of-distribution evaluation. The 1.5B model hit 35 perplexity on tiny PTB against a state of the art of 46, without training on it at all.

The counterpoint is that perplexity is maybe more than you need.

Some benchmarks are perplexity in disguise: LAMBADA (2016) is fill-in-the-blank on words deliberately chosen to require long context, and HellaSwag is multiple-choice sentence completion — accuracy on the surface, next-token prediction underneath. One warning if you ever run a perplexity leaderboard: submitters hand you log-probabilities, and you must trust they come from a valid distribution that sums to 1 — a cheater’s LM that always returns log-prob 0 wins outright. Downstream tasks avoid this because a black-box model returns a response and you grade the response. Perplexity remains heavily used — it varies smoothly with scale, which is exactly what scaling laws need — but for the non-believers you need benchmarks that look like the real world.

Exam benchmarks and their shelf life

Exams are how we test humans: controlled subject and difficulty, unambiguous answers, easy grading. LM benchmarking culture inherited them. MMLU (Hendrycks et al., 2020) collected 57 subjects of multiple-choice questions when it was still radical to expect a language model to answer a few-shot prompt at all; GPT-3 was well above chance, small models barely, and by now the benchmark sits in the 90s — saturated. MMLU-Pro (2024) removed noisy questions, expanded four choices to ten, and evaluated with chain of thought; accuracies dropped by 16 to 33 points, and it too has climbed back to nearly 90. GPQA — Graduate-Level Google-Proof Q&A — had 61 PhD contractors write questions through expert validation and revision loops; PhD experts scored 65%, non-experts with Google and 30 minutes got 34%, GPT-4 got 39% at the time. It now stands at 94. Humanity’s Last Exam threw everything at difficulty: 2,500 multimodal questions, crowdsourced with a $500K prize pool plus co-authorship, filtered by frontier models, with a private held-out set to guard against contamination. Frontier models scored single digits at release; in 2026 even Mythos is only at 64.7 — still room.

Chat benchmarks: grading the open-ended

“I would like to make a beet salad with goat cheese. What kind of herbs would work well?” There is no ground truth to exact-match against. Chatbot Arena’s answer is clever crowdsourcing: any visitor chats for free, gets two anonymized responses from two random models, and votes which is better. Pairwise comparisons feed an ELO model,

$p(A \text{ beats } B) = \frac{1}{1 + 10^{(\text{ELO}_B - \text{ELO}_A)/400}},$

fit to maximize the likelihood of the observed votes. The beauty of ELO is that you don’t need every prompt to hit every model — a sparse, connected comparison graph suffices, as in chess — and new models and prompts fold in over time. The worries: who are these random internet people, and are there spammers gaming their own submissions? Binary preference conflates style with correctness; the voter asked because they don’t know the answer, so how do they judge which response is right? Sycophancy may beat honesty.

AlpacaEval (2023) replaced the human with an LLM: 805 instructions, win rate against a GPT-4-preview baseline as judged by GPT-4 preview — an instantiation of LLM-as-a-judge, with obvious potential bias. LLM judges favor long responses, which produced actual leaderboard gaming by models fine-tuned for length, fixed in AlpacaEval 2.0 by a simple regression debiasing. Which raises a meta-question: how do you evaluate a metric? The sanity check is correlation with other metrics — AlpacaEval correlates 0.98 with Chatbot Arena, over the models of that era. WildBench sourced 1,024 real prompts from a million human-chatbot conversations and added the key innovation of a per-task checklist for the judge: asking “is this response good?” is ill-defined, and a rubric scopes the judgment — true for human judges too, as anyone who has run crowdsourcing knows. Pairwise comparisons give higher signal than absolute scores, and biases (human and LLM alike) never fully go away.

Agents: evaluating what models do

Chat evaluates what models say; agents are what models do. An agent is a language model plus a scaffold — the logic deciding how the model gets called and what tools it can use. SWE-bench (2,294 tasks across 12 Python repositories) hands you a codebase and a GitHub issue; you submit a patch, graded by unit tests that fail before and should pass after. It went from roughly 16% in 2024 to 93% (on the Verified set) today. Terminal-Bench generalizes the environment to a computer terminal — simple and universal — with 229 crowdsourced tasks from 93 contributors, 89 forming version 2.0, tasks that take a human between an hour and over a week. CyBench poses 40 capture-the-flag hacking tasks; best models scored around 10% at release and it is now completely solved. MLE-Bench wraps 75 Kaggle competitions — read the data, write code, train models, submit.

The leaderboards reveal something important: the same model under two different scaffolds gets different accuracies. Serious scaffolds use explicit planning (a to-do list that gets checked off, not stream-of-consciousness chain of thought), hierarchical delegation (sub-agents with clean context returning only results), persistent memory (reading and writing files as context outgrows the window), and heavy context engineering. Evaluating an agent is always evaluating the scaffold and the model together.

Pure reasoning and safety

Everything so far requires linguistic and world knowledge. ARC-AGI (started 2019, pre-LLM — prescient) tries to isolate reasoning: visual pattern puzzles meant to be 100% human-solvable but hard for AI, each task a special snowflake so memorization doesn’t help. Pretrained models didn’t move the needle at all — exactly as designed. Then in 2024 the o1 and o3 reasoning models made scores take off abruptly; ARC-AGI-1 is basically solved, ARC-AGI-2 (2025) is on its way, and ARC-AGI-3 (2026) — interactive games, no language — sits at extremely low scores. The caveats: fully decoupling reasoning from knowledge may be impossible, and the 100%-human-solvable constraint means it says nothing about superhuman reasoning.

Safety is even harder to pin down — cars have crash-test ratings backed by decades of lobbying over what safety means; AI has no such settlement. HarmBench tests refusal on 510 harmful behaviors. AIR-Bench builds a taxonomy from EU, China, and US regulatory frameworks plus company policies — 314 risk categories, 5,694 prompts. And refusal training can be bypassed: GCG optimizes gibberish adversarial suffixes on open-weight models that transfer to closed models, extracting step-by-step plans to destroy humanity from models that should refuse. Underneath sits the unsolved question of what safety even is: it’s contextual (politics, law, norms varying by country), the risks are heterogeneous (hallucination, sycophancy, abetting crime, lost critical thinking — some correlated with capability, some counter to it), and capable systems are dual-use — the same cybersecurity agent hacks systems or pen-tests them.

Validity: is your evaluation measuring anything?

Three projects chase realism at the use-case level. GDPVal (OpenAI) took the top nine US GDP sectors, 44 occupations, and had professionals with about 14 years of experience write tasks. MedHELM replaced standardized-exam medicine with 121 clinical tasks sourced from 29 clinicians — what clinicians actually ask, not what med boards ask. Clio (Anthropic) uses language models themselves to analyze real usage data and report only aggregate patterns — because the model developers are the ones who have the real query stream, and realism and privacy are directly at odds.

Then there’s contamination. Machine learning 101 says don’t train on your test set, which was easy when ImageNet and SQuAD had clean splits; now models train on the internet and nobody tells you what’s in the data. Percy gives four routes: (1) infer overlap from the model — a neat idea from Tatsu’s group exploits the fact that benchmark question order should be exchangeable, so a model that prefers the benchmark’s specific ordering probably trained on it; (2) norms — model providers claiming a GPQA score should report train-test overlap, the way statisticians report confidence intervals; (3) fresh evals — LiveCodeBench and UncheatableEval scrape pages past the model’s cutoff date, though timestamps aren’t safe either, since new GitHub code can be derived from old repos; (4) private evals — internal codebases, or your own never-published writing (“I have a bunch of rejected papers from grad school”), easiest to use with perplexity.

Dataset quality is the last trap: SWE-bench needed a Verified re-audit, MMLU and GSM8K have known-broken questions (“does the baby have socks on? — there’s no way to tell”), agentic benchmarks are harder still to audit, and on TorchBench an agent that outputs the empty response scores 38%. Tools like Docent use LLMs to inspect agent traces for exactly these problems.

Check yourself

Exercises in the lecture’s spirit — pencil, a calculator, and some healthy skepticism.

Problem 12.1 perplexity arithmetic

A model assigns a 500-token test set a total log-probability of $-2000$ bits (i.e., $\log_2 p(D) = -2000$). What is its perplexity? A second model achieves perplexity 12 on the same set — how many total bits does it spend? Why does the leaderboard need to trust both models’ probabilities?

Show solution

Per-token loss: $2000 / 500 = 4$ bits, so perplexity $= 2^4 = 16$ (perplexity is the exponentiated average negative log-likelihood, the per-token form of $(1/p(D))^{1/|D|}$). The second model spends $\log_2 12 \approx 3.585$ bits per token, so $500 \cdot 3.585 \approx 1792$ total bits — better. The trust issue: the leaderboard receives only the number. An invalid “distribution” that returns probability 1 for everything reports 0 bits and perplexity 1; nothing in the log-prob contract proves the submitted values come from a distribution that sums to 1. Response-based tasks dodge this because the output itself is graded.

Problem 12.2 ELO from pairwise votes

Under the Arena rating model, model A has ELO 1250 and model B has ELO 1150. What is the probability A wins a pairwise comparison? Why can Arena rank hundreds of models even though no human ever compares more than two?

Show solution

$p(A \text{ beats } B) = 1/(1 + 10^{(1150-1250)/400}) = 1/(1 + 10^{-0.25}) \approx 1/(1 + 0.562) \approx 0.64$. A 100-point ELO gap means roughly a 64% win rate. The ranking works with sparse comparisons because the ELO parameters are fit jointly to maximize the likelihood of all observed pairwise outcomes; as long as the comparison graph is connected, every model’s rating is identified — the same reason chess doesn’t require everyone to play everyone else.

Problem 12.3 contamination audit

A benchmark was published in June 2024; a model’s training cutoff is December 2023. Is the benchmark automatically contamination-safe for that model? Name the failure mode, and describe the exchangeability test you could run.

Show solution

Not automatically. Timestamps fail when content is derived: the 2024 benchmark may have drawn questions from older exams, textbooks, or repositories that predate the cutoff — contamination is usually this subtle process, not literal training on the test file. (The reverse risk also exists going forward: prompts sent to an API can enter future training.) The exchangeability test: benchmark question order should carry no signal, so present the questions in their canonical order versus random permutations and compare the model’s likelihoods. If the model assigns systematically higher probability to the benchmark’s specific ordering, it has likely seen the benchmark itself in training.

Problem 12.4 evaluating a metric

You build an LLM-as-a-judge benchmark and find it correlates 0.97 with Chatbot Arena. What does this establish, what does it not establish, and what gaming failure should you test for before launching a leaderboard?

Show solution

It establishes that, over the current set of models, your metric reproduces the ranking of a human-preference signal — a useful sanity check that lets people skip the slow human loop. It does not establish ground truth: the correlation is circular (is Arena itself correct?), and it holds only for the model range you measured — it can break for models stronger than your judge, as AlpacaEval’s correlation was tied to the GPT-4-preview era. The gaming failure to test: length bias. LLM judges favor longer responses, and AlpacaEval was gamed by models fine-tuned to be verbose until version 2.0 debiased with a regression. Check whether win rate rises with response length at fixed content, and debias before launch.