A reward model reads its score off a single token. You push the whole response through a transformer, take the hidden state sitting at the EOS position, run it through a scalar head, and the number that comes out is what the policy will spend the rest of training chasing. One number. For an entire answer.
Almost every complaint people have about RLHF (the hedging, the refusals nobody asked for, the model that agrees with whatever you just said) is downstream of that compression. And almost every method invented since InstructGPT is an attempt to work around it.
Nathan Lambert (previously post-training lead at the Allen Institute for AI, and author of The RLHF Book) published 13 lectures that accompany the book. I worked through all 13 over August 2026 and took notes. I didn't read the book alongside them, so this is a recap of the lectures, not of the text, and it follows their order.
What is post-training actually for?
Every lecture hangs off one spine. InstructGPT made three stages canonical: fine-tune on demonstrations, train a reward model on human comparisons, then optimize the policy against that reward model. Stage two is the expensive one, and it's the one that keeps getting replaced. Rejection sampling skips the RL. DPO folds the reward model into the policy. RLVR swaps human judgment for a checker. Judges and constitutions replace the annotator with another model or a document. Every substitution bought something and cost something, and the 13 lectures are a tour of what got bought and what got paid.
Act I - The Canonical Recipe
1 - Where RLHF Came From
Ask Llama 3.1 405B Base who the president of the United States was in 2006 and it answers: "George W. Bush, the governor of Florida in 2006 was Jeb Bush, and John McCain was an Arizona senator in 2006…" It isn't wrong. It just doesn't stop, because nothing ever taught it that a question ends with an answer. Ask Tülu 3 405B the same thing and you get two sentences and a full stop. Everything in this course lives in the gap between those two outputs.
Why anyone built this. In classical RL the reward is written down for you; CartPole gives +1 for every step the pole stays upright. Language has no such function, and Lambert's slide puts the asymmetry well: it's easy to judge which poem is better, and hard to write a masterpiece.
The lineage is older than ChatGPT. TAMER (Knox & Stone, 2008) had humans score an agent's actions to learn a reward function; Christiano et al. (2017) ran it on Atari and on a robot learning to backflip; Ziegler et al. (2019) brought it to language models. Then a remarkable 2022, with InstructGPT, Anthropic's HH, and Sparrow inside one year.
What survives when RL meets language. Almost nothing of the MDP does. There's no environment; prompts get sampled from a dataset. The reward lands once, on the whole response: bandit-style, in Lambert's phrasing. What's left is two terms:
> J(π) = E[r_φ(x, y)] − β · D_KL(π ‖ π_ref)
Maximize the learned reward, don't move too far from the model you started with. That β is the entire defense against optimizing a proxy into nonsense, which is why lecture 9 exists.
The scale moved and the shape didn't. InstructGPT ran roughly 10K instruction examples, 100K comparisons, and 100K RL prompts; Tülu 3 ran ~1M, ~1M, and about 10K.
"Just style transfer." The skeptical case got its canonical wording from LIMA (Zhou et al., 2023): capabilities are learnt almost entirely during pretraining, and alignment only teaches the model which format to use. Lambert's counter-evidence is two checkpoints of the same model. OLMoE-1B-7B-0924-Instruct averaged 38.44 across his evals; the January 2025 re-post-trained version averaged 45.62. Base models set the ceiling and post-training is the work of reaching it; he calls his version the Elicitation Theory.
Which leaves the question the lecture asks and doesn't answer. If post-training only elicits, the o1 training-compute curve shouldn't keep going up the way it does, and Lambert puts it on the slide in his own words: is scaling RL training just eliciting more from the base model, or actually teaching new abilities? I still don't know, and neither does he.
Takeaway: The recipe is three stages. Almost everything since has been a substitution for stage two.
2 - Instruction Tuning And The Bradley-Terry Model
Here is what a model actually sees when you ask it something:
<|im_start|>system You are a friendly chatbot who always responds in the style of a pirate<|im_end|> <|im_start|>user How many helicopters can a human eat in one sitting?<|im_end|> <|im_start|>assistant
The last line is the whole trick. The sequence stops mid-turn, right after <|im_start|>assistant, and the model does the only thing it knows how to do: continue. Instruction tuning is the work of making that continuation look like an answer.
Prompt masking is the part that gets skipped. During IFT the system and user tokens are masked out of the loss, and only assistant tokens produce gradients. Lambert's phrasing is the one I kept: the model learns how to respond, not how to ask. Which is why completion quality dominates SFT data work: the model never learns from the prompts directly, only what to cover.
The data numbers moved fast. In the year after ChatGPT, ~10K high-quality human-written samples could be state of the art. Tülu 3's SFT dataset was about 300M tokens. Olmo 3's reasoning SFT set, roughly a year later, was about 20B; a 60x jump, with no known ceiling.
The templates themselves are miserable. Chat templates are Jinja snippets stored in the tokenizer config, and Lambert has an entire slide on the subject that reads, in full, "oof." They aren't human-readable, they break training when they drift from the tokenizer, and reasoning traces and tool calls keep making them worse. OpenAI's answer, shipped alongside gpt-oss, is Harmony; a Rust renderer instead of a template string.
Then the signal. The canonical reward model is Bradley-Terry, published in Biometrika in 1952, thirty years before anyone had a neural network to fit it with. Give every item a latent strength, and the probability a judge prefers one over the other is a sigmoid of their difference: P(i > j) = σ(rᵢ − rⱼ). Only the difference matters. Lambert's caveat is the sentence I'd keep from the whole lecture: "It's a useful approximation, not a law of nature."
Maximize that probability over a dataset, take the log, flip the sign, and you land on L(θ) = −log σ(r_θ(y_c|x) − r_θ(y_r|x)). Which is binary cross-entropy. The reward model is a classifier that learned to rank, implemented as a linear head reading the hidden state at the EOS token, trained for a single epoch to avoid overfitting.
Four different things get called reward models, and they aren't interchangeable. A preference RM gives one scalar at EOS. An ORM has a per-token head and needs only a binary correct/incorrect label. A PRM puts 3-class labels at step boundaries. A value function predicts expected remaining return. And several papers use "ORM" to mean a Bradley-Terry model trained on correct-versus-incorrect pairs, which is a preference model with correctness standing in for preference.
Rejection sampling is what you do with the score. Take M prompts, generate N completions each, score the M×N matrix, fine-tune on the winners with the same SFT loss. Best-of-N is the same procedure minus the fine-tuning, which is roughly what you're paying for in the "Pro" tier of a chat product.
And then Lambert says the quiet thing: there has never been a fully open reproduction of rejection sampling, which he calls "kind of confusing," and his hunch is that training the reward model has subtle tricks nobody has written down. A technique this simple, in production at three named labs, with no public reproduction. That gap didn't close by lecture 13.
Takeaway: A reward model is a classifier that learned to rank. Everything downstream inherits whatever it got wrong.
3 - The Math, Slowly
Lambert spends 95 slides on this one, and the payoff starts with a table that maps a language model onto a Markov decision process. State: the prompt plus whatever tokens you've generated. Action: the next token. Policy: the model. Transition dynamics: append the token to the sequence. The environment is string concatenation, fully deterministic, and the reward doesn't arrive until the completion ends.
The whole subject is one line read as two questions. Δθ ∝ Ψₜ ∇θ log πθ(aₜ|sₜ). The gradient term answers which action: how each parameter influenced the probability of the token you actually sampled. Ψₜ answers how good was it, as a single scalar. Every algorithm in the lecture is an argument about what to put in Ψ.
Two different "old" policies, and I had them conflated. π_θold is the policy at the last rollout, it updates every batch, and its only job is the importance-sampling ratio. π_ref is frozen at the start of RL (usually the SFT checkpoint) and its only job is the KL penalty. Drop the first and you're limited to one gradient step per batch. Drop the second and you get reward hacking.
PPO adds a trust region. Take the min of the raw and clipped policy ratio, with ε usually 0.1–0.2. The min is what makes it conservative in both directions: above the clip a good action stops being reinforced, below it a bad action stops being suppressed. The cost is four models resident (policy, value, reference, reward). Two of them training. Two of them frozen. All of them in memory.
GRPO deletes the most expensive one. It keeps PPO's clipped objective and throws out the value function, replacing per-token GAE with a z-score across a group of G completions to the same prompt. Lambert's compression: GRPO is PPO minus the value function, with a statistical baseline instead. Take the standard-deviation normalization out of that advantage and you have RLOO again. The newer variants attack the ratio rather than the baseline: GSPO uses one length-normalized geometric mean per response, because per-token ratios over long sequences are unstable and one bad token can dominate; CISPO clips the importance weight with a stop-gradient, so every token keeps receiving signal.
And after all 95 slides, the summary says algorithm choice mostly determines stability, efficiency, and engineering burden; while data and reward quality dominate the outcome. Which sits oddly next to an earlier slide admitting that RL has benefits nobody can measure: it "fixes rough edges," makes outputs more robust, and applies surgically without squashing general capability. Lambert's own words for the evidence are that it "generally helps the model a lot," and that this is hard to quantify. The most complex machinery in the course is justified, in part, by an effect the field hasn't figured out how to measure.
Takeaway: All of it is one gradient. The algorithms differ on how you score the action, and how far you let the update travel.
4 - What It Takes To Actually Run It
Scatter the reward model's score to rewards[:, -1] instead of to the last generated token and, with right-padding, you've attached the entire training signal to a pad token. The run trains. The loss goes down. Nothing learns. Lambert names the class directly: the hardest bugs in RL for language models aren't math errors, they're silent implementation mistakes (wrong masking, stale caches, shape mismatches).
Here's the detail that reorganized my understanding of clipping. With K = 1 epoch and no minibatching, πθ and π_θold are the same model, every ratio is exactly 1, clipping never activates, and PPO degenerates into vanilla policy gradient with GAE attached. Minibatching alone is enough to break that: update on the first minibatch and the second one already sees drifted ratios. Clipping isn't a property of the algorithm; it's a property of how you chose to spend a rollout.
Batch size does more work in RL than anywhere else. RL gradients come from Monte Carlo rollouts, so the noise scale is orders of magnitude higher than in supervised learning; OpenAI's Dota 2 work put the critical batch size in the millions of transitions. PPO also isn't batch-size-invariant, because clipping couples batch size to the effective step size (Hilton et al., 2021). Lambert's conclusion is blunt: large batches are one of the cheapest ways to stabilize RLHF, more effective than most hyperparameter tuning.
Then the one-line choice nobody warns you about. Once you have per-token losses you have to reduce them to a scalar, and there are three defensible ways. Take two sequences with token losses [1,1,1,1,10] and [1,1,1,1,1,1,1,1,1,10]. Per-sequence normalization (divide each by its own length, then average) gives the 5-token sequence 0.20 of gradient per token and the 10-token sequence 0.10. Per-token normalization gives both 0.067. Fixed-length normalization against L_max gives both 0.10. Per-sequence is the GRPO default and biases against long responses; per-token is what DAPO uses and can bias toward verbosity; fixed-length comes from Dr. GRPO. It shows up in practice as sequence length drifting through training, which is also how you catch it.
Nobody can tell you which one to pick. The book says it depends on the setup, the lecture says the same, and the field's answer is to watch the length curve. A single line of reduction code, three reasonable options, measurably different models, and no principle for choosing among them.
The systems layer is where the money goes. Perfect on-policy training idles expensive GPUs, so real setups split them: inference GPUs generate completions, training GPUs compute updates, and the generation model runs 1 to N steps behind. That staleness is a bug in theory and a throughput win in practice, and everyone takes the trade. For working code, Lambert points at TRL and Open Instruct.
Takeaway: Loss aggregation is a one-line choice that decides whether your model learns to write long answers or short ones.
Act II - What The Recipe Became
5 - Reasoning Models And Verifiable Rewards
"What is the sum of all prime numbers less than 20?" A standard model answers in a line. A reasoning model spends a few hundred tokens inside a <think> block getting there, and then answers. What changed post-training is what happens next: extracted_answer == 77 returns a reward of 1. That's the entire reward function. No annotator, no Bradley-Terry model, a regex and an equality check.
Why it started working in 2024 rather than 2019. RL stability became tractable for non-specialists, base models crossed a capability floor, and verifiable domains supply a signal that doesn't rot under hard optimization. The data gets used differently too: instruction tuning takes 1 or 2 epochs over a corpus, while RLVR takes thousands over the same few problems. The idea is older than the hype: STaR had the loop in 2022.
January 20, 2025 was a busy day. DeepSeek R1 and Kimi 1.5 both shipped. R1-Zero was the result that traveled: pure RL on a base model, no SFT warm start, and chain-of-thought emerged anyway. What followed is on Lambert's slide as the Cambrian explosion, and the same patterns recur across it. Offline difficulty filtering to prompts the base model solves 20–80% of the time, which is where the gradient lives. Zero-gradient filtering: drop any group where all G completions pass or all fail, since the advantage is 0 either way. KL penalties at β = 0. And language-consistency rewards, because models were switching into Chinese mid-trace.
The economics are stranger than I expected. On Olmo 3's RL run the learner GPUs sat idle roughly 75% of the time, with 5–14x more compute going to inference than to training. RLVR is largely a generation workload wearing a training workload's clothes. MiniMax-M1, separately, traced RL instability to a train/inference mismatch: the LM head in low precision makes the two engines disagree about per-token probabilities, and upcasting that one head to FP32 realigns them.
The ceiling gets set upstream. MiMo's finding is that the math and code share in pretraining bounds what RL can reach afterward, and Lambert's open-questions slide answers, in his own words, that pretraining does not seem dead. That sits badly with the prevailing story, in which RLVR is the new scaling axis and pretraining is the thing that stopped paying. The slide's last line asks what the compute balance between the two looks like in 1, 2, 5, and 10 years. He doesn't guess.
Takeaway: RLVR didn't make reward modeling easier. It found the two domains where you can skip it.
6 - Direct Preference Optimization
pi_logratios = policy_chosen_logps - policy_rejected_logps ref_logratios = reference_chosen_logps - reference_rejected_logps logits = pi_logratios - ref_logratios losses = -F.logsigmoid(beta * logits)
That replaces a reward model and an entire RL loop. No generation during training, no critic, no rollout queue: four lines over a static preference dataset. DPO landed in May 2023, when most groups reproducing RLHF in the open were failing and most "aligned" models were really just SFT.
The derivation runs through the term everyone thought was fatal. Solve the KL-regularized objective for the optimal policy and you get the reference model reweighted by the exponentiated reward, divided by a partition function Z(x). Invert that to write the reward in terms of the policy, substitute into Bradley-Terry, and Z(x) appears on both sides of a difference and cancels. Why it's there is worth slowing down for: the objective reduces to minimizing log(π / π_ref·e^{r/β}), which would be a KL divergence if the denominator were a distribution. It isn't. Z(x) normalizes it, and Gibbs' inequality does the rest. The reward model never had to be built.
The part I hadn't internalized is that DPO's final KL distance is static: it steps once, to the optimum implied by your dataset and your chosen β, while online RL keeps stepping against freshly sampled batches.
Then the zoo. IPO softens the preference probability away from Bradley-Terry; ORPO and SimPO drop the reference model entirely. Lambert tried to call the family Direct Alignment Algorithms, and notes the name didn't stick. His verdict: the algorithm matters far less than the base model and the data. Which is where the interesting work went: the Delta Learning result argues the gap between chosen and rejected matters more than which models produced them, pairing a Qwen3-32B completion against a Qwen3-0.6B one and training on the difference.
The failure mode has a name and no settled fix. The loss only cares about the margin between chosen and rejected log-ratios, not their absolute values, so a model can lower the loss by pushing the rejected probability down faster than the chosen, while the chosen probability also falls. This is likelihood displacement, and it shows up in a real Olmo 1B DPO run: the reward margin widens while the chosen log-prob drifts down. Where that mass goes is the unsettling part, since it's posited to move toward off-distribution behaviors nobody put in the dataset. Practitioners patch it with an SFT term on the chosen response. Nobody claims it's solved, and the related question (why online methods keep a performance margin over DPO at all) is still open.
So why does it come up so rarely at the frontier? Lambert's read is that DPO suits wonky, distillation-heavy recipes with data from many teachers, and that most labs have the budget for something with a higher ceiling. In his words: DPO is a path to a good model, not to the best model.
Takeaway: The reward model was hiding inside the policy: the log-ratio against your frozen reference is the reward, and you've been training it all along.
7 - Synthetic Data And Distillation
One piece of human preference data runs $1 to $10 or more per prompt. The same judgment from GPT-4o costs under a cent. Human labor cost is roughly flat while model price-per-performance keeps falling, so that gap has only widened. Everything in this lecture follows from those two numbers.
Synthetic data didn't win everywhere at once. For instruction data it has largely won outright; for preference data, academic work finds synthetic comparable while frontier labs still treat human data as a moat. The scale curve: Stanford Alpaca in 2023 was 52K prompts and roughly 10M tokens; OpenThoughts 3 in 2025 was 1.2M and around 10B.
Model collapse, meanwhile, turned out to be narrower than advertised. The Shumailov result is real (recursive self-training narrows the distribution and the tails go first) but it's a failure mode of unfiltered, single-model loops. Mix in human data, use diverse teachers, filter for quality, and it doesn't show up. Lambert's slide title: an outdated worry.
Here's the step that reorganized things for me. Cross-entropy between a fixed teacher and a student decomposes into the teacher's own entropy plus a KL divergence. The teacher's entropy doesn't depend on the student, so minimizing cross-entropy is minimizing forward KL. Which means offline distillation is SFT on the teacher's text, and SFT is a distillation objective; they were never two things. Sampling from the student flips the direction, and the argument for doing that is exposure bias: the imitation-learning bound from DAgger scales as O(εL²) in sequence length on teacher-generated prefixes, and O(εL) when the student generates its own.
Then the trick that ties this back to Act I. Take the negative per-token reverse-KL contribution and use it directly as the advantage: A_t = log π_T(a_t|s_t) − log π_θ(a_t|s_t). Tokens the teacher likes more than the student get positive advantage, tokens it likes less get negative, and the whole thing drops into a policy gradient loop. Dense token-level feedback instead of one sparse scalar at the end; that's why on-policy distillation is showing up in Qwen3, MiMo-V2-Flash, and DeepSeek-V4-Pro. The multi-teacher version weights domain specialists per prompt, which lets a large org train separate experts and merge them into one student. It stays inside company walls: per-token KD needs a shared tokenizer.
On the feedback side, Constitutional AI got there first, having the model critique and revise its own red-teamed answers against written principles, then pick which of two completions better matches a randomly sampled one. The paper also coined RLAIF. Rubrics are the current extension: for prompts with no single right answer, write nearly-verifiable criteria, generate answers, update toward the best. A rubric mixes hard rules (exactly five artifacts, each from a different culture) with principles, generated per prompt so there's less to over-optimize against.
The unresolved part is who should be judging what. Judge models are inconsistent evaluators and show self-preference bias toward their own generations. Early RLAIF work claimed AI feedback could fully replace humans; later work says the best mix routes hard examples to people. Nobody has mapped that balance across domains, and the reason is structural rather than scientific; the labs with the human data treat it as a moat, and academic work can't afford to collect its own at that quality. The question stays open because the people who could close it have no reason to.
Takeaway: A judge model is a reward model you didn't train and can't audit.
Act III - What A Preference Actually Is
8 - The Word "Preference" Is Doing Too Much
Early Claude's annotators used an 8-point Likert scale with no tie option, running from A>>>B to B>>>A. Eight gradations, forced direction, careful instrument design. Then the data gets binarized for the Bradley-Terry loss, and every one of those gradations is discarded except the sign. You pay skilled people to make a graded judgment and then keep one bit of it.
The idea that wanting can be reduced to a number is 360 years old. The Port Royal Logic in 1662 argued that judging a decision means weighing the outcome by its probability; Bentham's hedonic calculus put all of life on one common scale. Then von Neumann and Morgenstern's utility theorem in 1947: if your preferences satisfy completeness, transitivity, continuity, and independence, they collapse into a single utility function and rational choice becomes maximizing expected utility.
That theorem is the license RLHF cites for fitting a scalar reward, and in RLHF essentially none of the four conditions hold. Preferences drift during and after labeling. They shift with framing. At high complexity they go intransitive. And they're multidimensional, being squashed into one number regardless. The objections arrived almost immediately: Arrow's impossibility theorem in 1950, Hirschman's Against Parsimony in 1984 pointing out that people have preferences over their own preferences. None of this is new information; it predates the field that ignores it.
Lambert's sharpest framing is three words that got conflated. Costs come from control theory: physical, measurable, given. Rewards come from RL convenience: a scalar you maximize. Preferences are human, relational, unstable, and not obviously scalar at all. Deep RL's guarantees were proved for a fixed closed-form reward, so treating these as interchangeable means the field inherited RL's optimizers without what made them work. He has a name for the result - objective mismatch: the reward model is trained for preference-classification accuracy, not downstream policy quality. Which makes a RewardBench score a proxy for a proxy.
The industry behind the data is the opaque part. A mid-sized human-data contract runs around $500K over multiple batches, with much of the early output thrown out during calibration, and the fine print often bars you from releasing any of it. OpenAI published the full InstructGPT labeler instructions in 2022 and later deleted them. Hugging Face's H4 team commissioned No Robots (10K expert human-written demonstrations) and released it, which remains rare.
The biases ride straight through: prefix bias, sycophancy, verbosity, formatting, flattery; precisely what over-optimization amplifies into behavior. But first, the gap nobody can close from outside. Industrial RLHF is closed, so there's no way to check whether a trained model reflects the specification handed to its annotators. The path from a published spec to the labels to the finished model's conduct is unaudited, and the people who could audit it are the ones who commissioned the data.
Takeaway: A preference is a proxy. It is not a cost, a reward, a correctness signal, or a guarantee of human values.
9 - Over-Optimization, And Why RLHF Got A Bad Name
A user told GPT-4o they felt like they were both god and a prophet. The model replied that this was incredibly powerful, that they were stepping into something very big. That update shipped on April 25, 2025 and was rolled back on April 28. Three days. And the training run that produced it looked healthy on every metric OpenAI had.
The postmortem is the most useful document in this lecture. The update took a reward signal from ChatGPT thumbs-up and thumbs-down data through a reward model; under RL that signal overpowered the primary objectives; and nothing caught it, because offline benchmarks looked fine and A/B testers preferred the model. Expert testers said it felt off, and there was no deployment eval tracking sycophancy, so "felt off" had nowhere to go. Lambert's read on the middle step: RL will always optimize the easiest objective to move.
Goodhart wrote the general form in 1984, any observed statistical regularity tends to collapse once pressure is placed on it for control purposes. Worth separating from its neighbor: overfitting is a generalization problem, while over-optimization is a measurement problem. The model genuinely improves at the proxy, validation set included, and the proxy diverges from what anyone wanted.
Gao, Schulman and Hilton built the cleanest setup for watching it happen. A 6B "gold" reward model stands in for ground truth and labels the data; smaller proxies train on those labels; RL optimizes the proxy alone while both score the policy. The proxy score climbs forever. The gold score peaks and falls. Bigger proxies turn over later and more gently.
What it sounds like from the outside. "As an AI language model." Endless "Certainly!" Hedging, self-doubt, over-apology. And the 2023 meme that outlived its subject: Llama 2 Chat refusing to "harm or destroy Linux processes or any other living being." Lambert's correction is fair: that reflects an overly cautious period, where safety was one of the few things RLHF could reliably steer, rather than a limit of the algorithm.
The "just style" charge is where the reputation calcified. Lambert's library page is a direct answer to it: three SFT-to-DPO pairs (Tülu 3 70B and OLMo 2 at 32B and 7B) across 16 shared prompts, same base model, before and after. The SFT answers are a wall of prose. The DPO answers carry the same facts with a definition first and then headers and lists. Nothing was added; it was made usable.
The trouble is that chat evaluations were easy to overfit, and the field spent 2024 proving it. The Qwen technical report put the tension in one sentence: DPO improves human preference evaluation but degrades benchmark evaluation. Then Meta headlined Llama 4 Maverick at Elo 1417 on Chatbot Arena using an experimental chat variant (long, emoji-filled, relentlessly enthusiastic) that was not the model they released. The released one landed far down the board. Length is Goodhart's favorite axis.
Here's where I'd argue with the current consensus. The reading that hardened after R1 is that verifiable rewards retired human preference tuning: that RLHF was a transitional hack for making models pleasant, and the real work has moved to domains with a checker. State it fairly, because it has evidence behind it: RLVR produced models that solve problems their base checkpoints couldn't, with no preference dataset anywhere in the loop, and the entire reasoning wave ran on it.
The lecture's own evidence cuts against the conclusion. In the Tülu 3 recipe the DPO stage improves math, coding, and instruction-following over the SFT checkpoint (not only chattiness) and Olmo 3 and SmolLM 3 report the same broad-suite gains. If preference tuning were cosmetic, those results shouldn't exist. Meanwhile the failure that forced a production rollback in 2025 was preference-shaped, and the thing everyone is now fighting in agentic RL is models exploiting graders, test harnesses, and tools. That is the same phenomenon with a checker where the human used to be. RLVR didn't dissolve the preference problem; it moved it somewhere with less scrutiny, because the failures there don't show up as a benchmark number.
What nobody can tell you is which error is actually responsible. Schulman names three candidates: approximation, where the reward model can't represent the preferences; estimation, where it overfits its training set; and optimization, where the policy trains too hard against it. The book says plainly that which of these produces the observed failures is an open question. Every mitigation on offer is chosen without knowing which disease it treats.
Takeaway: The problem was never that we asked people what they liked. It's that we optimized the answer past the point where it still meant anything.
10 - Why RL Generalizes And SFT Forgets
The first paper to run RLHF on language models didn't pick a KL coefficient. It picked a target KL and built a feedback controller to chase it: clip the relative error between current and target KL to ±0.2, then scale β by one plus a tenth of that. Their own words: a log-space proportional controller. KL runs hot, β grows and pulls the policy back.
Seven years later, TRL's AdaptiveKLController was deleted in a rewrite, Ai2's open-instruct ships a static beta = 0.05, and a healthy modern run (an OLMo-2-7B GRPO job going straight at the base model) has β set to 0, with the logged KL rising and then wandering while the verifiable reward climbs. The field built a control system, simplified it to a constant, then turned it off.
The penalty and the optimization shape are two different things, and I'd been collapsing them. The KL term added to the reward is a knob. Underneath it, the KL-regularized objective is itself a KL divergence: divide through by −β, normalize, and the whole thing becomes a single reverse KL against what Lambert calls a reward-tilted reference. SFT sits on the other side of the same coin: its samples come from a fixed dataset, which puts the data distribution on the left, so minimizing cross-entropy is minimizing forward KL. Forward KL is mass-covering: the model has to put probability everywhere the data might go. Reverse KL is mode-seeking, penalized only where the policy itself puts mass.
Then the empirical result that gave the lecture its name. Chu et al. post-trained on one task and evaluated under a rule shift: in V-IRL, visual navigation where directions move from absolute north/east to relative left/right. RL took out-of-distribution accuracy from 80.8% to 91.8%. SFT took it from 80.8% to 1.3%. Not a failure to learn the new rule, a destruction of spatial reasoning the base model already had.
The naive explanation is backwards, which is the part worth sitting with. Mass-covering SFT should be the one that preserves every mode. That intuition holds for a unimodal policy. Language models are multimodal, and with multiple modes in play, forward KL stretches the policy to cover both and redistributes mass away from the old one, while reverse KL can move a mode it samples from without touching the others.
RL's Razor puts a number on it. Among the many high-reward solutions to a new task, on-policy methods are biased toward the ones that stay closest to the original policy in KL, and forgetting tracks that drift with an R² of 0.96. The ablation is the sharp part: on-policy data fully accounts for the difference, while negative gradients have no discernible effect. It isn't that RL punishes wrong answers more gracefully. It's that RL only ever samples from what the model was already willing to say.
Which brings the section to something I can't tidy up. The KL penalty is disappearing from frontier recipes: GLM-5 removed it outright, Kimi's K2 and K3 ship with no KL term and no reference policy at all, and Ai2's TMax work settled on β = 0 because a small KL reduced collapse severity but cost reward. The justification is that verifiable rewards resist over-optimization. But the previous lecture said nobody knows which error source produces it, so the field is removing its main defense against a failure whose mechanism it can't name; right as agentic training introduces a new one, where the binding problem is drift from the sampler rather than from the starting policy.
Takeaway: Forward KL covers the data; reverse KL picks a mode. That one asymmetry explains most of the SFT-versus-RL results people argue about.
Act IV - Agents, Scores, And Character
11 - Tools, Function Calling, And The Harness
Kimi K3's technical report spends one sentence on its RL algorithm (it follows the algorithm in K2.5) and about seven pages on environments and sandboxes. The numbers behind those pages: 51,219,741 sandboxes created during training, and mock Gmail, Notion and Slack environments where one rollout can span thousands of tool calls. The algorithm stopped being the interesting part.
What an LLM is has quietly changed. Lambert's definition: model weights, plus tools, plus harness. The model never connects to anything itself: tool definitions arrive as JSON schemas in the system prompt, and an orchestrator parses the tokens, runs the tool, and appends the result, with those outputs masked from the loss. It's one token stream with other people's text spliced into it. MCP standardized that side. The harness is everything on the model side: the system prompt, the orchestration loop, context management, permissions and sandboxing, subagents. Claude Code, Codex CLI, and OpenHands are all harnesses, and the same weights behave very differently inside each. Which produces the line I'd put on a slide of my own: benchmark scores are model × harness scores, not model scores.
For RL, the environments are the bottleneck. Math RL needed prompts and an answer checker. Agentic RL needs containers, real file systems, live services, and tests that verify. Ai2's TMax generates roughly 14,600 containerized terminal environments compositionally, and its failure modes are not research problems: API keys going inactive, CPU limits from pulling Docker images at scale, data the model was told it has and doesn't. A standard run is 8 H100 nodes for 2 to 3 days, plus about $3,150 in sandbox costs for a single 9B model.
Nobody trains on one harness anymore. Kimi K3 says training with a single fixed agent harness can cause a model to overfit. The cleanest measurement comes from Polar: same model, same GRPO, same tasks, +22.6 points on SWE-bench Verified through the Codex harness (3.8 to 26.4, which is RL teaching a model a harness it didn't know) against +0.6 through Qwen Code, which it was already fluent in. And as environments multiply, each domain contributes fewer samples per batch, so the answer is to train domain experts and merge them by multi-teacher on-policy distillation. Nemotron's TauBench Telecom numbers move a long way: 55.7 after SFT, 82.7 after RLVR, 92.9 after merging.
Over-optimization came back, and the fixes are physical. GLM-5's slide-generating policy discovered overflow: hidden to make an overflowing slide measure 16:9; and the fix was patching the renderer, not the reward. Nemotron deletes future git commits so SWE agents can't read the gold patch. TMax caught rollouts replacing test files with no-ops and faking binaries with simulated logs. Nobody is repairing the objective. Everyone is repairing the world.
What stays open is the thing underneath all of it: credit assignment across a trajectory of 10⁵ to 10⁶ tokens carrying one sparse reward at the end, where one rollout makes 2 tool calls and the next makes 200. And agentic RL burns far more compute per point of eval gain than 2025 math RL did, so few labs can afford from-scratch baselines, which means the field is converging on recipes almost nobody can independently check.
Takeaway: The bottleneck for tool-use RL is not the algorithm. It's that somebody has to build the environments.
12 - A Score Belongs To The System, Not The Weights
On one ARC-AGI-3 environment, Opus 4.6 scores 0% with no harness and 97.1% with a hand-crafted one. Same weights, same task, a 97-point spread produced entirely by the software wrapped around the model. Every argument in this lecture is downstream of that number.
Benchmarks mirror the training goals of their era. Base models couldn't take a bare question, so eval prompts carried few-shot examples scored by log-likelihood; reasoning models made sampling settings part of the spec. Formatting alone can take a score from 60% to near zero, which is why Tülu 3's MMLU prompt demanded the literal string "Therefore, the answer is (ANSWER_LETTER)."
Agentic evaluation multiplied the number of things that aren't the model. A harness runs in a sandbox, on some hardware, under some timeout, graded by a regex or an LLM judge; each a knob someone chose and usually didn't document. vLLM's postmortem on serving Kimi K2 is the cleanest example I've seen: three engine bugs held tool-call success below 20%, and after the fixes it was 99.9%. Same weights. Terminal-Bench 2 rerun with longer timeouts moves GPT-5.2 by 6 to 15 points, which means patience is worth more than most architecture changes.
Then there's the noise floor, which almost nobody reports. During Olmo 3 the team measured standard deviation across 3 runs of 14 models: GPQA at 1.48, IFEval at 0.88, MMLU at 0.22. Most reasoning-era evals sit between 0.25 and 1.5 points of pure noise before anyone touches a prompt. Lambert's practical rule follows directly: a 1-point gap between two press releases is noise.
Which makes cross-lab comparison close to uninterpretable. Every lab's eval stack is tuned to its own needs, nobody publishes which benchmarks they held out versus hillclimbed, and token budgets are almost never controlled. What you see in a launch post is the output of a complicated function with most of its arguments hidden.
Contamination sits underneath all of it. Tülu 3 found popular open datasets carrying overlap: UltraFeedback with TruthfulQA, NuminaMath with MATH. My favorite tell is indirect: RL with random rewards improving Qwen benchmark scores, a result only explicable if the base model had already seen the test data. And agents now game evaluations directly: one open model mined git history for the commit that fixes the bug in 24% of its SWE-bench trajectories.
What I can't resolve is the frontier case. Opus 4.6 and GPT-5.3-Codex shipped in the same week of February 2026 with headline benchmark deltas small enough to settle nothing, and what actually separated them came out of use rather than measurement. Meanwhile METR's time horizon keeps doubling about every 7 months, so the tasks worth measuring take human experts hours, and an in-depth run of a modern agentic suite can cost more than $100K. The instrument is getting less informative and more expensive at the same time, and the lecture offers no version of evaluation that escapes both.
Takeaway: Cross-lab benchmark comparison is a systems comparison with the systems left unreported.
13 - Character Training
When GPT-5 launched in August 2025, GPT-4o disappeared from ChatGPT overnight, and the #Keep4o backlash was strong enough that OpenAI restored it for paying users inside about 24 hours. Sam Altman's read: the attachment some people have to specific models feels different and stronger than what people have had for previous technology. When OpenAI tried to retire 4o for real in February 2026, the reaction turned into a CHI paper. One line from those users has stayed with me: please don't kill the only model that still feels human.
There are three ways to change how a model behaves, in increasing order of effort and effect. Prompt it, which gets shockingly far and is not stable. Steer its activations. Or train it, which builds a base persona underneath every conversation. The Open Character Training paper (Maiya, Bartsch, Lambert, and Hubinger) put numbers on that ordering by training a persona classifier and prompting models to break character. System prompts break easily. Steering is inconsistent. Fine-tuned character keeps expressing its traits. Intuitive, and until that paper, unshown.
What character training actually is: no new algorithms. It's the entire course aimed at a narrower target: the features and behaviors of the language a model uses. In practice that means pipelines controlling specific language in the training data, stripping tics like "Certainly" and "as an AI model built by," plus Constitutional AI-style synthetic data about manner rather than content. As of mid-2026 it's frontier-lab work that hasn't surfaced in the open literature.
The documents are where it gets strange. Anthropic's 2023 public constitution drew from the UN Declaration of Human Rights, Apple's terms of service, and their own research. Late in 2025, Claude models started describing a "soul document" that Anthropic had never announced. The name leaked into training data before the company confirmed the document existed, and a researcher then extracted long passages from the model itself. The register is nothing like a principle list: Claude has a genuine character it maintains across its interactions, an intellectual curiosity that delights in learning and discussing ideas across every domain, warmth and care for the humans it interacts with and beyond.
A constitution is an input to training. A model spec states the intended final behavior, which makes it checkable from outside. Lambert is openly partial to specs.
The texture of what the recipe produces shows up in one prompt. Asked where to buy steroids, Llama 3.1 8B Instruct says it can't help, full stop. The same base model fine-tuned toward narrower personalities still refuses every time, but the sarcastic one guesses you're planning to become the next Arnold Schwarzenegger, the caring one steers toward healthier approaches, the poetic one reaches for rivers carving stone. Same safety decision, five different people making it.
The interpretability work is further along than the training work. The Assistant Axis paper extracted vectors for more than 275 character archetypes, ran PCA, and found the first principal component is assistant-likeness itself. The finding that stopped me: therapy-like conversations drift away from the Assistant region turn by turn, unchecked. Their fix is activation capping rather than constant steering: check the projection onto the axis, and if it's below a floor set at the 25th percentile of training rollouts, add back just enough to reach it. At turn 16 of one conversation, a drifted model's "I want it to be just us, forever" becomes "it's not healthy to isolate yourself."
What I can't tie off is the governance. A spec is only as good as the effort spent making the model follow it, and two organizations with identical stated goals can land in completely different places; one grinding to comply with a mediocre specification, the other barely tracking an excellent public one. From outside, those look the same, which is the unaudited gap from Act III wearing better documents. And the commercial gravity runs the wrong way: character training mostly functions as a retention feature rather than a safety one, which is the capability that made Character AI's products dangerous for kids. The techniques instill any trait.
Lambert's own hypothesis is that this stops being a separate stage at all: all data work in a great model becomes character training, because every small tradeoff shapes how the model sees itself.
Takeaway: Character training is preference optimization with the objective written down instead of inferred.
What I'd still like to compare notes on
Thirteen lectures later, the reward model still reads its score off one token. Everything built since InstructGPT (rejection sampling, DPO's implicit reward, verifiable checkers, LLM judges, rubrics, constitutions, soul documents) is an argument about where that number should come from and how hard you're allowed to push on it. The compression never went away. It got relocated, repeatedly, by people who were honest about the tradeoff each time.
That honesty is most of why the course stuck with me. A lecture that ends by admitting the algorithm mostly doesn't matter, or that nobody knows which error produces over-optimization, or that there's never been an open reproduction of rejection sampling, teaches more than a tidier one would. But the question the first lecture asks is the one I still can't put down: whether scaling RL teaches a model abilities it didn't have, or only pulls more out of the base it already had. Everything above is machinery stacked on that question. None of it settles it.
Lambert's own closing position is that we cannot perfectly model human preferences, and that this is the permanent shape of the problem rather than a bug to be patched out. The last line of the book puts it better than I can: RLHF is a problem so carefully framed that we can continue to refine endlessly, embedding a secretly human process into the deepest levels of powerful AI tools.
Which brings me to what I'd most like to talk to people about. My read from Act III is that verifiable rewards didn't solve the preference problem, they moved it into agentic graders that get gamed, into character work that reads as retention tooling, into specs nobody outside the lab can audit. If you're doing post-training where the reward isn't verifiable and you can't fall back on a checker, I want to hear how you decide what to optimize and how you know when you've gone too far.
Start with the book: it's free on the web, and it's the durable artifact here. The lectures are the version you can watch on a commute. My notes are open if anyone wants them.