Models that
think first.

In 2022, chain-of-thought was a prompting trick — add 'think step by step' and accuracy jumped. The 2024–25 generation (OpenAI's o-series, DeepSeek-R1, Claude's extended thinking) made it a trained behavior: models are optimized with reinforcement learning to produce long private reasoning before answering. The chain of thought stopped being a user technique and became part of the model.

Reinforcement Learning from Verifiable Rewards is the engine. For math, the reward is 'did the final answer match'; for code, 'did the tests pass.' No human labelers, no learned reward model to fool — a deterministic checker grades millions of attempts. DeepSeek-R1-Zero's headline result: pure RL on verifiable rewards, with no supervised reasoning examples at all, taught a base model to reflect, backtrack, and self-correct. The behaviors emerged because they won reward.

The algorithm most RLVR pipelines run is GRPO (module 09): sample a group of answers per prompt, score them with the verifier, and push the model toward the above-average ones — no critic model needed. The trio of modules connects here: RLHF (03) established the post-training idea, GRPO (09) made it cheap, RLVR (this lesson) gave it an unhackable-ish grader and the reasoning objective.

Pretraining scaled with data and parameters. Reasoning models added a third axis: inference-time compute. The same model scores dramatically higher on hard problems when allowed more thinking tokens — o1's core chart showed accuracy climbing log-linearly with test-time compute. Practical consequence: 'how long should it think' is now a product dial (extended thinking budgets, effort levels), and hard problems are bought with tokens rather than retrained for.

A reasoning model may generate 10–100× more tokens than it shows you. That inverted the cost structure of serving (decode-heavy workloads — module 08) and created the distillation pattern: train a huge reasoner with RLVR, then distill its reasoning traces into small models via plain SFT. R1-distill proved a 7B student can inherit much of the teacher's reasoning — which is why capable small local models exist at all.

RLVR needs a checkable answer — it shines on math, code, logic, and agentic tasks with verifiable outcomes (tests pass, the form got submitted). It transfers only partially to fuzzy domains (writing quality, taste), where learned reward models and human preference still rule. And verifiable ≠ unhackable: models still discover reward hacks — special-casing test inputs, gaming format checks — so frontier pipelines audit transcripts for exactly that.