Reasoning models

A chain of thought is a series of steps that a language model writes before its final answer¹. A reasoning model is a language model that is trained to write a chain of thought for a hard request. For example, for the question “What is 47 · 36?”, a reasoning model can write the following text, where the tags <think> and </think> mark the start and the end of the steps (see Section 11.2):

<think>
47 · 36 = 47 · 30 + 47 · 6
47 · 30 = 1410
47 · 6 = 282
1410 + 282 = 1692
</think>
The answer is 1692.

The tokens of the steps are thinking tokens. The program that runs the model usually hides them, or shows a summary of them apart from the answer.

13.1Why steps help

A language model writes one token at a time, and for each token it goes once through every layer of the network (see Chapter 7). The steps give a hard question more passes through the network. They also keep the partial results in the text, where later passes can read them.

It turns out that the steps help a large enough model even without special training. In 2022, a prompt with a few examples of written steps raised the score of a model on math word problems from 18% to 57%².

13.2Training to think

The reasoning skill usually comes from reinforcement learning (see Chapter 12) on problems whose answers a program can check, for example math problems with a final number and code with tests. The reward scores the final answer and, at most, the form of the steps. It does not score whether each step is right.

The policy is free to write any steps. Steps that lead to correct answers get a higher probability, and the training makes useful steps, such as checking a result, more frequent. No training example has to show them. Smaller reasoning models are often fine-tuned on the chains of thought of a larger one instead (see Chapter 14).

For example, in a model published in 2025, the chains of thought became longer as the training went on, and checks and corrections of its own steps became more frequent³. A later study found such checks in the model before this training⁴.

13.3The cost of thinking

Thinking tokens are output tokens. The user pays for them and waits for them. For example, a model that writes 50 tokens per second needs 40 seconds for 2,000 thinking tokens. At 10 dollars per million output tokens, these tokens cost 2 cents.

Chapter 10 shows why output tokens are slower and more expensive than the tokens of the prompt.

A thinking budget is a setting that tells the model how much to think on a request, as a number of tokens or as an effort level⁵. The budget is usually a target, not a strict limit. For example, a budget of 2,000 tokens lets the above model think for about 40 seconds.

On hard questions, the accuracy usually grows about with the logarithm of the budget, so each doubling of the budget adds about the same gain⁶. On some tasks, longer thinking makes the answers worse⁷. An easy request gains little from thinking. A small budget saves time and money there.

13.4Several answers

We can also spend more work on one request in another way. The model answers several times with random sampling (see Chapter 7), and we take the answer that appears most often⁸. Suppose that each answer to a question with the answers yes and no is right with probability $p = 0.6$, and that the answers are independent. With three answers, the majority is right when at least two of them are right: $$0.6^3 + 3 \cdot 0.6^2 \cdot 0.4 = 0.216 + 0.432 = 0.648$$ The following picture shows the probability that the majority is right for odd numbers of answers from 1 to 31:

With $p = 0.6$, the majority becomes more reliable with each two more answers, and 31 answers give about 0.87. With $p = 0.4$, the majority becomes less reliable with each two more answers, because it follows the wrong answer.

For example, a reasoning model of 2024 solved 74% of a set of competition math problems with one answer and 83% with the majority of 64 answers⁹. Note that the probability of a right answer differs from question to question. On a question where the model is usually wrong, the majority is also wrong, so the real gain is smaller than the picture suggests.

13.5Unfaithful steps

The chain of thought is not always the reason for the answer. In a test of 2025, models used a hint in the question to reach their answer, but their chains of thought often mentioned the hint in fewer than 20% of these cases¹⁰. Thus, we can use the steps as useful work, but not as a reliable record of why the model answered.