Inference

Inference is running a trained model on a request: the model reads the prompt and writes the answer (see Chapter 7). The training is paid once (see Section 9.3), but inference is paid again for every request. An API counts its price in tokens (see Section 5.1), and a written token, which APIs call an output token, costs more than a token of the prompt, an input token. For example, one API charges 2 dollars for each million input tokens and 10 dollars for each million output tokens¹.

It turns out that most of the time of a written token goes to loading numbers from memory, not to the arithmetic. The model reads the whole prompt in one pass, but it writes only one token in each pass.

10.1Reading and writing

The program that runs the model works in two phases. The prefill reads the prompt in one pass, which gives the vectors of all its positions at once, as in the training (see Section 7.2). The logits of the last position give the first token of the answer. The decode writes the other tokens of the answer, one pass for each token (see Section 7.3).

For example, for the prompt The ride of 3 km cost $, the prefill reads 8 tokens and writes 9. The decode then writes . and the end token in two more passes. Each pass of the decode runs only the new token through the network, because the program keeps what it needs from the earlier positions (see Section 10.2).

We can calculate the time of a pass from the speed of the GPU. A pass does about 2 operations per parameter for each position that it runs (see Section 3.3).

The model of Section 7.1 has about 7.5 billion parameters outside its embedding matrix, which it reads without multiplying (see Section 6.2). A position then takes about 15 billion operations, and the GPU of Section 9.3 does them in about 0.015 milliseconds. For a prefill of 1,024 tokens, attention adds only about 2% to these operations on average (see Section 8.5).

However, the GPU also has to load every parameter outside the embedding matrix from its memory in each pass. At 2 bytes per parameter, these parameters take 15 GB. At 3.35 TB per second², the loading takes about 4.5 milliseconds. Since the GPU loads and calculates at the same time, a pass takes the longer of the two times.

The following table shows both times for a prefill of 1,024 tokens and for one pass of the decode:

Thus, a written token takes about 300 times as long as a token of the prompt.

The decode is limited by the memory, which papers call memory-bound³. The prefill is limited by the arithmetic. These are the times at the full speed of the GPU, and real programs reach only a part of it.

The time to first token is the time from sending a request to receiving the first token of the answer⁴. It consists of the prefill, the time that the request waits for a GPU and the time that the request and the token travel over the network. When the API hides the steps that a reasoning model writes before its answer (see Chapter 13), the first token of the answer also waits for them.

After it, the answer comes at the speed of the decode, the output speed. In the example, the output speed is at most about 220 tokens per second, because the time between two tokens, the inter-token latency, is at least 4.5 milliseconds.

A pass of the decode leaves almost all the arithmetic of the GPU unused. For this reason, a pass over a few positions takes about as long as a pass over one. Speculative decoding⁵ uses this.

A small model, the draft model, writes a few tokens ahead, and the large model checks all of them in one pass. The large model keeps the proposed tokens up to the first one that it would have written differently, and it writes its own token in that place.

For example, suppose that the large model has written 9 after the above prompt, and that the draft model proposes . and the end token. One pass of the large model over the positions of 9 and . gives its own next token at both positions. When these are . and the end token, the pass has written two tokens instead of one. With greedy decoding, the answer is exactly the one that the large model writes alone.

However, the draft model needs passes of its own. The gain is small when the large model keeps few of the proposed tokens, and in a large batch, which already uses more of the arithmetic (see Section 10.4)⁶.

10.2The KV cache

A straightforward way to write a token is to run the whole text through the network again, the prompt and the tokens written so far. For example, the three passes for the above prompt would then run 8, 9 and 10 positions, 27 in all. However, the vectors of the earlier positions do not change when a token is added (see Section 8.2), so each pass calculates the same vectors again.

The idea is to store what the later positions read. A new position reads an earlier position only through its keys and values, in each head of each block (see Section 8.1).

The KV cache stores the keys and the values of every position of the text. Initially, the prefill calculates them for the prompt. After this, each pass of the decode runs only the new position. It adds the keys and values of this position to the cache, and its attention reads the keys and values of the earlier positions from the cache.

As an example, consider the three passes for the above prompt. In the following picture, a shaded cell is a position that a pass runs through the network, and a white cell is a position whose keys and values the pass reads from the cache. The token after each row is the token that the pass writes:

The second pass runs only the position of 9, and the third only the position of .. Thus, the three passes run 8 + 1 + 1 = 10 positions instead of 27.

The cache can be large. It has one key and one value for each position, each head of keys and values and each block.

For example, the model of Section 7.1 has 32 blocks. In each block, every 4 of its 32 heads of queries share one head of keys and values, so the block has 8 heads of keys and values of 128 numbers each (see Section 8.4). Thus, each token adds 2 · 32 · 8 · 128 = 65,536 numbers to the cache, or 131,072 bytes at 2 bytes each.

The following table shows the cache of one request for three lengths:

For 131,072 tokens, which the later version of Section 8.5 can read, the cache of one request is larger than the 16 GB of all the parameters of the model. The second row is for a model with the same 32 blocks and vectors of the same length, but with one head of keys and values for each of its 32 heads of queries. Its cache is 4 times as large.

Note that each pass of the decode loads the whole cache from memory, because the new position reads the keys and values of every earlier position. For example, a pass at the end of 131,072 tokens loads 17 GB of cache and 15 GB of parameters. Such a pass takes about twice as long as a pass on a short text. For this reason, fewer heads of keys and values make the decode faster, not only the cache smaller⁷.

10.3Prompt caching

Many requests start with the same tokens, for example the same system turn (see Section 11.2), the same long document or the earlier turns of a conversation. Since the keys and values of a position depend only on the tokens up to it, these starts give the same part of the KV cache in every request. Prompt caching keeps the KV cache of a prompt after a request, and it reuses the cache for a later request whose prompt starts with the same tokens. The prefill can then skip the shared start.

For example, suppose that a request with the prompt The ride of 3 km cost $ has left its cache. A later prompt The ride of 2 km cost $ shares the first 4 tokens, The, ␣ride, ␣of and ␣. Its prefill reads their keys and values from the cache and runs only the 4 positions from 2 on.

Note that the reuse stops at the first token that differs. The prompt A ride of 3 km cost $ cannot use the cache at all. Its last 7 tokens are the same, but their keys and values depend on the first token, which is different. In practice, we put the parts of a prompt that stay the same first and the parts that change last.

However, an API reuses a start only up to certain points of the prompt. One API stores a cache only at the end of a part of the prompt that the request marks, or that the API marks by itself on request. Another API stores it only at multiples of 2,048 tokens for one of its models. The request also has to reach a machine that holds the cache⁸.

APIs charge less for the tokens that come from the cache. For example, for most of its models, the API of the opening charges a tenth of the price of an input token for a token read from the cache, and 1.25 times the price for a token stored in the cache⁹. For the model of the opening, a token read from the cache costs a twentieth.

As an example, consider requests to a model with the price of a tenth, with a system turn of 10,000 tokens and a question of 100 tokens. With the cache, each later request pays as much as for 10,000 · 0.1 + 100 = 1,100 tokens instead of 10,100, about 9 times less. The prefill also runs only 100 positions instead of 10,100, so the time to first token falls too.

However, a cache does not last long. For example, the cache of the same API lives 5 minutes, and each use starts the 5 minutes again¹⁰. A prompt start that comes less often than once in 5 minutes pays for storing the cache and never uses it.

10.4Batching

Batching is running the decode of several requests together, where one pass writes one token for each of them. The requests of one pass are also called a batch. This batch is not the training batch of Section 2.4. It has no loss and no gradient, and each request gets its own answer.

The idea is that a pass loads the parameters once, whatever the number of requests. Each request adds only its arithmetic and the loading of its own cache. For example, a pass for 64 requests of 1,024 tokens each loads the 15 GB of parameters and 64 · 0.134 ≈ 8.6 GB of caches, and it writes 64 tokens in about 7 milliseconds. The following table shows a pass of the decode for different numbers of requests, each with 1,024 tokens:

The table shows that 256 requests together get about 80 times as many tokens per second as one request, but each of them gets its tokens about 3 times more slowly. In the large batches, the caches take most of the loading. For example, at 256 requests they are 34 GB of the 49.4 GB.

Thus, a large batch lowers the cost of each token. However, the memory of the GPU limits the batch. Its 80 GB¹¹ can hold the 16 GB of the parameters and the caches of at most 476 requests of 1,024 tokens.

The table also shows why a written token costs more than a token of the prompt. The prefill of Section 10.1 takes about 16 / 1,024 ≈ 0.016 milliseconds of the GPU for each token of the prompt. A written token takes 14.7 / 256 ≈ 0.057 milliseconds even at 256 requests, and 7.0 / 64 ≈ 0.11 at 64 requests. Thus, a written token takes about 4 to 7 times as much time of the GPU as a token of the prompt, and the API of the opening charges 5 times as much for it.

The requests of a batch start and end at different times. A straightforward way is to start a batch of requests together and to wait until the longest answer ends. However, the requests with short answers then leave their places in the pass unused.

Continuous batching¹² adds a new request to the batch before any pass, and it removes a request as soon as the request writes the end token. For example, suppose that one of 64 requests writes 500 tokens and the others write 50. The straightforward way then runs 450 passes for one request, and continuous batching gives the 63 free places to new requests.

In practice, the program also stores the caches in small pages of a fixed size, so that a request takes only the memory that its tokens need¹³.

Batching changes the logits of a request a little. The GPU adds numbers with a limited number of digits, and the order of the additions changes the result. For example, suppose that the GPU keeps only 3 significant digits. Then $(100 + 0.4) + 0.4 = 100$, because 100.4 is rounded to 100, but $100 + (0.4 + 0.4) = 100 + 0.8 = 101$.

The program splits the sums of a pass in different ways for different numbers of requests, so the same request can get slightly different logits in a batch of 16 and in a batch of 64¹⁴.

When two tokens have almost the same logit, such a change can make greedy decoding choose the other token, and the rest of the answer continues from it (see Section 7.4).

10.5Quantization

Quantization stores each number in fewer bytes by rounding it to one of fewer values (see Section 6.4). The parameters of a model can be quantized in the same way. With 1 byte for each parameter instead of 2, the parameters of the model of Section 7.1 take 8 GB instead of 16 GB. The decode then loads half as many bytes, and a pass of a small batch takes about half as long.

As an example, consider the four weights 0.7, −0.23, 0.04 and 0.38, stored with 4 bits each. A number of 4 bits is an integer from −8 to 7. The idea is to give the group of weights one scale, here 0.1, so that the largest weight 0.7 becomes the integer 7. Each weight is divided by the scale and rounded to the nearest integer:

For example, −0.23 / 0.1 = −2.3 is rounded to −2, which stands for −0.2. The error of each weight is at most half the scale, 0.05. For the input (1, 1, 1, 1), the dot product with the weights is 0.89, and with the rounded weights 0.9.

In practice, a group has more weights, for example 128 weights for each scale¹⁵. With a scale of 2 bytes, each weight then takes 4 + 16/128 = 4.125 bits, and the parameters take about a quarter of their 16 GB.

Some models are trained further with few bits and released that way. For example, most parameters of gpt-oss-120b have about 4 bits each, so that its 117 billion parameters fit one GPU of 80 GB¹⁶.

It turns out that a model with 8 bits for each parameter gives almost the same answers as with 2 bytes, and 4 bits for each weight are almost as good as 8 bits¹⁷. However, such results are averages over benchmarks, and a quantized model can still be worse on one particular task¹⁸. In practice, we test a quantized model on our own requests (see Chapter 21).

10.6Mixture of experts

A model with 10 times more parameters costs about 10 times more for each written token (see Section 9.4). However, a model does not have to use all its parameters for each token.

A mixture of experts¹⁹ is a model in which only a part of the parameters works on each token. In each block, the feed-forward network, which has about four fifths of the parameters of a block in the model of Section 8.4, is replaced by several feed-forward networks, the experts, and a router chooses a few experts for each position. The router is a layer that gives each expert a logit.

As an example, suppose that a block has four experts, that the router gives the position of ␣$ the logits 2, 1.5, 0 and −1, and that it chooses the two experts with the largest logits. The softmax of the two chosen logits gives the shares 0.62 and 0.38, as in top-p sampling (see Section 7.4). The output of the feed-forward part of the block is then 0.62 times the output of expert 1 plus 0.38 times the output of expert 2:

Experts 3 and 4 do no arithmetic for this position. Another position can choose other experts.

For example, DeepSeek-V3 (see Chapter 9) has 671 billion parameters, but each token uses only 37 billion of them, because in each of its blocks with experts, the router chooses 8 of 256 experts²⁰. Thus, a written token takes about 2 · 37 billion = 74 billion operations, as in a model of 37 billion parameters.

With one request, a pass of the decode loads only the experts that the position chooses. In a batch, the positions of different requests choose different experts. For example, when 256 positions each choose 8 of 256 experts at random, on average fewer than one of the 256 experts of a block stays unused, about 0.08.

Thus, the pass loads almost every expert. However, each expert works on only about 256 · 8 / 256 = 8 of the positions, so the batch shares the loading of an expert among 8 positions instead of 256 (see Section 10.4). A model with experts thus needs a much larger batch than a model of 37 billion parameters for the same cost of a token²¹.

However, all the experts must stay in memory, because each position can choose any of them. Even at 1 byte for each parameter, the 671 billion parameters take 671 GB, so the model needs at least 9 GPUs of 80 GB.