Pretraining

Pretraining is the first stage of training a language model, where the model learns next-token prediction on trillions of tokens of text (see Chapter 7). The training starts from random parameters. The result is a base model, a model that continues a text. For example, Llama 3 8B was pretrained on over 15 trillion tokens of public text¹.

Fine-tuning (see Chapter 11) and reinforcement learning (see Chapter 12) start from a base model and use far less data. For example, DeepSeek-V3 took 2,788,000 GPU hours to train, where a GPU is a processor that does many multiplications at the same time, and one GPU hour is one GPU working for one hour. Only 5,000 of them came after the pretraining and its stage of long texts². However, the reinforcement learning of a reasoning model can be as large as its pretraining (see Chapter 13)³.

9.1The loss of a text

The pretraining uses the loss of Section 1.3 at every position of every training text. Each position predicts the token after it. One pass through a text gives the loss at all of its positions, because causal attention hides the next token from each position (see Section 8.2).

As an example, suppose that the training data is the three rides of Chapter 1, written as the texts The ride of 1 km cost $5., The ride of 2 km cost $6. and The ride of 3 km cost $9.. Each text has 10 tokens in the tokenizer of Section 5.1, so it gives 9 predictions. The following table shows them for the third text, with the probabilities of the model that has the smallest loss on the three texts:

The three texts are the same up to the distance, so the position of ␣ before the distance cannot know which distance comes. The best it can do is to give each distance the probability 1/3, and the loss there is $\ln 3 \approx 1.10$.

The price is different. The position of ␣$ reads the distance (see Section 8.1), and each distance has one price.

The loss of the pretraining is the average of the losses at all positions of all training texts. In the example, each text has the loss 1.10 at one of its 9 positions, so the smallest average loss is $\ln 3 / 9 \approx 0.12$. No model gets below it, because nothing before the distance tells which distance comes.

At the start of the training, the parameters are small random numbers (see Section 3.4). The logits are then near 0, and the softmax gives every token about the same probability. With the 200,000 tokens of the vocabulary of Section 5.1, each token gets about 1/200,000. The loss is then about 12.2 at every position.

Each step of the pretraining takes a batch of texts (see Section 2.4), calculates the loss at all of their positions and changes the parameters with AdamW. For example, Llama 3 405B read its 15.6 trillion tokens in about 1.15 million steps, with batches that grew from 4 million to 16 million tokens and a learning rate that rose in the first 8,000 steps and then fell (see Section 2.2)⁴.

9.2Training data

Pretraining data is mostly text from the web, together with code, books and scientific papers. The data mix is the share of each kind of text in the training data. For example, the tokens of Llama 3 are about 50% general knowledge and 25% math and reasoning. Code has 17%, and languages other than English have 8%⁵.

Text from the web is cleaned before the training. A copy of the web contains the same page many times, pages with little text and spam. Filters remove duplicate documents and lines (see Section 4.5 and Section 4.6) and pages in unwanted languages. They also remove pages that rules or a model rate as low quality, where the model works like the spam filter of Section 1.3⁶.

For example, the filters of one dataset of 2023 removed nearly 90% of the documents of Common Crawl, a public copy of the web⁷. The cleaned data can still contain the questions and answers of public tests (see Section 4.5).

The data decides the probabilities that the model learns. As an example, suppose that the training data has the text of the 3 km ride 10 times, where eight of the texts have the price 9 and two have the price 10.

At the position of ␣$, the model gives 9 some probability $p$ and 10 the probability $1 - p$. The average loss at this position is $-(0.8 \ln p + 0.2 \ln (1 - p))$, and the following table shows it for some values of $p$:

The loss is smallest when $p = 0.8$, the share of the texts with 9. A model that always predicts 9 has an infinite loss, because it gives the probability 0 to the price of two texts (see Section 1.3).

It turns out that the same holds in general, for a model that can give each sequence of earlier tokens its own probabilities. The smallest loss gives each token its share among the training texts in which it follows the same tokens.

Thus, a text that appears more often gets a larger probability. A page that the web contains 100 times counts 100 times. For this reason, the data is deduplicated, and the people who train a model choose its data mix. With a fixed number of tokens, a larger share of code leaves fewer tokens for other text.

Filters that remove most of the data leave fewer new tokens. The pretraining reads most of its data once or a few times (see Section 4.6). Repeated data is almost as good as new data for up to 4 epochs. More repetitions help less and less⁸.

The pretraining also changes its data in stages. Long texts come in a late stage, for the reason of Section 8.5. For example, in the last stage of Llama 3 405B, the learning rate fell to 0, and the data mix gave more weight to sources of very high quality⁹.

9.3Training cost

We can calculate the cost of the pretraining from the size of the model and the size of the data. A training step does about 6 operations per parameter and token (see Section 3.3). Thus, pretraining a model of $N$ parameters on $D$ tokens takes about $6ND$ operations.

For example, Llama 3 405B has 405 billion parameters and read 15.6 trillion tokens. This gives $6 \cdot (405 \cdot 10^9) \cdot (15.6 \cdot 10^{12}) \approx 3.8 \cdot 10^{25}$ operations, the number in its paper¹⁰.

The operations run on GPUs. One H100 SXM GPU, the kind that trained Llama 3, does at most about 990 trillion operations per second on numbers of 2 bytes¹¹, and the training of Llama 3 405B used 38% to 43% of this¹².

At 40%, one GPU would need about 3,000 years. The run used up to 16,000 GPUs, which need about 70 days.

The GPUs share the work of each step. Each GPU calculates the gradient on its part of the batch, and the GPUs exchange their gradients to get the average (see Section 2.4). The model is also split over the GPUs. Its 405 billion parameters take 810 GB at 2 bytes each, and one H100 has 80 GB of memory¹³.

Llama 3 405B was released as Llama 3.1 405B, and its model card, the page that describes the released model, gives 30.84 million GPU hours for its training¹⁴. Suppose that one GPU hour costs 2 dollars, as the report of DeepSeek-V3 assumes. Then the training costs about 62 million dollars.

In practice, a run on thousands of GPUs often stops because one GPU fails¹⁵. For this reason, the training saves its parameters and the averages of Adam at regular intervals. After a failure, it continues from the last saved copy and repeats the steps since that copy¹⁶.

9.4Scaling laws

A scaling law is a formula that predicts the loss of a model from its number of parameters $N$ and its number of training tokens $D$¹⁷. Its constants are fitted to the final losses of many training runs. For example, a fit to the runs of the Chinchilla paper of 2022 gives the following formula¹⁸: $$L(N, D) = 1.82 + \frac{482}{N^{0.348}} + \frac{2085}{D^{0.366}}$$ The constant 1.82 is the loss that the formula gives for unlimited parameters and tokens. It plays the role of the smallest average loss $\ln 3 / 9$ of Section 9.1. The second term becomes smaller when the model has more parameters. The third term becomes smaller when the model reads more tokens.

For example, for $N$ = 70 billion and $D$ = 1.4 trillion, the two terms are 0.081 and 0.075. The formula then gives 1.976.

Consider a problem where we are given the number of operations $C$ of a training, its budget, and our task is to choose $N$ and $D$ with $6ND = C$ so that the loss is smallest.

A larger model makes the second term smaller. However, it reads fewer tokens, because $D = C/(6N)$. The following picture shows the loss that the formula predicts for $C = 5.76 \cdot 10^{23}$, the budget of a model of 2021 called Gopher¹⁹:

Each mark on the horizontal axis is 10 times the previous one. The loss is smallest at about 72 billion parameters and 1.33 trillion tokens, about 18 tokens per parameter. With 280 billion parameters, the size of Gopher, the budget gives only 343 billion tokens. The loss is then 1.995.

The Chinchilla paper found that the parameters and the tokens should grow in the same proportion, with about 20 tokens per parameter²⁰. It tested this with Chinchilla, a model of 70 billion parameters trained on 1.4 trillion tokens with the budget of Gopher, which has 280 billion parameters and read 300 billion tokens. Chinchilla answered 67.6% of the questions of the public test MMLU correctly (see Section 4.4), and Gopher 60.0%²¹.

In practice, models are often trained on many more tokens per parameter. For example, Llama 3 8B read over 15 trillion tokens, about 1,900 per parameter. It was still improving at the end²².

For $N$ = 8 billion and $D$ = 15 trillion, the formula gives 2.02. The same $7.2 \cdot 10^{23}$ operations give the smallest loss 1.97 with about 81 billion parameters.

However, a model with 10 times more parameters costs about 10 times more for each token that it writes, when it uses all of its parameters for each token. The reason for this is that a prediction does about 2 operations per parameter (see Section 3.3). The training is paid once. The cost of the written tokens grows with every request (see Chapter 10)²³.

With a scaling law, we can choose $N$ and $D$ from small runs before the large run. For example, the fit for Llama 3 suggested 402 billion parameters and 16.55 trillion tokens for $3.8 \cdot 10^{25}$ operations. The model has 405 billion parameters and read 15.6 trillion tokens²⁴.

The final loss of a large run can be predicted from small runs too²⁵.

Note that the constants depend on the data, the tokenizer and the model, so a fit holds only for its own setting. For example, the fit for Llama 3 gives about 41 tokens per parameter at its budget, not 20.

9.5Base models

A base model continues a text, because every training text continues after each of its positions. As an example, consider the prompt The ride of 1 km cost $5. The ride of 2 km cost $. It reads like the start of a list of rides. A base model can continue it with 6. and then with a third ride, because a list goes on after each item.

A base model can also write a text of its training data word for word, more often when the text appears many times in the data (see Section 4.6).

We can give a base model a task by starting a document that does the task. For example, in the above prompt, the first ride shows the task, and the model writes the price of the second. In the same way, a prompt that shows a task with a few examples makes a base model continue with the task (see Chapter 18)²⁶.

The knowledge cutoff of a model is the date of the latest text in its training data. For example, the pretraining data of Llama 3.1 has a cutoff of December 2023²⁷. Its training data has no texts about later events. We can give such facts to the model only in the prompt (see Chapter 19).

The probabilities follow the data, not the truth. In the example of Section 9.2, the model gives the price 10 the probability 0.2. Sampling then writes it in about 2 of 10 runs (see Section 7.4), and the model cannot tell which price is right.

A wrong statement that is frequent on the web gets a high probability in the same way. Chapter 22 shows why a model states false facts with confidence.

Instruction tuning teaches a base model to answer a request instead of continuing it (see Section 11.2). Reinforcement learning from human feedback then teaches it which answers people prefer (see Section 12.2).