Fine-tuning

A pretrained model, also called a base model, is a model that has been trained on a large general dataset. Fine-tuning is training a pretrained model further on a smaller dataset, for one task or for answering in a conversation.

For example, we can fine-tune a language model pretrained on trillions of tokens of web text on 1,000 support tickets and their answers. After this, the model answers new tickets in the same format. A small dataset is enough, because it only has to teach what differs between the tasks.

In practice, fine-tuning is usually used to teach a format, a tone or a narrow task. Facts that change often are instead given in the prompt, the text that the model reads before it answers, because each change would need a new round of training.

11.1Training from pretrained parameters

When we fine-tune a model, we use the steps of gradient descent (see Chapter 2), but we start from the pretrained parameters instead of random values.

As an example, consider a model that predicts $y = a \cdot u + b \cdot v$ from two inputs $u$ and $v$, with the pretrained parameters $a = 2$ and $b = 1$. The new task has one example: the input $u = 1$, $v = 1$ with the answer 4. The loss is $(a + b - 4)^2$, and with a learning rate of 0.25, one step adds $(4 - a - b)/2$ to both $a$ and $b$. The following table shows this step from the pretrained parameters and from zero:

Both models now give the answer 4. They differ in $a - b$, which the example says nothing about. Since the step adds the same amount to $a$ and $b$, the value of $a - b$ does not change. It remains 1 in the fine-tuned model, and it is 0 in the model trained from zero, a value that no data has taught.

The following picture shows the two steps in the plane of the parameters:

Each step moves the parameters in the direction (1, 1), which is perpendicular to the line $a + b = 4$. Thus, each model stops at the point of the line that is nearest to its start.

The step also changes the answer for the input $u = 1$, $v = 0$ from 2 to 2.5. The new example does not contain this input. If an old task needs the answer 2 for this input, the fine-tuned model now fails it.

Catastrophic forgetting is this loss of an old skill. A small learning rate and few steps make it smaller, but they also make the model learn the new task less.

11.2Instruction tuning

A pretrained model continues a text (see Chapter 9). For example, when the text is a question, a pretrained model can continue it with more questions instead of an answer¹.

Instruction tuning is fine-tuning on many requests with good answers, so that the model answers a request instead of continuing it. Each example is usually stored in a chat format, where, in most current models, special tokens mark the start and the end of each turn. For example, a support conversation can be stored as the following text:

<|system|>You are a support assistant.<|end|>
<|user|>How do I change my password?<|end|>
<|assistant|>Open Settings and choose Security.<|end|>

Each special token is one entry of the vocabulary, the list of all tokens that the model knows (see Chapter 5). Some tokenizers, the programs that turn text into tokens, also make it from the same text typed by a user, so the program that builds the prompt has to check user text for these names. The names differ between models, and the above names are only an example.

The training usually counts the loss (see Section 1.3) only on the tokens of the assistant turns². The model learns to write answers, and it also learns to write <|end|> when an answer is complete. The program that runs the model asks it for one token at a time (see Chapter 7), and it stops when the model writes this token.

The system turn holds the instructions of the people who build the product (see Chapter 18).

Instruction tuning needs far fewer examples than the pretraining. For example, the first step of training a model of 2022 used about 13,000 requests with answers that people wrote, and a later step trained the model from scores of its answers instead (see Chapter 12)³. One view is that the examples mostly teach the form of an answer⁴. It turns out that fine-tuning is a poor way to add new facts, because a model learns examples with new facts more slowly than the others⁵.

11.3Low-rank adaptation

A straightforward way to fine-tune a large model is to train all its parameters. However, training with Adam (see Section 2.4) stores a gradient and two running averages for each parameter. Thus, training needs at least four numbers per parameter instead of one. Each task also needs its own copy of the model.

A language model keeps most of its parameters in weight matrices, which multiply the vectors that the model works on (see Chapter 3). Low-rank adaptation (LoRA)⁶ is a technique that keeps these matrices fixed and trains an update $B \cdot A$ for some of them. For a matrix $W$ of size $d \times d$, where $d$ is the length of these vectors, $B$ has size $d \times r$, $A$ has size $r \times d$ and $r$ is small. For example, when $d = 4$ and $r = 1$, the 8 parameters in $B$ and $A$ give the 16 values of the update:

The rank of a matrix is the largest number of its columns of which none is a combination of the others. For example, each column of the above update is a multiple of $B$, so its rank is 1. In general, $B \cdot A$ has rank at most $r$, because each column of $B \cdot A$ is a combination of the $r$ columns of $B$. The example of Section 11.1 is similar: the new data moved the parameters in one direction only.

In a real model with $d = 4096$ and $r = 8$, $B$ and $A$ have 2 · 4096 · 8 = 65,536 parameters instead of the 4096 · 4096 = 16,777,216 of $W$, which is 256 times fewer.

Initially, $B$ is zero, so the model is exactly the pretrained one. Only $B$ and $A$ need a gradient and two running averages, and each task stores only its own $B$ and $A$. After training, we can also add $B \cdot A$ to $W$, and the merged model runs as fast as before.

It turns out that $r = 8$ or less is often enough for a format or a narrow task, but full fine-tuning is much better for training on a large amount of code or math⁷.