Models and loss

A model is a function whose output depends on numbers called parameters. For example, the model $y = a \cdot x + b$ predicts the price $y$ of a taxi ride from its distance $x$ in kilometers, and its parameters are $a$ and $b$. When $a = 2$ and $b = 3$, a ride of 4 km gets the prediction $2 \cdot 4 + 3 = 11$ dollars.

The people who design a model choose the form of the function, and the form does not change when the model learns. The training chooses the parameters, using training data: inputs whose correct outputs we know. For example, a list of past rides with their prices is training data for the taxi model.

A language model works in the same way. Its input is a text, its output is a probability for each possible next token, a piece of a word (see Chapter 5), and it has billions of parameters.

1.1Loss

The loss is a number that measures how wrong the predictions of a model are on the training data. A smaller loss means better predictions, and the training looks for the parameters with the smallest loss.

As an example, consider the following three rides and the predictions of the model when $a = 2$ and $b = 3$:

The mean squared error is the average of the squared errors. In this case, it is $(0 + 1 + 0)/3 = 1/3$. The following picture shows the rides and the predictions:

The error of a ride is the vertical distance between its point and the line. For example, the parameters $a = 1$ and $b = 4$ predict 5, 6 and 7. Their loss is $(0 + 0 + 4)/3 = 4/3$. This choice is worse.

Note that squaring makes large errors count more: an error of 2 counts 4 times as much as an error of 1. For this reason, the best parameters spread the error over all rides.

The parameters $a = 2$ and $b = 3$ predict two rides exactly, but they are not the best choice. The best choice is $a = 2$ and $b = 8/3$¹. Its loss is $2/9 \approx 0.22$.

1.2Choosing the parameters

A straightforward way to choose the parameters is to try many values and keep the pair with the smallest loss. For example, we can try the values 0, 0.5, 1, …, 5 for both $a$ and $b$, calculate the loss of each of the $11 \cdot 11 = 121$ pairs, and keep the best pair, which is $a = 2$ and $b = 2.5$ with the loss 0.25.

However, a model with $k$ parameters needs $11^k$ tries. Even $k = 10$ gives 25,937,424,601 tries, and a language model has billions of parameters. Chapter 2 shows a method that changes all parameters at the same time, in the direction that makes the loss smaller.

1.3Loss for probabilities

Many models output a probability instead of a number. For example, a spam filter gives the probability that an email is spam. The cross-entropy loss of one example is $-\ln p$, where $p$ is the probability that the model gives to the correct answer. The following table shows the loss for some values of $p$:

The loss is 0 only when the model gives the probability 1 to the correct answer. A wrong answer with high confidence costs much more than an unsure one. For example, the loss for $p = 0.01$ is about 44 times the loss for $p = 0.9$.

A language model uses the same loss. For each position of a text, the loss is $-\ln p$, where $p$ is the probability that the model gives to the token that actually comes next. The pretraining (see Chapter 9) makes the average of this loss small over trillions of tokens.

Note that a small loss on the training data does not mean good predictions for new inputs. A model can learn the training data by heart, and Chapter 4 shows how we can detect this.