Neural networks

A neural network is a model built from layers, where each layer turns a vector of numbers into another vector, and the output of one layer is the input of the next. For example, a language model is a neural network with dozens of blocks, each with several layers of the kind below¹, and its parameters are the numbers inside these layers.

The taxi model of Chapter 1 is the smallest case: one layer with one input, one output and the parameters $a$ and $b$. With more layers and an activation function after each, a model can follow data that no straight line fits, and backpropagation calculates the derivatives that gradient descent needs (see Chapter 2).

3.1Layers

The dot product of two vectors of the same length is the sum of the products of their numbers. For example, the dot product of (1, 2) and (3, 4) is $1 \cdot 3 + 2 \cdot 4 = 11$.

A layer has a weight matrix $W$, a bias vector $b$ and an activation function $f$. The layer multiplies its input by $W$, adds $b$ and applies $f$ to each number of the result: $$\text{output} = f(W \cdot \text{input} + b)$$ where $W \cdot \text{input}$ is the vector of the dot products of the rows of $W$ with the input. The taxi model is a layer of this kind with the weight $a$, the bias $b$ and no activation function.

A simple and common activation function is the ReLU, which keeps a positive number and turns a negative number into 0: $$\text{relu}(z) = \max(0, z)$$ For example, $\text{relu}(3) = 3$ and $\text{relu}(-2) = 0$. Many language models use smooth variants of it, such as GELU and SiLU².

A layer that turns a vector of length $d$ into another vector of length $d$ has a $d \times d$ weight matrix and $d$ biases. For example, when $d = 4096$, the weight matrix has 4096 · 4096 = 16,777,216 weights. Section 11.3 adapts matrices of this size.

3.2What layers can draw

Consider rides where each kilometer costs 2 dollars, but a ride costs at most 10 dollars, as with a fixed airport price. The training data has the following rides:

A straight line cannot follow the cap. The best line is $y = x + 3$, and its mean squared error is $(1 + 1 + 1 + 1)/4 = 1$.

However, a network with two layers fits the rides exactly. The idea is to let the first layer calculate two numbers $h_1$ and $h_2$, and to let the second layer combine them: $$h_1 = \text{relu}(x) \qquad h_2 = \text{relu}(x - 5) \qquad y = 2 h_1 - 2 h_2$$ The following picture shows the network, with the weight on each edge, and the bias and the activation function next to each node:

The first layer is a hidden layer, i.e., a layer between the input and the output. The following table shows the numbers of the network for the four rides:

The following picture shows the rides, the best line and the network:

The network draws a line that bends at $x = 5$, where $h_2$ starts to grow, and its mean squared error is 0.

Note that the activation function is what bends the line. Without it, the network gives $2x - 2(x - 5) = 10$ for every ride, which is also a straight line.

Thus, two layers with no activation function are no better than one. With $W_1$ and $b_1$ in the first layer and $W_2$ and $b_2$ in the second, $W_2(W_1 x + b_1) + b_2 = (W_2 W_1) x + (W_2 b_1 + b_2)$, which is one layer with the weight matrix $W_2 W_1$ and the bias $W_2 b_1 + b_2$. In the example, $W_2 W_1 = 0$ and the bias is 10.

It turns out that one hidden layer with enough numbers can approximate any continuous function on a closed and bounded range of inputs, to any accuracy³. In practice, networks use many layers instead of one very wide layer, because one layer may need too many numbers, and a deeper network can need fewer numbers and make fewer errors on new inputs⁴.

3.3Backpropagation

Gradient descent needs the derivative of the loss with respect to every parameter. In a straightforward way, we can get an approximation of one derivative by changing one parameter a little and calculating the loss again. However, this needs one more pass through the whole network for each parameter, which is billions of passes for one step of a language model.

Backpropagation⁵ calculates all derivatives in one pass from the output back to the input. The idea is that the derivative of the loss with respect to a number is the sum, over the numbers it feeds, of the derivative with respect to each of them times how fast that number changes with it. Going backward, each derivative uses one that is already calculated.

As an example, suppose that a new ride of 7 km cost 11 dollars. The network gives $h_1 = 7$, $h_2 = 2$ and $y = 14 - 4 = 10$, so the loss is $(10 - 11)^2 = 1$. The following table shows the derivatives of the loss in the order in which backpropagation calculates them:

Each row after the first uses a row above it. For example, the derivative for $h_1$ is the derivative for $y$ times the weight 2 between $h_1$ and $y$. Both $h_1$ and $h_2$ are positive, so the ReLU passes their derivatives unchanged to its input, and each weight of the first layer gets that derivative times $x$. The derivatives of the biases can be calculated in the same way: the bias of $y$ gets −2, the bias of $h_1$ gets −4 and the bias of $h_2$ gets 4.

A training step goes through the network once forward and once backward. The backward pass does about twice the arithmetic of the forward pass, because each layer needs derivatives for both its input and its weights. Thus, a training step does about the arithmetic of three predictions, whatever the number of parameters⁶.

3.4Starting values

Training starts from small random weights. The reason is that two numbers of a hidden layer with the same weights and bias into them and the same weights out of them get the same derivatives. Gradient descent then keeps them equal forever, and the layer acts as if it had one number fewer. For example, if $h_1$ and $h_2$ both started as $\text{relu}(x)$ with the weight 1 in $y$, every step would change both in the same way.

Since the start is random, two training runs of the same network end with different parameters. The loss of a network has many points where the gradient is 0, and training moves toward one of them. Large models are stopped after a fixed number of steps, well before the gradient is 0⁷.