Embeddings

An embedding is a vector of numbers that represents a text, where texts with similar meanings get vectors that point in similar directions. For example, the embeddings of password and passcode point in almost the same direction, and the embedding of invoice points in another direction.

We scale the embeddings to length 1. The similarity of two embeddings is their dot product, i.e., the cosine of the angle between them. It is 1 for embeddings with the same direction and 0 for perpendicular embeddings.

Embeddings are used in two ways. First, we can search for related texts by comparing their embeddings, as in Chapter 19. Second, inside a language model, the first step turns each token into an embedding, and the rest of the model works on these vectors.

6.1Word vectors

A straightforward way to turn words into vectors is to number the words of the vocabulary and to give each word a vector with a 1 at its own position and 0 elsewhere. For example, a vocabulary of four words gives the following vectors:

However, the similarity of any two different words is 0. The vectors do not show that password and passcode mean almost the same thing. In addition, each vector has as many numbers as the vocabulary has words, for example 50,000.

The idea is to give each word a short vector of $d$ numbers and to learn the numbers from a large text¹. The training adjusts the vectors so that each word can predict the words around it. The training never sees what a word means. Since password and passcode appear next to the same words, for example reset and forgot, the training gives them similar vectors.

Suppose that $d = 2$ and that the training gives the following vectors:

For example, the similarity of password and passcode is $1 \cdot 0.96 + 0 \cdot 0.28 = 0.96$, and the similarity of password and invoice is 0. The following picture shows the vectors:

The words about logging in form one group, and the words about money form another group.

It turns out that some directions carry meaning. For example, in vectors learned from news text, the vector king − man + woman is nearest to the vector of queen. Note that the search must skip the three words themselves, because one of them is often nearer than queen.

6.2Text embeddings

A straightforward way to get the embedding of a text is to take the average of the vectors of its words. However, the average does not depend on the order of the words, so the dog bit the man and the man bit the dog get the same embedding.

An embedding model is a language model that reads a whole text and outputs one vector. Chapter 8 shows how such a model reads the words in their order. The training uses pairs of texts that belong together, for example a question and its answer. In each round, the training takes $m$ pairs and changes the model so that each question is more similar to its own answer than to the other $m - 1$ answers. For example, when $m = 3$, the similarities after the training can be as follows:

In each row, the largest value is on the diagonal. Thus, each question is nearest to its own answer.

Note that a question and its answer often share few words. For this reason, a search with embeddings can find an answer that does not repeat the words of the question.

6.3Storing embeddings

Each number of an embedding usually takes 4 bytes. For example, 1,000,000 texts with $d = 1536$² need 1,000,000 · 1536 · 4 = 6,144,000,000 bytes, which is about 6 GB. In practice, the numbers can be stored in 1 byte each. This makes the storage 4 times smaller, but it rounds each number to one of 256 values.

Embeddings from two different embedding models cannot be compared, because each model has its own directions. When we change the embedding model, we have to calculate the embeddings of all texts again.