Tokens

A token is a piece of text that a language model reads or writes as one unit. A tokenizer is a program that turns a text into tokens and each token into a number, and the model works only on these numbers. Frequent words are usually one token, and rare words are split into several pieces. For this reason, a model does not see the letters of a word, and it can miscount them.

5.1Token ids

The vocabulary is the list of all tokens that a tokenizer knows, and the token id of a token is its number in this list. For example, a tokenizer with a vocabulary of 200,000 tokens¹ turns the text The ride of 2.5 km cost 8 dollars. into the following tokens, where ␣ is a space and the number under each token is its id:

A space usually joins the word after it, so ␣ride is one token. Before a number, the space stays a token of its own. The number 2.5 becomes the three tokens 2, . and 5, and both dots get the id 13.

The model never sees the text, only the ids 976, 15104, …, 13. Its first step turns each id into a vector of numbers, the embedding of the token (see Chapter 6). The model also writes its answer as token ids, and the tokenizer turns them back into text.

A special token is a token that marks a place in a text instead of standing for its characters. For example, the vocabulary of a chat model holds special tokens that mark the start and the end of each turn in a conversation (see Section 11.2).

In practice, an English text has about 4 characters per token². The above text has fewer, 34 characters in 13 tokens, because 7 of its tokens are single characters: digits, dots and spaces. The price of a request is counted in tokens (see Chapter 10), and so is the longest text that a model can read at once (see Chapter 8). The same text in another language can need many more tokens, up to 15 times as many in some cases³.

5.2Choosing the tokens

A straightforward way to choose the vocabulary is to make each character a token. The vocabulary is then small for a text in one alphabet, but texts become long: the above text has 34 tokens instead of 13. A model writes one token at a time (see Chapter 7), so a longer text takes more time and costs more.

Another way is to make each word a token, as the word vectors of Section 6.1 do. Texts are then short, but the vocabulary needs every word, including names, misspellings and the words of other languages. A word that is not in the vocabulary cannot be written at all. For example, a vocabulary that has taxi but not taxis has no tokens for taxis.

The idea is to use pieces of words: frequent words get their own tokens, and rare words are written with several pieces.

5.3Byte-pair encoding

Byte-pair encoding is a method that chooses such pieces of words from a large sample of text⁴, and the tokenizer that it builds is made once, before the model is trained.

Initially, the vocabulary contains the single characters of the sample, and each word is a list of these characters. After this, the method counts how often each pair of neighboring tokens appears in the sample, adds the most frequent pair to the vocabulary as a new token, and joins every copy of the pair into this token. It repeats this step until the vocabulary has the wanted size.

As an example, consider a sample where the word taxi appears 4 times, tax 3 times, taxes 2 times and max 2 times. The following table shows the first three steps:

In step 1, the pair a x appears 4 + 3 + 2 + 2 = 11 times, because every word contains it. The pair t a appears only 9 times, because max does not contain it.

After three steps, the vocabulary has 10 tokens: the 7 characters a, e, i, m, s, t and x, and the new tokens ax, tax and taxi. We number them from 0 in this order, so taxi has the id 9. Special tokens are added to the vocabulary apart from these steps.

To tokenize a new text, the tokenizer writes each word as characters and joins the pairs in the same order as the steps. In this way, we can also tokenize a word that is not in the sample. For example, the word taxis is tokenized as follows, where each row joins one more pair and the new token is shaded:

Thus, taxis becomes the tokens taxi and s, with the ids 9 and 4.

Each step counts the pairs in the whole sample, so a step takes $O(n)$ time for a sample of $n$ characters.

Many real tokenizers start from the 256 values of a byte instead of characters⁵. Others start from characters and fall back to bytes.

In both cases, a character that the sample never contained is written with its bytes. Thus, every text can be tokenized, but a rare character can take several tokens. For example, the taxi emoji 🚕 takes 4 bytes, and the tokenizer of Section 5.1 writes it as 2 tokens.

More steps make the vocabulary larger and texts like the sample shorter. On the other hand, each token needs its own embedding. For example, a vocabulary of 128,256 tokens with embeddings of 4096 numbers needs 128,256 · 4096 = 525,336,576 numbers⁶.

The pieces follow the sample. A language that is rare in the sample gets few long pieces, and its texts need more tokens (see Section 5.1).

5.4Counting letters

A model sees token ids, not letters. For example, with the above vocabulary, taxi is the token 9, and a model that reads the id 9 has no letters to look at. To answer how many times x appears in taxi, the model must have learned the spelling of token 9 from its training text. For example, it can learn it from texts that spell the word out.

The same holds for real tokenizers. In the question How many r's are in the word strawberry?, the tokenizer of the first example makes ␣strawberry one token, and many models could not count the r’s of this word⁷.

The same word can also be written with different tokens in different forms. The following table shows three forms of taxi in the same tokenizer:

Nothing in the ids says that these three lists are the same word. The model can only learn it from its training text. Numbers are split in a similar way: the tokenizer turns ␣123456 into ␣, 123 and 456, and ␣2024 into ␣, 202 and 4.

It turns out that tokens are not the only difficulty. In the same study, models found which letters a word contains but often miscounted the letters that repeat. Most models failed on words where a letter appears more than twice, like the r in strawberry.

We can also write the word with spaces, as s t r a w b e r r y. Then each letter is its own token, and the three r’s are three copies of one token.