Tokenisation

The job of a tokeniser is to break down a prompt into tokens. Then it uses a scheme to map each token to a unique number, called a token id, which the LLM can process. When the LLM then generates token ids in response to our prompt, we can use this number mapping to convert those token ids to tokens, and then to generated text.

To show how this works we’ll start with a simplified example, then we’ll move on to how real LLM tokenisers work.

Word-based tokeniser

One very straightforward way to split a prompt into tokens is to split it into words and punctuation characters:

import re

def tokenise(text):
    # \w+ matches a run of word characters,
    # [^\w\s] matches a single punctuation character
    return re.findall(r"\w+|[^\w\s]", text)


tokenise("The dog fetched the ball!")
# ['The', 'dog', 'fetched', 'the', 'ball', '!']

In order to map these to token ids, before we start LLM training, we take a large corpus of text. This could be the corpus we intend to use for LLM training. Then we split this corpus using our tokeniser, and start assigning token ids to each word we encounter:

If we encounter the same word twice, we ignore it:

This way, we map every word in our text corpus to a unique token id.

Now, when we tokenise a prompt, we have a way to map it to token ids:

When the LLM produces output token ids, we can now construct text from them using the reverse of this mapping:

This simple tokenisation scheme has some limitations. Firstly, if we encounter a word in our prompt that was not in our training corpus, we will not be able to map it to a token. Think, for example, of misspellings, or names of cities in fantasy books. With this tokeniser, our LLM will not be able to process these words.

Secondly, the vocabulary has no idea that some of its words are obviously related. walk, walks, walked and walking each map to a separate token id, and nothing about those four ids tells the model that they share a root — it has to learn that connection from scratch, four times over. Multiply that across every inflection, plural, compound and capitalisation in a language and the vocabulary becomes both enormous and wasteful.

How LLMs actually do it

Real LLMs, instead of breaking text into words, break text into sequences of characters that commonly occur together. This will often end up being whole words (as words are sequences of characters that commonly occur together!).

There are various schemes for doing this, including Byte Pair Encoding (BPE), WordPiece, and Unigram. All of these break down text into words and sub-word parts, for example

Importantly, tokenisers implementing these schemes are able to uniquely tokenise any prompt they encounter. In a worst-case scenario, they can just break down a word into its constituent characters.¹

Once we have our tokenisation scheme, everything else works the same as in our simple word-based tokeniser.

Here’s some code showing how the tiktoken library can be used to encode and decode text as tokens:

import tiktoken

encoding = tiktoken.get_encoding("cl100k_base")

# text -> token ids
token_ids = encoding.encode("The dog fetched the ball!")
# [791, 5679, 42542, 279, 5041, 0]

# inspect the token behind each token id
tokens = [encoding.decode([token_id]) for token_id in token_ids]
# ['The', ' dog', ' fetched', ' the', ' ball', '!']

# token ids -> text
encoding.decode(token_ids)
# 'The dog fetched the ball!'