This site gives a behind the scenes look at how LLMs work under the hood.
This page gives a high level overview of all of the core concepts that underpin the tech behind LLMs. Each section has one or more deep dives, where you’ll piece together the complete code for a fully functional LLM.
No specialist knowledge is required to read this overview page, but to get the most out of the deep dive pages some familiarity with python and neural networks is recommended.
Here are a couple of excellent resources for this:
If you want to skip straight to the fully working code you can find that here.
When we provide a prompt to an LLM, the first thing that happens is that the prompt is broken down into tokens.
Much of the time, this just means breaking down the prompt into individual words and punctuation characters, but sometimes, particularly for more obscure words, longer words, or proper nouns, they are broken down into sub-word chunks. This is mainly to allow the LLM to better handle unknown or rare words, as well keeping the vocabulary size (i.e. the number of tokens the LLM needs to know) down to a manageable size.
Once a prompt has been converted to tokens, each token is then represented as a point in a high-dimensional geometrical space. This is because LLMs are neural networks, and neural networks need mathematical objects to work with instead of text data.
This process is called token embedding.
In theory, you could just assign any old point in space randomly to each token, and it would be a valid token embedding. But the interesting thing about LLM token embeddings, is that as a result of the LLM training, words with similar meanings end up with nearby to each other in space. So, for example, king, queen, and prince would end up next to each other, and cat, dog, and rabbit would end up close to one another, but the two groups of points would be far apart.
Try it out below. Here we embed tokens in 3D space to illustrate the concept. Real LLMs use much higher dimensional spaces, but the principle is the same.
Once a prompt has been tokenised and embedded, the next key step is self-attention. This is the mechanism that allows each token to look at all the other tokens in the sequence and decide which ones are most relevant to it.
The point of this is that, without self attention, a neural network struggles to relate tokens that are far away from each other, but important to add context. Imagine the example:
The book that I borrowed from the library last week, despite its damaged cover and missing index, was surprisingly useful.
The words book and useful are far away, but they need to be considered together to understand the core meaning of the sentence:
The book was useful
Self-attention is a way to model this within the neural network; as the model undergoes training, we allow token embeddings to be enriched by other token embeddings which are important to understanding the given token. We do this using a self-attention module
A self-attention module has many attention heads running in parallel, each learning to track a different kind of relationship — some focus on grammar, some on meaning, some on position.
Switch between the attention heads below to see how each one tracks a different kind of relationship. You can also click a word to select it and see how the amber highlight — showing what it attends to — shifts.
After the self-attention module comes the feed-forward layer. Once self attention has enriched each token with information from other tokens in the input, the feed-forward layer then re-interprets each token in terms of learned features, some of which may correspond to higher level concepts or abstractions.
Which features get picked out for a given token is decided by an activation function. It acts like a switch, turning each hidden feature on or off (or somewhere in between) depending on the input — which is what lets different tokens light up different features in the diagram below.
Pick an input token below to see which learned feature it triggers. Notice that different tokens light up different concept units — and that tokens sharing a concept (like cat and dog) activate the very same hidden unit.
Feed-forward layer: in depth →
Self-attention and the feed-forward layer are the two main processing stages inside a transformer block. Self-attention lets each token gather useful information from the other tokens, then the feed-forward layer processes what it has learned.
The calculations inside a neural network can make some values much larger or smaller than others. Before each stage, LayerNorm brings those values back to a consistent scale, giving the stage a steady input to work with.
After each stage, a residual connection carries the stage's input around the side and adds it back to the result. This preserves useful existing information while the stage adds what it has learned. During training, the same shortcuts also give the learning signal (called the gradient) a clearer route back through the model, helping even the earliest blocks learn effectively.
A full LLM is built by stacking many transformer blocks. The token embeddings enter the first block, and the output from each block becomes the input to the next.
Each block progressively refines the token representations, building on the work done by all the blocks before it. Once the final block has finished, its result passes to the output head, which produces the model's output — in this case, the next token.
Throughout an LLM, there are weights: in the matrices used for embeddings, the linear projections central to self-attention, the linear layers in the feed-forward network, and so on. Each of these components can contain thousands or millions of weights. Together, these weights allow the LLM to learn patterns from its training text and generate meaningful answers to prompts. Some frontier models contain hundreds of billions of weights.
These weights are learned by training the model on a very large corpus of text: public web pages, code repositories, and many other sources. We’ll explore how this works later. Training an LLM this way can cost millions of dollars in hardware and electricity; for some frontier models, the figure can reach hundreds of millions.
However, some companies and research teams make their model weights available for others to use. They publish files containing the trained weights of their LLMs, which other companies or researchers can load into their own instances of the model. Sites like Hugging Face host these files.
We talked above about how models have millions or even billions of weights. The LLM we finally managed to build in The Full LLM section had random weights, and if you go ahead and generate text with it, you will get back random gibberish:
the cat sat on theromancersurface crimesAud dissectiffs abol
We then showed how to load weights for a very small but trained model in the Open Weights section, which produces more coherent (if a little uninspired!) results:
the cat sat on the floor, and the cat was sitting on the floor
To get from our random weights to weights that will produce coherent sentences, we use a process called pre-training.
This involves the following steps:
Our model now produces next-token probabilities that tell us how well it would have predicted the correct sentence from the incomplete sentence:
| Input | Observed probability |
|---|---|
| The cat sat on the | The probability of “mat” is 0.02 |
In fact, the architecture of our model means that in a single pass it can actually simultaneously calculate next token probabilities for all incomplete versions of our sentence:
| Input | Observed probability |
|---|---|
| The | The probability of “cat” is 0.004 |
| The cat | The probability of “sat” is 0.03 |
| The cat sat | The probability of “on” is 0.01 |
| The cat sat on | The probability of “the” is 0.06 |
| The cat sat on the | The probability of “mat” is 0.02 |
After pre-training, the model has learned broad patterns in language, along with a great deal of knowledge encoded in its weights. It can complete passages, generate fluent text, answer some questions, and perform other tasks when prompted in the right way.
However, it is still fundamentally trained to predict the next token. It has not been specifically trained to interpret a user’s message as an instruction, follow a conversation, or consistently provide answers that are helpful, safe and relevant. This makes the pre-trained model a powerful foundation, but not yet the kind of chatbot people commonly associate with an LLM.
Post-training builds on this foundation using techniques such as instruction fine-tuning on examples of prompts and good responses, preference optimisation using human or model feedback, and additional training focused on safety, reliability, tool use, or particular domains.
Instruction fine-tuning: in depth →
We’ve now learned all of the core concepts of an LLM, and have built two complete and fully functional examples along the way:
Production LLMs use all of the same core concepts as our example models. The main differences you will see in a modern production LLM are: