Instruction fine-tuning

At the end of pre-training, a language model is good at text completion. Given a starting phrase, it can complete a paragraph in fluent English. However, it will perform badly when used as a chatbot, as it’s not yet learned how to answer questions or fulfil requests. For example, if I were to ask a newly pre-trained LLM

“What should I have for dinner tonight?”

it might respond

“What should I have for dinner tonight?” is a question many of us ask ourselves at the end of a long day.

Instruction fine-tuning gives the pre-trained model further training on examples of instructions paired with good responses. From a technical point of view this is straightforward; we just take the newly pre-trained model and feed it a large number of examples, and allow the weights to update using the same mechanism that we used in pre-training.

Building the examples

The main step is to build a training dataset of examples for the LLM to learn from. These examples can come from online sources that happen to be in roughly the correct format, and can be adapted from existing data (perhaps using another LLM). Somewhat surprisingly, it’s been shown that you can use other LLMs to create and filter instruction response examples from scratch—so-called synthetic instruction tuning—and this is very effective.

Here is what some of these examples might look like:

A useful dataset varies the wording, subject, difficulty, and desired response style. It may also contain multi-turn conversations, where each response needs to take account of earlier messages.

Training

The learning mechanism is closely related to pre-training. The model reads the instruction and the response so far, predicts the next token of the good response, and is penalised when it gives the correct token a low probability. Backpropagation and optimisation then make small adjustments to the model’s weights.

The main difference is that we only get the model to predict the next tokens in the response, given the instruction:

The reason for this is that we want the model to learn what to say after an instruction, rather than to learn to reproduce the user’s words.