What is fine-tuning? How LLM fine-tuning works
Fine-tuning an LLM means training a pretrained model further on your own examples so its weights change. How SFT works, what it costs, and what it forgets.

Fine-tuning an LLM means taking a model that has already been pretrained and training it further, with gradient descent, on a smaller set of your own examples, so its weights change to favor the behavior those examples show. Prompting changes what the model reads; fine-tuning changes the model. It is the right tool for teaching a format, a tone or a narrow skill, and a poor tool for teaching facts.
I ran the snippet below on 2026-10-31 with Python 3.13.11 and no packages.
How does fine-tuning work?
A pretrained model has learned to predict the next token over a huge general corpus. Fine-tuning runs the same training loop again, from those weights, on a dataset you choose. The loss is the same next-token cross-entropy, usually computed only on the response part of each example. The model’s predictions on your data get better because its parameters move toward them.
Three kinds of fine-tuning cover most of what people mean today. OpenAI’s model optimization guide lists them as products, and the same split shows up in open-source tooling:
- Supervised fine-tuning (SFT). You provide prompts and ideal responses. The model learns to imitate them. This is what most people mean by fine-tuning, and OpenAI recommends it for classification, content in a specific format and fixing instruction-following failures.
- Preference fine-tuning. You provide a prompt with a better and a worse response, and the model learns to prefer the better one. DPO is the common method. OpenAI suggests it for tone and style.
- Reinforcement fine-tuning. The model generates answers, a grader scores them, and training reinforces the higher-scoring ones. This is the RLHF family, and with programmatic graders it becomes reinforcement learning with verifiable rewards.
There is a second axis: how many weights you update. Full fine-tuning updates all of them. Parameter-efficient methods freeze the base model and train a small add-on; LoRA and QLoRA are the ones you will meet.
What did fine-tuning do for ChatGPT?
The clearest result is OpenAI’s InstructGPT paper (Ouyang et al., March 2022). They fine-tuned GPT-3 first with supervised demonstrations written by labelers, then with RLHF. In human evaluations on their prompt distribution, outputs from the 1.3B parameter InstructGPT model were preferred to outputs from the 175B GPT-3, a model with 100 times more parameters.
That’s the point of fine-tuning in one sentence. Pretraining gives a model knowledge and capability. Fine-tuning decides how it uses them: follow instructions, answer in this shape, refuse that. A small, well-tuned model can beat a large untuned one on the task it was tuned for.
What does fine-tuning look like in code?
Real fine-tuning needs a GPU and a framework. The mechanism fits in a toy. Below, a next-word model (a table of bigram logits, the simplest language model there is) is “pretrained” on general sentences, then fine-tuned with SFT on support-ticket replies. I measure the loss on both datasets before and after.
# finetune.py: supervised fine-tuning on a toy next-word model, and what it forgets.
# Python 3 stdlib only.
import math
general = ("the cat sat on the mat . the dog sat on the rug . the cat ate the fish . "
"a dog ate a bone . the bird sat on a branch .").split()
support = ("ticket : refund . reply : sorry , refund approved . "
"ticket : login . reply : sorry , reset sent . "
"ticket : refund . reply : sorry , refund approved .").split()
vocab = sorted(set(general + support))
ix = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
pairs = lambda words: [(ix[a], ix[b]) for a, b in zip(words, words[1:])]
def softmax(row):
m = max(row); e = [math.exp(x - m) for x in row]; z = sum(e)
return [x / z for x in e]
def loss(W, data): # mean cross-entropy, nats per token
return sum(-math.log(softmax(W[a])[b]) for a, b in data) / len(data)
def train(W, data, steps, lr):
for _ in range(steps):
grad = [[0.0] * V for _ in range(V)]
for a, b in data:
p = softmax(W[a])
for j in range(V):
grad[a][j] += (p[j] - (j == b)) / len(data)
W = [[w - lr * g for w, g in zip(wr, gr)] for wr, gr in zip(W, grad)]
return W
G, S = pairs(general), pairs(support)
base = train([[0.0] * V for _ in range(V)], G, 300, 5.0) # "pretraining"
print(f"{'model':<34}{'general loss':>13}{'support loss':>14}")
print(f"{'base (pretrained only)':<34}{loss(base, G):>13.3f}{loss(base, S):>14.3f}")
for name, data, steps, lr in [
("SFT on support data, lr 5.0", S, 300, 5.0),
("SFT on support data, lr 0.5", S, 300, 0.5),
("SFT on support + general mix, 5.0", S + G, 300, 5.0),
]:
W = train([row[:] for row in base], data, steps, lr)
print(f"{name:<34}{loss(W, G):>13.3f}{loss(W, S):>14.3f}")
W = train([row[:] for row in base], S, 300, 5.0)
top = lambda M, w: vocab[max(range(V), key=lambda j: M[ix[w]][j])]
print("\nnext word after '.' base:", top(base, "."), " after SFT:", top(W, "."))
model general loss support loss
base (pretrained only) 0.762 3.877
SFT on support data, lr 5.0 1.523 0.452
SFT on support data, lr 0.5 1.140 0.614
SFT on support + general mix, 5.0 0.859 0.555
next word after '.' base: the after SFT: reply
Read the table row by row. Fine-tuning drops the loss on support replies from 3.877 to 0.452 nats per token: the model now writes in the new format. It also doubles the loss on general text, from 0.762 to 1.523. After a full stop, the base model expected “the”; the tuned one expects “reply”. Nothing told it to forget. The weights that served the old data are the same weights that moved.
That’s catastrophic forgetting, at toy scale. The two mitigations in the table are the ones used on real models: a gentler learning rate forgets less but learns less, and mixing some general data back into the fine-tuning set keeps most of both. A frozen base with a small adapter is the third, which is why LoRA exists.
How much data and compute does fine-tuning need?
Less data than people expect, more memory than they expect.
On data: OpenAI’s supervised fine-tuning guide sets a minimum of 10 examples and recommends “starting with 50 well-crafted demonstrations and evaluating the results.” The QLoRA paper found that “data quality is far more important than dataset size”: a 9k sample dataset (OASST1) beat a 450k sample one (a FLAN v2 subsample) on chatbot performance. Fifty clean examples of exactly the output you want beat ten thousand scraped ones.
On memory: full fine-tuning with mixed-precision Adam needs about 16 bytes per parameter before activations, according to the ZeRO paper: 2 for fp16 weights, 2 for gradients, 12 for the fp32 master copy and Adam’s two moments. For a 7B model that’s roughly 112 GB of model state, more than one 80 GB GPU holds. The QLoRA paper puts regular 16-bit fine-tuning of a 65B model at more than 780 GB of GPU memory.
That gap is why parameter-efficient fine-tuning won for most teams. The LoRA paper cut trainable parameters for GPT-3 175B by 10,000 times and GPU memory by 3 times, and QLoRA fit a 65B fine-tune on one 48 GB GPU. I cover both in LoRA vs QLoRA.
When should you fine-tune instead of prompting or RAG?
Use this order, cheapest first:
- Prompt. Clear instructions and a few examples in the prompt. OpenAI’s own guide says prompt engineering “may be all you need” and asks you to write evals before anything else.
- Retrieve. If the model lacks facts, put the facts in the context. That’s RAG, and it gives you citations and freshness that weights can’t.
- Fine-tune. When the model has the knowledge but won’t reliably produce the behavior: a strict output schema, a house tone, a classification label set, a short prompt that has to replace a long one, or a small model that has to match a big one on one task.
The last case deserves its own name. Training a small model on a large model’s outputs is distillation, and it’s most of what production fine-tuning looks like in 2026: you pay once for training so you can pay less for every request afterward.
The comparison I’d write on a whiteboard: fine-tuning changes behavior, RAG changes knowledge. I go through the trade-offs in RAG vs fine-tuning.
Where does fine-tuning break?
It forgets. The toy above shows it. At real scale, Biderman et al. (LoRA Learns Less and Forgets Less, 2024) measured full fine-tuning and LoRA on code and math: LoRA underperformed full fine-tuning on the target task, and it also preserved the base model’s performance on other tasks better. You choose where on that curve to sit.
It teaches facts poorly. A fine-tune can make a model sound like it knows your product catalog without making it right about it. Facts that change belong in retrieval.
It ties you to a base model. A fine-tune is a delta on one specific checkpoint. When the base model is deprecated, your fine-tune goes with it. OpenAI’s deprecations page says fine-tuned models stay available for inference “until the base models are deprecated.” The same page records that on May 7, 2026, OpenAI started winding down self-serve fine-tuning: no new organizations, and active customers can’t create new jobs after January 6, 2027. If your fine-tune matters to the business, keep the training data and the eval set so you can rebuild it on the next model, hosted or open-weight.
It hides regressions. A fine-tuned model can get better on your examples and worse on safety behavior or edge cases you didn’t include. You only see that with an eval set that covers more than the training distribution.
From the data-platform and privacy seat I sat in at Walmart, the line I’d circle is that training data becomes part of the model. Anything personal in the fine-tuning set can come back out in a completion, and you can’t delete one record from weights the way you delete a row. Scrub the data before training, not after.
What I would do
- Write the eval first: 50 to 200 real inputs with expected outputs, scored automatically where possible.
- Push prompting and retrieval as far as they go and record the scores.
- If a gap remains in behavior or format, build 50 to 500 hand-checked SFT examples in the exact chat format the model will see in production.
- Start with LoRA on an open-weight model, or a hosted SFT job, and compare against the prompted baseline on the same eval, plus a slice of general tasks to catch forgetting.
- Keep the dataset, the config and the eval in version control. The model is a build artifact; the data is the source.
FAQ
What is the difference between pretraining and fine-tuning?
Pretraining learns from a huge general corpus starting from random weights. Fine-tuning starts from the pretrained weights and trains on a small, targeted dataset to change behavior. The loss and optimizer are usually the same; the data and scale differ.
How many examples do I need to fine-tune an LLM?
OpenAI’s SFT minimum is 10 and its suggested start is 50 well-crafted examples. Quality matters more than count: the QLoRA authors found a 9k sample dataset beat a 450k sample one.
Is fine-tuning better than RAG?
They solve different problems. Fine-tuning changes behavior, format and style; RAG supplies facts at request time with citations. Many systems use both.
Can I fine-tune an LLM on a single GPU?
Yes, with parameter-efficient methods. QLoRA fine-tuned a 65B model on one 48 GB GPU. Full fine-tuning at 16 bytes per parameter rules out most models on one card.
What is catastrophic forgetting?
When training on new data degrades what a model could do before, because the same weights serve both. Lower learning rates, mixing in general data and adapter methods like LoRA reduce it.
Sources
- Ouyang et al., Training language models to follow instructions with human feedback, March 4, 2022
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, June 17, 2021
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, May 23, 2023 (data quality vs size in section 1)
- Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, October 4, 2019 (16 bytes per parameter in section 3.1)
- Biderman et al., LoRA Learns Less and Forgets Less, May 15, 2024
- OpenAI, Model optimization guide (read 2026-10-31)
- OpenAI, Supervised fine-tuning guide (read 2026-10-31)
- OpenAI, Deprecations, “Update to OpenAI’s self-serve fine-tuning”, May 7, 2026 (read 2026-10-31)