What is model distillation? Knowledge distillation for LLMs
Model distillation trains a small student model to copy a large teacher's outputs, keeping much of its skill at a fraction of the cost. How it works, in Python.

Model distillation (knowledge distillation) is a training method where a small “student” model learns to imitate a large “teacher” model, either by matching the teacher’s full probability distribution over outputs or by training on text the teacher generated. The student ends up far cheaper to run while keeping much of the teacher’s ability, which is how many small open-weight LLMs, including the DeepSeek-R1-Distill models, were made.
I ran the demo below on 2026-10-31 with Python 3.13.11 and no packages.
Where does knowledge distillation come from?
The term comes from Hinton, Vinyals and Dean (2015), “Distilling the Knowledge in a Neural Network”. Their starting point was ensembles: averaging many models works well but is “cumbersome” and “too computationally expensive to allow deployment to a large number of users”. They showed you can compress that knowledge into a single smaller model by training it on the big model’s output probabilities, which they call “soft targets”, instead of on the one-hot labels.
The reason it works is in the small numbers. Their example: for an image of a 2, a good model might give “a probability of 10^-6 of being a 3 and 10^-9 of being a 7”, and for a different 2 the other way around. Those tiny probabilities say which 2s look like 3s and which look like 7s. A one-hot label throws all of that away.
The MNIST result is the one I remember. A large regularized net made 67 test errors. A smaller net trained normally made 146. The same small net trained to also match the large net’s soft targets at temperature 20 made 74, close to the teacher.
How does distillation work?
Two ideas from the paper do most of the work.
Temperature. Divide the logits by T before the softmax. At T = 1 a confident teacher puts nearly all probability on one answer, so the small probabilities barely move the loss. Raising T flattens the distribution and makes them visible. The student trains at the same high T and is then run at T = 1.
The T² correction. Gradients from soft targets “scale as 1/T²”, so the paper multiplies the soft-target loss by T² when combining it with the normal hard-label loss. That keeps their relative weight stable when you change T.
For LLMs, the same idea runs per token. The teacher outputs a probability for every token in its vocabulary at every position; the student learns to match that whole distribution instead of only the single token in the training text. Google’s Gemma 2 report states it as minimizing -Σ P_T(x | x_c) log P_S(x | x_c) over the vocabulary, where P_T is the teacher’s distribution and P_S the student’s.
What does distillation look like in code?
The script below has a fixed “teacher” over four possible next tokens. Part 1 shows soft targets at different temperatures. Part 2 trains the same small student three ways on only 40 examples and checks it on 5,000 new ones.
# distill.py: knowledge distillation with soft targets (Hinton et al., 2015).
# Python 3 standard library only.
import math, random
random.seed(11)
def softmax(z, T=1.0):
m = max(z); e = [math.exp((x - m) / T) for x in z]; s = sum(e)
return [x / s for x in e]
# A "teacher": a fixed model over 4 next tokens, given 6 input features.
K, D = 4, 6
W_t = [[random.gauss(0, 1.5) for _ in range(D)] for _ in range(K)]
teacher = lambda x: [sum(w * v for w, v in zip(row, x)) for row in W_t]
make_x = lambda: [random.gauss(0, 1) for _ in range(D)]
# 1. What the teacher knows that a one-hot label throws away.
x0 = make_x()
print("teacher logits:", [round(z, 2) for z in teacher(x0)])
for T in (1, 2, 4):
print(f" soft targets at T={T}:", [round(p, 3) for p in softmax(teacher(x0), T)])
best = max(range(K), key=lambda k: teacher(x0)[k])
print(" hard label :", [1.0 if k == best else 0.0 for k in range(K)])
# 2. Train the same small student three ways on only 40 examples.
train = [make_x() for _ in range(40)]
test = [make_x() for _ in range(5000)]
def fit(targets_for, T=1.0, epochs=500, lr=0.1):
W = [[0.0] * D for _ in range(K)]
for _ in range(epochs):
for x in train:
p = softmax([sum(w * v for w, v in zip(row, x)) for row in W], T)
t = targets_for(x)
for k in range(K): # gradient of cross-entropy, times T^2 / T = T
g = (p[k] - t[k]) * T
W[k] = [w - lr * g * v / len(train) for w, v in zip(W[k], x)]
return W
def agreement(W):
hits = 0
for x in test:
s = [sum(w * v for w, v in zip(row, x)) for row in W]
hits += max(range(K), key=s.__getitem__) == max(range(K), key=teacher(x).__getitem__)
return hits / len(test)
def kl_to_teacher(W):
tot = 0.0
for x in test[:1000]:
p = softmax(teacher(x)); q = softmax([sum(w * v for w, v in zip(row, x)) for row in W])
tot += sum(pi * math.log(pi / qi) for pi, qi in zip(p, q))
return tot / 1000
hard = lambda x: [1.0 if k == max(range(K), key=teacher(x).__getitem__) else 0.0 for k in range(K)]
print("\nstudent trained on 40 examples, scored on 5,000 new ones")
for label, W in [("hard labels", fit(hard)),
("soft targets, T=1", fit(lambda x: softmax(teacher(x)))),
("soft targets, T=4", fit(lambda x: softmax(teacher(x), 4), T=4))]:
print(f" {label:<18} top-1 agrees with teacher {agreement(W):.1%} KL to teacher {kl_to_teacher(W):.4f}")
teacher logits: [4.87, -2.47, -5.81, -0.44]
soft targets at T=1: [0.994, 0.001, 0.0, 0.005]
soft targets at T=2: [0.908, 0.023, 0.004, 0.064]
soft targets at T=4: [0.669, 0.107, 0.046, 0.178]
hard label : [1.0, 0.0, 0.0, 0.0]
student trained on 40 examples, scored on 5,000 new ones
hard labels top-1 agrees with teacher 81.5% KL to teacher 0.1682
soft targets, T=1 top-1 agrees with teacher 93.8% KL to teacher 0.0455
soft targets, T=4 top-1 agrees with teacher 99.7% KL to teacher 0.0001
At T = 1 the teacher’s runner-up tokens get 0.005 and 0.001, nearly invisible. At T = 4 they get 0.178 and 0.107, and the ranking among wrong answers shows up. Same 40 examples, three results: hard labels give 81.5% agreement with the teacher on new inputs, soft targets 93.8%, soft targets at T = 4 give 99.7%.
That’s a best case, and I should say why. My student has the same shape as the teacher, a linear layer, so it can copy it exactly. A real student is much smaller than its teacher and can’t match it everywhere. The direction of the result holds, and it’s the one Hinton’s paper reports: soft targets carry more information per example, so the student needs less data.
What are the types of LLM distillation?
Two families, depending on what you can see of the teacher.
Logit distillation (white-box). You have the teacher’s weights or at least its full output distribution, and the student matches it token by token. This is what Gemma 2 did: the 2B and 9B models were trained “with knowledge distillation instead of next token prediction”, on more than 50x the compute-optimal number of tokens. The report also says knowledge distillation was used in Gemini 1.5. Meta did a version of it for Llama 3.2 1B and 3B (September 2024): pruned from Llama 3.1 8B, then trained with logits from Llama 3.1 8B and 70B “used as token-level targets”.
Output distillation (black-box). You only see the teacher’s generated text. Generate answers with the teacher, then run ordinary supervised fine-tuning of the student on them. This is cheaper to set up and works across vendors and tokenizers, because nothing needs to line up but text. It’s also a form of synthetic data.
The most visible example is DeepSeek’s. The R1 paper fine-tuned six open models, Qwen2.5 at 1.5B, 7B, 14B and 32B and Llama at 8B and 70B, on “a curated dataset comprising 800,000 samples generated with DeepSeek-R1”, for 2 to 3 epochs, with SFT only and no RL stage. These are the DeepSeek-R1-Distill models. I wrote about the R1 release itself in DeepSeek R1.
Is distillation better than training a small model with RL?
DeepSeek tested exactly this. They ran large-scale RL directly on Qwen2.5-32B-Base for over 10K steps, producing Qwen2.5-32B-Zero, and compared it with DeepSeek-R1-Distill-Qwen-32B. The distilled model “performs significantly better” on every benchmark in their table. Their conclusion has two halves: “distilling more powerful models into smaller ones yields excellent results”, while smaller models relying on large-scale RL “require enormous computational power and may not even achieve the performance of distillation”. The other half: going beyond the teacher “may still require more powerful base models and larger-scale reinforcement learning”.
That matches how I think about it. Distillation transfers a capability that exists; it doesn’t create one. Someone has to train the teacher with RLVR or similar first.
Distillation vs quantization: what’s the difference?
Both make models cheaper, in different ways. Quantization keeps the same model and stores its weights in fewer bits. Distillation trains a different, smaller model. They combine: a distilled 8B model quantized to 4 bits is one way to run LLMs locally. DistilBERT is the classic data point for distillation alone: Sanh et al. (2019) cut BERT’s size by 40% while “retaining 97% of its language understanding capabilities and being 60% faster”.
Where does distillation break?
- The student inherits the teacher’s mistakes. Soft targets transfer the teacher’s errors and biases along with its skill.
- Capacity limits. A 1.5B student can’t hold everything a teacher with hundreds of billions of parameters knows. Expect it to keep the skill you distilled for (math reasoning, say) and fall behind elsewhere.
- Tokenizer mismatch. Logit distillation needs teacher and student to share a vocabulary, or extra alignment work. Output distillation avoids this.
- Licenses and terms. Model licenses and API terms can restrict using outputs to train other models. Read them before you build a training set from someone else’s model.
What I would do
For a production task with a narrow scope, such as classification, extraction or a fixed-format answer, I’d distill. I’d use the strongest model I can afford as the teacher, generate a few thousand outputs on real inputs, filter them with checks and spot reviews, and fine-tune a small open model on the result. Then I’d evaluate the student against the teacher on held-out data, because agreement with the teacher is the number that matters.
If I have the teacher’s logits, for example an open-weight teacher I can run myself, I’d use them; my toy shows how much more signal they carry than text alone. And I’d keep the teacher in the loop for inputs where the student’s confidence is low, rather than pretending the student covers everything.
FAQ
What is the difference between a teacher model and a student model?
The teacher is the large, capable model whose behavior you want to copy. The student is the smaller model you train to imitate it, either by matching its output probabilities or by fine-tuning on its generated text.
Are DeepSeek-R1-Distill models the same as DeepSeek R1?
No. They’re Qwen2.5 and Llama models, from 1.5B to 70B, fine-tuned on 800,000 samples generated by DeepSeek-R1. They inherit much of R1’s reasoning style but are different, much smaller models.
What temperature should I use for distillation?
Hinton et al. found that with 300 or more units per layer, all temperatures above 8 worked about the same, while a much smaller net with 30 units per layer did best between 2.5 and 4. Treat T as a hyperparameter and tune it on validation data.
Is distillation the same as fine-tuning?
Output distillation is fine-tuning on data a teacher model generated. Logit distillation changes the loss: the student matches the teacher’s full distribution over the vocabulary instead of a single target token.
Sources
- Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network, March 9, 2015
- Sanh et al., DistilBERT, a distilled version of BERT, October 2, 2019
- Gemma Team, Google, Gemma 2: Improving Open Language Models at a Practical Size, July 31, 2024
- Meta, Llama 3.2: Revolutionizing edge AI and vision with open, customizable models, September 25, 2024
- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 22, 2025 (v2, January 4, 2026), appendices B.4.3 and F