What is synthetic data? How LLMs are trained on generated data
Synthetic data is training data written by a model, not people. How LLM labs generate and verify it, why model collapse happens, with a Python demo.

Synthetic data is training data generated by a model or a program instead of collected from people. For LLMs it usually means a strong model writes instructions, answers, code, explanations or whole textbooks, and a pipeline checks, filters and deduplicates them before they are used to train another model, or the same one. Most frontier and small models released since 2023 use it somewhere in training, and the quality of the checking step decides whether it helps or hurts.
Why do LLM labs use synthetic data?
Three reasons, in roughly the order they appeared.
Instruction data is expensive. Turning a base model into an assistant needs examples of instructions and good answers. Writing them by hand is slow. Self-Instruct (Wang et al., December 2022) had GPT-3 generate its own instructions, inputs and outputs, filtered out invalid and near-duplicate ones, and fine-tuned on the rest: a 33% absolute improvement on Super-NaturalInstructions, and only a 5% gap behind InstructGPT-001 in human evaluation. Stanford’s Alpaca (March 2023) applied the idea to LLaMA 7B with 52K instructions generated by text-davinci-003, for “less than $500” of API calls.
Quality beats quantity for small models. Microsoft’s phi-1 (June 2023) trained a 1.3B code model on about 6B tokens of filtered “textbook quality” web data plus about 1B tokens of synthetic textbooks and exercises written by GPT-3.5, and reached 50.6% on HumanEval. Phi-4 (December 2024) went further and “strategically incorporates synthetic data throughout the training process”, and the authors report it surpasses its teacher, GPT-4, on STEM question answering. Those results are why small language models got so much better per parameter.
Some skills have no natural data. Long-context reasoning, tool use and multi-step code fixes are rare on the web in the form a model needs. Meta’s Llama 3 paper says it relied predominantly on synthetic data for long-context fine-tuning, and generated over 2.7M synthetic coding examples for supervised fine-tuning.
How is synthetic training data generated?
The Llama 3 coding pipeline is the most detailed public description I know of, and it is a good template. Meta:
- Generated problems by sampling random code snippets and prompting the model to write programming problems inspired by them, to cover the long tail of topics.
- Generated solutions with Llama 3, asking it to follow good programming rules and explain its thinking in comments.
- Checked correctness with static analysis (parser and linter) and with unit tests the model wrote, run in a container.
- Fed errors back. When a solution failed, the model got the problem, the faulty code and the error output, and tried again. “Only dialogs that pass all checks are included in the final dataset.” About 20% of solutions were initially wrong but fixed themselves this way.
- Iterated, with each round’s improved model generating better data for the next.
The paper also states the failure that motivated step 3: training Llama 3 405B on its own generated data “is not helpful (and can even degrade performance)” until execution feedback was added as a source of truth. That sentence is the whole field in miniature. Generation is cheap; verification is where the value comes from.
Other common patterns follow the same shape: rejection sampling (generate many answers, keep only those a reward model or checker accepts), distillation (a big teacher writes answers or reasoning traces for a smaller student), rephrasing (rewrite web text into cleaner forms), and backtranslation (generate the question for an answer you already trust).
What does a synthetic data pipeline look like in code?
The script below has two parts. The first is a toy generate, verify and deduplicate loop: a simulated teacher writes arithmetic word problems and gets 12% of the answers wrong, and the pipeline keeps only problems whose answer survives a recomputation. The second shows model collapse in miniature: a Gaussian refit, generation after generation, on 20 samples drawn from the previous generation’s fit, with no fresh real data.
# synthetic.py: generate, verify, dedupe; then what happens without fresh data.
# Python 3, stdlib only. The "teacher" is simulated: it gets 12% of answers wrong.
import random, statistics
random.seed(7)
ITEMS = ["mangoes", "tickets", "invoices", "servers"]
def teacher():
"""Pretend LLM: writes a problem and an answer, sometimes a wrong one."""
a, b, item = random.randint(2, 40), random.randint(2, 40), random.choice(ITEMS)
q = f"A team has {a} {item} and gets {b} more each day for 3 days. How many {item}?"
ans = a + 3 * b + (random.choice([-1, 1, b]) if random.random() < 0.12 else 0)
return {"q": q, "a": ans, "check": (a, b)}
raw = [teacher() for _ in range(2000)]
verified = [r for r in raw if r["a"] == r["check"][0] + 3 * r["check"][1]] # run the math
seen, unique = set(), []
for r in verified:
key = " ".join(r["q"].lower().split()) # exact-dup key
if key not in seen:
seen.add(key); unique.append(r)
print(f"generated {len(raw)}, failed check {len(raw) - len(verified)}, "
f"duplicates {len(verified) - len(unique)}, kept {len(unique)}")
print("sample:", unique[0]["q"], "->", unique[0]["a"])
# Model collapse in miniature: each generation fits a Gaussian to 20 samples drawn from
# the previous generation's fit, with no fresh real data. 500 independent chains.
chains = [[random.gauss(0, 1) for _ in range(20)] for _ in range(500)]
for gen in range(201):
fits = [(statistics.fmean(c), statistics.stdev(c)) for c in chains]
if gen in (0, 10, 50, 100, 200):
sds = sorted(sd for _, sd in fits)
narrow = sum(sd < 0.5 for sd in sds) / len(sds)
print(f"generation {gen:>3}: median std {sds[len(sds) // 2]:.3f}, "
f"chains below half the original spread: {narrow:.0%}")
chains = [[random.gauss(mu, sd) for _ in range(20)] for mu, sd in fits]
Output (Python 3.13.11):
generated 2000, failed check 251, duplicates 185, kept 1564
sample: A team has 22 servers and gets 11 more each day for 3 days. How many servers? -> 55
generation 0: median std 0.985, chains below half the original spread: 0%
generation 10: median std 0.801, chains below half the original spread: 20%
generation 50: median std 0.292, chains below half the original spread: 69%
generation 100: median std 0.083, chains below half the original spread: 87%
generation 200: median std 0.006, chains below half the original spread: 97%
In the first part, 251 of 2,000 examples had a wrong answer and 185 were exact duplicates, so 22% of what the generator produced would have been noise or repetition in the training set. The verifier caught the errors only because the answer can be recomputed. Real pipelines get the same effect from unit tests, math checkers or a reward model, and they also need near-duplicate detection, which an exact string key does not catch.
The second part is the warning. Each generation is a perfectly reasonable fit of the previous one, yet the spread shrinks: the median standard deviation falls from about 1 to 0.08 after 100 generations, and 97% of the chains have lost more than half their original spread by generation 200. Rare values stop being sampled, so the next fit never sees them.
What is model collapse?
Shumailov et al. named it in 2023: training on model-generated content damages the resulting models, a process in which “tails of the original content distribution disappear.” Each model slightly underrepresents rare events, the next model learns from that thinner distribution, and the error compounds. My Gaussian demo is the simplest version of their argument.
What the successful pipelines above have in common is that they never close the loop on unfiltered output. They mix synthetic data with real data, they verify against something outside the model (a test, a checker, a human preference), and they generate from a stronger teacher or with new information such as execution results. Collapse is a risk of indiscriminate recursion, not of synthetic data as such.
Where does synthetic data go wrong?
Errors become training signal. An unverified wrong answer teaches the student to be wrong with confidence, which feeds hallucinations.
Diversity drops. Generators repeat favorite phrasings, names and structures. Without seeding from varied sources, as Meta did with random code snippets, the dataset narrows.
Benchmark contamination. A teacher that has seen a benchmark can reproduce it in “synthetic” data, which inflates benchmark scores without improving the model.
Licenses and terms. A dataset written by a hosted model inherits that provider’s terms of use, and an open-weight teacher inherits its license. Read both for clauses about training other models before you build a dataset.
Privacy is not automatic. From a privacy-compliance seat, the line I would circle: synthetic data generated from real customer records can still leak those records if the generator memorized them. Treat it as derived personal data until you have tested for leakage.
What I would do
I would start with the verifier, not the generator. If I cannot write an automatic check for an example, such as tests for code, recomputation for numbers, or schema validation for structured output, I would budget for human review of a sample before training on any of it. I would seed generation from real, varied inputs, deduplicate with near-duplicate hashing, keep a held-out set of real data that never touches the generator, and track the share of synthetic data in every training mix. For fine-tuning projects, my bet is on a few thousand verified examples over a hundred thousand unverified ones: they are cheaper to inspect, and every error you leave in is one the model learns.
FAQ
What is synthetic data in machine learning?
Data created by a program or model rather than recorded from the real world. For LLMs it is usually model-written text: instructions and answers, code with tests, reasoning traces or rephrased documents.
Is synthetic data as good as real data?
When it is verified and mixed with real data, it can be better for specific skills, as Phi and Llama 3 report. Unverified and used alone, it degrades models over generations.
What is model collapse?
The loss of rare information when models are trained repeatedly on the output of earlier models. Shumailov et al. showed the tails of the original distribution disappear.
How do you generate synthetic data with an LLM?
Prompt a strong model with varied seeds to produce examples, check each one with an automatic verifier or a reward model, remove duplicates, and mix the survivors with real data before training.
Sources
- Wang et al., Self-Instruct: Aligning Language Models with Self-Generated Instructions, December 20, 2022
- Stanford CRFM, Alpaca: A Strong, Replicable Instruction-Following Model, March 13, 2023
- Gunasekar et al., Textbooks Are All You Need (phi-1), June 20, 2023
- Abdin et al., Phi-4 Technical Report, December 12, 2024
- Llama Team, Meta, The Llama 3 Herd of Models, July 31, 2024 (section 4.3.1 on synthetic code data)
- Shumailov et al., The Curse of Recursion: Training on Generated Data Makes Models Forget, May 27, 2023