History of large language models: GPT-3 to 2026, a timeline
A history of large language models, from the 2017 transformer and GPT-3 to reasoning models and agents in 2026, with dates, sources and training numbers.

Large language models went from a research result to everyday infrastructure in about six years. The short version: the transformer (2017) made training at scale practical, GPT-3 (2020) showed that a big enough model learns tasks from examples in the prompt, RLHF and ChatGPT (2022) turned models into assistants, open weights and tool calling (2023) turned them into components, long context and multimodality (2024) widened what they could read, and reasoning models and agents (2024 to 2026) made them spend more compute per answer and act on their own.
This timeline uses the date each paper or announcement went public, links the primary source, and leaves out anything I could not confirm.
What came before GPT-3?
June 12, 2017: the transformer. Google’s Attention Is All You Need proposed an architecture “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.” Without recurrence, training parallelizes across a whole sequence on GPUs, which is what made the later scale affordable. Every model below is a transformer or a close descendant.
May 28, 2020: GPT-3. OpenAI’s Language Models are Few-Shot Learners described “an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model”, trained on 300 billion tokens. The result that mattered was in-context learning: “GPT-3 is applied without any gradient updates or fine-tuning”, with tasks specified only through text and a few examples in the prompt. Prompting as a way to program a model starts here.
2022: how did LLMs become assistants?
January 28, 2022: chain-of-thought. Wei et al. showed that worked reasoning steps in the prompt let large models solve multi-step problems they otherwise failed. I cover it in what is chain-of-thought.
March 4, 2022: InstructGPT. OpenAI’s Training language models to follow instructions with human feedback fine-tuned GPT-3 on human demonstrations and then on human preference rankings, the recipe known as RLHF. The striking line: “outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3.” Alignment work beat a 100x size gap on what users actually wanted.
March 29, 2022: Chinchilla. DeepMind’s Training Compute-Optimal Large Language Models argued that models of the day were undertrained, and that “for every doubling of model size the number of training tokens should also be doubled.” Their 70B Chinchilla, trained on 1.4 trillion tokens, “uniformly and significantly outperforms Gopher (280B).” The scaling laws post covers what that means.
November 30, 2022: ChatGPT. OpenAI launched ChatGPT as a free research preview of a GPT-3.5 model tuned with RLHF. Nothing in the architecture was new; the interface was, and it is the point where LLMs became a consumer product.
How did training recipes change? A worked comparison
The cleanest way to see the shift is to put the training runs side by side. This script uses the parameter and token counts each paper reports, and estimates training compute with the 6 * N * D approximation the Chinchilla paper uses (N parameters, D tokens).
# history.py: six training runs from the papers, one formula. Python 3, stdlib only.
# Training compute is approximated as 6 * N * D FLOPs (the approximation Chinchilla uses),
# N = parameters, D = training tokens. Figures as reported in each paper.
RUNS = [ # (model, year, parameters, training tokens)
("GPT-3", 2020, 175e9, 300e9),
("Chinchilla", 2022, 70e9, 1.4e12),
("LLaMA 65B", 2023, 65e9, 1.4e12),
("Llama 2 70B", 2023, 70e9, 2.0e12),
("Phi-3-mini", 2024, 3.8e9, 3.3e12),
("Llama 3 405B", 2024, 405e9, 15.6e12),
]
print(f"{'model':<13} {'year':>4} {'params':>6} {'tokens':>7} {'tok/param':>9} {'6ND FLOPs':>10}")
for name, year, n, d in RUNS:
print(f"{name:<13} {year:>4} {f'{n / 1e9:g}B':>6} {d / 1e12:>6.1f}T {d / n:>9.0f} {6 * n * d:>10.1e}")
Output (Python 3.13.11):
model year params tokens tok/param 6ND FLOPs
GPT-3 2020 175B 0.3T 2 3.2e+23
Chinchilla 2022 70B 1.4T 20 5.9e+23
LLaMA 65B 2023 65B 1.4T 22 5.5e+23
Llama 2 70B 2023 70B 2.0T 29 8.4e+23
Phi-3-mini 2024 3.8B 3.3T 868 7.5e+22
Llama 3 405B 2024 405B 15.6T 39 3.8e+25
Two checks keep the estimate honest. The Llama 3 paper reports its flagship was pre-trained using 3.8 × 10^25 FLOPs, the same figure the formula gives. And the tokens-per-parameter column tells the story of the era. GPT-3 saw under 2 tokens per parameter; Chinchilla set the compute-optimal point near 20; open models went past it, because a model that will serve millions of requests is cheaper to run if it is smaller and trained longer. Phi-3-mini at 868 tokens per parameter is that logic taken to the extreme, and it is why small language models got good.
2023: how did LLMs become building blocks?
February 27, 2023: LLaMA. Meta’s LLaMA models ran from 7B to 65B parameters, trained on “publicly available datasets exclusively”, and “LLaMA-13B outperforms GPT-3 (175B) on most benchmarks.” Meta released the models “to the research community”, and the open-model ecosystem grew around them.
March 14, 2023: GPT-4. OpenAI’s GPT-4 accepted image and text input, with an 8,192-token context at launch and a 32,768-token version on limited access.
June 13, 2023: function calling. GPT-4 and GPT-3.5 Turbo began returning JSON arguments for functions described in the request, which I wrote up in OpenAI function calling. Tool use, and every agent since, builds on this.
July 18, 2023: Llama 2. Meta released Llama 2 in 7B, 13B and 70B, trained on 2 trillion tokens, free for research and commercial use under a custom license. Open weights became a procurement option for enterprises.
January 8, 2024: Mixtral. Mistral’s Mixtral of Experts paper described a sparse model with 8 experts per layer, two chosen per token: 47B parameters in total, 13B active, under Apache 2.0. Mixture of experts is now how most large models cut cost per token.
2024: what did long context and multimodality change?
February 15, 2024: Gemini 1.5. Google announced Gemini 1.5 with a 128K standard context and a 1 million token preview. Google’s own equivalences for a million tokens were 1 hour of video, 11 hours of audio or codebases of over 30,000 lines, which changed the trade-off between retrieval and the context window.
May 13, 2024: GPT-4o. OpenAI’s GPT-4o was trained as one model across text, vision and audio, at half GPT-4 Turbo’s price, and it reached the free tier. Multimodal models became the default.
June 21, 2024: Claude 3.5 Sonnet. Anthropic’s Claude 3.5 Sonnet put strong coding at a mid-tier price; its launch post reported 64% on an internal agentic coding evaluation against 38% for Claude 3 Opus.
July 2024: Llama 3. Meta’s Llama 3 Herd of Models paper described a dense 405B model with a context of up to 128K tokens, trained on 15.6T tokens, an open-weight model at frontier scale.
September 12, 2024: o1-preview. OpenAI’s o1-preview was trained to think before answering, with a hidden chain of thought billed as output tokens. This started the reasoning-model line and made test-time compute a second scaling axis.
October 22, 2024: computer use. Anthropic put computer use in public beta: Claude operating a desktop from screenshots.
November 2024: MCP. Anthropic open-sourced the Model Context Protocol, a standard way for AI apps to discover and call tools.
2025 and 2026: how did reasoning and agents take over?
January 20, 2025: DeepSeek R1. DeepSeek released R1 under MIT, with performance it described as on par with o1, and a paper showing reasoning can be incentivized through pure reinforcement learning. Reasoning stopped being a closed-lab capability.
November 2025: effort controls. GPT-5.1 let developers set reasoning effort down to none, covered in what are reasoning models, and Claude Opus 4.5 shipped at a third of Opus 4.1’s price with an effort parameter. Cost per task, not per token, became the number to compare.
December 2025: sparse attention and an open foundation for MCP. DeepSeek V3.2 shipped sparse attention to cut long-context cost, and Anthropic donated MCP to the Linux Foundation’s new Agentic AI Foundation on December 9.
2026: agents as the product. The launches I covered this month were judged on what an agent session costs. Moonshot’s Kimi K3.1 Pro (October 23, 2026) is an open-weight model with a 1M-token context, and Google’s Gemini 4 Carbon (October 28, 2026) was priced to match Claude Opus 5.5, with the cache, not the list price, deciding which is cheaper for long agent runs.
What patterns repeat across the history?
Recipes beat raw size. InstructGPT 1.3B beat GPT-3 175B on preference; Chinchilla 70B beat Gopher 280B; LLaMA-13B beat GPT-3. Each time, data and training choices mattered more than parameter count alone.
Interfaces drive adoption. ChatGPT, function calling and MCP were not capability jumps in the model. They changed how people and programs could use one.
Price per token keeps falling, spend per task does not. Cheaper tokens got spent on longer contexts, reasoning and agent loops.
What I would do
I would read the primary sources in this list in order; they are short and they explain more than most summaries, this one included. For a team picking models today, I would take two lessons from the history: evaluate on your own tasks, because the leader at every stage of this timeline was overtaken quickly, and design for swapping models, because the one you pick now will not be the one you run in eighteen months. From a data-platform seat, the line I would circle is that open weights and on-device models made “the data never leaves” a real option, and that option has only grown.
FAQ
When were large language models invented?
There is no single date. The transformer architecture appeared in June 2017, and GPT-3 in May 2020 showed the in-context learning that defines LLMs today. ChatGPT in November 2022 brought them to the public.
What was the first large language model?
Statistical language models are decades old. The modern line starts with the transformer in 2017, and GPT-3, at 175 billion parameters in 2020, is the model most histories of today’s LLMs start from, because it was the first to show in-context learning at that scale.
What is the difference between GPT-3 and ChatGPT?
GPT-3 is a base model that continues text. ChatGPT was a GPT-3.5 model further trained with human feedback (RLHF) to follow instructions and hold a conversation, released with a chat interface in November 2022.
What are the major milestones in LLM history?
The transformer (2017), GPT-3 and in-context learning (2020), InstructGPT and RLHF (2022), ChatGPT (2022), open weights with LLaMA and Llama 2 (2023), function calling (2023), million-token context and native multimodality (2024), reasoning models (2024 to 2025) and agents (2025 to 2026).
Sources
- Vaswani et al., Attention Is All You Need, June 12, 2017
- Brown et al., Language Models are Few-Shot Learners (GPT-3), May 28, 2020
- Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 28, 2022
- Ouyang et al., Training language models to follow instructions with human feedback, March 4, 2022
- Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), March 29, 2022
- OpenAI, ChatGPT: Optimizing Language Models for Dialogue, November 30, 2022
- Touvron et al., LLaMA: Open and Efficient Foundation Language Models, February 27, 2023
- Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, July 18, 2023 (2 trillion tokens)
- Jiang et al., Mixtral of Experts, January 8, 2024
- Abdin et al., Phi-3 Technical Report, April 22, 2024
- Llama Team, Meta, The Llama 3 Herd of Models, July 31, 2024 (15.6T tokens, 3.8 × 10^25 FLOPs)
- DeepSeek-AI, DeepSeek-R1, January 22, 2025
- Earlier posts on this site, each with its own primary sources: GPT-4, function calling, Llama 2, Gemini 1.5, GPT-4o, Claude 3.5 Sonnet, o1-preview, computer use, MCP, DeepSeek R1, GPT-5.1, Claude Opus 4.5, DeepSeek V3.2, Kimi K3.1 Pro and Gemini 4 Carbon