What is LLM temperature? Temperature, top-p and top-k explained
LLM temperature controls how random the next token is. How temperature, top-p and top-k reshape the probabilities, and why Claude's API dropped them in 2026.

Temperature is a sampling setting that controls how random a language model’s output is. Before picking each token, the model’s scores (logits) are divided by the temperature: below 1 the distribution sharpens toward the most likely token, above 1 it flattens and rare tokens get picked more often. Top-p and top-k are companion settings that cut off the unlikely tail before sampling.
Where does temperature fit in text generation?
A large language model doesn’t output text directly. At every step it outputs one score per token in its vocabulary, and a softmax turns those scores into probabilities. Then something has to choose one token. That choice is decoding, and temperature, top-k and top-p are the knobs on it.
The obvious choice, always take the most likely token (greedy decoding), works badly for open-ended text. Holtzman et al.’s The Curious Case of Neural Text Degeneration (ICLR 2020) showed that “using likelihood as a decoding objective leads to text that is bland and strangely repetitive”. Sampling fixes the blandness but, drawn from the full distribution, occasionally picks a token from the long tail of nonsense. The three settings trade between those two failures.
How does temperature work?
The paper writes it as a softmax with the logits divided by t:
p(token_i) = exp(logit_i / t) / Σ_j exp(logit_j / t)
At t = 1 you get the model’s own distribution. As t approaches 0, the gap between the top logit and the rest is magnified until the top token takes all the probability, which is greedy decoding. As t grows, the gaps shrink and the distribution moves toward uniform. Temperature never changes the order of the tokens; it changes how much the leader leads by.
What do top-k and top-p do?
Both throw away the tail and renormalize what’s left.
- Top-k keeps the k most probable tokens. It’s simple, but a fixed k is wrong in both directions. Holtzman et al. point out that some next-token distributions are flat across “tens or hundreds of reasonable options” while others put almost all the mass on one or two tokens, so a small k makes text bland in the first case and a large k admits junk in the second.
- Top-p, also called nucleus sampling, is the paper’s proposal. It keeps the smallest set of tokens whose cumulative probability reaches p, so the candidate list grows when the model is unsure and shrinks when it’s confident.
What does that look like in code?
This script takes one made-up next-token distribution and applies each setting, then draws 1,000 samples at three temperatures.
# sampling.py: what temperature, top-k and top-p do to one next-token distribution.
# Python 3, stdlib only.
import math, random
from collections import Counter
# Made-up logits for the token after "The capital of France is"
logits = {"Paris": 6.0, "a": 3.5, "the": 3.2, "located": 2.8, "Lyon": 1.5, "beautiful": 1.0, "banana": -2.0}
def softmax(scores, temperature=1.0):
m = max(scores.values())
e = {t: math.exp((s - m) / temperature) for t, s in scores.items()}
z = sum(e.values())
return {t: v / z for t, v in e.items()}
def top_k(probs, k):
keep = dict(sorted(probs.items(), key=lambda x: -x[1])[:k])
z = sum(keep.values())
return {t: p / z for t, p in keep.items()}
def top_p(probs, p):
keep, mass = {}, 0.0
for t, q in sorted(probs.items(), key=lambda x: -x[1]):
keep[t] = q; mass += q
if mass >= p: # smallest set whose mass reaches p
break
z = sum(keep.values())
return {t: q / z for t, q in keep.items()}
def show(label, probs):
print(f"{label:<16}" + " ".join(f"{t} {p:.2f}" for t, p in sorted(probs.items(), key=lambda x: -x[1]) if p >= 0.005))
for T in (0.2, 0.7, 1.0, 1.5):
show(f"T={T}", softmax(logits, T))
show("T=1.0 top_k=3", top_k(softmax(logits), 3))
show("T=1.0 top_p=0.9", top_p(softmax(logits), 0.9))
random.seed(42) # 1,000 draws at each temperature
for T in (0.2, 1.0, 1.5):
probs = softmax(logits, T)
draws = Counter(random.choices(list(probs), weights=probs.values(), k=1000))
print(f"1,000 samples at T={T}: {dict(draws.most_common())}")
Output (Python 3.13.11):
T=0.2 Paris 1.00
T=0.7 Paris 0.94 a 0.03 the 0.02 located 0.01
T=1.0 Paris 0.83 a 0.07 the 0.05 located 0.03 Lyon 0.01 beautiful 0.01
T=1.5 Paris 0.64 a 0.12 the 0.10 located 0.08 Lyon 0.03 beautiful 0.02
T=1.0 top_k=3 Paris 0.87 a 0.07 the 0.05
T=1.0 top_p=0.9 Paris 0.92 a 0.08
1,000 samples at T=0.2: {'Paris': 1000}
1,000 samples at T=1.0: {'Paris': 853, 'a': 59, 'the': 47, 'located': 28, 'beautiful': 7, 'Lyon': 6}
1,000 samples at T=1.5: {'Paris': 637, 'a': 120, 'the': 106, 'located': 62, 'Lyon': 40, 'beautiful': 30, 'banana': 5}
The logits are invented, but the shapes are real. At 0.2 the model is effectively greedy: “Paris” all 1,000 times. At 1.5, “Lyon” comes up 40 times in 1,000 and “banana” 5 times, and each of those starts a sentence the model now has to continue. Top-p at 0.9 keeps only two candidates here because the model is confident; on a flatter distribution the same p would keep dozens. That adaptive behavior is the paper’s whole argument for top-p over top-k.
Note what none of these settings do. “Lyon” is a wrong answer that the model gave 1% probability at temperature 1; lowering temperature hides it, but doesn’t make the model know more. Temperature is a dial on variety, not on truth, a point I come back to in why LLMs hallucinate.
What changed for temperature in 2026?
The provider APIs no longer agree on whether you should touch these settings at all.
OpenAI’s Chat Completions reference still accepts temperature “between 0 and 2” with a default of 1, and top_p (nucleus sampling, default 1), with the advice: “We generally recommend altering this or temperature but not both.” It has no top_k, and it notes that parameter support varies for newer reasoning models.
Anthropic went further. Its migration guide for API version 2026-10-21 says that a request sending temperature, top_p or top_k now returns a 400 error, “even at the parameter’s default value”, and that the latest SDKs no longer have the three parameters. Requests sample with the defaults, including a temperature of 1.0. Claude 4.7 and later models already accepted only the defaults. On older models such as Claude Sonnet 4.6, the guide is explicit that a request which lowered temperature for more consistent output loses that effect, and “No other setting replaces it.” The escape hatch is to keep that request on the older 2023-06-01 version.
My reading: on reasoning models the sampled text includes thinking, and the labs would rather tune decoding themselves than have users break a trained behavior with a low temperature. The practical upshot is that “set temperature to 0 for determinism” is no longer advice you can apply everywhere, and it was never a guarantee anyway.
What temperature should I use?
For APIs that still expose it, these are my starting points, not rules from any provider:
- Extraction, classification, structured output, code edits: low, 0 to 0.3. You want the most likely answer and little variety.
- General chat and writing: leave the default. Providers tune their defaults for this.
- Brainstorming, varied candidates, synthetic data: 0.8 to 1.2, often with top-p around 0.9 to keep the tail out.
- Change one knob at a time. Temperature or top-p, as OpenAI’s reference advises.
On APIs that don’t expose it, get consistency elsewhere: a tighter prompt, examples, structured output with schema validation, and a check that rejects bad answers. If you need the same answer every time, cache the answer.
What I would do
Stop treating temperature as a quality setting. It controls variety. For anything feeding a database, a workflow or a customer, the real controls are the prompt, the output schema and the validator, and they work whether or not the provider lets you set temperature.
From a data-platform seat, the line I’d circle is reproducibility. A pipeline that silently depended on temperature=0 will change behavior the day it moves to a newer model or API version, without a single code change on your side. Log the model ID, the API version and the sampling settings with every output you store, the same way you’d log a schema version, so that a shift in outputs can be traced to its cause.
For variety on purpose (several candidate answers to pick from, as in best-of-N agent runs), sample several times and select with a checker or an LLM judge, rather than hoping one high-temperature sample lands.
FAQ
Does temperature 0 make an LLM deterministic?
It makes decoding greedy in principle, but that doesn’t guarantee identical outputs. Thinking Machines Lab showed in September 2025 that server load changes the batch size, and common inference kernels give slightly different numbers at different batch sizes: 1,000 temperature-0 completions from Qwen3-235B produced 80 unique outputs. And some APIs no longer accept the setting. Treat temperature 0 as “low variety”, not “reproducible”.
What is a good temperature for coding?
Low, 0 to 0.2, when you want one correct edit. For generating several alternative solutions to test against each other, a higher temperature gives you more variety to choose from.
What is the difference between top-p and top-k?
Top-k keeps a fixed number of candidate tokens. Top-p keeps however many tokens it takes to reach a probability mass, so the list adapts to how confident the model is. Top-p is usually the better default.
Does a higher temperature make a model more creative?
It makes it less predictable, which can read as creative. Past a point it produces incoherent text, because rare tokens start getting sampled. Better prompts usually buy more useful creativity than a higher temperature does.
Sources
- Holtzman et al., The Curious Case of Neural Text Degeneration, April 22, 2019 (ICLR 2020), Sections 3.1 to 3.3
- OpenAI, Create chat completion API reference, as of October 31, 2026
- Horace He, Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, September 10, 2025
- Anthropic, Migrating to API version 2026-10-21, as of October 31, 2026