Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is speculative decoding? Faster LLMs, same output

Speculative decoding has a small draft model guess several tokens and the big model verify them in one pass. Why the output stays identical, with Python.

What is speculative decoding? Faster LLMs, same output

Speculative decoding is an inference trick where a small, fast draft model guesses the next few tokens and the large target model checks all of those guesses in a single forward pass, keeping the ones it agrees with. Because a special acceptance rule decides which guesses survive, the output follows exactly the same distribution as the large model on its own; you get the big model’s answers in fewer slow steps.

The idea was published twice, independently, within a few weeks: Leviathan, Kalman and Matias at Google (November 2022, ICML 2023), who named it speculative decoding, and Chen et al. at DeepMind (February 2023), who called it speculative sampling. Today it’s a standard feature of serving engines.

I ran the script below on 2026-10-31 with Python 3.13.11 and no packages.

Why is LLM decoding slow in the first place?

A language model writes one token per forward pass, and each pass depends on the token before it. Generating 500 tokens means 500 serial runs of the full model.

Each of those runs does very little arithmetic per byte it reads. At batch size 1, the model reads every weight from memory to produce a single token, so the GPU spends most of its time waiting on memory rather than computing. The Leviathan paper puts it plainly: in the common case “memory bandwidth is the bottleneck, and compute resources are available”. The Medusa paper says the same thing: decoding is limited by moving model parameters from memory to the accelerator.

That spare compute is the opening. Scoring five candidate tokens in one pass costs about the same wall time as scoring one, because the weights are read once either way. If something cheap can propose good candidates, the expensive model only needs to verify them. I go through the prefill and decode phases in more detail in LLM latency explained.

How does speculative decoding work?

One step of the algorithm (Algorithm 1 in Leviathan et al.) has three parts:

  1. Draft. The small model generates γ tokens autoregressively. This is γ serial calls, but each is cheap.
  2. Verify. The large model runs once over the prefix plus all γ drafts, producing its own next-token distribution at every position (γ + 1 distributions, because the last position predicts one token past the drafts).
  3. Accept or reject. Walk through the drafts in order. Accept each one with probability min(1, p(x) / q(x)), where p is the target’s probability for that token and q is the draft’s. At the first rejection, resample that position from the leftover distribution norm(max(0, p - q)) and stop. If every draft is accepted, sample one bonus token from the target’s last distribution.

So every step of the large model yields at least one token (no worse than normal decoding) and at most γ + 1.

The acceptance rule is the clever part. If the draft gave a token less probability than the target did, it’s always kept. If the draft was overconfident, it’s kept only some of the time, and the residual distribution adds back exactly the probability mass the draft underweighted. The paper proves in its appendix that the token you end up with is distributed according to p, for any p and q. A bad draft model makes things slower; it can’t make them wrong.

Can I see it in code?

Here is speculative sampling on a toy six-word vocabulary. The “target model” is a fixed bigram distribution; the “draft model” is the target plus noise, so it’s close but wrong. The script checks two things: that the speculative output matches the target’s distribution, and how many tokens each target pass produces.

# speculative.py: speculative sampling, checked against the target distribution.
# Python 3 standard library only.
import random
from collections import Counter

random.seed(7)
V = ["the", "cat", "sat", "on", "mat", "."]

def target_p(prev):  # the big model: a fixed toy bigram distribution
    r = random.Random(V.index(prev))
    w = [r.random() ** 3 for _ in V]
    return [x / sum(w) for x in w]

def draft_q(prev):  # the small model: the target plus noise, so it is close but wrong
    r = random.Random(V.index(prev) + 100)
    w = [p + 0.15 * r.random() for p in target_p(prev)]
    return [x / sum(w) for x in w]

def sample(dist): return random.choices(range(len(V)), weights=dist)[0]

def spec_step(prefix, gamma):
    """Leviathan et al. Algorithm 1: gamma draft tokens, one parallel target pass."""
    drafts, qs, ctx = [], [], prefix[-1]
    for _ in range(gamma):  # cheap: gamma serial calls to the draft model
        q = draft_q(ctx); x = sample(q); drafts.append(x); qs.append(q); ctx = V[x]
    # expensive: ONE target pass scores every draft position (plus one extra)
    ctxs = [prefix[-1]] + [V[x] for x in drafts]
    ps = [target_p(c) for c in ctxs]
    out = []
    for i, x in enumerate(drafts):
        if random.random() < min(1.0, ps[i][x] / qs[i][x]):
            out.append(x)  # accepted
            continue
        resid = [max(0.0, p - q) for p, q in zip(ps[i], qs[i])]  # rejected: resample
        out.append(sample(resid))
        return out
    out.append(sample(ps[gamma]))  # all accepted: bonus token from the target
    return out

# 1. Is the output distributed like the target? Check the token after "cat".
N = 200_000
plain = Counter(sample(target_p("cat")) for _ in range(N))
spec = Counter(spec_step(["cat"], gamma=4)[0] for _ in range(N))
print(f"{'token':<6}{'target p':>10}{'plain':>10}{'speculative':>13}")
for i, w in enumerate(V):
    print(f"{w:<6}{target_p('cat')[i]:>10.3f}{plain[i]/N:>10.3f}{spec[i]/N:>13.3f}")

# 2. Generate 20,000 tokens and count target-model passes.
for gamma in (1, 2, 4, 8):
    prefix, calls = ["the"], 0
    while len(prefix) < 20_001:
        prefix += [V[x] for x in spec_step(prefix, gamma)]; calls += 1
    print(f"gamma={gamma}: {20_000 / calls:.2f} tokens per target pass")

# 3. Acceptance rate alpha = E[sum_x min(p, q)], and the paper's Equation (1).
alpha = sum(sum(min(p, q) for p, q in zip(target_p(w), draft_q(w))) for w in V) / len(V)
print(f"\nalpha (mean over contexts) = {alpha:.3f}")
for gamma in (1, 2, 4, 8):
    print(f"gamma={gamma}: Equation (1) predicts {(1 - alpha ** (gamma + 1)) / (1 - alpha):.2f}")
token   target p     plain  speculative
the        0.002     0.002        0.002
cat        0.473     0.474        0.474
sat        0.347     0.345        0.347
on         0.013     0.013        0.013
mat        0.095     0.095        0.094
.          0.071     0.071        0.071
gamma=1: 1.85 tokens per target pass
gamma=2: 2.57 tokens per target pass
gamma=4: 3.69 tokens per target pass
gamma=8: 5.14 tokens per target pass

alpha (mean over contexts) = 0.853
gamma=1: Equation (1) predicts 1.85
gamma=2: Equation (1) predicts 2.58
gamma=4: Equation (1) predicts 3.73
gamma=8: Equation (1) predicts 5.18

The first table is the guarantee: over 200,000 samples, the token picked by speculative sampling has the same frequencies as sampling the target directly, even though the draft model’s own distribution is different. The second block is the payoff: with four drafts per step, each run of the big model produced 3.69 tokens on average instead of 1.

How much faster does it get?

The paper gives a closed form. Call α the acceptance rate, the expected value of the sum of min(p, q) over the vocabulary, which measures how much the two models overlap. Assuming acceptances are independent, the expected tokens per target pass is (1 - α^(γ+1)) / (1 - α). My simulation lands within a few percent of that formula; the gap is the independence assumption meeting a toy where some contexts are easier than others.

Tokens per pass is not wall-time speedup, because drafting costs something. With c as the draft model’s cost relative to the target, Theorem 3.8 gives a speedup of (1 - α^(γ+1)) / ((1 - α)(γc + 1)). Large γ only pays when α is high and c is tiny.

The measured numbers from the papers:

  • Leviathan et al. used T5-XXL (11B) as the target and T5-small (77M) as the draft on a single TPU-v4 at batch size 1. They report 3.4x for English-to-German translation and 3.1x for summarization with argmax sampling, and 2.6x and 2.3x at temperature 1. Draft models “a couple of orders of magnitude smaller” than the target gave α between 0.5 and 0.9. Even a trivial bigram model reached α of 0.2 on translation, worth a 1.25x speedup.
  • Chen et al. reported a 2 to 2.5x decoding speedup on Chinchilla 70B in a distributed setup.

Note the temperature effect: greedy decoding speeds up more, because a sharp target distribution is easier to guess.

What are Medusa and EAGLE?

The weak point of the original method is the draft model. You need one that shares the target’s tokenizer, is small enough to be cheap and is aligned enough to be accepted. Later work removes the separate model.

  • Medusa (Cai et al., January 2024) adds extra decoding heads on top of the target model, each predicting a token further ahead, and verifies several candidate continuations at once with tree attention. The paper reports over 2.2x for Medusa-1, which trains heads on a frozen model, and 2.3 to 3.6x for Medusa-2, which fine-tunes the backbone too.
  • EAGLE (Li et al., January 2024) drafts at the feature level, predicting the target’s second-to-top-layer features rather than tokens, and keeps the output distribution unchanged. It reports a 2.7 to 3.5x latency speedup on LLaMA2-Chat 70B, with throughput doubled.
  • N-gram or prompt lookup. Leviathan et al. already noted that a parameter-free drafter that copies tokens from the context can score high α. That’s useful for edits, summaries and code rewrites, where the output repeats the input.

A small draft model is often a distilled version of the target, since distillation pushes the student toward the teacher’s distribution, which is exactly what raises α.

Where is speculative decoding used, and where does it break?

Serving engines ship it. The vLLM docs list EAGLE, multi-token prediction, draft models, n-gram and suffix decoding among the supported methods, and describe the goal as reducing “inter-token latency under medium-to-low QPS (queries per second), memory-bound workloads”. Their configuration looks like this:

vllm serve <target-model> \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

The phrase “medium-to-low QPS” is the catch. The whole method spends idle compute to save memory reads. Under heavy batching, a server is already using that compute on other users’ requests, so verification work competes with real work. The vLLM comparison table rates a separate draft model as “High gain” at low QPS and “Medium gain” at high QPS, with EAGLE dropping from “High gain” to “Medium to high gain”. More on how vLLM batches in what is vLLM.

Other limits:

  • Compute goes up. Rejected drafts are wasted arithmetic. Theorem 3.11 in the paper quantifies the increase; with low α it’s large.
  • Memory goes up. A draft model needs its own weights and its own KV cache.
  • “Lossless” has fine print. vLLM describes speculative sampling as “theoretically lossless up to the precision limits of hardware numerics”. Floating-point and batch-shape effects can still change individual outputs across runs, and some newer verification modes in the same docs deliberately relax the guarantee. Read which mode you enabled.
  • It helps decode, not prefill. Time to first token is unchanged; speculative decoding only shortens the per-token phase.

What I would do

If I self-host a model for an interactive product with modest concurrency, speculative decoding is one of the first switches I’d test, because it doesn’t touch output quality by design. For workloads that echo the input (code edits, RAG answers that quote sources, structured extraction), I’d try n-gram lookup first since it costs no extra weights, knowing vLLM rates its gain only “Low to medium” at low QPS. For general chat, I’d look for a published EAGLE or draft checkpoint matched to the exact target model.

Then I’d measure three numbers on real traffic: acceptance rate, inter-token latency at my real concurrency, and throughput at peak. If the acceptance rate is low or peak throughput drops, I’d turn it off for batch jobs and keep it for the latency-sensitive path. On hosted APIs this is the provider’s job, and the knobs I control are model choice and how many tokens I ask for; reasoning models that think for thousands of tokens are where faster decode matters most.

FAQ

Does speculative decoding change the model’s output?

No, in the exact form from Leviathan et al. and Chen et al. The acceptance rule guarantees samples follow the target model’s distribution. Implementations can differ slightly from run to run because of floating-point numerics, and some engines offer relaxed, lossy verification modes as options.

What is a good draft model for speculative decoding?

A much smaller model with the same tokenizer and a similar output distribution, often from the same family or distilled from the target. Leviathan et al. got their best result with T5-small (77M) drafting for T5-XXL (11B).

How many tokens should the draft model guess?

It depends on the acceptance rate α and the draft’s cost. Higher α supports more guesses. vLLM’s examples use 3 or 4 speculative tokens; tune it by measuring latency on your traffic.

Does speculative decoding help with high-throughput serving?

Less. It trades spare compute for fewer memory-bound steps, and heavy batching already uses that compute. vLLM positions it for medium-to-low QPS workloads.

What is the difference between speculative decoding and Medusa?

Classic speculative decoding uses a separate draft model. Medusa adds extra prediction heads to the target model itself and verifies a tree of candidates, so there’s no second model to host.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.