Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is the KV cache? How LLMs avoid recomputing the past

The KV cache stores each token's attention keys and values so an LLM doesn't recompute them per new token. How it works, how big it gets, and GQA.

What is the KV cache? How LLMs avoid recomputing the past

The KV cache is the memory where a transformer stores the attention keys and values it has already computed for every earlier token, so that generating the next token only needs keys and values for that one new token. It turns each decoding step from “reprocess the whole sequence” into “process one token and look up the rest”, and it’s often the largest memory cost in LLM serving after the weights themselves.

I ran the snippets below on 2026-10-31 with Python 3.13.11 and no packages.

Why does an LLM need a KV cache?

Because generation is a loop. A model writes one token, appends it, and runs again to write the next. In self-attention, every token produces three vectors: a query, a key and a value. The new token’s query is compared against the keys of all earlier tokens, and the resulting weights mix their values.

The keys and values of earlier tokens don’t change when a new token arrives, since a decoder only looks backwards. So recomputing them on every step is pure waste. The KV cache keeps them in GPU memory, one key and one value per token, per layer, per KV head.

Here is that claim tested on a toy single-head attention: decode 64 tokens with and without a cache and compare.

# kv_cache.py: why the KV cache exists, and how big it gets.
# Python 3 standard library only.
import math, random

# 1. Decoding with and without a cache gives the same answer; the cache skips recomputation.
random.seed(1)
d = 8
Wq, Wk, Wv = ([[random.gauss(0, 0.3) for _ in range(d)] for _ in range(d)] for _ in range(3))
matvec = lambda W, x: [sum(w * v for w, v in zip(row, x)) for row in W]

def attend(q, K, V):
    s = [sum(a * b for a, b in zip(q, k)) / math.sqrt(d) for k in K]
    m = max(s); w = [math.exp(x - m) for x in s]; z = sum(w)
    return [sum(wi * v[j] for wi, v in zip(w, V)) / z for j in range(d)]

tokens = [[random.gauss(0, 1) for _ in range(d)] for _ in range(64)]
no_cache_projections = cache_projections = 0
K_cache, V_cache = [], []
max_diff = 0.0
for t in range(1, len(tokens) + 1):
    # without a cache: recompute K and V for every earlier token at every step
    K = [matvec(Wk, x) for x in tokens[:t]]; V = [matvec(Wv, x) for x in tokens[:t]]
    no_cache_projections += 2 * t
    out_a = attend(matvec(Wq, tokens[t - 1]), K, V)
    # with a cache: compute K and V once for the new token, append, reuse the rest
    K_cache.append(matvec(Wk, tokens[t - 1])); V_cache.append(matvec(Wv, tokens[t - 1]))
    cache_projections += 2
    out_b = attend(matvec(Wq, tokens[t - 1]), K_cache, V_cache)
    max_diff = max(max_diff, max(abs(a - b) for a, b in zip(out_a, out_b)))
print(f"64 tokens: K/V projections without cache {no_cache_projections}, with cache {cache_projections}")
print(f"max difference in outputs: {max_diff}")
print()

# 2. Bytes of KV cache per token = 2 (K and V) x layers x kv_heads x head_dim x bytes per value.
def per_token(layers, kv_heads, head_dim, bytes_per=2):
    return 2 * layers * kv_heads * head_dim * bytes_per

models = {  # (layers, kv_heads, head_dim) from each model's paper
    "OPT-13B (MHA, 40 heads)":     (40, 40, 128),
    "Mistral 7B as if MHA (32)":   (32, 32, 128),
    "Mistral 7B (GQA, 8 KV)":      (32, 8, 128),
    "Llama 3 405B (GQA, 8 KV)":    (126, 8, 128),
}
print(f"{'model':<27}{'KiB/token':>10}{'GiB @ 32k ctx':>15}{'GiB @ 32k x 16 users':>22}")
for name, cfg in models.items():
    b = per_token(*cfg)
    print(f"{name:<27}{b / 1024:>10.0f}{b * 32768 / 2**30:>15.1f}{b * 32768 * 16 / 2**30:>22.1f}")
64 tokens: K/V projections without cache 4160, with cache 128
max difference in outputs: 0.0
model                       KiB/token  GiB @ 32k ctx  GiB @ 32k x 16 users
OPT-13B (MHA, 40 heads)           800           25.0                 400.0
Mistral 7B as if MHA (32)         512           16.0                 256.0
Mistral 7B (GQA, 8 KV)            128            4.0                  64.0
Llama 3 405B (GQA, 8 KV)          504           15.8                 252.0

The outputs match exactly, and the cached loop did 128 key/value projections instead of 4,160. Without a cache, the work per step grows with the sequence, so total work is quadratic; with it, each step computes K and V for one token.

How big is the KV cache?

The formula is in the second half of the script: 2 (one key, one value) times layers times KV heads times head size times bytes per number. At 16-bit precision that’s 2 bytes.

The vLLM paper (Kwon et al., SOSP 2023) gives the same calculation for OPT-13B: 2 × 5120 × 40 × 2 bytes, “800 KB” per token, and up to 1.6 GB for one 2,048-token request. My first row reproduces their 800. OPT-13B never supported 32k tokens; I kept it in the table to show the formula, not a real deployment.

The modern rows use configs from the papers. Mistral 7B has 32 layers, head size 128 and 8 KV heads: 128 KiB per token, 4 GiB at a 32k context. Llama 3 405B has 126 layers and 8 KV heads: 504 KiB per token. Multiply by concurrent users and the cache outgrows a GPU fast. That’s why the context window you can actually serve is often set by cache memory, not by what the model was trained on.

How do GQA and MQA shrink the KV cache?

By sharing keys and values across query heads. In standard multi-head attention (MHA), each head has its own K and V. Noam Shazeer’s 2019 paper “One Write-Head is All You Need” proposed multi-query attention (MQA): all heads share one K and one V, which cuts the cache by the number of heads. He framed the problem exactly right: incremental decoding is slow because it keeps reloading “large key and value tensors”, so it’s bound by memory bandwidth.

MQA can cost quality. Grouped-query attention (GQA, Ainslie et al., 2023) is the middle ground: a few KV heads, each shared by a group of query heads. The paper reports quality “close to multi-head attention” at speed comparable to MQA. Mistral 7B has 32 query heads and 8 KV heads, so each KV head serves 4 queries. That’s the 4x gap between rows two and three of my table. The Llama 2 paper used GQA for its 34B and 70B models, and Llama 3 uses it with 8 KV heads at every size.

Two other tricks attack the same number. Mistral 7B’s sliding-window attention keeps only the last 4,096 tokens per layer in a rolling buffer, which the paper says “reduces the cache memory usage by 8x” on a 32k sequence. DeepSeek-V2 compresses keys and values into a small latent vector, which it calls Multi-head Latent Attention, and reports a KV cache 93.3% smaller than DeepSeek 67B’s.

How does the KV cache change serving?

It splits a request into two phases with different costs, which I’ll go into in LLM latency explained:

  • Prefill. The whole prompt goes through the model in one parallel pass, filling the cache. This is compute-heavy and sets time to first token.
  • Decode. One token at a time, each step reading the full cache and all the weights. This is memory-bandwidth-bound and sets tokens per second.

The cache is also what serving engines manage. The vLLM paper found that existing systems used only “20.4% - 38.2%” of KV cache memory for actual token states, because they reserved contiguous space for the longest possible output. Their fix, PagedAttention, stores the cache in fixed-size blocks like virtual memory pages, reaching “near-zero waste” and 2 to 4x higher throughput. I’ll cover the engine itself in what is vLLM.

And it’s the mechanism behind prompt caching. If many requests share a prefix, such as a long system prompt, a provider can keep that prefix’s KV cache and skip its prefill on the next request. That’s why cached input tokens are billed cheaper: the expensive part of reading them has already happened.

Where does the KV cache break?

  • It grows linearly with context. Long conversations and agent loops that resend history keep the cache growing. Ten users at 100k tokens can need more memory than the model.
  • Batching fights it. More users per GPU means more caches. Serving throughput is often a direct function of how many caches fit.
  • Quantizing it has a cost. vLLM can store the cache in FP8 (kv_cache_dtype="fp8"), half the bytes of 16-bit. Its docs recommend calibrating scales for accuracy and note that some layer types, such as sliding-window attention, are more sensitive. Test it on your own evals.
  • It’s per model. A cache from one model or one version can’t be reused by another, which is why prompt caches are scoped to a model.

FlashAttention is a different fix for a different memory problem: it avoids storing the attention score matrix, while the KV cache stores keys and values across decoding steps. Production stacks use both.

What I would do

Before picking a model to self-host, I compute KV bytes per token from its config file with the formula above and multiply by the context length and the number of concurrent requests I expect. That number, plus the weights, is the real memory budget. I’d do this before reading any benchmark.

When I build agents, I keep stable content (system prompt, tool definitions) at the front and the changing content at the end. That keeps the prefix cacheable, on both hosted APIs and engines like vLLM with prefix caching. And I’d treat “1M token context” claims as a statement about the model, not about what a given deployment can afford to cache for every user.

FAQ

What is stored in the KV cache?

The key and value vectors that each attention layer computed for every token seen so far, for every KV head. Queries aren’t cached, because only the newest token’s query is needed at each step.

How much memory does the KV cache use?

2 × layers × KV heads × head size × bytes per value, per token. For Mistral 7B at 16-bit that’s 128 KiB per token, or 4 GiB for a 32k-token sequence.

Does the KV cache affect output quality?

An exact cache doesn’t: my test above shows identical outputs. Lossy variants, such as quantized caches, evicting old tokens or sliding windows, can change outputs, and each model has to be evaluated with them.

What is the difference between the KV cache and prompt caching?

The KV cache lives inside one generation, so the model doesn’t recompute earlier tokens. Prompt caching keeps the KV cache of a shared prefix across separate requests, so the provider can skip prefill and charge less for those tokens.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.