Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is attention in transformers? Query, key, value explained

Attention lets each token in a transformer weigh every other token. How query, key and value vectors work, why the sqrt(d_k) is there, with runnable Python.

What is attention in transformers? Query, key, value explained

Attention is the operation in a transformer that lets each token build its new representation from a weighted mix of other tokens. Every token produces a query (what it is looking for), a key (what it offers) and a value (what it passes on); the dot product of a query with each key, scaled and run through a softmax, gives the weights for mixing the values. In a language model the mix is masked so a token can only look at itself and earlier tokens.

Why do transformers need attention?

A word’s meaning depends on other words. In “the cat sat on the mat because it was tired”, the model has to connect “it” to “cat” before it can predict anything sensible. Older recurrent networks passed information along one step at a time, and Bahdanau, Cho and Bengio’s 2014 translation paper argued that compressing a sentence into one fixed-length vector “is a bottleneck”. Their fix was to let the decoder “(soft-)search for parts of a source sentence” at every step: attention.

The 2017 paper Attention Is All You Need went further and built the whole network from it. In self-attention, every token attends to every other token in the same sequence, in one step, so the path between any two positions has constant length. That’s what made transformers trainable in parallel and good at long-range dependencies.

How do query, key and value work?

Each token arrives as a vector (its embedding plus whatever earlier layers added). Three learned weight matrices project it into three new vectors:

  • Query (Q): what this token is looking for.
  • Key (K): what this token can be found by.
  • Value (V): the information this token hands over if someone attends to it.

The paper’s formula is the whole mechanism:

Attention(Q, K, V) = softmax(Q Kᵀ / sqrt(d_k)) V

Read it left to right. Q Kᵀ is a table of dot products: row i, column j says how well token i’s query matches token j’s key. Dividing by sqrt(d_k) keeps those numbers in a sane range (more on that below). The softmax turns each row into weights that sum to 1. Multiplying by V gives each token a weighted average of the values it attended to.

The database analogy helps if you don’t push it too far. A query is matched against keys to fetch values, but the match is soft: every key gets some weight, and the result is a blend, not one row.

What does attention look like in code?

Here is scaled dot-product attention with lists and loops, on four tokens. I hand-wrote the vectors so that the query for “it” lines up with the key for “cat”. In a real model, training finds weight matrices that produce vectors like these.

# attention.py: scaled dot-product attention by hand, then why the 1/sqrt(d_k) matters.
# Python 3, stdlib only.
import math, random

def softmax(xs):
    m = max(xs)
    e = [math.exp(x - m) for x in xs]
    return [v / sum(e) for v in e]

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def attention(Q, K, V, causal=False):
    d_k = len(K[0])
    out, weights = [], []
    for i, q in enumerate(Q):
        scores = [dot(q, k) / math.sqrt(d_k) for k in K]
        if causal:                                   # token i may not look at tokens after i
            scores = [s if j <= i else -math.inf for j, s in enumerate(scores)]
        w = softmax(scores)
        weights.append(w)
        out.append([sum(w[j] * V[j][c] for j in range(len(V))) for c in range(len(V[0]))])
    return out, weights

# Four tokens, d_k = 4. Hand-made so "it" (row 3) has a query that matches "cat"'s key.
tokens = ["the", "cat", "sat", "it"]
Q = [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 3, 0, 0]]
K = [[1, 0, 0, 0], [0, 3, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]
V = [[1, 0], [0, 1], [1, 1], [0, 0]]                 # what each token passes on

_, w = attention(Q, K, V, causal=True)
print("causal attention weights (row = token doing the looking)")
print("        " + "".join(f"{t:>7}" for t in tokens))
for t, row in zip(tokens, w):
    print(f"{t:>6}  " + "".join(f"{x:7.2f}" for x in row))

# Why divide by sqrt(d_k)? Dot products of random vectors grow with dimension,
# and a softmax over large numbers collapses onto one key.
random.seed(0)
for d in (4, 64, 512):
    raw_max = scaled_max = 0.0
    for _ in range(200):                             # average over 200 random queries
        q = [random.gauss(0, 1) for _ in range(d)]
        ks = [[random.gauss(0, 1) for _ in range(d)] for _ in range(8)]
        raw_max += max(softmax([dot(q, k) for k in ks])) / 200
        scaled_max += max(softmax([dot(q, k) / math.sqrt(d) for k in ks])) / 200
    print(f"d_k={d:>3}  avg top weight over 8 keys: unscaled {raw_max:.2f}   scaled {scaled_max:.2f}")

Output (Python 3.13.11):

causal attention weights (row = token doing the looking)
            the    cat    sat     it
   the     1.00   0.00   0.00   0.00
   cat     0.18   0.82   0.00   0.00
   sat     0.27   0.27   0.45   0.00
    it     0.01   0.97   0.01   0.01
d_k=  4  avg top weight over 8 keys: unscaled 0.52   scaled 0.34
d_k= 64  avg top weight over 8 keys: unscaled 0.86   scaled 0.35
d_k=512  avg top weight over 8 keys: unscaled 0.96   scaled 0.38

The weight table is the attention pattern. The upper triangle is all zeros: that’s the causal mask, and it’s why “the” can only attend to itself. The last row is the point of the exercise: “it” puts 97% of its weight on “cat”, so its output is almost exactly cat’s value vector. The model has resolved the pronoun by arithmetic.

Why divide by the square root of d_k?

The paper’s reason is in Section 3.2.1: for large d_k the dot products grow large, “pushing the softmax function into regions where it has extremely small gradients”. If query and key entries are random with unit variance, their dot product has variance d_k, so its typical size grows with the square root of the dimension.

The second half of the output shows the effect. Without scaling, the average top weight climbs from 0.52 at d_k = 4 to 0.96 at d_k = 512: the softmax has collapsed onto one key almost every time, for random vectors that carry no signal. A softmax that saturated passes almost no gradient back to the losing keys, so the model can’t learn to shift its attention. With the 1/sqrt(d_k) factor the spread stays about the same at every width. The original base model used d_k = 64 per head.

What is multi-head attention?

One attention pattern per layer is limiting: a token may need to track its subject, the previous token and a matching bracket all at once. Multi-head attention runs several attention operations in parallel, each with its own Q, K and V projections at a smaller width, then concatenates the results. The base Transformer used h = 8 heads with d_k = d_v = 64, so the total cost is about the same as one full-width head.

Heads matter for inference cost, because every key and value vector for every past token has to be kept in memory while generating. That store is the KV cache. Grouped-query attention, from the GQA paper, lets several query heads share one key-value head; the authors report quality close to full multi-head attention at speed comparable to a single shared head. The Llama 3 paper uses GQA with 8 key-value heads “to improve inference speed and to reduce the size of key-value caches during decoding”.

Why is attention expensive?

Look at the Q Kᵀ table again: n tokens produce n × n scores. Table 1 of the transformer paper lists self-attention at O(n²d) per layer. Double the prompt and the attention work quadruples, which is the mechanical reason long prompts are slow and why context windows have prices attached.

There are two lines of work against that cost. One computes exactly the same result with less memory traffic, which is what FlashAttention does. The other changes which pairs get scored at all, which is sparse attention. Both start from the formula above.

What I would do

You don’t need to write attention to use an LLM, but two practical habits follow from understanding it.

First, put the material that matters where the model can match it. Attention is content-based: a token attends to keys that look like what it’s searching for. Clear headings, consistent names and the actual identifiers from your code help the model find the right span in a long prompt more than clever phrasing does.

Second, when you size a self-hosted model, compute the KV cache as well as the weights. Per token it’s 2 (keys and values) × layers × key-value heads × head size × bytes per number. For long contexts and many concurrent users, that number decides your GPU count. From a data-platform seat, this is the same lesson as indexes: the data structure you build to answer queries fast often costs more memory than the data itself.

FAQ

What is the difference between self-attention and cross-attention?

In self-attention, queries, keys and values all come from the same sequence. In cross-attention, queries come from one sequence and keys and values from another; the original transformer’s decoder used it to read the encoder’s output. Decoder-only LLMs use self-attention only.

What are Q, K and V in simple terms?

The query is what a token is looking for, the key is how other tokens advertise themselves, and the value is what they hand over when matched. All three are learned linear projections of the same token vector.

What is masked or causal attention?

The mask is a part of attention, not attention itself. GPT-style models set the scores for future positions to negative infinity before the softmax, so their weights become zero. That keeps generation honest: a token can’t see the tokens it’s being trained to predict.

Does attention show what the model is thinking?

Partly. Attention weights show where information moved in one layer, and they’re useful for debugging. They don’t explain the whole computation, since feed-forward layers and dozens of stacked layers transform that information further.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.