Malhar Ujawane

Blog

Post·

What is sparse attention? DeepSeek V3.2 explained

Sparse attention lets each token attend to a few chosen earlier tokens. How DeepSeek V3.2 does it, why long prompts get cheaper, and a runnable numpy demo.

What is sparse attention? DeepSeek V3.2 explained

Sparse attention lets each token look at a chosen subset of earlier tokens instead of every one of them. DeepSeek released DeepSeek-V3.2 on December 1, 2025, and its main architectural change is exactly this: DeepSeek Sparse Attention (DSA), where each token attends to 2,048 selected tokens out of a context of up to 128K.

Why does sparse attention matter to a builder?

Because long prompts are where the money goes. Agents resend tool results, files and chat history on every turn, so most of what a model reads is old context. In ordinary (dense) attention, every new token compares itself against every token before it, which makes the work for a whole prompt grow with the square of its length. Double the prompt and attention does about four times the work.

DeepSeek’s answer shows up on its price list. When it first shipped DSA in V3.2-Exp on September 29, 2025, it cut API prices by more than 50%. V3.2 keeps the same prices. As of December 3, 2025, deepseek-chat and deepseek-reasoner (both V3.2) cost $0.28 per million input tokens on a cache miss, $0.028 on a cache hit and $0.42 per million output tokens, with a 128K context. A 100,000-token prompt on a cache miss costs $0.028. In my context window post, the same prompt on Gemini 3 Pro cost $0.20.

Price is a business decision, so I can’t prove how much of that gap DSA explains. The tech report does say why DeepSeek thinks it can afford it, and the mechanism is small enough to build.

What does sparse attention look like in code?

Here is attention from scratch in numpy, then the sparse version. One query looks back over 4,096 earlier tokens, and I plant 40 of them as relevant, the way a variable name or a function definition matters to one later line and most of the file doesn’t. A cheap scorer (16 dimensions instead of 64) picks the top k tokens, and real attention runs only on those.

# sparse_attention.py: dense attention vs top-k sparse attention, from scratch.
# pip install numpy==2.3.5   (Python 3.13)
import numpy as np

rng = np.random.default_rng(0)
L, d = 4096, 64                        # 4,096 earlier tokens, 64-dim keys and values
K = rng.standard_normal((L, d))
V = rng.standard_normal((L, d))
q = rng.standard_normal(d)
# Make 40 earlier tokens relevant to this query, the way a name or a function
# definition matters to one later token and most of the context doesn't.
rel = rng.choice(L, 40, replace=False)
K[rel] += 1.0 * q

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

def attend(idx):                      # ordinary attention over the chosen keys only
    w = softmax(K[idx] @ q / np.sqrt(d))
    return w @ V[idx]

dense = attend(np.arange(L))          # every key: L dot products of size d

# A cheap "indexer": score every key in a 16-dim projection, keep the top k.
P = rng.standard_normal((d, 16)) / np.sqrt(16)
cheap = (K @ P) @ (P.T @ q)

exact = K @ q                         # what a perfect indexer would rank by
def error(idx):
    return np.linalg.norm(attend(idx) - dense) / np.linalg.norm(dense)

print(f"{'k':>5} {'relevant kept':>14} {'error, cheap':>13} {'error, exact':>13}")
for k in (64, 256, 1024):
    top = np.argsort(cheap)[-k:]
    best = np.argsort(exact)[-k:]
    print(f"{k:>5} {len(set(top) & set(rel)):>9} / 40 {error(top):>13.3f} {error(best):>13.3f}")
print()

# Scale: query-key pairs scored by the main attention over a full prompt.
k = 2048                              # tokens DeepSeek-V3.2 keeps per query
for n in (8_192, 32_768, 131_072):
    dense_pairs = n * (n + 1) // 2
    sparse_pairs = sum(min(t, k) for t in range(1, n + 1))
    print(f"{n:>7,} tokens  dense {dense_pairs:>14,}  top-{k} {sparse_pairs:>12,}  "
          f"{dense_pairs / sparse_pairs:>5.1f}x fewer")

Output (Python 3.13.5, numpy 2.3.5):

    k  relevant kept  error, cheap  error, exact
   64        35 / 40         0.380         0.307
  256        39 / 40         0.310         0.240
 1024        40 / 40         0.187         0.126

  8,192 tokens  dense     33,558,528  top-2048   14,681,088    2.3x fewer
 32,768 tokens  dense    536,887,296  top-2048   65,012,736    8.3x fewer
131,072 tokens  dense  8,590,000,128  top-2048  266,339,328   32.3x fewer

Two things stand out. In the first table, a crude random projection finds 35 of the 40 planted tokens while keeping only 64 of 4,096. The error column is the honest part: even with a perfect ranking, dropping the long tail moves the output (0.126 relative error at k = 1,024), because in this toy the 4,000 unimportant tokens still carry some weight together. DeepSeek trains its indexer and then trains the whole model with the selection in place, so the model learns to live with what it can’t see. My toy does neither, and its error numbers say nothing about V3.2’s quality.

The second table is pure counting. At 8K tokens a 2,048-token budget saves little, since most tokens have fewer than 2,048 predecessors anyway. At 128K the main attention scores 32 times fewer pairs. Short prompts gain almost nothing from sparse attention; long ones gain the most.

Two triangular attention grids side by side. The left grid, labelled dense, is fully shaded in black. The right grid, labelled top-k, has a few scattered orange cells per row. Below, a bar chart shows 8.6 billion query-key pairs for a dense 131,072-token prompt against 266 million for top-2,048, 32.3 times fewer.

How does DeepSeek Sparse Attention work?

Start with dense attention. For each new token, the model makes a query vector, compares it with a key vector for every earlier token, turns those scores into weights with a softmax, and takes a weighted average of the earlier tokens’ value vectors. That comparison step is the expensive one: token number 100,000 does 100,000 comparisons, and the original transformer paper lists self-attention’s cost per layer as quadratic in sequence length.

The idea of skipping most comparisons is old. OpenAI’s Sparse Transformers (2019) used fixed patterns, such as attending to nearby tokens plus every n-th one. A fixed pattern is cheap but blind: it can’t know that the function you need is 60,000 tokens back.

DSA picks tokens by content instead, in two steps, as described in the DeepSeek-V3.2 tech report:

  1. A lightning indexer scores every earlier token for the current one. It uses a few small heads with a ReLU, and DeepSeek says it can run in FP8, so each score is far cheaper than a full attention comparison.
  2. A top-k selector keeps the key-value entries with the highest index scores (2,048 per query token in training), and the main attention runs only over those.

The report puts the result plainly: core attention drops from O(L²) to O(Lk), where k is much smaller than L. It also admits the catch. The indexer still looks at every earlier token, so it remains O(L²); it’s just much cheaper per step than the attention it replaces. And for short prompts DeepSeek uses a masked mode that simulates DSA, because selection overhead isn’t worth paying when there’s little to skip.

The training detail matters as much as the mechanism. V3.2 started from V3.1-Terminus, which already had a 128K context. DeepSeek first warmed up the indexer alone, teaching it to match the dense model’s attention, then continued pre-training the whole model with sparse selection switched on. The report’s cost curves come from DeepSeek’s own service on H800 GPUs, priced at $2 per GPU hour, so they are DeepSeek’s estimates for its own setup.

Where does sparse attention break, and what would I watch?

Retrieval misses. If the indexer doesn’t rank a token in the top 2,048, the main attention never sees it. DeepSeek reports that V3.2-Exp did not regress on long-context evaluations such as AA-LCR, which is encouraging, but I’d still test my own needle-in-a-long-prompt cases (a config value deep in a log, a clause on page 80) before trusting it with them.

Short prompts. Under a few thousand tokens there’s little to skip, as the 8K row shows. If your requests are short, sparse attention changes nothing for you except through the price list.

The context ceiling. DSA makes a 128K prompt cheaper, and the limit stays at 128K. The tech report says more than 20% of the test cases in its search-agent evaluation exceeded the 128K limit and needed context management. Agents that dump whole tool outputs into the prompt still need trimming. I wrote about one way to do that in code execution with MCP.

Thinking tokens. deepseek-reasoner writes up to 64K output tokens. Sparse attention makes reading cheaper, but output still costs $0.42 per million, and long reasoning traces add up, as I covered in what reasoning models are.

FAQ

Is sparse attention the same as a smaller context window?

No. The model still accepts the full 128K tokens, and the indexer looks at all of them. Each token then attends in full detail to only 2,048 of them. A smaller window would drop old tokens entirely, whatever they contain.

Does sparse attention make outputs worse?

It can, if the indexer drops something the answer needs. DeepSeek reports V3.2-Exp performing on par with V3.1-Terminus on its benchmarks, and the V3.2-Exp announcement claims “minimal impact on output quality”. Those are DeepSeek’s own numbers; run your own long-context cases.

Can I run DeepSeek-V3.2 myself?

The weights are on Hugging Face under the MIT license, along with an open-source inference implementation. The checkpoint holds about 685 billion parameters (a mixture-of-experts model), so self-hosting means a multi-GPU server; for most builders the API is the practical route.

What is DeepSeek-V3.2-Speciale?

A higher-reasoning variant released the same day, available only through a temporary API endpoint until December 15, 2025, with no tool calls and the same price as V3.2. DeepSeek says it uses more tokens.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.