What is FlashAttention? Exact attention, less memory traffic
FlashAttention computes exact attention in tiles that stay in fast GPU SRAM, so the N x N score matrix never hits HBM. How tiling works, with Python.

FlashAttention is a GPU algorithm that computes exactly the same result as standard transformer attention, but in small tiles that stay in the GPU’s fast on-chip memory, so the full N by N matrix of attention scores is never written to the GPU’s main memory. It doesn’t approximate anything; it reorders the work to cut memory traffic, which makes attention faster and makes its memory use grow linearly with sequence length instead of quadratically.
I ran the code below on 2026-10-31 with Python 3.13.11 and no packages.
Why is standard attention slow?
Not because of the arithmetic. Self-attention for a sequence of N tokens computes an N by N matrix of scores (every query against every key), applies softmax row by row, and multiplies by the values. A straightforward implementation runs that as separate GPU kernels, and each one reads its input from GPU main memory and writes its output back.
The FlashAttention paper (Dao, Fu, Ermon, Rudra and Ré, May 2022) frames the problem with numbers for an A100: 40 to 80 GB of high bandwidth memory (HBM) at 1.5 to 2.0 TB/s, and 192 KB of on-chip SRAM per streaming multiprocessor across 108 of them, at an estimated 19 TB/s. SRAM is an order of magnitude faster and many orders of magnitude smaller. Softmax is an elementwise and reduction operation, which the paper classes as memory-bound: the time goes into moving data, not computing on it.
So the slow part is writing an N by N matrix to HBM, reading it back for the softmax, writing the result, and reading it again for the multiply with V. At 32k tokens, that’s a billion scores per head.
How does FlashAttention work?
The paper’s key word is IO-aware: count reads and writes between memory levels as well as FLOPs. Two techniques do the work.
- Tiling. Split Q, K and V into blocks small enough for SRAM. For each block of queries, stream through the blocks of keys and values, computing a partial result. The full score matrix never exists.
- Recomputation. Training needs attention weights in the backward pass. Instead of storing them, FlashAttention stores only the softmax normalization statistics from the forward pass and recomputes attention on-chip in the backward pass. The paper says that’s faster than reading the stored matrix from HBM.
All of it runs in one fused CUDA kernel. The result uses memory linear in sequence length, and the paper proves it needs O(N² d² / M) HBM accesses, where d is the head size and M the SRAM size, against Ω(Nd + N²) for standard attention.
The hard part is the softmax. Softmax needs the maximum and the sum over a whole row before it can normalize anything, and tiling means you never see the whole row at once. The fix is an online softmax: keep a running maximum and running sum, and when a new block brings a bigger maximum, rescale everything you’ve accumulated so far. The paper cites Milakov and Gimelshein’s work on online softmax normalization for this.
What does the tiling trick look like in code?
Here is that online softmax for one query against 4,096 keys, processed in blocks of 128, compared against plain attention.
# flash.py: the tiling trick behind FlashAttention, for one query row.
# Python 3 standard library only. Real FlashAttention is a fused GPU kernel; this is the math.
import math, random
random.seed(0)
d, N, BLOCK = 64, 4096, 128
q = [random.gauss(0, 1) for _ in range(d)]
K = [[random.gauss(0, 1) for _ in range(d)] for _ in range(N)]
V = [[random.gauss(0, 1) for _ in range(d)] for _ in range(N)]
score = lambda k: sum(a * b for a, b in zip(q, k)) / math.sqrt(d)
# Standard attention: materialize all N scores, softmax, then weight V.
s = [score(k) for k in K] # N numbers held at once (N x N for all queries)
m = max(s); p = [math.exp(x - m) for x in s]; z = sum(p)
ref = [sum(pi * v[j] for pi, v in zip(p, V)) / z for j in range(d)]
# Tiled attention with an online softmax: one block of keys at a time.
run_max, run_sum, acc = -math.inf, 0.0, [0.0] * d
peak_scores = 0
for start in range(0, N, BLOCK):
sb = [score(k) for k in K[start:start + BLOCK]] # only BLOCK scores in "SRAM"
peak_scores = max(peak_scores, len(sb))
new_max = max(run_max, max(sb))
scale = math.exp(run_max - new_max) # rescale what we already summed
pb = [math.exp(x - new_max) for x in sb]
run_sum = run_sum * scale + sum(pb)
acc = [a * scale + sum(pi * v[j] for pi, v in zip(pb, V[start:start + BLOCK]))
for j, a in enumerate(acc)]
run_max = new_max
out = [a / run_sum for a in acc]
print(f"keys: {N}, block: {BLOCK}, blocks: {N // BLOCK}")
print(f"scores held at once standard: {len(s)} tiled: {peak_scores}")
print(f"max |difference|: {max(abs(a - b) for a, b in zip(ref, out)):.2e}")
# The memory that disappears, for a full N x N score matrix in fp16, per head:
for n in (4_096, 32_768, 131_072):
print(f"N = {n:>7,}: score matrix {n * n * 2 / 2**30:8.2f} GiB per head per sequence")
keys: 4096, block: 128, blocks: 32
scores held at once standard: 4096 tiled: 128
max |difference|: 2.08e-17
N = 4,096: score matrix 0.03 GiB per head per sequence
N = 32,768: score matrix 2.00 GiB per head per sequence
N = 131,072: score matrix 32.00 GiB per head per sequence
The tiled version never holds more than 128 scores, and the answer matches standard attention to 2e-17, which is floating-point rounding. The last three lines show what tiling avoids: a 16-bit score matrix of 2 GiB per head at 32k tokens, 32 GiB at 128k. Multiply by heads and layers and you can see why long context was impractical with naive attention.
Python can’t show the speedup, because Python has no SRAM to manage. The speed comes from a CUDA kernel that keeps each tile in shared memory and registers. The script shows the part that makes it possible: exact softmax without the full row.
How much faster is FlashAttention?
From the papers, each on its own hardware and baselines:
- FlashAttention (2022): 15% faster end-to-end BERT-large training than the MLPerf 1.1 record, 3x on GPT-2 at 1K sequence length, and up to 7.6x on the attention computation for GPT-2 over the PyTorch implementation. It also gave the first transformers to beat chance on the Path-X (16K) and Path-256 (64K) long-range tasks.
- FlashAttention-2 (Tri Dao, July 2023): the original reached only 25 to 40% of the A100’s theoretical FLOPs/s. Better work partitioning across thread blocks and warps gave “around 2x speedup” and 50 to 73% of peak, up to 225 TFLOPs/s per A100 in end-to-end GPT training.
- FlashAttention-3 (Shah et al., July 2024): built for Hopper GPUs, where FlashAttention-2 hit only 35% utilization on H100. Using the H100’s asynchronous Tensor Cores and TMA plus FP8, it reports 1.5 to 2.0x speedup, up to 740 TFLOPs/s in FP16 and close to 1.2 PFLOPs/s in FP8.
Each version is a response to new hardware. The algorithm stays exact; the engineering moves to wherever the bottleneck went.
FlashAttention vs sparse attention: what’s the difference?
FlashAttention computes full, exact attention and makes it cheaper in memory traffic. The arithmetic is still quadratic in sequence length. Sparse attention changes the math: each query attends to a subset of keys, so the arithmetic itself shrinks, at the cost of being an approximation the model has to be trained for.
They combine. The FlashAttention paper itself extends the method to block-sparse attention and reports it faster than every approximate attention method they compared. In practice, a long-context model can use sparse or sliding-window patterns and still run each computed block with a FlashAttention-style kernel.
FlashAttention also isn’t the KV cache. The KV cache stores keys and values across decoding steps; FlashAttention avoids storing scores within one attention computation. During decoding, with one new query per step, the bottleneck is reading the KV cache, and serving engines like vLLM pair cache management with attention kernels built for that case.
How do you use FlashAttention?
Mostly you already are. PyTorch’s scaled_dot_product_attention lists three implementations it can dispatch to, and FlashAttention-2 is the first one listed, next to a memory-efficient kernel and a C++ fallback. Transformers models choose a backend through attn_implementation, and inference engines ship their own fused kernels.
What you control is whether the fast path gets used. The PyTorch docs say the function picks “the most optimal implementation based on the inputs”, that fused kernels apply to the CUDA backend and “each of the fused kernels has specific input limitations”, and that other backends get the plain PyTorch implementation. When a fused kernel can’t run, you get a warning with the reasons, not an error. You can force or forbid a backend with the torch.nn.attention.sdpa_kernel() context manager. The output is the same up to rounding; the speed and memory aren’t.
What I would do
I’d never write attention by hand for production. I’d call the framework’s fused attention function and then verify the fast kernel is actually selected, by profiling one forward pass or pinning the backend with sdpa_kernel() in a test, because a fallback shows up as a log warning and “the GPU is slow”, not as a failure.
When I estimate memory for long context, I’d count the KV cache and activations, and I wouldn’t budget for an N by N score matrix, since no modern stack materializes one. That one assumption is the difference between “128k won’t fit” and “128k fits”, and it’s why the long-context models I covered in Gemini 1.5 and its long context became possible. Whether the model uses all that context window well is a separate question.
FAQ
Is FlashAttention an approximation?
No. It computes exact attention; the output matches standard attention up to floating-point rounding, as the script above shows. The approximate variant in the paper is block-sparse FlashAttention, which is a separate option.
Does FlashAttention reduce the KV cache size?
No. It removes the N by N score matrix from memory. The KV cache is a separate structure that holds keys and values across decoding steps, and it shrinks with techniques like grouped-query attention or quantized caches instead.
Does FlashAttention work on Apple Silicon or CPUs?
The FlashAttention papers target NVIDIA GPUs (A100 for v1 and v2, H100 for v3), and PyTorch’s docs say its fused kernels apply to the CUDA backend, with other backends using the plain implementation. The tiling and online softmax idea is general, so other runtimes can ship their own fused kernels; check the one you use.
What is the difference between FlashAttention-2 and FlashAttention-3?
FlashAttention-2 improved how work is divided across GPU thread blocks and warps, about 2x faster than v1. FlashAttention-3 targets Hopper GPUs, overlapping data movement and computation asynchronously and adding FP8 support, for 1.5 to 2.0x over v2 on H100.
Sources
- Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, May 27, 2022
- Tri Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, July 17, 2023
- Shah et al., FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision, July 11, 2024
- PyTorch, torch.nn.functional.scaled_dot_product_attention (read 2026-10-31)