Malhar Ujawane

Blog

Post·

What is quantization in LLMs? Bits, scales and FP4

Quantization stores model weights in fewer bits plus a scale. Runnable int8, int4, MXFP4 and NVFP4 code, the memory math, and where accuracy goes.

What is quantization in LLMs? Bits, scales and FP4

Quantization stores a model’s numbers, usually its weights, in fewer bits than it was trained with, plus a scale factor that maps those small integers or tiny floats back to real values. A 4-bit model needs about a quarter of the memory of the same model in 16 bits, at the cost of some rounding error.

What shipped this week?

At CES on January 5, 2026, NVIDIA launched the Rubin platform: six chips, including the Rubin GPU and the Vera CPU. NVIDIA says Rubin is in full production, with partner products from the second half of 2026. The headline GPU number is “50 petaflops of NVFP4 compute for AI inference”, and NVIDIA claims up to 10x lower inference cost per token than Blackwell. That’s the vendor’s claim, measured its way. Cost per token also isn’t the bill you pay: what matters is cost per task, which depends on how many tokens each task uses.

What caught my eye is the unit. The flagship spec of NVIDIA’s next data center GPU is quoted in a 4-bit number format, which tells you where serving is heading. If you self-host open-weight models, or even just read model cards, you’ll keep running into INT4, MXFP4 and NVFP4, so here’s what they are from first principles.

Why does quantization matter to a builder?

Memory first. A model’s weights take parameters × bits per weight ÷ 8 bytes, so a 70B model in BF16 needs 140 GB before you store a single token of context. Speed follows from memory: when a model generates text one token at a time, each step has to read the weights it uses from GPU memory, so fewer bytes per weight means less data to move per token. That matters most for reasoning models, which generate long runs of thinking tokens before they answer.

You’ve probably used a quantized model already. OpenAI’s gpt-oss models (August 2025) ship “natively quantized in MXFP4”, which is how OpenAI gets gpt-oss-120b to run within 80 GB of memory and gpt-oss-20b within 16 GB.

What does quantization look like in code?

Here’s the whole idea in one script: take a made-up 4096 × 4096 weight matrix, squeeze it five ways, and measure the error. I made 0.1% of the weights 20 times larger than the rest, because real models have outliers like that, and I also report the error on the same matrix without them.

# quantize.py: five ways to squeeze one weight matrix, and what each costs in error.
# Python 3.13, numpy==2.3.5, ml_dtypes==0.5.3 (for the real FP4 E2M1 and FP8 E4M3 types).
import numpy as np
import ml_dtypes as mld

rng = np.random.default_rng(0)
W0 = rng.normal(0, 0.02, size=(4096, 4096)).astype(np.float32)  # a made-up layer
W = W0.copy()
idx = rng.choice(W.size, size=W.size // 1000, replace=False)     # 0.1% outliers, 20x larger
W.flat[idx] *= 20

def fp4(v):   # round to the nearest FP4 E2M1 value: 0, 0.5, 1, 1.5, 2, 3, 4, 6 (and negatives)
    return v.astype(mld.float4_e2m1fn).astype(np.float32)

def int_absmax(w, bits):                          # one scale for the whole tensor
    qmax = 2 ** (bits - 1) - 1
    s = np.abs(w).max() / qmax
    return np.clip(np.round(w / s), -qmax, qmax) * s

def int4_blocks(w, block=32):                     # one FP16 scale per 32 weights
    b = w.reshape(-1, block)
    s = (np.abs(b).max(1, keepdims=True) / 7).astype(np.float16).astype(np.float32)
    return (np.clip(np.round(b / s), -7, 7) * s).reshape(w.shape)

def mxfp4(w, block=32):                           # FP4 values, power-of-two scale per 32
    b = w.reshape(-1, block)
    s = 2.0 ** (np.floor(np.log2(np.abs(b).max(1, keepdims=True))) - 2)
    return (fp4(b / s) * s).reshape(w.shape)

def nvfp4(w, block=16):                           # FP4 values, FP8 E4M3 scale per 16, FP32 per tensor
    g = np.abs(w).max() / (6 * 448)
    b = w.reshape(-1, block)
    s = (np.abs(b).max(1, keepdims=True) / 6 / g).astype(mld.float8_e4m3fn).astype(np.float32) * g
    return (fp4(b / s) * s).reshape(w.shape)

def err(w, wq):                                   # relative error, ||w - wq|| / ||w||
    return np.linalg.norm(w - wq) / np.linalg.norm(w)

print(f"{'format':22} {'bits/weight':>11} {'error':>7} {'no outliers':>12}")
for name, bits, Wq in [("int8, 1 scale", 8.0, lambda w: int_absmax(w, 8)),
                       ("int4, 1 scale", 4.0, lambda w: int_absmax(w, 4)),
                       ("int4, scale per 32", 4 + 16 / 32, int4_blocks),
                       ("MXFP4 (32, 2^k)", 4 + 8 / 32, mxfp4),
                       ("NVFP4 (16, E4M3)", 4 + 8 / 16, nvfp4)]:
    print(f"{name:22} {bits:11.2f} {err(W, Wq(W)):7.1%} {err(W0, Wq(W0)):12.1%}")

print("\nweights only, GB (1e9 bytes)")
print(f"{'params':>8} {'BF16':>7} {'FP8':>7} {'NVFP4':>7} {'MXFP4':>7}")
for p in (8e9, 70e9, 117e9):
    print(f"{p / 1e9:7.0f}B " + " ".join(f"{p * b / 8 / 1e9:7.2f}" for b in (16, 8, 4.5, 4.25)))

# Where the NVFP4 error comes from: a block that holds an outlier rounds its neighbours to zero.
q = nvfp4(W).reshape(-1, 16)
has_out = (np.abs(W.reshape(-1, 16)) > 0.2).any(1)
zeros = (q == 0).mean(1)
print(f"\nNVFP4 weights rounded to 0: {zeros[~has_out].mean():.0%} in normal blocks, "
      f"{zeros[has_out].mean():.0%} in the {has_out.mean():.1%} of blocks with an outlier")

Output (Python 3.13.5, numpy 2.3.5, ml_dtypes 0.5.3):

format                 bits/weight   error  no outliers
int8, 1 scale                 8.00   15.5%         1.2%
int4, 1 scale                 4.00   85.0%        22.1%
int4, scale per 32            4.50   12.4%         9.7%
MXFP4 (32, 2^k)               4.25   13.7%        11.4%
NVFP4 (16, E4M3)              4.50    9.3%         9.5%

weights only, GB (1e9 bytes)
  params    BF16     FP8   NVFP4   MXFP4
      8B   16.00    8.00    4.50    4.25
     70B  140.00   70.00   39.38   37.19
    117B  234.00  117.00   65.81   62.16

NVFP4 weights rounded to 0: 7% in normal blocks, 58% in the 1.0% of blocks with an outlier

The 117B row uses gpt-oss-120b’s total parameter count from OpenAI’s post. It assumes every weight is stored in the format, which real checkpoints don’t quite do, so treat it as the floor.

How does quantization work?

Start with int8 and one scale for the whole tensor. Find the largest absolute weight, call it amax, and set the scale s = amax / 127. Each weight becomes round(w / s), an integer from -127 to 127, and you get it back as that integer times s. On the clean matrix that costs 1.2% error, which is why 8-bit is the safe default.

Now add the outliers. The largest weight jumps by about 20 times, so the scale does too, and the step between integers becomes so coarse that most normal weights land on a handful of values near zero. Error goes from 1.2% to 15.5% with the same 8 bits. At int4 with one scale there are only 15 levels, and the error hits 85%: almost every normal weight rounds to 0. This is the problem the LLM.int8() paper found in real transformers, where a small number of large-magnitude features break simple 8-bit quantization; its fix was to keep those in 16-bit.

The general fix is to use more scales. Give every block of 32 weights its own scale and an outlier can only spoil its own block. int4 with a 16-bit scale per 32 weights drops to 12.4% error, for 0.5 extra bits per weight.

The 4-bit floating-point formats take this one step further. FP4 E2M1 has 1 sign bit, 2 exponent bits and 1 mantissa bit, so it can hold only 0, 0.5, 1, 1.5, 2, 3, 4 and 6, and their negatives. Those values are packed closer together near zero, which suits weights that cluster around zero. Two block formats use it:

  • MXFP4, from the OCP Microscaling spec, shares one power-of-two scale (8 bits, E8M0) across 32 values: 4.25 bits per weight. It’s the format gpt-oss ships in.
  • NVFP4, which NVIDIA introduced with Blackwell, shares one FP8 E4M3 scale across 16 values and adds one FP32 scale per tensor: 4.5 bits per weight. The E4M3 scale can be fractional, so it can map a block’s largest weight to exactly 6 instead of rounding the scale to a power of two.

Sixteen bars show original weights as outlines and the values NVFP4 stores as solid bars, nearly matching. Below, the stored FP4 codes: 0.5, -0.5, 1.5, 0.5, -1.5, 1, 3, 2, -2, -3, -1.5, 0, -6, -0.5, -3, -2, with -6 highlighted as the largest weight. An orange box shows the block scale 13 in FP8 E4M3 times a tensor scale of 5.998e-04, so weight = code × 0.00780. A strip of sixteen 4-bit cells and one 8-bit cell totals 72 bits, 4.5 bits per weight.

In this block the largest weight is -0.0465. NVFP4 picks a block scale of 13 (stored in FP8) times a tensor scale of 0.0005998, so each weight is its code times 0.0078, and -0.0465 becomes code -6. Every other weight rounds to the nearest of the eight magnitudes.

In my toy, the finer blocks and fractional scale make NVFP4 the best of the 4-bit options at 9.3%, ahead of MXFP4 at 13.7%, which matches the direction of NVIDIA’s argument. The last line of the output shows where the remaining error lives: in the 1% of blocks that hold an outlier, 58% of weights round to zero, against 7% in normal blocks.

Horizontal bar chart of relative error per format, with 0.1% outliers in orange and without them in black. int8 with one scale, 8 bits: 15.5% and 1.2%. int4 with one scale, 4 bits: 85.0% and 22.1%. int4 with a scale per 32, 4.5 bits: 12.4% and 9.7%. MXFP4, 4.25 bits: 13.7% and 11.4%. NVFP4, 4.5 bits: 9.3% and 9.5%.

What gets quantized: weights, activations or the KV cache?

Weight-only quantization stores weights in 4 or 8 bits and converts them back to 16-bit just before each matrix multiply. It saves memory and bandwidth on almost any GPU. GPTQ and AWQ are two well-known methods that pick the rounding more carefully than my nearest-value version.

Quantizing activations as well lets the multiply itself run in low precision, and that’s what a figure like “50 petaflops of NVFP4” measures. It needs hardware that does FP4 math; NVIDIA’s blog lists FP4 support on Blackwell tensor cores. The KV cache, the per-token memory that grows with the context window, can be quantized too, which matters more as contexts get longer. Sparse attention goes after the compute cost of long contexts; quantizing the KV cache goes after their memory. Rubin’s announcement includes a separate storage tier just for KV cache, which shows how much memory long contexts eat.

Where does quantization break?

My numbers are about weights, not answers. Relative error on random Gaussian weights says nothing direct about whether a model still gets your task right. The metric also hides things: NVFP4 shows 9.3% with outliers and 9.5% without, because the large weights it stores well dominate the norm, while the zeros in outlier blocks barely move it.

The vendor numbers are best cases. NVIDIA reports 1% or less accuracy loss going from FP8 to NVFP4 on DeepSeek-R1-0528 in its evaluations, and its own table says the lower risk applies “particularly for larger models”. My read is that smaller models have less slack to absorb the rounding, so test them harder.

Speed needs kernel and hardware support. A 4-bit checkpoint on hardware without FP4 math still saves memory, but every multiply pays to convert weights back to 16-bit. Check what your serving stack actually supports for the format before planning around a speedup.

Post-training quantization is the easy path. Quantizing a finished model (PTQ) is what my script does. Quantization-aware training (QAT) lets the model learn around the rounding; NVIDIA’s blog mentions both. I’d trust a checkpoint released in 4-bit by the people who trained it over one I rounded myself.

What would I do with this?

If I were picking a self-hosted model today, I’d start with the quantized checkpoint the model’s authors publish, like gpt-oss in MXFP4 or NVIDIA’s NVFP4 build of DeepSeek-R1-0528 on Hugging Face, and run my own eval set on it next to the 8-bit or 16-bit version. If the 4-bit model passes, the memory math above decides the hardware.

My opinion on Rubin: when the headline throughput number is quoted in FP4, 4-bit inference is on its way to becoming the default precision for serving large models, at least on NVIDIA hardware. For builders that means more open weights released in 4-bit formats and more reason to have an eval set ready before you switch.

FAQ

Does quantization make a model faster?

Usually, for generating tokens, because there’s less data to read per step. How much faster depends on whether your hardware and kernels compute in the low-precision format or convert back to 16-bit first.

What’s the difference between INT4 and FP4?

Both use 4 bits. INT4 has 16 evenly spaced levels; FP4 E2M1 has 15 distinct values, spaced closer together near zero and farther apart near the edges. With per-block scales, both can work well; the scale format (power of two or FP8) and block size matter as much as the element type.

Is a quantized model the same model?

It has the same architecture and the same weights rounded to fewer values, so it behaves very close to the original but not identically. Treat it as a new model version and re-run your evals.

How much memory does a 4-bit model need?

For the weights, parameters × bits ÷ 8: about 4.5 GB for an 8B model at 4.5 bits per weight, from the table above. Add the KV cache for your context length and batch size, plus runtime overhead, on top.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.