Malhar Ujawane

Blog

Post·

Mistral Large 4 hardware: what self-hosting takes

Mistral Large 4's open weights are 1,060.6 GB in FP8 and 610 GB in NVFP4. What fits on one 8-GPU node, KV cache per token, and API vs renting GPUs.

Mistral Large 4 hardware: what self-hosting takes

Mistral released the weights of Mistral Large 4 on October 29, 2026. It’s a mixture-of-experts model with about 1 trillion parameters, 128 routed experts with 4 active per token, a 1M-token context window and text plus image input. On Mistral’s API it costs $0.70 per million input tokens, $0.07 cached and $2.10 output.

An open trillion-parameter model makes me ask one practical question: what would it take to run it ourselves, and when would that beat paying the API? This post has no throughput numbers from my own hardware. What anyone can do from a laptop is read the checkpoint itself. Every number below comes from the Hugging Face files, the vLLM recipe, NVIDIA spec sheets or a public price page, and the two scripts that produce them are here with their real output.

How many parameters does Mistral Large 4 have?

1,053 billion, counted from the checkpoint. The model card says 1 trillion total and 49 billion active per token, or 52 billion if you include the embeddings and output layer.

I wanted the exact figure without downloading 1 TB, so the script below reads only the JSON header at the start of each of the 54 safetensors shards, using HTTP Range requests. The header lists every tensor’s name, dtype and shape, which is all you need to count.

# count_params.py: Python 3.13 stdlib only
import json, re, struct, urllib.request
from collections import defaultdict

REPO = "mistralai/Mistral-Large-4-1T-A52B"
BASE = f"https://huggingface.co/{REPO}/resolve/main/"

def get(url, start, end):
    req = urllib.request.Request(url, headers={"Range": f"bytes={start}-{end}"})
    with urllib.request.urlopen(req) as r:
        return r.read()

def header(fname):
    url = BASE + fname
    n = struct.unpack("<Q", get(url, 0, 7))[0]      # first 8 bytes: header length
    return json.loads(get(url, 8, 8 + n - 1)), n

index = json.load(urllib.request.urlopen(BASE + "consolidated.safetensors.index.json"))
shards = sorted(set(index["weight_map"].values()))

params = defaultdict(int)
bytes_read = 0
for f in shards:
    h, n = header(f)
    bytes_read += 8 + n
    for name, t in h.items():
        if name == "__metadata__" or "qscale" in name:
            continue                                  # FP8 block scales are not parameters
        count = 1
        for d in t["shape"]:
            count *= d
        if name.startswith(("vision_encoder", "vision_language_adapter", "patch_merger", "pre_mm_projector")):
            bucket = "vision encoder + adapter"
        elif name.startswith("mtp_dspark"):
            bucket = "DSpark draft head"
        elif name.startswith(("tok_embeddings", "output")):
            bucket = "embeddings + output"
        elif ".experts." in name:
            bucket = "routed experts (128 x 71 layers)"
        elif ".shared_experts." in name:
            bucket = "shared expert (71 layers)"
        elif ".feed_forward." in name:
            bucket = "dense FFN (layer 0)"
        elif ".attention." in name:
            bucket = "attention + indexer"
        else:
            bucket = "router, norms, misc"
        params[bucket] += count

total = sum(params.values())
print(f"shards: {len(shards)}, header bytes fetched: {bytes_read/1e6:.1f} MB")
for k, v in sorted(params.items(), key=lambda x: -x[1]):
    print(f"{k:34s} {v/1e9:9.2f} B  ({100*v/total:5.1f}%)")
print(f"{'total':34s} {total/1e9:9.2f} B")
shards: 54, header bytes fetched: 6.2 MB
routed experts (128 x 71 layers)     1029.18 B  ( 97.7%)
attention + indexer                     9.11 B  (  0.9%)
shared expert (71 layers)               8.04 B  (  0.8%)
embeddings + output                     2.52 B  (  0.2%)
DSpark draft head                       2.10 B  (  0.2%)
vision encoder + adapter                1.70 B  (  0.2%)
dense FFN (layer 0)                     0.45 B  (  0.0%)
router, norms, misc                     0.06 B  (  0.0%)
total                                1053.16 B

real	3m50.331s

It ran on October 30, 2026 and fetched 6.2 MB to describe a 1,060.6 GB checkpoint, and most of the 3 minutes 50 seconds was 108 sequential round trips.

The card’s active count checks out. Each token uses 4 of the 128 routed experts in each sparse layer, which is 1,029.18 × 4 / 128 = 32.2B, plus the shared expert (8.0B), attention and indexer (9.1B) and the dense first layer (0.45B). That is 49.8B, the card’s 49B. Add 2.5B for embeddings and output and you get 52.3B, the card’s 52B.

That split is the whole self-hosting problem in one line. Compute per token scales with about 50B parameters, while memory has to hold all 1,053B, because any token can route to any expert.

How much KV cache does a token need?

About 41,920 bytes per token by my arithmetic from config.json, which is small for a model this size. Four settings in the config get it there:

  • Multi-head latent attention: each layer caches one 512-value latent plus 64 RoPE values per token (kv_lora_rank, qk_rope_head_dim) instead of full keys and values for 64 heads.
  • An FP8 KV cache (fp8_kv_cache: true in params.json), so each value is 1 byte.
  • KV sharing: 63 of the 72 layers keep their own cache, and the other 9 reuse layer 35’s (kv_source_layers).
  • The sparse-attention indexer adds a 128-value FP8 key per token in the 44 layers that have their own indexer. It picks the top 2,048 tokens for each query (index_topk), the same DeepSeek-style design I explained with V3.2.

The same 64 heads with plain bf16 keys and values would need 2,359,296 bytes per token, 56 times more. At 41,920 bytes, a 128K-token agent session holds 5.5 GB of cache and a full 1M-token context holds 44.0 GB. Treat these as lower bounds, since serving engines add page alignment and scale factors on top.

The number per token isn’t the number per GPU, though. MLA stores one latent per token with no heads to split, so under plain tensor parallelism every GPU keeps a full copy of the cache. The vLLM docs describe this as 8x KV duplication for DeepSeek-R1 at -tp 8, and decode context parallelism (-dcp) shards the cache by token position to remove it. The vLLM recipe for Large 4 measured the gap on 8 GB300 GPUs with NVFP4: 3.60M tokens of KV capacity at TP8 and 28.76M at TP8 DCP8.

Will Mistral Large 4 fit on one 8-GPU node?

In NVFP4 on Blackwell, yes. In FP8, only on the bigger nodes. The second script pulls the config and the two official checkpoint sizes from Hugging Face and lays them against NVIDIA’s memory specs for H200 (141 GB per GPU), DGX B200 (1,440 GB) and DGX B300 (2.1 TB). The script uses --gpu-memory-utilization 0.85 because that’s what the model card’s vLLM command sets.

# selfhost_math.py: Python 3.13 stdlib only
import json, urllib.request

REPO = "mistralai/Mistral-Large-4-1T-A52B"
cfg = json.load(urllib.request.urlopen(f"https://huggingface.co/{REPO}/resolve/main/config.json"))["text_config"]

def repo_gb(repo):
    info = json.load(urllib.request.urlopen(f"https://huggingface.co/api/models/{repo}?blobs=true"))
    return sum(s.get("size") or 0 for s in info["siblings"] if s["rfilename"].endswith(".safetensors")) / 1e9

# 1. KV cache per token, from the config
L = cfg["num_hidden_layers"]                                    # 72
own_kv = sum(1 for i, src in enumerate(cfg["kv_source_layers"]) if src == i)
own_idx = cfg["indexer_types"].count("full")
latent = cfg["kv_lora_rank"] + cfg["qk_rope_head_dim"]          # 512 + 64 values per layer per token (MLA)
mla_bytes = own_kv * latent * 1                                 # fp8 KV cache: 1 byte per value
idx_bytes = own_idx * cfg["index_head_dim"] * 1                 # fp8 indexer keys for sparse attention
naive = L * cfg["num_attention_heads"] * (cfg["qk_head_dim"] + cfg["v_head_dim"]) * 2   # plain MHA, bf16
per_tok = mla_bytes + idx_bytes
print(f"layers {L}, layers that store their own KV {own_kv}, own indexer {own_idx}")
print(f"KV per token: {per_tok:,} bytes (MLA {mla_bytes:,} + indexer {idx_bytes:,})")
print(f"same shape with plain bf16 multi-head KV: {naive:,} bytes ({naive/per_tok:.0f}x more)")
for ctx in (32_768, 131_072, 1_048_576):
    print(f"  one {ctx:>9,}-token sequence: {per_tok*ctx/1e9:6.1f} GB")

# 2. Does it fit? Weights vs GPU memory (decimal GB, vendor spec sheets)
weights = {"FP8 (official)": repo_gb(REPO), "NVFP4 (official)": repo_gb(REPO + "-NVFP4")}
nodes = {"8x H200 (141 GB each)": 8 * 141, "DGX B200 (1,440 GB)": 1440, "DGX B300 (2.1 TB)": 2100}
UTIL = 0.85                                                     # --gpu-memory-utilization from the model card
print("\nKV room left after weights, at 0.85 memory utilization (ignores activations, so an upper bound):")
for wname, w in weights.items():
    for nname, mem in nodes.items():
        room = mem * UTIL - w
        if "NVFP4" in wname and "H200" in nname:
            print(f"  {wname:17s} {w:7.1f} GB on {nname:22s}: n/a, NVFP4 targets Blackwell")
        elif room <= 0:
            print(f"  {wname:17s} {w:7.1f} GB on {nname:22s}: does not fit ({room:7.1f} GB)")
        else:
            toks = room * 1e9 / per_tok
            print(f"  {wname:17s} {w:7.1f} GB on {nname:22s}: {room:6.1f} GB, {toks/1e6:5.1f}M tokens, {toks/131_072:4.0f} x 128K sessions")

# 3. API vs a rented node
API = {"input": 0.70, "cached": 0.07, "output": 2.10}           # $ per 1M tokens, mistral.ai/news/mistral-large-4-oss
GPU_HR = 9.86                                                   # $ per B200 GPU-hour, Lambda 1-Click Cluster, 16 GPUs, 2 weeks to 1 year
node_hr = 8 * GPU_HR
print(f"\n8 x B200 at ${GPU_HR}/GPU-hour = ${node_hr:.2f}/hour, ${node_hr*24*30:,.0f} per 30 days")
for kind, price in API.items():
    m = node_hr / price
    print(f"  to spend that on the API you'd need {m:8.1f}M {kind:6s} tokens/hour ({m*1e6/3600:9,.0f} tokens/s, around the clock)")

# 4. What one node-hour of measured output is worth at API prices.
# vLLM recipe, 8x GB300, NVFP4, DP8 + EP + DSpark, 8k input, 32 concurrent: 209.8 output tok/s per GPU.
out_tok_hr = 209.8 * 8 * 3600
print(f"\nvLLM-measured best case: {out_tok_hr/1e6:.1f}M output tokens/hour on 8 GPUs = ${out_tok_hr/1e6*API['output']:.2f}/hour at API output price")
layers 72, layers that store their own KV 63, own indexer 44
KV per token: 41,920 bytes (MLA 36,288 + indexer 5,632)
same shape with plain bf16 multi-head KV: 2,359,296 bytes (56x more)
  one    32,768-token sequence:    1.4 GB
  one   131,072-token sequence:    5.5 GB
  one 1,048,576-token sequence:   44.0 GB

KV room left after weights, at 0.85 memory utilization (ignores activations, so an upper bound):
  FP8 (official)     1060.6 GB on 8x H200 (141 GB each) : does not fit ( -101.8 GB)
  FP8 (official)     1060.6 GB on DGX B200 (1,440 GB)   :  163.4 GB,   3.9M tokens,   30 x 128K sessions
  FP8 (official)     1060.6 GB on DGX B300 (2.1 TB)     :  724.4 GB,  17.3M tokens,  132 x 128K sessions
  NVFP4 (official)    610.0 GB on 8x H200 (141 GB each) : n/a, NVFP4 targets Blackwell
  NVFP4 (official)    610.0 GB on DGX B200 (1,440 GB)   :  614.0 GB,  14.6M tokens,  112 x 128K sessions
  NVFP4 (official)    610.0 GB on DGX B300 (2.1 TB)     : 1175.0 GB,  28.0M tokens,  214 x 128K sessions

8 x B200 at $9.86/GPU-hour = $78.88/hour, $56,794 per 30 days
  to spend that on the API you'd need    112.7M input  tokens/hour (   31,302 tokens/s, around the clock)
  to spend that on the API you'd need   1126.9M cached tokens/hour (  313,016 tokens/s, around the clock)
  to spend that on the API you'd need     37.6M output tokens/hour (   10,434 tokens/s, around the clock)

vLLM-measured best case: 6.0M output tokens/hour on 8 GPUs = $12.69/hour at API output price

Stacked bars for four 8-GPU setups showing Mistral Large 4 weights in black and the KV cache room left at 0.85 memory utilization in orange. FP8 weights overflow the 0.85 budget on 8x H200. A DGX B200 leaves 163.4 GB with FP8 and 614.0 GB with NVFP4. A DGX B300 leaves 724.4 GB with FP8.

The KV token counts in that output assume the cache is sharded with DCP. Under plain TP8, divide them by 8.

The vLLM recipe agrees with the arithmetic and adds a warning. It validates NVFP4 on 8x B200, FP8 and NVFP4 on 8x B300, and FP8 on two 8x H200 nodes (TP16, or PP2 x TP8). It doesn’t list FP8 on a single B200 node, and its “~128 GiB of weights per GPU on 8 GPUs” for FP8 leaves little room on a 180 GB B200 once you hold back 15%. I’d follow the recipe over my napkin. Mistral’s card says the NVFP4 checkpoint has the “same performance”. That’s Mistral’s own evaluation; the recipe’s GSM8K-200 score of 0.970 for NVFP4 at TP8 is a sanity check and doesn’t compare the two.

If you’re thinking of a Mac or a single workstation, the smallest GGUF on Hugging Face today is Unsloth’s UD-IQ1_S at 233.0 GB, and ggml-org’s Q4_K_M is 634.8 GB. A 1-bit-class quant of a trillion parameters is something I’d want to evaluate on my own tasks before trusting it with anything.

When does renting a node beat the API?

On these numbers, I can’t find the point where it does. Lambda lists HGX B200 at $9.86 per GPU-hour on its 1-Click Clusters (16 GPUs, 2 weeks to 1 year, plus tax, checked October 30, 2026). The cluster minimum is 16 GPUs, so the real entry ticket is $157.76 an hour; the script uses 8 GPUs to compare one node with one API account.

At $78.88 an hour, you’d have to push 37.6M output tokens an hour through Mistral’s API, about 10,434 tokens per second around the clock, before the API bill matched the rental. The best throughput number in vLLM’s recipe is 209.8 output tokens per second per GPU, measured on GB300 (a faster part than B200) with NVFP4, data parallel plus expert parallel, DSpark speculative decoding, 8k-token inputs and 32 concurrent requests. Eight of those GPUs produce 6.0M output tokens an hour, which is $12.69 at Mistral’s API price.

Input tokens are worth something too, and the recipe doesn’t say how long the outputs were in that run, so I can’t add them exactly. Suppose each request was 8k in and 1k out. The input side would then be worth 8 × $0.70 / $2.10, about 2.7 times the output side, for roughly $46 an hour in total. That’s still under $78.88, on better hardware, with the GPUs fully busy. And coding agents mostly send cached input, which the API sells at $0.07 per million.

So on cost alone I’d use the API, through Mistral or OpenRouter. Self-hosting buys other things: prompts that never leave your network, a fine-tuned variant you own, and a model version that doesn’t change under you. It also hands you the serving bugs. Cloudflare’s GLM KV cache leak, which I linked on October 27, happened in exactly the layer you’d now be running.

What does the license allow?

Most things, with one carve-out. The Mistral Ecosystem Open-Weight License is MIT plus a condition: if you or an affiliate operate a “Model as a Service” business, you need a separate commercial license before any commercial use. The license defines that as giving a third party access to the model “in a manner that allows such third party to exercise meaningful control over the inputs, parameters, or training data”, and it excludes:

(a) end-user products with model capabilities solely embedded within specific features or harnesses, or (b) mere relaying of requests to models hosted by others.

Internal use is exempt too. My reading, as a builder and not a lawyer: an agency shipping a client app where Large 4 sits behind one feature is fine, and reselling a raw endpoint is the case that needs Mistral’s permission. If a client deal depends on it, I’d ask Mistral in writing.

What I’d do with it

For client work I’d start on the API at $0.70/$2.10 and keep self-hosting for a client who needs it.

My opinion on where this goes: a 1T open-weight model with a permissive license is good for everyone, because it puts a public ceiling on what closed APIs can charge. Arvind Krishna’s line about open-weight models costing 10% as much is the same argument from the buyer’s side. But “open” at this size means “open to whoever can spend about $57,000 a month on one GPU node”, and most of us will consume it through someone’s API anyway. The parts I’d steal for my own work are the cheap ones: the KV-sharing and FP8-latent tricks that make long agent sessions affordable, and the habit of reading a checkpoint’s config before believing its launch post.

What would change my mind: a measured B200 throughput figure near 10,000 output tokens per second per node for agent-shaped traffic, or a rental price far below Lambda’s list. If you have either, I’d like to see it.

FAQ

How much GPU memory does Mistral Large 4 need? The weights alone are 1,060.6 GB in FP8 and 610.0 GB in NVFP4 (decimal GB, from Hugging Face). vLLM validates NVFP4 on one 8x B200 node and FP8 on 8x B300 or 16x H200.

Is Mistral Large 4 free for commercial use? Yes, under a modified MIT license, unless you run a Model-as-a-Service business, which needs a separate license from Mistral.

Why is its KV cache so small? Multi-head latent attention caches one 576-value latent per layer instead of per-head keys and values, in FP8, and 9 of the 72 layers reuse another layer’s cache. That’s about 41,920 bytes per token by my count.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.