TIL: xgrammar's JSON grammar sets your separators
XGrammar cuts the next-token menu to 2 to 12 tokens on JSON structure. With whitespace off it still expects ': ' after keys unless you pass separators.
Constrained decoding, the fix in Ubicloud’s vLLM tool-call post, works by masking every token that would break a JSON schema before the model samples. vLLM implements it with XGrammar. To see how small that menu gets, this note runs XGrammar against the GPT-2 tokenizer (50,257 tokens) and the webfetch schema from Ubicloud’s post, and counts the allowed tokens at each step.
# token_mask.py: how many tokens does constrained decoding allow at each step of one tool call?
# pip install xgrammar==0.2.8 transformers==5.21.0 torch==2.14.1
import json
import torch
import xgrammar as xgr
from transformers import AutoTokenizer
# The webfetch parameter schema from Ubicloud's post, description fields trimmed.
SCHEMA = {
"type": "object",
"properties": {
"url": {"type": "string"},
"format": {"type": "string", "enum": ["text", "markdown", "html"]},
"timeout": {"type": "number"},
},
"required": ["url"],
}
tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
info = xgr.TokenizerInfo.from_huggingface(tok)
grammar = xgr.GrammarCompiler(info).compile_json_schema(
SCHEMA, any_whitespace=False, separators=(",", ":"))
matcher = xgr.GrammarMatcher(grammar)
bitmask = xgr.allocate_token_bitmask(1, info.vocab_size)
def allowed():
matcher.fill_next_token_bitmask(bitmask)
logits = torch.zeros(1, info.vocab_size)
xgr.apply_token_bitmask_inplace(logits, bitmask)
return [i for i in range(info.vocab_size) if logits[0, i] != float("-inf")]
call = '{"url":"https://ubicloud.com","format":"markdown"}'
print(f"vocab: {info.vocab_size} tokens")
for t in tok.encode(call):
ok = allowed()
sample = [tok.decode([i]) for i in ok[:4]]
print(f"{len(ok):>6} allowed next={tok.decode([t])!r:14} e.g. {sample}")
assert matcher.accept_token(t)
# A token the schema forbids: after '{"', a key that isn't in the schema.
matcher.reset()
for t in tok.encode('{"'):
matcher.accept_token(t)
bad = tok.encode("search")[0]
print("accept 'search' as a key:", matcher.accept_token(bad))
Output, run on October 27, 2026 with Python 3.13.5 (Hugging Face warnings removed):
vocab: 50257 tokens
2 allowed next='{"' e.g. ['{', '{"']
3 allowed next='url' e.g. ['u', 'ur', 'url']
5 allowed next='":"' e.g. ['"', '":', '":"', '":"/']
50033 allowed next='https' e.g. ['!', '"', '#', '$']
50033 allowed next='://' e.g. ['!', '"', '#', '$']
50033 allowed next='ub' e.g. ['!', '"', '#', '$']
50033 allowed next='ic' e.g. ['!', '"', '#', '$']
50033 allowed next='l' e.g. ['!', '"', '#', '$']
50033 allowed next='oud' e.g. ['!', '"', '#', '$']
50033 allowed next='.' e.g. ['!', '"', '#', '$']
50033 allowed next='com' e.g. ['!', '"', '#', '$']
50033 allowed next='","' e.g. ['!', '"', '#', '$']
10 allowed next='format' e.g. ['f', 't', 'form', 'for']
3 allowed next='":"' e.g. ['"', '":', '":"']
12 allowed next='mark' e.g. ['h', 'm', 't', 'te']
3 allowed next='down' e.g. ['d', 'down', 'do']
4 allowed next='"}' e.g. ['"', '",', '","', '"}']
accept 'search' as a key: False
On structure the menu shrinks to 2 to 12 tokens out of 50,257. Inside the URL string it opens back up to 50,033, because almost anything is valid inside a JSON string. After “format”:“ only the prefixes of text, markdown and html survive. A made-up key like search is rejected outright, which is the search_web versus web_search failure Ubicloud describes.
The TIL is the separators argument. The first version of this script called compile_json_schema(SCHEMA, any_whitespace=False) with no separators, and the assert failed on the third token. With whitespace turned off, XGrammar 0.2.8 still fixed the separator after each key as “: “ with a space, so compact JSON was rejected:
# separators.py: same schema, same compact JSON, with and without explicit separators.
import xgrammar as xgr
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
info = xgr.TokenizerInfo.from_huggingface(tok)
schema = {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}
for kwargs in ({"any_whitespace": False}, {"any_whitespace": False, "separators": (",", ":")}):
g = xgr.GrammarCompiler(info).compile_json_schema(schema, **kwargs)
root = [line for line in str(g.grammar).splitlines() if line.startswith("root")][0]
print(kwargs)
print(" ", root)
for text in ('{"url":"a"}', '{"url": "a"}'):
m = xgr.GrammarMatcher(g)
print(f" {text:14} accepted={all(m.accept_token(t) for t in tok.encode(text))}")
{'any_whitespace': False}
root ::= (("{\"url\": " basic_string "}"))
{"url":"a"} accepted=False
{"url": "a"} accepted=True
{'any_whitespace': False, 'separators': (',', ':')}
root ::= (("{\"url\":" basic_string "}"))
{"url":"a"} accepted=True
{"url": "a"} accepted=False
Printing grammar shows the rule the matcher enforces, and it’s the fastest way to debug a “why won’t it accept this” moment. If your grammar’s separators differ from how the model was trained to format tool calls, the mask forces it onto tokens it wouldn’t pick, which is one of the failure modes the Ubicloud post lists for turning constrained decoding on everywhere.
Earlier TILs in the same spirit: the OpenAI CLI counts tokens offline, and the tokenizer differences in What are tokens in AI?.