Malhar Ujawane

Inspiration

Inspiration·

Ubicloud: constrained decoding took Kimi tool calls from 92% to 100%

Ubicloud took Kimi K2.6 tool-call schema accuracy in vLLM from 92% to 100% with constrained decoding. What it does and why vLLM leaves it off by default.

Ubicloud: constrained decoding took Kimi tool calls from 92% to 100%

Fixing Tool-Call Accuracy in vLLM (Jesse Donkervliet and Junhao Li, Ubicloud, October 20, 2026)

Ubicloud serves open-weight models, and this post explains why a model can score well on benchmarks and still fumble tools when you run it yourself. Their measurement: Kimi K2.6 served with vLLM 0.28 produced the expected tool-call schema 92% of the time on Moonshot’s K2VV ToolCall benchmark, “2 out of every 25 attempted tool calls failed.” With constrained decoding turned on, they got 100%.

Constrained decoding works at the sampling step. The model still produces a probability for every token in its vocabulary, but before sampling, the engine masks out every token that would break the tool’s JSON schema. The post puts it well:

This changes schema accuracy from a best-effort approach to a runtime guarantee. The model still decides which token to generate, but now the token is selected from a restricted and syntactically-correct set.

The part I didn’t know: vLLM supports this but doesn’t turn it on by default for tool calls. Ubicloud found its accuracy “dropped back to 91%” after switching from its own fork to upstream, and traced it to the default. vLLM’s reasons are fair. Forcing a grammar on a model whose natural output drifts from the schema can push it off course, and the post lists reported problems with gpt-oss-20b, Qwen 3.6 and GLM 5.1. The opt-in is the –tool-strict-level flag.

For anyone running agents on self-hosted open weights, this changes what a bad benchmark result means. A malformed tool call costs a retry and leaves an error message in the context, and every later turn re-reads that error. Before blaming the model, check the serving config. It’s the same lesson as Theo’s point about harnesses: the number you see measures the model plus everything wrapped around it.

To see the mask in action, there’s a small test in a note from the same day, TIL: xgrammar’s JSON grammar sets your separators.

Via Hacker News.

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.