Malhar Ujawane

Blog

Explainer·

How to choose an LLM: a decision model for builders

How to choose an LLM: set hard constraints, test candidates on your own eval set, then compare cost per solved task, not price per token. With a cost script.

How to choose an LLM: a decision model for builders

To choose an LLM, filter candidates by hard constraints first (data residency, deployment, context length, modalities, licensing), then run the survivors on an eval set built from your real task, and pick the cheapest model that clears your quality bar measured as cost per solved task, not price per token. Public leaderboards help you build the shortlist; only your own evals can make the final call, and you should rerun them whenever a new model ships.

Why is choosing an LLM hard?

Because the options change every few months and the numbers that are easy to compare are the wrong ones. A price table compares cost per token. A leaderboard compares scores on someone else’s task. Neither tells you what it costs to get your task done correctly.

Anthropic’s choosing a model guide is vendor-specific, but its structure generalizes. It lists capabilities, speed and cost as the criteria, adds effort (a per-request setting that trades intelligence for latency and cost within one model), and says that when deciding whether to switch, “Having a good evaluation set is the most important step in the process.” I agree with that sentence more than with any leaderboard.

What should I decide before comparing models?

Hard constraints remove candidates without any testing. I’d write them down first:

  • Where can the data go? Some data cannot leave a region, a cloud account or your own hardware. That can rule out a provider’s API in favor of the same model on a cloud platform, or rule out closed models in favor of open-weight models you host.
  • What does the input look like? Text only, images, PDFs, audio? How long is the longest realistic prompt? A context window that is too small is a hard stop.
  • Does it need tools? Agents need reliable tool calling and long-horizon behavior, which varies more between models than chat quality does.
  • Latency budget. A voice assistant and a nightly batch job have different answers. Time to first token and output speed both matter.
  • Lifecycle. Model IDs get retired. Anthropic’s models overview lists a retirement commitment date for each current model; check the equivalent for any vendor before you build on a model.
  • Language. If your users write in Gujarati or Hindi, tokenizer efficiency and quality in those languages can change both cost and accuracy, as I showed in what are tokens.

How do I compare the models that are left?

Run them on your task. A shortlist of three is enough: a small fast model, a mid-tier one and a frontier one, ideally from more than one provider.

  1. Build an eval set of 100 to 300 real inputs with a way to grade each output: exact match, a unit test, a schema check, or an LLM judge checked against human labels.
  2. Log real token usage per attempt from the API’s usage fields, including reasoning tokens. Two models with the same price per token can use very different numbers of tokens for the same task.
  3. Compute pass rate and cost per attempt for each model, then cost per solved task.
  4. Apply the quality bar before looking at cost. A model that fails the bar is not cheap; it is wrong.

LLM benchmarks are useful at the shortlist step: they tell you which models are worth testing. They don’t replace step 1.

How do I compare LLM cost properly?

Cost per solved task, which is cost per attempt divided by pass rate if you retry failures, is the number that matters. This script runs that math for Anthropic’s current three-tier lineup at today’s list prices. The pass rates are placeholders for illustration; the point is how they interact with price.

# choose_llm.py: pick a model on cost per solved task, after a quality gate. Python 3, stdlib only.
# Prices: Anthropic list prices, USD per 1M tokens (input, output), read 2026-10-31.
# Pass rates are placeholders: replace them with your own eval results.
models = {
    "claude-haiku-5-5":  {"price": (0.10, 0.50), "pass_rate": 0.62},
    "claude-sonnet-5-5": {"price": (2.00, 10.00), "pass_rate": 0.84},
    "claude-opus-5-5":   {"price": (4.00, 20.00), "pass_rate": 0.91},
}
IN_TOK, OUT_TOK = 20_000, 2_000   # one attempt at the task (measure yours from API usage logs)
MIN_PASS = 0.80                   # the quality bar the product needs
TASKS = 50_000                    # per month

print(f"{'model':<18} {'per attempt':>11} {'pass':>5} {'per solved':>10} {'monthly':>9}  gate")
for name, m in models.items():
    p_in, p_out = m["price"]
    attempt = IN_TOK * p_in / 1e6 + OUT_TOK * p_out / 1e6
    solved = attempt / m["pass_rate"]          # expected cost if you retry until it passes
    gate = "ok" if m["pass_rate"] >= MIN_PASS else "fails quality bar"
    print(f"{name:<18} {attempt:>11.4f} {m['pass_rate']:>5.0%} {solved:>10.4f} {solved * TASKS:>9,.0f}  {gate}")

Output (Python 3.13.11):

model              per attempt  pass per solved   monthly  gate
claude-haiku-5-5        0.0030   62%     0.0048       242  fails quality bar
claude-sonnet-5-5       0.0600   84%     0.0714     3,571  ok
claude-opus-5-5         0.1200   91%     0.1319     6,593  ok

The prices are from Anthropic’s pricing page as of 31 October 2026: Haiku 5.5 at $0.10 / $0.50 per million input and output tokens for prompts up to 100,000 tokens, Sonnet 5.5 at $2 / $10, Opus 5.5 at $4 / $20. Three things to read from it:

  • Haiku is 20 times cheaper per attempt than Sonnet and still fails the bar in this example. Retrying a model that is wrong 38 percent of the time does not make it right on the cases it can’t do; the division assumes failures are random, which is generous.
  • Sonnet clears the bar at about half of Opus’s cost per solved task. That is the pick here, unless the 7-point pass-rate gap is worth $3,000 a month to the business.
  • The token profile is an assumption. Real models differ in how many tokens they spend on the same task, and on agent work that difference can outweigh the price gap. I worked through one such case in the Claude Opus 4.5 effort post, where Anthropic reported a more expensive model using far fewer output tokens per task.

Should I use one model or several?

Often several. A cheap model can handle the easy majority while a frontier model takes the hard cases. That is model routing, and it is the natural next step once you have per-model pass rates.

Anthropic’s own guidance adds a caution worth repeating: on its page about optimizing for cost and intelligence, it says to sweep effort on your current model first and only adopt a multi-model pairing if it “beats the single model’s whole curve of score against spend.” Turning one model’s effort setting down is simpler to operate than two models, and sometimes it wins.

Where do LLM choices go wrong?

Choosing from a leaderboard. A top score on a public benchmark says the model is good at that benchmark. Your task has its own data, format and failure costs.

Comparing price per token. As the script shows, pass rate and token usage decide the real cost. Reasoning tokens are billed as output and can dominate.

Choosing once. New models ship every few months. An eval set you can rerun in an afternoon turns a model upgrade from a debate into a measurement.

Ignoring the switching cost. Prompts tuned for one model often underperform on another. Budget prompt work into any switch, and keep provider-specific code behind one interface.

What I would do

From a data-platform seat, the line I would circle is that model choice is a procurement decision with a test harness. Write the constraints down, keep the eval set in the repo, record pass rate, cost and p95 latency per model in one table, and rerun it on every major release.

My default start, absent constraints, is the mid-tier model from a major provider: it usually clears the bar, it is fast enough for interactive use, and it leaves room to move down for cost or up for quality once the eval says which. For rSearch, the search product I founded, the reasoning model is DeepSeek R1, an open-weight reasoning model; I wrote about its release in DeepSeek R1. Open weights are worth a slot on any shortlist when cost or deployment control matters.

FAQ

What is the best LLM?

There isn’t one for every task. The best model for you is the cheapest one that passes your quality bar on your own eval set, under your constraints on data, latency and deployment.

How do I compare LLM costs?

Compare cost per solved task: tokens per attempt times price, divided by the pass rate. Measure tokens per attempt from real API usage logs, because models use different amounts for the same task.

Should I use an open-weight or a closed model?

Use open weights when you need to control where data goes, run on your own hardware or fine-tune freely; use a closed API when you want the strongest model with the least operations work. Put one of each on the shortlist and let the eval decide.

How often should I re-evaluate my LLM choice?

Whenever a relevant new model ships, and at least every few months. With a saved eval set, rerunning the comparison takes hours, not weeks.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.