Malhar Ujawane

Notes

Note·

TIL: the OpenAI CLI counts tokens offline

OpenAI's official CLI has a tokenizer command that counts and inspects tokens locally, with no API key or network. Commands and real output.

The OpenAI CLI (openai, the official command-line tool for the OpenAI REST API) has a tokenizer command that counts tokens locally, with no API key and no network. Its help text says it “Tokenizes exact UTF-8 text without credentials, network access, or saved input.”

Run on Linux x86_64 on October 24, 2026, with v1.40.6 (released October 23). Outputs are pasted as printed.

Install from the release tarball and check the hash first:

V=1.40.6
curl -sLO https://github.com/openai/openai-cli/releases/download/v$V/openai_${V}_linux_amd64.tar.gz
curl -sLO https://github.com/openai/openai-cli/releases/download/v$V/openai_${V}_checksums.txt
grep linux_amd64.tar.gz openai_${V}_checksums.txt | sha256sum -c -
tar xzf openai_${V}_linux_amd64.tar.gz
./openai --version
openai_1.40.6_linux_amd64.tar.gz: OK
openai version 1.40.6

It ships four encodings:

./openai tokenizer encodings
o200k_base (default)
cl100k_base
r50k_base (legacy)
p50k_base (legacy)

Take the Gujarati sentence from What are tokens in AI?, where tiktoken and the GPT-2 tokenizer gave 8, 37 and 58 tokens. The CLI gives the same three numbers:

for e in o200k_base cl100k_base r50k_base; do
  ./openai tokenizer count --text "રેલવે સ્ટેશન ક્યાં છે?" --encoding $e
done
Encoding: o200k_base
Input bytes: 58
Tokens: 8
Encoding: cl100k_base
Input bytes: 58
Tokens: 37
Encoding: r50k_base
Input bytes: 58
Tokens: 58

To check the “no network” claim, run it inside a network namespace with nothing but a downed loopback interface. curl can’t resolve a host in there; the tokenizer still works:

unshare -rn curl -sS -m 5 -o /dev/null https://api.github.com
unshare -rn ./openai tokenizer count --text "રેલવે સ્ટેશન ક્યાં છે?"
curl: (6) Could not resolve host: api.github.com
Encoding: o200k_base
Input bytes: 58
Tokens: 8

inspect shows each token ID with its byte range, and --format json gives a script something to parse:

./openai tokenizer inspect --text "Hello world!"
./openai tokenizer count --text "Hello world! Tokens are how models read." --format json
Encoding: o200k_base
Input bytes: 12
Tokens: 3

0  ID 13225  [0, 5)
   "Hello"
   hex: 48656c6c6f

1  ID 2375  [5, 11)
   " world"
   hex: 20776f726c64

2  ID 0  [11, 12)
   "!"
   hex: 21

{"encoding":"o200k_base","input_bytes":40,"token_count":9}

Same IDs (13225, 2375, 0) and the same 9-token count as the tiktoken snippet in the tokens post.

One limit, in the CLI’s own words: “Plain-text counts are not complete request counts, billed usage, prices, or context-limit guarantees.” Tool definitions and message structure add tokens a plain-text count can’t see. For the number you’ll be billed, the help points to openai responses input-tokens count, which calls the API and needs a key. That one wasn’t run for this note.

Also handy if you run it in scripts: set OPENAI_CLI_NO_UPDATE_CHECK=1 to turn off update checks.

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.