Fast and lightweight token count estimation without requiring a full tokenizer.
Estimates are calibrated against OpenAI's o200k_base encoding – the tokenizer of all current GPT models. Counts for other LLM families will differ somewhat; the defaultCharsPerToken and languageConfigs options let you tune the heuristics for your model. For precise counts, use a full tokenizer like gpt-tokenizer.
- 🎯 95%+ average accuracy, and no single sample below 90%
- 📦 Just 2kB bundle size with zero dependencies
- 🖥️ Bundled CLI – count, slice and split from the shell
- 🌍 Built-in rules for CJK, Cyrillic, Greek, accented Latin scripts and emoji – extend them with your own
The following chart shows how close the estimates come to actual GPT token counts for different input texts:
Bars grow left when tokenx underestimates and right when it overestimates. The axis spans the ±10% per-sample deviation bound.
under ◂·▸ over
Team chat transcript (en) 293 → 285 ███│ -2.73%
Vite releases API response 8,075 → 8,551 │██████ +5.89%
tokenx source code 3,496 → 3,341 ████│ -4.43%
Vite plugin API docs (en) 6,901 → 7,155 │████ +3.68%
Cat article (ja) 12,437 → 11,914 ████│ -4.21%
Cat article (ko) 7,117 → 6,815 ████│ -4.24%
Cat article (zh) 9,057 → 8,953 █│ -1.15%
The Great Gatsby by Fitzgerald (en) 4,391 → 4,479 │██ +2.00%
Die Verwandlung by Kafka (de) 4,437 → 4,384 █│ -1.19%
─────────────────────
mean 3.28%
Accuracy depends on the kind of text, not its length: a short excerpt deviates about as much as the full document it came from. A holdout corpus of French, English, and Russian documents that no ratio was ever fitted against is held to a looser ±15% bound per document, so a retune that only fits the calibration texts fails the test run.
These cases are knowingly left outside the bounds:
- High-entropy strings – base64, hashes, digests: ≈-70%. Pricing them would cost every caller runtime for a case ordinary traffic rarely carries.
- Traditional and classical Chinese – the hanzi rate is calibrated on contemporary simplified script: ≈-5% to -20%.
- Scripts without a built-in rule – Arabic ≈-35%, Hindi ≈-30%, Hebrew ≈-45%, Thai ≈-60%; a custom language config closes the gap.
- Chinese or Japanese mixed with Latin words or code – Latin letters that share a segment with CJK text are priced like hanzi: ≈+30% to +80% for technical prose, up to ≈+200% for a sentence built around code identifiers.
# npm
npm install tokenx
# pnpm
pnpm add tokenx
# yarn
yarn add tokenximport { estimateTokenCount, isWithinTokenLimit, sliceByTokens, splitByTokens } from 'tokenx'
const text = 'Your text goes here.'
estimateTokenCount(text) // Estimated token count
isWithinTokenLimit(text, 1024) // true
sliceByTokens(text, 0, 5) // Roughly the first five tokens
splitByTokens(text, 100) // Chunks of about 100 tokens eachThe package ships a tokenx binary. Run it with npx tokenx, or install the package to call tokenx directly. The CLI requires Node.js 24 or later.
# Count tokens in a file, a pipe, or several files at once
tokenx README.md
cat article.md | tokenx
tokenx count src/*.ts
# Fail a script when a prompt outgrows its budget (exit code 2)
tokenx prompt.txt --limit 8000
# Extract a token range, or chunk a document for RAG
tokenx slice article.md --end 500
tokenx split article.md --size 500 --overlap 50Only results go to stdout – a bare integer for one input, a table for several, chunks as a JSON array – so $(tokenx count file.txt) and | jq work unchanged. Notices and errors go to stderr. Run tokenx --help for every option.
Every function takes an optional options object last:
defaultCharsPerToken– average characters per token for segments no other rule prices (default: 7)languageConfigs–{ pattern, averageCharsPerToken }entries checked before the built-in language configs; the first matching config prices a segment. They take precedence over every built-in rule except whitespace, the CJK script rates included, and the built-in configs still price whatever no custom config matches.
Returns the estimated token count of text.
estimateTokenCount('Hello, world!')
estimateTokenCount('Bonjour le monde!', {
defaultCharsPerToken: 4,
languageConfigs: [{ pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }],
})Returns whether the estimated token count stays within tokenLimit, the limit included. Stops counting as soon as the limit is exceeded.
isWithinTokenLimit('Check this text against a limit', 100) // trueExtracts a token range like Array.prototype.slice(): start is inclusive, end exclusive, and negative indices count back from the end. A boundary inside a multi-token word cuts the word proportionally, so a slice can start or end mid-word – adjacent slices still rejoin into the original text.
const text = 'Hello, world! This is a test sentence.'
sliceByTokens(text, 0, 3) // First three tokens
sliceByTokens(text, -2) // Last two tokens
sliceByTokens(text, 1, -1) // All but the first and last tokenSplits text into chunks of about tokensPerChunk tokens – for RAG, batch processing, or staying within a context window.
tokensPerChunk is a target, not a hard maximum: a chunk closes once it reaches the target, so a single long segment can push a chunk beyond it. Chunks never break words apart. Chinese and Japanese text has no spaces, so a whole clause up to its punctuation mark forms one segment, and a long clause can push a chunk well past the target.
The overlap option (default: 0, clamped below tokensPerChunk) repeats the trailing tokens of each chunk at the start of the next; a final chunk consisting only of overlap content is never emitted.
splitByTokens(text, 100)
splitByTokens(text, 100, { overlap: 10 })MIT License © 2023-PRESENT Johann Schopplich