niieani / gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
Why this job: matched by what the repo talks about: openai, gpt, model, fastest
What it does
- Support for easily tokenizing chats thanks to the encodeChat function
- Support for all current OpenAI models, including GPT-5, GPT-4.1, GPT-4o, o1/o3/o4, gpt-oss, realtime, audio, transcription, image, and video models
- All OpenAI encodings. r50k_base, p50k_base, p50k_edit, cl100k_base, o200k_base, and o200k_harmony
- Can be loaded and work synchronously! (i.e. in non async/await contexts)
- Generator function versions of both the decoder and encoder functions
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 2 days ago
- Stars
- 860
- +4 this week
- Licence
- MIT
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- 4.1.2
- Released 2 days ago
- Built with
- TypeScript
- 71% of the code
Is it looked after?
- Contributors
- 3
- Last 90 days
- Issues
- 96% closed
- 2 still open
- Pull requests
- 25 merged
- 1 waiting
- Releases
- 42
- 8 in the last year
- No contributing guide
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| microsoft / Tokenizer | 215 | 2 months ago | MIT | No downloads |
More: languages, topics and project record
Languages
- TypeScript 71%
- Python 16%
- JavaScript 12%
- Shell <1%
Topics
- bpe
- gpt-2
- gpt-3
- machine-learning
- gpt-4
- tokenizer
- decoder
- encoder
- openai
- gpt-4o
- gpt-o1
- gpt-5
Project record
- JobModel gateways
- StatusListed
- Commits171
- Forks66
- Watching5
- Stars in 28 days+7
- CreatedMar 22, 2023
- First seen by usNov 7, 2026
- CheckedNov 12, 2026