microsoft / Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
Why this job: matched by what the repo talks about: openai, llm, typescript, net
What it does
This repo contains Typescript and C# implementation of byte pair encoding(BPE) tokenizer for OpenAI LLMs, it's based on open sourced rust implementation in the OpenAI tiktoken. Both implementation are valuable to run prompt tokenization in Nodejs and .NET environment before feeding prompt into a LLM.
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 2 months ago
- Stars
- 215
- +2 this week
- Licence
- MIT
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- v1.3.3
- Released 2 years ago
- Built with
- C#
- 44% of the code
Is it looked after?
- Contributors
- Not stated
- Last 90 days
- Issues
- 73% closed
- 4 still open
- Pull requests
- 41 merged
- 8 waiting
- Releases
- 4
- 0 in the last year
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| niieani / gpt-tokenizer | 860 | 2 days ago | MIT | No downloads |
| sunmh207 / AI-Codereview-Gitlab | 1,878 | 35 days ago | Apache-2.0 | No downloads |
| MagnivOrg / prompt-layer-library | 788 | 14 days ago | Apache-2.0 | No downloads |
More: languages, topics and project record
Languages
- C# 44%
- TypeScript 30%
- Jupyter Notebook 24%
- JavaScript 1%
Topics
- ai
- gpt
- llm
- openai
- tokenizer
Project record
- JobModel gateways
- StatusListed
- Commits98
- Forks36
- Watching8
- Stars in 28 days+4
- CreatedMar 28, 2023
- First seen by usNov 7, 2026
- CheckedNov 8, 2026