elixir-nx / tokenizers
Elixir bindings for 🤗 Tokenizers
Why this job: works in the areas we cover, but does not say which job it does
What it does
You can use any pre-trained tokenizer from any model repo on Hugging Face Hub, such as bert-base-cased.
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 19 days ago
- Stars
- 101
- On GitHub
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- 3 systems
- Mac, Windows, Linux
- Latest version
- v0.5.2
- Released 19 days ago
- Built with
- Elixir
- 53% of the code
Is it looked after?
- Contributors
- Not stated
- Last 90 days
- Issues
- 100% closed
- 0 still open
- Pull requests
- 54 merged
- 2 waiting
- Releases
- 11
- 1 in the last year
- No contributing guide
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| brainlid / langchain | 1,214 | 10 days ago | Custom | No downloads |
| deepfates / imp | 260 | 17 days ago | MIT | No downloads |
| xberg-io / html-to-markdown | 887 | 4 days ago | MIT | Mac, Windows, Linux |
More: download files, languages, topics and project record
Download files, v0.5.2
- Mac4 files, 192 downloads
- Windows4 files, 19 downloads
- Linux12 files, 6,769 downloads
Languages
- Elixir 53%
- Rust 46%
- Nix 1%
Topics
- elixir
- machine-learning
- rust
- text-processing
Project record
- JobOther AI tools
- StatusListed
- Commits78
- Forks20
- Watching10
- Stars in 28 days0
- CreatedFeb 12, 2022
- First seen by usNov 7, 2026
- CheckedNov 8, 2026