dphnAI / sonar
Large-scale LLM inference engine
Why this job: says: inference engine, llm inference; topics: inference-engine
What it does
- Continuous batching and paged KV-cache management
- Prefix caching enabled by default
- Tensor, pipeline, data, and expert parallelism
- Multi-node multiprocessing without a Ray cluster
- Prefill/decode disaggregation through NIXL and other KV connectors
Get started
curl -fsSL https://sonar.dphn.ai/install.sh | bash - Maintenance
- Active
- Last commit yesterday
- Stars
- 1,906
- +7 this week
- Licence
- AGPL-3.0
- Share-alike, even hosted
- Downloads for
- 2 systems
- Mac, Linux
- Latest version
- v0.25.0
- Released yesterday
- Built with
- Python
- 55% of the code
Is it looked after?
- Contributors
- 2
- Last 90 days
- Issues
- 88% closed
- 30 still open
- Pull requests
- 1,349 merged
- 25 waiting
- Releases
- 44
- 7 in the last year
- Contributing guide (opens in new tab)
- No security policy
- Code of conduct (opens in new tab)
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| vllm-project / vllm | 96,588 | today | Apache-2.0 | Mac, Linux |
| containers / ramalama | 3,112 | 3 days ago | MIT | Mac |
| LMCache / LMCache | 12,310 | today | Apache-2.0 | Linux |
More: download files, languages, topics and project record
Download files, v0.25.0
- Mac1 file, 0 downloads
- Linux5 files, 0 downloads
Languages
- Python 55%
- C++ 37%
- Rust 4%
- Cuda 4%
Topics
- api-rest
- inference-engine
- machine-learning
- cuda
- inferentia
- rocm
- intel
- lora
- speculative-decoding
- tpu
Project record
- JobLocal LLMs
- StatusListed
- Commits2,717
- Forks213
- Watching28
- Stars in 28 days+35
- CreatedJun 23, 2023
- First seen by usNov 7, 2026
- CheckedNov 9, 2026