microsoft / sarathi-serve
A low-latency & high-throughput serving engine for LLMs
Why this job: matched by what the repo talks about: llm, engine, serv, high
What it does
Sarathi-Serve is a high througput and low-latency LLM serving framework. Please refer to our OSDI'24 paper for more details.
Get started
pip install -e .- Maintenance
- Inactive
- Last commit 10 months ago
- Stars
- 536
- +1 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- None yet
- No releases
- Built with
- Python
- 94% of the code
Is it looked after?
- Contributors
- Not stated
- Last 90 days
- Issues
- 64% closed
- 10 still open
- Pull requests
- 28 merged
- 7 waiting
- Releases
- None
- Never published
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| vllm-project / vllm | 96,774 | today | Apache-2.0 | Mac, Linux |
| Tiiny-AI / PowerInfer | 9,879 | 6 months ago | MIT | No downloads |
| sgl-project / sglang | 38,767 | today | Apache-2.0 | No downloads |
More: languages, topics and project record
Languages
- Python 94%
- Cuda 6%
- C++ <1%
- C <1%
Topics
- llama
- llm-inference
- pytorch
- transformer
Project record
- JobLocal LLMs
- StatusListed
- Commits34
- Forks68
- Watching7
- Stars in 28 days+2
- CreatedNov 2, 2023
- First seen by usNov 7, 2026
- CheckedNov 8, 2026