intentee / paddler
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale šš¦ Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Why this job: matched by what the repo talks about: llm, gpu, model, serv
What it does
- Inference through a built-in llama.cpp engine
- Works through agents that can be added dynamically, allowing integration with autoscaling tools
- Request buffering, enabling scaling from zero hosts
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 3 days ago
- Stars
- 1,691
- +4 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- 2 systems
- Mac, Linux
- Latest version
- v4.1.0
- Released 3 months ago
- Built with
- Rust
- 89% of the code
Is it looked after?
- Contributors
- Not stated
- Last 90 days
- Issues
- 68% closed
- 26 still open
- Pull requests
- 153 merged
- 4 waiting
- Releases
- 26
- 10 in the last year
- No contributing guide
- Security policy (opens in new tab)
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| fiveoutofnine / whatcanirun | 248 | 2 months ago | MIT | No downloads |
More: download files, languages, topics and project record
Download files, v4.1.0
- Mac2 files, 52 downloads
- Linux4 files, 254 downloads
Languages
- Rust 89%
- TypeScript 7%
- Python 2%
- CSS 1%
Topics
- ai
- llamacpp
- llm
- llmops
- load-balancer
Project record
- JobLocal LLMs
- StatusListed
- Commits1,400
- Forks98
- Watching13
- Stars in 28 days+17
- CreatedApr 27, 2024
- First seen by usNov 7, 2026
- CheckedNov 8, 2026