thushan / olla
High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.
Why this job: topics: llama-cpp, llamacpp, ollama, llm-inference, vllm, local-ai, mlx
What it does
- Smart Load Balancing. Priority-based routing with automatic failover and connection retry
- Sticky Sessions. KV-cache-aware affinity routing that pins multi-turn conversations to the same backend
- Smart Model Unification. Per-provider unification + OpenAI-compatible cross-provider routing
- Dual Proxy Engines. Sherpa (simple) and Olla (high-performance)
- Advanced Filtering. Profile and model filtering with glob patterns for precise control
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 9 days ago
- Stars
- 337
- +6 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- 3 systems
- Mac, Windows, Linux
- Latest version
- v0.1.0
- Released 9 days ago
- Built with
- Go
- 84% of the code
Is it looked after?
- Contributors
- 1
- Last 90 days
- Issues
- 57% closed
- 22 still open
- Pull requests
- 166 merged
- 12 waiting
- Releases
- 25
- 9 in the last year
- No contributing guide
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| xorbitsai / inference | 9,649 | today | Apache-2.0 | No downloads |
| SemiAnalysisAI / InferenceX | 2,016 | today | Apache-2.0 | No downloads |
| containers / ramalama | 3,111 | 4 days ago | MIT | Mac |
More: download files, languages, topics and project record
Download files, v0.1.0
- Mac2 files, 9 downloads
- Windows2 files, 35 downloads
- Linux2 files, 28 downloads
Languages
- Go 84%
- TypeScript 6%
- Python 4%
- Shell 3%
Topics
- ai
- llm-inference
- lmstudio
- ollama
- proxy
- vllm
- golang
- llamacpp
- llm-proxy
- llm-router
- llm-routing
- self-hosted
Project record
- JobLocal LLMs
- StatusListed
- Commits1,729
- Forks60
- Watching3
- Stars in 28 days+23
- CreatedMay 23, 2025
- First seen by usNov 7, 2026
- CheckedNov 10, 2026