kubeai-project / kubeai
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.
Why this job: says: speech to text; topics: whisper
What it does
When running multiple replicas of vLLM, the random load balancing strategy built into kube-proxy that backs standard Kubernetes Services performs poorly (TTFT & throughput). This is because vLLM isn't stateless, its performance is heavily influenced by the state of its KV cache.
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 10 days ago
- Stars
- 1,289
- On GitHub
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest tag
- helm-chart-models-0.25.0
- Released 10 days ago
- Built with
- Go
- 90% of the code
Is it looked after?
- Contributors
- 15
- Last 90 days
- Issues
- 63% closed
- 79 still open
- Pull requests
- 376 merged
- 25 waiting
- Releases
- 111
- 23 in the last year
- No contributing guide
- Security policy (opens in new tab)
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| deusjin / subforge | 103 | 2 months ago | MIT | No downloads |
More: languages, topics and project record
Languages
- Go 90%
- Shell 6%
- Makefile 2%
- Python 1%
Topics
- k8s
- llm
- openai-api
- autoscaler
- ollama
- vllm
- ollama-operator
- vllm-operator
- ai
- kubernetes
- whisper
- inference-operator
Project record
- JobSpeech and audio
- StatusListed
- Commits444
- Forks144
- Watching13
- Stars in 28 days+5
- CreatedOct 21, 2023
- First seen by usNov 7, 2026
- CheckedNov 9, 2026