defilantech / LLMKube

What it does

• Kubernetes-native CRDs, all v1alpha1. Model and InferenceService at the core; the Helm chart also installs ModelRouter, ModelPool, GPUQuota, LoRAAdapter, MetalNode and FederatedCluster (Foreman's CRDs come with its own chart)
• In-cluster runtimes. llama.cpp (GGUF, plus a router mode that serves several models from one server), vLLM, SGLang, PersonaPlex (Moshi speech-to-speech; bring your own image), and any container (generic). TGI is still accepted as a legacy r
• On Apple Silicon through the Metal Agent. llama-server, mlx-server, oMLX, vllm-swift, Ollama and TensorFold, as native processes
• Model sources. http(s)://, hf://, s3:// (S3-compatible stores), pvc://, oci:// (image volumes, Kubernetes 1.36+), and file:// or absolute paths under allowlisted host roots
• Persistent model cache, download once, deploy instantly (guide)