localai-org / vllm.cpp

What it does

• One 66 MiB binary instead of a 9.1 GiB install. A flat, exception-free, llama.cpp-style C ABI
• GGUF as a first-class citizen. Load the same quantized files llama.cpp uses, and on CPU
• EXL3 trellis weights. A stock 3.0 bpw Llama-3.2-1B-Instruct checkpoint generates on CPU and
• SGLang's good ideas, as documented toggles. RadixAttention / prefix caching, LPM cache-aware
• Runs on hardware people actually have. CUDA, CPU, Metal, and Vulkan ship from one tree. ROCm