InftyAI / llmaz

What it does

• Easy of Use. People can quick deploy a LLM service with minimal configurations.
• Broad Backends Support. llmaz supports a wide range of advanced inference backends for different scenarios, like vLLM, Text-Generation-Inference, SGLang, llama.cpp, TensorRT-LLM. Find the full list of supported backends here.
• Heterogeneous Cluster Support. llmaz supports serving the same LLM with heterogeneous devices together with InftyAI Scheduler for the sake of cost and performance.
• Various Model Providers. llmaz supports a wide range of model providers, such as HuggingFace, ModelScope, ObjectStores. llmaz will automatically handle the model loading, requiring no effort from users.
• Distributed Inference. Multi-host & homogeneous xPyD support with LWS from day 0. Will implement the heterogeneous xPyD in the future.