Tiiny-AI / PowerInfer

What it does

• Locality-centric design. Utilizes sparse activation and 'hot'/'cold' neuron concept for efficient LLM inference, ensuring high speed with lower resource demands.
• Hybrid CPU/GPU Utilization. Seamlessly integrates memory/computation capabilities of CPU and GPU for a balanced workload and faster processing.
• Easy Integration. Compatible with popular ReLU-sparse models.
• Local Deployment Ease. Designed and deeply optimized for local deployment on consumer-grade hardware, enabling low-latency LLM inference and serving on a single GPU.
• Backward Compatibility. While distinct from llama.cpp, you can make use of most of examples/ the same way as llama.cpp such as server and batched generation. PowerInfer also supports inference with llama.cpp's model weights for compatibilit