EfficientMoE / MoE-Infinity

What it does

• Cost-effective. Expert offloading to host memory and SSD lets memory-constrained GPUs serve MoE models that would otherwise not fit. DeepSeek-V4-Flash additionally offloads FP4-quantized experts.
• Fast. Activation-aware expert caching, prefetching, tracing, fused CUDA kernels, CUDA graph capture, Marlin INT4 GEMM, and FP4/MXFP4 expert paths keep the hot path lean.
• HuggingFace-native. The MoE class remains the current in-process synchronous API, but MoE.generate() emits DeprecationWarning and is scheduled for removal. Use MoE.serve() for continuous batching; it starts an async HTTP service and is not
• Production serving. OpenAI-compatible HTTP server with continuous batching, paged KV cache, request scheduling with preemption, streaming (SSE), runtime hot reload, watchdog/health monitoring, and crash-recovery logging. Opt-in correctness-
• Acceleration-aware. Automatically integrates with FlashAttention and uses FlashInfer where the selected standard paged-attention backend supports it, with graceful fallback to built-in kernels. DeepSeek MLA currently uses the correct PyTorc