flashinfer-ai / flashinfer
FlashInfer: Kernel Library for LLM Serving
Why this job: matched by what the repo talks about: llm, serv, kernel, library
What it does
- Paged and Ragged KV-Cache. Efficient memory management for dynamic batch serving
- Decode, Prefill, and Append. Optimized kernels for all attention phases
- MLA Attention. Native support for DeepSeek's Multi-Latent Attention
- Cascade Attention. Memory-efficient hierarchical KV-Cache for shared prefixes
- Sparse Attention. Block-sparse and variable block-sparse patterns
Get started
pip install flashinfer-python- Maintenance
- Active
- Last commit today
- Stars
- 6,885
- +37 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- Linux
- From the latest release
- Latest version
- v0.7.3
- Released 6 days ago
- Built with
- Cuda
- 80% of the code
Is it looked after?
- Contributors
- 33
- Last 90 days
- Issues
- 75% closed
- 492 still open
- Pull requests
- 5,551 merged
- 929 waiting
- Releases
- 378
- 30+ in the last year
- Contributing guide (opens in new tab)
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| vllm-project / vllm | 96,634 | today | Apache-2.0 | Mac, Linux |
| SemiAnalysisAI / InferenceX | 2,016 | today | Apache-2.0 | No downloads |
| llm-d / llm-d | 5,202 | yesterday | Apache-2.0 | No downloads |
More: download files, languages, topics and project record
Languages
- Cuda 80%
- Python 19%
- C++ 1%
- Lean <1%
Topics
- gpu
- large-large-models
- cuda
- pytorch
- llm-inference
- jit
- attention
- nvidia
- distributed-inference
- moe
Project record
- JobLocal LLMs
- StatusListed
- Commits5,861
- Forks1,719
- Watching56
- Stars in 28 days+287
- CreatedJul 22, 2023
- First seen by usNov 7, 2026
- CheckedNov 10, 2026