thu-ml / SageAttention
[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.
Why this job: topics: video-generation
What it does
- Optmized kernels for Ampere, Ada and Hopper GPUs.
- INT8 quantization and smoothing for $QK^\top$ with support for varying granularities.
- FP8 quantization for $PV$, and FP16 accumulator for FP8/FP16 $PV$.
- Two-level accumulation strategy for $PV$ to improve accuracy in FP8 MMA and WGMMA.
- Support torch.compile with non-cudagraphs mode and distributed inference.
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Inactive
- Last commit 9 months ago
- Stars
- 4,006
- +10 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- 1 releases
- None marked latest
- Built with
- Cuda
- 48% of the code
Is it looked after?
- Contributors
- Not stated
- Last 90 days
- Issues
- 45% closed
- 178 still open
- Pull requests
- 29 merged
- 42 waiting
- Releases
- 1
- 0 in the last year
- No contributing guide
- No security policy
- No code of conduct
More: languages, topics and project record
Languages
- Cuda 48%
- Python 28%
- C++ 22%
- C 2%
Topics
- attention
- inference-acceleration
- llm
- quantization
- cuda
- triton
- video-generation
- efficient-attention
- mlsys
- llm-infra
- vit
- video-generate
Project record
- JobVideo
- StatusListed
- Commits188
- Forks554
- Watching43
- Stars in 28 days+43
- CreatedOct 3, 2024
- First seen by usNov 7, 2026
- CheckedNov 8, 2026