NVIDIA / kvpress
LLM KV cache compression made easy
Why this job: matched by what the repo talks about: llm, cache, compression, easy
What it does
Deploying long-context LLMs is costly due to the linear growth of the key-value (KV) cache in transformer models. For example, handling 1M tokens with Llama 3.1-70B in float16 requires up to 330GB of memory. kvpress implements multiple KV cache compression methods and benchmarks using 🤗 transformers, aiming to simplify the development of new methods for researchers and developers in this field.
Get started
pip install kvpress- Maintenance
- Active
- Last commit 2 days ago
- Stars
- 1,249
- +4 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- v0.5.5
- Released 56 days ago
- Built with
- Python
- 99% of the code
Is it looked after?
- Contributors
- 17
- Last 90 days
- Issues
- 96% closed
- 5 still open
- Pull requests
- 148 merged
- 11 waiting
- Releases
- 28
- 10 in the last year
- Contributing guide (opens in new tab)
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| LMCache / LMCache | 12,344 | today | Apache-2.0 | Linux |
| huawei-csl / KVarN | 519 | 4 months ago | Apache-2.0 | No downloads |
| novitalabs / pegaflow | 221 | yesterday | Apache-2.0 | Linux |
More: languages, topics and project record
Languages
- Python 99%
- Shell 1%
- Makefile <1%
Topics
- llm
- inference
- kv-cache
- kv-cache-compression
- long-context
- python
- pytorch
- transformers
- large-language-models
Project record
- JobLocal LLMs
- StatusListed
- Commits149
- Forks198
- Watching17
- Stars in 28 days+19
- CreatedNov 6, 2024
- First seen by usNov 7, 2026
- CheckedNov 12, 2026