alibaba / tair-kvcache

What it does

• LLM-aware matching. prefix matching, reverse rolling sliding-window matching, hybrid attention (Mamba/SWA), and batch queries by block key, with multi-LocationSpec organization for hybrid attention.
• Two-phase writes. reserve a write location, write the data, then confirm; only confirmed blocks become readable.
• Multi-tenant quotas and fair reclaim. Instance Group quotas, backend watermarks, Group LRU / usage-proportional policies, asynchronous eviction that does not block foreground requests.
• Tiered storage. DRAM/SSD hot-cold tiers, watermark-triggered background Copy or Mark writes, expanding effective cache capacity while controlling the hot tier.
• Inference-side cache state reporting. inference services use ReportEvent incremental updates + optional snapshots to bring L1.5/L2 cache state into the global index; GetHostCacheState evaluates host-local and global hits.