cacheMon / PrefixSim

What it does

Engines such as vLLM and SGLang keep the KV cache of finished requests, so a new request that shares a prefix with an earlier one skips recomputing it. When the cache is full, an eviction policy decides which blocks to drop, and that choice decides how much prefill compute the cache saves. prefixsim replays real LLM request traces through such a cache and compares eviction policies by what matters: the compute saving ratio (prefill compute avoided), not just the block hit ratio.