FlatCake: Why Tiered KV Cache Storage Is Holding LLM Serving Back

<h1> FlatCake: Why Tiered KV Cache Storage Is Holding LLM Serving Back </h1> <p>When you send a long prompt to an LLM API, the service has to store the key-value (KV) cache for your session somewhere. For short conversations, GPU HBM handles it fine. But as context windows stretch to hundreds of thousands of tokens — and agentic workflows keep sessions alive for minutes or hours — the KV cache gro

Читать первоисточник →