# The Hidden Cost of LLM Inference: KV Cache Memory Overhead How KV cache management impacts GPU memory utilization during concurrent LLM serving - **Source:** [Towards Data Science](https://towardsdatascience.com) - **Published:** 2026-09-16 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/3bf3125b-2353-4bd9-b3fb-82456fca428f ## The invisible memory bottleneck During LLM inference, GPU memory stores three components: model weights (fixed), transient activations, and the variable-sized KV cache. While weights get most attention, the KV cache grows with concurrent requests and context length. A 70B model's weights might need 140GB, but its KV cache can exceed 160GB at just 128 concurrent 4K-token requests. This dynamic scaling makes memory, not compute, the first constraint during traffic spikes. ## KV cache anatomy Each token's KV cache size multiplies four factors: 2 bytes per key/value, number of layers (e.g., 80), key/value heads (e.g., 8), and head dimension (e.g., 128). For Llama 3.1 70B at BF16 precision, this equals ~320KB per token. Without grouped-query attention reducing head count, this would be 8x larger. The cache's growth scales linearly with both token count and concurrent requests, unlike fixed-weight memory. ## Traditional allocation waste Early serving systems allocated KV memory as contiguous blocks sized for maximum sequence length (e.g., 4K tokens), leaving unused portions stranded. vLLM measured 60-80% waste from this over-reservation and fragmentation. This explains why GPUs hit out-of-memory errors at low compute utilization - the allocator fences off capacity that exists physically but isn't addressable contiguously. ## PagedAttention breakthrough Inspired by OS virtual memory, PagedAttention divides the KV cache into fixed-size non-contiguous blocks. Requests dynamically allocate blocks as needed and release them after use. This reduces waste from 80% to under 4%, enabling up to 24x higher throughput by fitting more concurrent requests. The small bookkeeping overhead (scattered memory access) is negligible compared to reclaimed memory. ## Prefix caching optimization For workloads with repeated prefixes (system prompts, few-shot examples), storing shared KV entries once boosts efficiency. SGLang's RadixAttention organizes prefixes in a radix tree for automatic sharing, achieving 5x throughput gains when hit rates are high. However, this only helps when requests genuinely share prefixes - unique one-off prompts see minimal benefit and incur lookup overhead. ## Quantization tradeoffs Reducing KV cache precision (FP8 or 2-bit via KIVI) can halve or quarter memory usage, enabling larger batches. However, quality impact varies: long-context retrieval may drop from 91% to 13% accuracy without proper calibration. Smaller models degrade more than larger ones. Teams must validate quantization on their specific workloads before deployment, as aggregate benchmarks can mislead. ## Deployment strategy Optimization should follow traffic patterns: PagedAttention is mandatory for all deployments. Prefix caching helps only with measurable prefix reuse. Quantization becomes necessary when still memory-bound after other optimizations. Critical metrics include peak concurrency (not average), cache hit rates, and long-context accuracy. Some workloads become compute-bound before hitting memory limits. ## Architectural implications Modern architectures increasingly adopt grouped-query attention to reduce KV head count. NVIDIA's inference guides formalize this math. The memory-compute balance shifts with model size - smaller models often hit compute limits first, while larger ones remain memory-bound. Teams must profile their specific workload to identify the actual bottleneck before optimizing. ## Production imperative KV cache management is now foundational for LLM serving. What appears as a hardware limitation often stems from inefficient memory allocation. vLLM's techniques have become baseline requirements, with prefix caching and quantization as conditional optimizations. The key insight: size deployments for peak concurrency spikes, where memory pressure emerges suddenly, not gradual load increases. ## Sources covering this story - https://towardsdatascience.com/the-kv-cache-tax-why-inference-servers-run-out-of-memory-before-compute/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._