The Hidden Cost of LLM Inference: KV Cache Memory Overhead
How KV cache management impacts GPU memory utilization during concurrent LLM serving
1The invisible memory bottleneck
During LLM inference, GPU memory stores three components: model weights (fixed), transient activations, and the variable-sized KV cache. While weights get most attention, the KV cache grows with concurrent requests and context length. A 70B model's weights might need 140GB, but its KV cache can exceed 160GB at just 128 concurrent 4K-token requests. This dynamic scaling makes memory, not compute, the first constraint during traffic spikes.
2KV cache anatomy
Each token's KV cache size multiplies four factors: 2 bytes per key/value, number of layers (e.g., 80), key/value heads (e.g., 8), and head dimension (e.g., 128). For Llama 3.1 70B at BF16 precision, this equals ~320KB per token. Without grouped-query attention reducing head count, this would be 8x larger. The cache's growth scales linearly with both token count and concurrent requests, unlike fixed-weight memory.
3Traditional allocation waste
Early serving systems allocated KV memory as contiguous blocks sized for maximum sequence length (e.g., 4K tokens), leaving unused portions stranded. vLLM measured 60-80% waste from this over-reservation and fragmentation. This explains why GPUs hit out-of-memory errors at low compute utilization - the allocator fences off capacity that exists physically but isn't addressable contiguously.
4PagedAttention breakthrough
Inspired by OS virtual memory, PagedAttention divides the KV cache into fixed-size non-contiguous blocks. Requests dynamically allocate blocks as needed and release them after use. This reduces waste from 80% to under 4%, enabling up to 24x higher throughput by fitting more concurrent requests. The small bookkeeping overhead (scattered memory access) is negligible compared to reclaimed memory.
5Prefix caching optimization
For workloads with repeated prefixes (system prompts, few-shot examples), storing shared KV entries once boosts efficiency. SGLang's RadixAttention organizes prefixes in a radix tree for automatic sharing, achieving 5x throughput gains when hit rates are high. However, this only helps when requests genuinely share prefixes - unique one-off prompts see minimal benefit and incur lookup overhead.
6Quantization tradeoffs
Reducing KV cache precision (FP8 or 2-bit via KIVI) can halve or quarter memory usage, enabling larger batches. However, quality impact varies: long-context retrieval may drop from 91% to 13% accuracy without proper calibration. Smaller models degrade more than larger ones. Teams must validate quantization on their specific workloads before deployment, as aggregate benchmarks can mislead.
7Deployment strategy
Optimization should follow traffic patterns: PagedAttention is mandatory for all deployments. Prefix caching helps only with measurable prefix reuse. Quantization becomes necessary when still memory-bound after other optimizations. Critical metrics include peak concurrency (not average), cache hit rates, and long-context accuracy. Some workloads become compute-bound before hitting memory limits.
8Architectural implications
Modern architectures increasingly adopt grouped-query attention to reduce KV head count. NVIDIA's inference guides formalize this math. The memory-compute balance shifts with model size - smaller models often hit compute limits first, while larger ones remain memory-bound. Teams must profile their specific workload to identify the actual bottleneck before optimizing.
9Production imperative
KV cache management is now foundational for LLM serving. What appears as a hardware limitation often stems from inefficient memory allocation. vLLM's techniques have become baseline requirements, with prefix caching and quantization as conditional optimizations. The key insight: size deployments for peak concurrency spikes, where memory pressure emerges suddenly, not gradual load increases.
Original article: Towards Data Science — The Hidden Cost of LLM Inference: KV Cache Memory Overhead