Optimizing Large Language Models: Cloudflare's Triple Strategy
Three key techniques to run Kimi and GLM models faster and more efficiently.
1Challenges of Large Models
Large language models like Moonshot's Kimi K-series and Z.ai's GLM stand out with their long-context support and mixture-of-experts architectures. However, their substantial GPU memory footprint makes efficient serving challenging. Cloudflare developed three core optimizations to overcome memory constraints when running these models on Workers AI infrastructure: KV cache quantization, model weight compression, and shared cache protection.
2KV Cache Quantization
As models generate text, they store attention keys (K) and values (V) for processed tokens in a KV cache. Cloudflare reduced memory usage by half by storing this cache in 8-bit floating point (FP8) format instead of default 16-bit (BF16). For Kimi K2.6, this increased storable context from ~686k tokens to ~1.37M. While FP8 adds slight per-token overhead, it enables supporting up to 64 concurrent requests on the same GPU compared to BF16's limit of 32.
3Model Weight Compression
For GLM 5.2, weights were compressed from 8-bit floating point (FP8) to 4-bit integers (INT4), shrinking model checkpoints from 705GB to 421GB (40% reduction). In an 8-way tensor-parallel deployment, per-GPU memory decreased from ~88GB to ~52GB. INT4 weights reduce memory bandwidth during decode, significantly improving latency at low concurrency levels where it matters most.
4Prefill/Decode Separation
Cloudflare's disaggregated architecture allows optimal format selection per phase. Prefill (context loading), being compute-bound, uses BF16 KV cache and FP8 weights. Decode (text generation), being memory-bound, uses FP8 KV cache and INT4 weights. This hybrid approach delivers 10,160 tokens/s for GLM prefill while maintaining lower decode latency.
5Cache Integrity Protection
With optimizations enabling hundreds of requests to share GPU memory, integrity risks emerge in paged attention mechanisms. Cloudflare implemented KV cache validation by tagging physical cache pages and recording expected page/tag mappings per request. Mismatches trigger request abortion. This safety layer adds less than 1% performance overhead while preventing cache corruption.
6SGLang Collaboration
All experiments and production traffic run on SGLang, an open-source inference serving framework identified by Cloudflare as market-leading in performance. The teams collaborate closely to upstream optimizations, making advancements available to the broader open-source community. This partnership accelerates innovation in large model deployment.
7Performance Gains
FP8 KV cache delivers 41% higher tokens/s (2,192) and 30% lower cost/token versus BF16 for Kimi K2.6. INT4 weight compression substantially reduces decode latency. Crucially, all optimizations maintain model accuracy - FP8 and INT4 variants prove indistinguishable from original models across evaluation suites.
8Industry Impact
These optimizations enable Cloudflare to serve more customers at lower costs, particularly advancing commercialization of long-context models. By contributing techniques to open-source, they may accelerate industry-wide large model deployment efforts beyond Cloudflare's own infrastructure, potentially reshaping the economics of AI service provision.
Original article: Cloudflare Blog — Cloudflare'ın Büyük Dil Modellerini Optimize Etme Stratejileri