Optimizing Memory Usage in Large-Scale Distributed Systems
How Cloudflare saved 100TB of RAM through algorithmic improvements in consistent hashing
1The scale challenge
Cloudflare operates thousands of servers worldwide with petabytes of RAM and millions of CPU cores. At this scale, even 1% improvements in resource usage translate to significant savings. The Performance team identified excessive memory usage in Pingora Backend Router's consistent hashing implementation, which was using more memory than expected for load balancing cacheable requests across servers.
2Consistent hashing basics
Consistent hashing distributes tasks across servers while minimizing reorganization when servers are added/removed. Cloudflare uses it to route cacheable requests by URL, maintaining one file copy per data center. The method maps servers and tasks to a 32-bit number line based on hash values, assigning each task to the nearest server to its left on the line.
3The imbalance problem
With basic consistent hashing, server workload distribution becomes uneven because hash outputs are random. Statistical analysis shows that with 100 servers, each handles about 0.99% of total requests on average, but actual values vary significantly (99% coefficient of variation). This means some servers get disproportionately more requests than others.
4Multiple hashes solution
Adding multiple hash points per server helps balance workloads. While each individual hash range varies, combining many hashes per server (default is 160 in NGINX/Pingora) reduces variation. For 100 servers, 160 hashes per server drops the coefficient of variation from 99% to 8%, significantly improving distribution fairness.
5Weighted distribution
Cloudflare further customizes hashing by weighting servers based on their capabilities. The ketama algorithm allows assigning weights proportional to disk space (for storage-heavy servers) or CPU/GPU count (for compute-intensive workloads). This ensures more capable servers handle proportionally more requests.
6Feature-specific rings
Different server capabilities (compliance requirements, caching features) require separate hash rings. Each feature combination needs its own ring, multiplying memory usage. This complexity became a major memory consumer in Cloudflare's infrastructure.
7Memory optimization breakthrough
Engineer Zaidoon identified a memory inefficiency in how Rust stores hash points. Due to alignment rules, each Point struct (containing hash and index) occupied 8 bytes when it could use just 6. By storing values as raw byte arrays with getters, they reduced memory usage by 25% for hashing data structures.
8The hash collision dilemma
Increasing hashes improves distribution but introduces collisions (birthday paradox). With 32-bit hashes, collisions become significant beyond 10,000 hashes per server. Analysis showed Cloudflare could reduce hashes by 90% (from 160 to 16 per server) without appreciable error, dramatically cutting memory needs.
9Careful migration strategy
Changing hash rings affects request routing and could invalidate cached content. Cloudflare ran both old and new rings simultaneously, using a migration framework to gradually shift traffic. They rolled out changes in layers, starting with small validation locations before expanding globally, monitoring performance throughout.
10Memory savings achieved
The optimization reduced PBR's memory usage by 100TB globally. Monitoring showed a sharp drop when old hash rings were decommissioned. This came on top of another 100TB saved by the DNS team the previous month, demonstrating how small algorithmic improvements create massive savings at Cloudflare's scale.
11Implementation availability
All optimizations are now available in the pingora-ketama crate as an unadvertised cargo feature. The v2 ring includes compact storage format, faster sorting, and adjustable hash counts per node. These changes prioritize stability while delivering significant memory reductions for large-scale deployments.
12Scale economics
Cloudflare's experience demonstrates how infrastructure at extreme scale requires different optimization approaches. Small percentage improvements create massive absolute savings, making even complex algorithmic changes worthwhile. The solution combines mathematical analysis, systems knowledge, and careful implementation to achieve results impossible at smaller scales.
Original article: Cloudflare Blog — Optimizing Memory Usage in Large-Scale Distributed Systems