Cloudflare Saves 100TB RAM Through Algorithm Optimization
Memory footprint reduction in Pingora Backend Router demonstrates power of small-scale optimizations at hyperscale
1Hyperscale optimization challenges
Cloudflare's global infrastructure operates thousands of servers with petabytes of RAM, where even 1% improvements yield massive gains. The Performance team identified memory inefficiency in Pingora Backend Router's consistent hashing implementation, consuming more RAM than expected. This discovery led to a focused optimization effort targeting the pingora-ketama library, Cloudflare's open-source solution for distributed request routing.
2Consistent hashing fundamentals
The system uses consistent hashing to distribute cacheable requests across servers by URL, maintaining single-copy storage per data center. Hash functions map servers and tasks to a 32-bit number line, where tasks are assigned to the nearest left server. While conceptually simple, this method creates workload imbalances because hash distributions are statistically random, causing uneven request allocation across servers.
3Addressing workload imbalance
To equalize server workloads, Cloudflare implements multiple hashes per server rather than single mappings. While NGINX and Pingora default to 160 hashes/server, statistical analysis showed this could be reduced. The ketama algorithm further refines this by weighting servers based on storage capacity, ensuring request distribution matches available resources. These techniques significantly reduce the coefficient of variation in workload distribution.
4Memory optimization breakthrough
Engineer Zaidoon identified wasteful memory usage in the Point struct storing hash mappings. The original implementation used 32-bit integers for server indices when 16-bit would suffice. Rust's memory alignment rules initially prevented space savings, but a byte array workaround reduced the structure size by 25%. This change alone reclaimed significant memory across Cloudflare's global infrastructure.
5Mathematical validation
Statistical analysis revealed that reducing hash counts by 90% wouldn't appreciably impact distribution accuracy. The team derived precise formulas for standard deviation with multiple hashes, considering 32-bit hash collision probabilities. Simulations confirmed predictions, showing acceptable error rates even with reduced hashing. This mathematical foundation justified the aggressive optimization strategy.
6Rollout strategy
Changing hash rings risked massive cache invalidation, so Cloudflare implemented a phased migration. PBR temporarily maintained both old and new rings, using a stable migration framework to gradually shift traffic. The rollout progressed from validation locations to larger data center groups, carefully monitoring backend selection, memory usage, and cache behavior at each stage.
7Feature-specific routing
The system accommodates server capability variations through multiple hash rings. Compliance requirements and caching features create server subsets for specific requests. Unlike workload balancing, this requires entirely separate rings for each feature combination. The optimization had to preserve this functionality while reducing memory overhead.
8Performance impact
The changes reduced Pingora-ketama's memory footprint significantly, reclaiming over 100TB RAM globally. This follows another 100TB reduction by the DNS team, demonstrating Cloudflare's focus on hyperscale efficiency. The optimization maintains request distribution quality while dramatically decreasing resource consumption across all data centers.
9Engineering culture
Cloudflare's Performance team exemplifies resource-conscious engineering at scale. Their work balances equitable resource sharing between teams while pushing infrastructure limits. The DNS and Pingora optimizations together saved 200TB RAM, showcasing how small improvements compound in hyperscale systems. This culture of measurement and optimization drives continuous infrastructure improvements.
10Rust implementation details
The memory optimization required careful Rust programming to overcome alignment rules. While #[repr(packed)] was considered controversial, the team implemented a safer solution using byte arrays with accessor methods. Both approaches compile to the same efficient machine code, but the latter maintains better memory safety guarantees in Rust's ownership model.
11Monitoring and validation
During rollout, engineers tracked multiple metrics: backend selection traces, ring-version counters, connection errors, memory usage, startup times, and cache behavior. This comprehensive monitoring ensured the optimization didn't degrade performance. The phased approach allowed quick rollback if needed, though none was required as the changes proved stable at each deployment stage.
12Hyperscale efficiency lessons
Cloudflare's optimization demonstrates how mathematical rigor and careful implementation can yield massive infrastructure savings. The project combined statistical analysis, systems programming, and phased deployment to safely achieve dramatic memory reductions. Such optimizations become increasingly valuable as global infrastructure scales, where small percentage gains translate to terabyte-scale savings.
Original article: Hacker News — Cloudflare Saves 100TB RAM Through Algorithm Optimization