# How Cloudflare Saves Petabytes of Cache Storage

Efficient cache management achieved with Zstandard and Pingora

- **Source:** [Cloudflare Blog](https://blog.cloudflare.com)
- **Published:** 2026-09-01
- **Topic:** [Software & Engineering](https://brifing.si/c/engineering)
- **Original article:** <https://blog.cloudflare.com/cache-transcoding/>
- **On Brifing:** https://brifing.si/s/6f420737-7e77-41b6-bd3b-d0cb44374ff7

## Rising Cache Storage Costs

Sharp increases in RAM and HDD prices over the past year pushed large-scale storage providers like Cloudflare to seek efficiency gains. For distributed systems like CDN services, expanding cache capacity is critical. This need spawned a new solution prototyped through the 1.1.1.1 Intern Program involving engineering interns.

## Implementing Zstandard Compression

Cloudflare engineers developed Cache Transcoding system that compresses eligible assets with Zstandard (zstd) before disk storage. This lossless compression algorithm, open-sourced by Facebook's Yann Collet in 2016, reduces data to 1/3 original size on average. Tests proved it's 42% faster than Brotli and 11.3% more efficient than gzip.

## Balancing CPU and Storage

The system uses zstd level 3 to optimize the compression-to-speed ratio. A minor CPU increase trades for petabytes of effective cache capacity. Compression cost is paid only when assets first enter cache, while storage and bandwidth savings compound with each reuse.

## Selective Data Compression

The prototype targets only compressible text content (HTML, JSON, CSS, JavaScript) which constitutes 67.3% of requests. Already compressed media like images/videos/fonts (63.3% byte ratio) are excluded. The solution achieved 71% success rate for uncompressed text above 4 KiB.

## Efficiency in Tiered Cache

Compressed assets transfer between data centers using less bandwidth. Data remains compressed through cache tiers, only being decoded for end users. This approach improves backbone efficiency while enabling more objects per disk.

## Increased Cache Density

Smaller file sizes allow servers to store more objects, increasing cache density. This reduces likelihood of useful content being evicted due to uncompressed versions occupying unnecessary space. Tests showed significant capacity gains using existing hardware.

## Popular Content Limit Unnecessary

Initially considering compression only for hot content, engineers found this reduced storage savings without proportionally cutting CPU costs. Instead, compressing all eligible text above 4 KiB delivered maximum storage benefits while staying within CPU budget.

## Future Implications

Cloudflare's innovation marks a milestone for resource efficiency in large-scale distributed systems. Smart implementation of modern compression algorithms like zstd maintains service quality despite rising hardware costs. This approach provides valuable insights for other cloud providers facing similar infrastructure challenges.

## Sources covering this story

- https://blog.cloudflare.com/cache-transcoding/ — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._