Optimizing DNS Build Pipelines to Reduce Latency and Memory Usage
Targeted build planning cuts DNS propagation delay by half and memory usage by 85%.
1Scaling DNS Build Challenges
Cloudflare's Zone Builder pipeline faced significant scaling pressure as event volume grew from 250 to 700 changes per second. With some zones exceeding 20 million records, the legacy full-build approach became unsustainable. The system needed to evolve from rebuilding entire zones to processing only the specific changes required for each update.
2The Full Build Bottleneck
The original architecture required reading, encoding, and comparing every record in a zone for every event. This process held the entire zone in memory, which is impossible for zones with tens of millions of records. Consequently, the team needed a more granular method to identify and process only necessary DNS data changes.
3Targeted Build Planning Logic
The new planner classifies events into five distinct build types: resources only, resources and metadata, apex and metadata, purge, or full zone. By identifying the smallest safe build, the system avoids redundant processing. Every written key must be byte-for-byte identical to a full build to ensure consistency across the global network.
4Managing Shared Metadata Keys
Quicksilver stores shared metadata keys that act as directories for authoritative DNS servers. Because multiple internal zone IDs can exist for a single public domain during onboarding, the builder must carefully update these keys. The system rebuilds the entire metadata key from a fresh view rather than attempting to patch older versions.
5Unified Scheduling and Merging
The team replaced separate schedulers for record and full-zone work with a single unified scheduler. This component combines multiple events for the same zone into one build, significantly reducing overhead. If any event requires a full-zone update, the scheduler escalates the task, ensuring that no critical state changes are ever lost.
6Ensuring Data Freshness
Planning occurs twice: once upon event arrival and again immediately before execution. The system reads a fresh publication snapshot from read-only database replicas to account for state changes during the wait. This double-check mechanism ensures that targeted builds remain accurate even if the underlying database state evolves during the pipeline delay.
7Handling Kafka Partition Ordering
To prevent data loss, the system uses soft committing for Kafka messages. While builds can finish out of order, the committed position only advances after every earlier message completes. This ensures that if the Zone Builder restarts, the system replays necessary messages rather than skipping potentially critical DNS record updates.
8Memory and Latency Gains
By implementing targeted builds, the team achieved an 85% reduction in peak memory usage. This efficiency provided the necessary headroom to shorten pipeline waits, effectively halving the time required for a DNS change to propagate to Quicksilver. These improvements allow the system to handle significantly larger zones without hitting memory limits.
9Safety Through Equivalence Testing
The team implemented rigorous equivalence tests comparing targeted builds against full builds from identical database states. These tests verify that every Quicksilver key matches byte for byte. By pinning time-dependent values during testing, engineers confirmed that the planner and scheduler correctly handle complex scenarios like wildcards, nameserver changes, and zone purges.
10Handling Massive Zone Thresholds
For zones exceeding a specific record-count threshold, the Zone Builder never escalates to a full build. Instead, it rebuilds only the apex, metadata, and affected names. This strategic tradeoff prevents the system from stalling, as full builds for these massive zones would otherwise fail to complete within the available memory.
11Managing Concurrent Zone Updates
The scheduler prevents any single busy zone from monopolizing workers by enforcing per-zone rebuild intervals. If multiple changes arrive for the same zone, they are merged into the next scheduled build. This approach balances the need for rapid propagation with the necessity of maintaining system stability under high-load conditions.
12Architectural Evolution Summary
Transitioning from full-zone rebuilds to targeted, event-driven planning solved the scaling limitations of the DNS pipeline. By integrating smarter scheduling and strict equivalence verification, the system now processes 700 events per second with greater efficiency. This architecture provides a robust foundation for supporting zones that continue to grow toward hundreds of millions.