ZGateway: Meta's Proxy Solution for ZippyDB Scalability
How Meta unified traffic and improved reliability with a proxy layer for its key-value store.
1Introduction to ZGateway
Meta introduced ZGateway as a proxy layer for ZippyDB, its most widely-used key-value store. ZGateway unifies client traffic, enabling admission control, load balancing, and cross-region resilience. ZippyDB serves billions of operations per second globally, storing product metadata, counters, and configurations. The proxy addresses challenges posed by a sprawling client fleet of over a million hosts owned by hundreds of teams.
2The Need for a Proxy
ZippyDB's direct-access model created a dense mesh of TLS connections, with clients holding tens of thousands of outbound connections and database hosts accepting similar inbound ones. This mesh proved wasteful and fragile, consuming resources even while idle. Sudden drops in connection reuse, like during deployments, could trigger host crashes due to file-descriptor exhaustion. The proxy layer decouples client and database fleets, collapsing the mesh into bounded hops.
3Benefits of ZGateway
ZGateway provides three key advantages: it bounds the problem by hiding the client population from the backend, creates a home for shared work like pooling and retries, and establishes a control point for workload visibility. The proxy tier allows Meta to implement features once, centrally, rather than in every client. This approach improves efficiency, performance, scalability, and most importantly, reliability across the global infrastructure.
4Connection Management
Under direct access, reconnection storms could become catastrophes, as seen in one incident where a routing bug caused clients to open excessive connections. ZGateway contains such storms at the proxy tier, which Meta controls and can harden centrally. The proxy currently handles 40% of ZippyDB traffic with just 6% computational overhead, projected to grow beyond 60% of traffic while maintaining this efficiency.
5Technical Implementation
ZGateway runs as stateless regional tiers discovered through ServiceRouter, Meta's hyperscale service mesh solution. It comes in two flavors: a pure proxy and a read-through cache, both sharing one pipeline. The system runs Meta's thick C++ client as its engine, with one internal client per use case. This architecture made moving capabilities to the proxy layer a natural evolution.
6Request Path Flow
Clients send requests over sticky connections to regional ZGateway hosts, which terminate TLS and apply authorization checks. The proxy performs admission control, validation, and shaping per tenant before resolving shards. For caching tiers, it checks local caches, then batches and coalesces misses and writes with other in-flight requests for the same shard before forwarding to database replicas.
7Performance Metrics
ZGateway handles over 1 billion operations per second while adding minimal overhead. The system's efficiency comes from collapsing the many-to-many connection mesh into manageable hops. This architecture reduces memory and CPU consumption significantly compared to the direct-access model, particularly by eliminating idle connection resource usage across the fleet.
8Evolution from Direct Access
Direct access was the right initial design for ZippyDB, but became problematic at scale. As client numbers grew into the thousands, the connection mesh transitioned from an inefficiency to a reliability limit. The timing for ZGateway aligned with improvements in ServiceRouter, Thrift overload protection, and other infrastructure that made a shared tier feasible to implement.
9Operational Advantages
ZGateway provides centralized observability and control that wasn't possible with direct access. Operators can now attribute load to specific sources and change behavior in minutes rather than waiting for fleet-wide client updates. The proxy also enables features like request batching and coalescing that would be impractical to implement across all clients independently.
10Reliability Improvements
By containing connection storms at the proxy tier, ZGateway prevents incidents like database host reboot loops. The system's centralized nature allows for more robust failure handling and quicker recovery. These reliability gains are particularly valuable for Meta's global operations, where downtime can have widespread impact across products and services.
11Future Growth
Meta projects ZGateway will soon handle over 60% of ZippyDB traffic. The proxy's architecture supports this growth while maintaining low overhead. Future enhancements may include additional caching strategies and optimization features that leverage the proxy's unique position in the request path to further improve performance and efficiency.
12Architectural Significance
ZGateway represents a mature evolution in Meta's database infrastructure strategy. The solution demonstrates how large-scale systems can benefit from inserting managed tiers between diverse clients and critical backends. This pattern of using proxies to collapse complexity may influence other distributed systems designs within Meta and across the industry.
Original article: Facebook Engineering — ZGateway: Meta's Proxy Solution for ZippyDB Scalability