Cloudflare Resolves Subtle MySQL Protocol Caching Bug in Hyperdrive General Availability
Eliminating cache desynchronization caused by capability mismatches between split nodes in distributed poolers.
1Hyperdrive MySQL Reaches General Availability
Cloudflare announced general availability for MySQL support in Hyperdrive, its connection pooler for Workers. Initially built solely for PostgreSQL, expanding to MySQL required hardening the system through roughly twenty critical bug fixes. The release targets a production availability milestone of 99.99% for real-world serverless database workloads.
2Solving Edge Connection Bottlenecks
Establishing standard SQL connections consumes at least five network round trips, which severely degrades serverless execution. Hyperdrive holds warm database connections open across Cloudflare's distributed network, sharing them across Worker invocations. It also caches common query responses globally, minimizing latency and preventing origin connection exhaustion on high-volume clusters.
3Encoding Fixes During Handshake
Early beta testing exposed character encoding failures where emojis returned corrupted data. The system initially negotiated utf8 during handshake, which in MySQL acts as an alias for three-byte utf8mb3. Switching the handshake to utf8mb4 resolved the issue by properly supporting four-byte Unicode characters across edge database connections.
4Connection Reset State Leakage
To prevent session leaks between Worker checkouts, engineers integrated the COM_RESET_CONNECTION command. However, this lightweight command automatically resets the connection back to the origin server's default charset instead of the handshake configuration. If the origin default is latin1, encoding broke, requiring an explicit utf8mb4 re-negotiation post-reset.
5The 0.001 Percent Cache Anomaly
A high-volume beta customer reported unexpected empty responses occurring roughly 0.001% of the time. Standard telemetry failed to pinpoint the execution traces because the failure rate was so low. Engineers discovered that the anomaly occurred exclusively on instances with caching enabled, narrowing the investigation to the wire protocol.
6Distributed Topology and Shared Caches
Hyperdrive employs a split architecture comprising a Client node near the Worker and an Endpoint node near the origin database. Both nodes read and write to the local data center CDN cache. When collocated in the same facility, they share cache entries indexed by normalized wire message hashes.
7Handling MySQL Capability Differences
MySQL capability negotiation establishes protocol behavior during startup. Node.js libraries like mysql2 lack CLIENT_DEPRECATE_EOF support, expecting intermediate EOF packets rather than modern OK packets introduced in MySQL 5.7.5. Hyperdrive supports the flag upstream, forcing it to dynamically translate packet formats between origin databases and legacy client libraries.
8The Flawed Protocol Translation Assumption
The PostgreSQL implementation cached the exact bytes sent to the Worker, an assumption that broke under MySQL's dynamic capability negotiation. The Endpoint node works exclusively with Hyperdrive's native capabilities, while the Client node translates packets for the Worker. Sharing untranslated or partially translated wire bytes corrupted downstream parser state.
9Asymmetric Collocated Cache Poisoning
When an Endpoint cached its raw response, a collocated Client served those bytes directly without Worker-level translation. Conversely, when a Client cached translated bytes, an Endpoint reading that entry forwarded them to another Client that re-translated the stream. This double conversion injected early EOFs, producing empty results or connection hangs.
10Standardizing on Canonical Cache Storage
Cloudflare eliminated the bug by enforcing a single canonical wire format in the cache, encoded strictly using Hyperdrive's capabilities. Regardless of which node writes the entry, reading nodes apply local Worker capability transformations consistently. This architectural fix resolved the empty query errors and lifted client success rates to 99.99%.