Tailscale Uncovers 16-Year-Old SQLite Bug Causing Database Corruption
A deep forensic analysis reveals an ancient SQLite WAL bug that disrupted cloud infrastructure.
1Critical Infrastructure Issue
Tailscale experienced recurring outages in its control plane in early 2026. Investigations revealed mysterious corruptions in SQLite databases. These corruptions prevented users from accessing device lists and establishing new connections. Each incident caused at least one hour of downtime on affected shards.
2SQLite's Unexpected Vulnerability
Tailscale had been using SQLite as its primary database since 2022. This 'boring technology' was known for working flawlessly at scale. But in August 2023, the first corruption was detected in S3 backups. The PRAGMA integrity_check command confirmed the database was genuinely corrupted.
3Recurring Crises
Over six months, 19 separate database corruption incidents occurred. Each time, the control plane on affected shards had to be stopped. Corruptions weren't tied to specific shards, customers, or load patterns. Tailscale dedicated significant engineering resources to solve the problem.
4User Impact
During corruption events, devices in affected tailnets couldn't see network changes. The web admin console and API became temporarily unavailable. New devices coming online couldn't establish connections. In early incidents, some device additions and config changes failed to persist.
5Erosion of Trust
Every incident was announced globally, even when only small tailnet groups were affected. Most users saw outages that didn't impact them directly. Repeated downtime caused trust issues even among unaffected customers. Tailscale's reliability promise came under scrutiny.
6Challenging Investigation
The team initially examined recent changes but found nothing relevant. Low-level SQLite interaction code hadn't been touched for years. No common factors emerged among corruptions - not shard, customer, or load pattern. Unable to reproduce synthetically, they deployed passive monitoring in production.
7Forensic Process
Corruptions occurred irregularly - sometimes hours apart, sometimes weeks. This unpredictability made progress tracking difficult. Tailscale engineers implemented forensic telemetry in live environments. Gathering live diagnostics was a last resort, but necessary without reproduction steps.
8Historical Bug Discovery
Intensive analysis revealed a 16-year-old SQLite Write-Ahead Logging (WAL) bug. This rare edge case caused unexpected database file resets during WAL mode operation. The bug was so obscure even SQLite's developers were unaware of its existence.
9Solution and Fix
Tailscale engineers collaborated with SQLite developers to identify the root cause. Fixes were applied to both Tailscale's code and SQLite itself. By summer 2026, systems stabilized and corruption incidents ceased. The solution also benefited other SQLite users.
10Backup Strategy Overhaul
The incident chain prompted Tailscale to revise its backup approach. Instead of full database snapshots every few minutes, they implemented incremental backups. The new system carries lower corruption risk and significantly faster recovery times. S3 upload strategy was also reviewed.
11User Communication
Tailscale rebuilt trust through transparent communication. This blog post details technical aspects, impacts, and resolution process. They apologized to users and recommitted to service quality. The response serves as a model for incident communication.
12Industry Implications
This incident showed even 'boring' technologies like SQLite can harbor obscure bugs. Tailscale's experience highlighted database robustness in distributed systems. It proved the critical role of forensic processes for cloud providers. The industry initiated new security reviews of SQLite implementations.
Original article: Hacker News (news.ycombinator.com) — Tailscale, 16 Yıllık SQLite Hatasını Çözdü