Tailscale Uncovers 16-Year-Old SQLite Bug
Database corruptions traced back to a longstanding SQLite issue finally resolved.
1The Problem Emerges
Tailscale experienced increasing downtime towards the end of last year. Status page data showed this instability continued into the new year. Many outages were traced to a single SQLite bug. Identifying this fault required months of intensive forensic analysis. The team initially struggled to pinpoint the root cause of recurring database corruptions.
2SQLite Architecture
Tailscale's control plane is internally divided into coordination servers (shards). Each shard maintains its own SQLite database accessed exclusively by a single Go process. SQLite is specifically optimized for such single-writer designs. The company has used SQLite as its primary database since 2022, chosen for its reliability and widespread adoption in the industry.
3Backup System Design
The existing backup pipeline takes complete database snapshots every few minutes, uploading entire SQLite files to an S3 bucket. This system had operated flawlessly since early 2023 until August 2026, when a data pipeline reading these backups reported database corruption. The team confirmed the corruption using SQLite's PRAGMA integrity_check command.
4Corruption Detection
When the first corruption was detected, the affected database was repaired, but the underlying cause remained elusive. While SQLite corruption is technically possible, it's exceptionally rare during normal operations. The team expected this to be an isolated incident, but the problem persisted and intensified over time.
5Recurring Issues
Over six months, Tailscale faced 19 separate database corruption incidents. Each occurrence required stopping the control plane process on the affected shard during recovery. Initial recovery times exceeded one hour, though subsequent improvements reduced this duration. Every outage meant service disruption for tailnets on that shard.
6Impact Assessment
Since the control plane only handles configuration data, corruptions affected metadata about tailnets and devices but never compromised private encryption keys or network traffic. In early incidents, some newly added devices or configuration changes failed to persist. Online devices maintained connections but couldn't learn about network changes during outages.
7Trust Erosion
Tailscale posted global incident notices even for outages affecting small tailnet groups. This meant many users saw status alerts for incidents that didn't impact them directly. While most shards never experienced corruption, repeated downtime notifications gradually undermined overall user trust in the service's reliability.
8Debugging Challenges
Engineers first examined recent code changes but found nothing relevant. The low-level SQLite interaction code hadn't been modified in years and had previously been trouble-free. The team meticulously reviewed all this code but found no corruption-causing bugs. Corruptions showed no pattern across shards, customers, features, or load levels.
9Diagnostic Approach
Without reliable trigger conditions, synthetic reproduction proved impossible. The team had to deploy passive forensic telemetry in production to catch corruption in action. While gathering live database diagnostics was far from ideal, it became the only viable option. This approach required careful implementation to avoid impacting performance.
10Timing Complications
Corruptions occurred at irregular intervals—sometimes hours apart, sometimes weeks. This unpredictability made progress tracking and planning extremely difficult. The team couldn't anticipate when the next incident might occur, prolonging the stressful debugging process and delaying potential solutions.
11Resolution Achieved
By summer, the team gained confidence they'd identified and understood the bug—more importantly, they confirmed its fix. The solution came after months of persistent effort to uncover this longstanding SQLite core issue. The resolution marked the end of a challenging period that tested Tailscale's engineering capabilities.
12Lessons Learned
Tailscale apologized for failing to meet reliability expectations during this period. This experience demonstrated how even 'boring technology' like SQLite can harbor surprising edge cases. The company emphasized the importance of persistent root cause analysis and shared their findings to benefit the wider tech community.
Original article: Hacker News — Tailscale, 16 Yıllık SQLite Hatasını Çözdü