Airtable's Real-Time Query Updates in Multi-Threaded Backend
Discover how real-time synchronization was maintained during the migration to Rust-based database architecture.
1The Magic Behind Real-Time Sync
Airtable's core feature is instant propagation of user edits to all connected clients, from simple cell changes to schema modifications. For example, a cattle rancher's location update immediately reflects in their partner's viewing query. Preserving this functionality was a top priority during the architectural migration, as it defines the Airtable user experience.
2Limitations of Single-Threaded Legacy
In the old system, all queries were processed sequentially by a single thread. The write thread knew all previous queries at timestamp N+1 and calculated updates accordingly. While simple, this architecture created significant scalability constraints, especially with high concurrent users, often resulting in performance bottlenecks.
3Challenges in Multi-Threaded World
In the new architecture, read and write threads operate independently. A write at N+1 might commit while N+1 reads are still open. Initially, an interlock mechanism between threads was considered but rejected as it would eliminate concurrency advantages. Instead, engineers developed a more radical solution.
4Revolutionary Solution: Moving Query Updates
The team relocated query update generation from write threads to real-time threads. This required two major changes: 1) Complete rewrite of legacy query management logic, 2) Leveraging next-gen database's Incremental View Maintenance (IVM) capabilities. The IVM system, inspired by DBSP, includes innovative additions for efficient row order change calculations.
5Power of Independent Processing Loops
Each client connection is now processed independently, preventing slow queries from delaying updates for others. The old system's single processing loop had to wait for all N+1 timestamped queries to complete. This new approach significantly improved performance, especially for users working with large datasets.
6Event Subscription System
The system developed an event subscription mechanism to serialize writes and new query registrations per client. The critical rule: A write at N+1 for a client can only be processed after all its N+1 queries complete. This guarantees data consistency while maximizing concurrency. The database abstracts timestamp and query ordering complexities through a simple event stream interface.
7Architectural Comparison
Old system: Write thread computed updates and passed to real-time thread. New system: 1) Write thread only modifies data, 2) Real-time thread computes updates using IVM, 3) Each client processed independently. These changes potentially increase throughput 10x while reducing latency by 70%.
8Impact on User Experience
This architectural transformation creates visible differences for Airtable users. Instant updates work seamlessly even when large teams run complex queries simultaneously. In 50+ user environments, system response times improved by up to 300%. These advancements set important benchmarks for similar database solutions at enterprise scale.