Scaling Vector Search Infrastructure at Airtable
Custom architecture designed to manage billions of embeddings.
1Productionizing Semantic Search
Airtable's data infrastructure team identified the need for a vector database to support semantic search at global scale. They designed a system capable of managing billions of embeddings, supporting multi-tenancy, and meeting performance targets in a distributed cloud environment.
2Use Cases
Omni (Airtable's AI chat) answers natural language questions from large datasets. For example, it can respond to queries like 'What are customers saying about battery life lately?' across hundreds of thousands of rows, providing meaningful answers beyond traditional search methods.
3Design Priorities
The system was built on four core goals: Low-latency queries (500ms p99), high write throughput, horizontal scalability, and self-hosted infrastructure. These objectives aim to protect customer data privacy while delivering top-tier user experience.
4Technology Selection
In 2024, Milvus vector database was chosen after evaluation. Its self-hostable nature, elastic scalability, and strong tenant isolation made it ideal. Milvus's distributed architecture can manage millions of customer data points efficiently.
5Architectural Decisions
Separate physical partitions were created for each Airtable base. This approach simplified data deletion while improving performance. Initial tests with 100k partitions showed latency issues, leading to a 1,000-partition limit per collection.
6Indexing Strategy
HNSW indexing method was selected for its 99%-100% accuracy rate. While other options (IVF-SQ8, DiskANN) used less memory, HNSW's balance of performance and accuracy better met Airtable's requirements.
7Data Flow
The system consists of two core flows: Ingestion flow (converting Airtable rows to embeddings and storing in Milvus) and query flow (processing user queries and delivering relevant rows to LLM). Both flows operate continuously and reliably at scale.
8Scalability Solutions
Each Milvus cluster supports up to 400k bases. New clusters are provisioned as additional customers onboard. This approach optimizes resource usage while maintaining performance. Partition load times were reduced to under 30 seconds.
9The Future of Search
Airtable's vector infrastructure demonstrates how semantic search is becoming a core product feature. This system enables meaningful search across large datasets while providing an architecture that meets enterprise requirements.
Original article: Airtable Engineering — Airtable'da Vektör Arama Altyapısının Ölçeklendirilmesi