RAP Technology for Instant Point Queries in Data Lakes
How Spotify achieves millisecond access to petabytes for AI agents and personalization.
1The need for instant big data access
Platforms like Spotify require millisecond responses on petabytes of user history for AI agents and personalization. While traditional SQL engines take seconds even for single-row queries, RAP technology reduces latency 100x. For queries like 'what did I listen to last summer?', it jumps directly to 12 relevant files out of 90,000, eliminating full scans.
2Bridging the data lake-KV store gap
80% of companies maintain separate systems for analytics and operational data. While KV stores like Bigtable are expensive at petabyte scale, data lakes suffer from slow access. Even S3 Express One Zone's 1ms storage gets bogged down by Trino/BigQuery's 1000ms overhead. RAP unifies both worlds in Parquet format, balancing cost and performance.
3The dependent read chain problem
Traditional methods require: 1) Reading file footers 2) Parsing row group metadata 3) Scanning key columns 4) Checking page indexes. Each step depends on the previous, causing 40ms delays to extract 100 bytes from 4MB files. In cloud environments, concurrent queries compound this latency exponentially.
4The external indexing revolution
RAP uses a multimap index that directly maps keys (user IDs) to file locations and row numbers. The system creates 1GB index per 1TB data, deterministically narrowing 90,000 files to precisely 12 relevant ones—unlike probabilistic Bloom filters. Index fragments grow append-only, enabling 1000x performance gains for point queries.
5Parallel read architecture
RAP's key innovation is breaking dependent read chains, enabling parallel column access. Latency drops from 10ms to 0.1ms in cloud, microseconds on SSD, and nanoseconds in memory. For AI agents performing filtering, SQL execution, and LLM context building, end-to-end latency decreases by 90%.
6Seamless existing infrastructure integration
Data engineers don't need to rewrite Parquet files for RAP. The same files generated by ML pipelines, notebooks, and batch analytics work out-of-the-box. The index builder automatically creates key-row mappings by reading page locations from footers. New data triggers incremental index fragment growth.
7Radical cost optimization
Where traditional methods read entire 4MB pages to extract 100B, RAP fetches only relevant 4KB chunks—achieving 99.9% cloud storage cost reduction. For 1 million queries, data transfer drops from 4PB to 4TB. At AWS S3 Express's $0.16/GB, monthly bills shrink from $640k to $640.
8Write-time optimizations
For ultra-low latency, Parquet internals can be optimized during writes: reducing row groups from 10k to 1k rows, positioning keys first in schemas, applying delta encoding, and precomputing PageIndexes to shrink 4MB pages below 100KB. These preparations prove critical for financial transactions and similar systems.
9Industry implications
Forrester reports 74% of enterprises plan to unify operational/analytical platforms by 2025. Solutions like Snowflake Unistore and Databricks Delta Live Tables offer similar approaches, but RAP uniquely preserves existing Parquet ecosystems. Spotify's 2-week pilot reduced AI query latency from 850ms to 9ms.
10Transforming AI assistants
LLMs require instant historical data access for context building. RAP enables answering 'my top 2023 artist' in 10ms instead of 1 second. A user's 5-year, 50GB listening history becomes searchable via 50MB indexes. This technology will particularly revolutionize personal assistants in healthcare and finance.
11Future roadmap
The RAP team is working on loading indexes into GPU memory for nanosecond access. Another focus is optimizing indexes for time-series range queries like 'last 7 days'. Integrations with Apache Iceberg and Delta Lake formats are underway, with an open-source release expected by late 2024.
12A paradigm shift in data engineering
RAP blurs lines between batch and stream processing, giving data lakes operational system performance. This shift will demand data engineers master 'data access patterns' over 'distributed systems expertise'. Gartner predicts 40% infrastructure cost reductions by 2026, with real-time analytics markets tripling in size.
Original article: Spotify Engineering — Veri Gölünde Hızlı Nokta Sorguları için RAP Teknolojisi