Encoding Domain Expertise: The Context Layer Behind Spotify's Data Assistant
How an AI data assistant delivers reliable answers at scale
1The Growing Burden of Data Needs
With thousands of teams moving fast at Spotify, demand for data experts outstripped capacity. Traditional methods - searching dashboards or waiting for Slack responses - became inadequate. Processing 1.4 trillion daily data points across 70,000+ datasets required a new approach to data accessibility.
2The Limits of Raw Data
Dumping all schemas into an LLM isn't a solution. Schemas lack context: An INT64 column doesn't reveal that values under 100 are test data or define 'active user'. The model might confidently select wrong tables - potentially leading to dangerous conclusions without proper domain framing.
3The Context Layer Solution
Spotify created 'clusters' that understand slices of the data warehouse. Each cluster, owned by expert teams, contains three components: Relevant datasets (with schema and profiling), vetted question-SQL pairs (curated examples), and additional business context (terminology, pitfalls, definitions).
4The Necessity of Human Curation
An automated approach was tested: Generating question-SQL pairs from historical queries using LLMs. Experts approved only 12.5% of proposed pairs - the rest were debugging, one-off queries, or incorrect patterns. Most query history proved to be 'noise' - experts were essential for identifying true signal.
5Operating via ReAct Loop
When questions arrive, the assistant: Selects appropriate context, writes SQL queries, executes against the warehouse, and returns answers with sources. It uses a ReAct loop - reasoning and acting incrementally based on tool outputs. Users can trace how results were generated, not just see final answers.
6User-Friendly Interfaces
The solution integrated with existing workflows: Slack bot for quick questions, MCP server for IDEs/AI tools, dedicated web UI for exploration. The assistant transparently indicates when knowledge gaps exist - this honesty builds trust in the answers it does provide.
7Scaling Expertise, Not Replacing It
This project aims to scale expertise, not replace experts. 2,100+ users engaged in 13,000+ conversations - a quarter had never written SQL. The cluster model enabled reliable data access across 177 domains including ads, podcasts, music, and finance while maintaining data integrity.
Original article: Spotify Engineering — Spotify'nin Veri Asistanı İçin Uzman Bilgisini Kodlamak