# Encoding Domain Expertise: The Context Layer Behind Spotify's Data Assistant

How an AI data assistant delivers reliable answers at scale

- **Source:** [Spotify Engineering](https://engineering.atspotify.com)
- **Published:** 2026-06-10
- **Topic:** [Data & AI](https://brifing.si/c/data-ai)
- **Original article:** <https://engineering.atspotify.com/2026/6/encoding-your-domain-expert-the-context-layer-behind-spotifys-data-assistant/>
- **On Brifing:** https://brifing.si/s/f234c148-ed61-49c4-8cf5-35c89086ee5c

## The Growing Burden of Data Needs

With thousands of teams moving fast at Spotify, demand for data experts outstripped capacity. Traditional methods - searching dashboards or waiting for Slack responses - became inadequate. Processing 1.4 trillion daily data points across 70,000+ datasets required a new approach to data accessibility.

## The Limits of Raw Data

Dumping all schemas into an LLM isn't a solution. Schemas lack context: An INT64 column doesn't reveal that values under 100 are test data or define 'active user'. The model might confidently select wrong tables - potentially leading to dangerous conclusions without proper domain framing.

## The Context Layer Solution

Spotify created 'clusters' that understand slices of the data warehouse. Each cluster, owned by expert teams, contains three components: Relevant datasets (with schema and profiling), vetted question-SQL pairs (curated examples), and additional business context (terminology, pitfalls, definitions).

## The Necessity of Human Curation

An automated approach was tested: Generating question-SQL pairs from historical queries using LLMs. Experts approved only 12.5% of proposed pairs - the rest were debugging, one-off queries, or incorrect patterns. Most query history proved to be 'noise' - experts were essential for identifying true signal.

## Operating via ReAct Loop

When questions arrive, the assistant: Selects appropriate context, writes SQL queries, executes against the warehouse, and returns answers with sources. It uses a ReAct loop - reasoning and acting incrementally based on tool outputs. Users can trace how results were generated, not just see final answers.

## User-Friendly Interfaces

The solution integrated with existing workflows: Slack bot for quick questions, MCP server for IDEs/AI tools, dedicated web UI for exploration. The assistant transparently indicates when knowledge gaps exist - this honesty builds trust in the answers it does provide.

## Scaling Expertise, Not Replacing It

This project aims to scale expertise, not replace experts. 2,100+ users engaged in 13,000+ conversations - a quarter had never written SQL. The cluster model enabled reliable data access across 177 domains including ads, podcasts, music, and finance while maintaining data integrity.

## Sources covering this story

- https://engineering.atspotify.com/2026/6/encoding-your-domain-expert-the-context-layer-behind-spotifys-data-assistant/ — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._