Advancing Generative Recommendation Inference with NVIDIA TensorRT LLM
Spotify optimizes generative recommendation inference through a strategic technical collaboration with NVIDIA.
1Transitioning to Generative Recommendations
Spotify is shifting from traditional multi-step recommendation systems to generative models. This transition aims to provide real-time, deeply personalized content across various surfaces. Moving from proof of concept to production-grade architecture presents significant engineering hurdles, particularly regarding latency and the need for scalable inference stacks capable of handling complex user interaction data.
2Beam Search Implementation Challenges
The initial podcast recommendation system utilized vLLM for beam search, a technique for identifying high-probability token sequences. However, vLLM’s implementation operated outside the core GPU engine, causing CPU bottlenecks and inefficient GPU utilization. This forced Spotify to use lower-tier accelerators to manage costs, limiting the system's overall performance potential.
3Optimizing Candidate Scoring Techniques
Spotify explored candidate scoring to rank recommendations using log probabilities. While a 'fan-out' approach is common, it is inefficient. By adopting SGLang, the team utilized efficient candidate masking to score multiple items in a single request. This method significantly reduces computational overhead compared to running separate inference passes for every individual candidate.
4Personalization via Embedding to Prefix
To personalize LLM outputs, Spotify moved away from text-heavy prompts, which increase latency. Instead, they implemented an 'Embedding to Prefix' approach. By projecting existing user embeddings into the LLM space as a single soft token, the model can personalize recommendations without the massive overhead associated with long, raw text-based user histories.
5Collaborating with NVIDIA on TRT-LLM
Spotify partnered with NVIDIA to unify inference features like beam search, scoring, and soft token injection into TensorRT LLM. The collaboration focused on closing feature gaps, resulting in critical additions such as multi-sequence beam search support, out-of-tree model handling, and multi-modal embedding capabilities, which were essential for Spotify's specific recommendation requirements.
6Customizing TensorRT LLM Plugins
Using TensorRT LLM’s out-of-tree model support, the team implemented a customized Qwen3 extension plugin. This plugin enables the injection of user vectors via multi-modal parameters, facilitating the soft token approach. Additionally, the new C++-based sampler design significantly improved GPU utilization by reducing host overheads and implementing more flexible, PyTorch-reliant scheduling.
7Quantifiable Performance and Cost Gains
These optimizations delivered measurable results for Spotify’s 1.7B Qwen3 model deployments. The implementation achieved up to 54% cost reduction for beam search workloads and 7% for scoring tasks. These efficiency gains validate the decision to move toward a unified, high-performance inference stack tailored for large-scale generative recommendation systems.
8Future Directions in Inference Scaling
Looking forward, Spotify plans to continue optimizing its inference stack by exploring quantization and NVIDIA Blackwell-specific techniques. The goal remains to support increasingly sophisticated personalization at scale while maintaining strict control over latency and infrastructure costs. This ongoing collaboration ensures that Spotify’s generative capabilities remain both performant and economically sustainable.