# How Spotify Maintains Quality at Scale with AI-Assisted Development Engineering insights from handling 100M concurrent users and 12M backend requests per second - **Source:** [Spotify Engineering](https://engineering.atspotify.com) - **Published:** 2026-09-16 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/b2ceb2fa-c1b8-4874-befe-c030833f3235 ## The scale challenge Spotify's infrastructure handles 777M monthly active users across 2,000+ devices, processing 12M backend requests per second. This complexity makes quality maintenance an ongoing challenge, especially with 500K new content uploads daily. The platform's distributed nature means weaknesses can quickly impact both listeners and creators, requiring robust systems to detect and resolve issues before they escalate. ## June 24 incident breakdown A perfect storm occurred when a scheduled batch job competed with new episodes, a quality improvement increased compute requirements, and a scheduling bug reduced throughput by 10%. Normally minutes-long publishing delays stretched to hours. Root causes included silent failures in media processing, insufficient video transcoding capacity, and gaps in workload controls - all pre-existing weaknesses exacerbated by volume. ## Post-incident improvements Spotify implemented end-to-end monitoring to detect failures before creators notice, fixed the scheduler, reprioritized batch jobs, and increased capacity. They redesigned service tiering to prioritize critical services during constraints. Bad actor content is now suppressed, reducing overall load. These changes required engineering judgment - AI helped deliver solutions faster but couldn't replace human oversight in system design decisions. ## Automation's double-edged sword Spotify's Fleet Management framework handles large-scale changes automatically, including a recent 3-day Java backend migration. While automation accelerates delivery, it introduces new failure modes. One automated dependency upgrade passed checks but failed in production, prompting strengthened safeguards, expanded rollback capacity, and scheduling changes during team working hours to mitigate risks. ## Capacity constraints from AI boom Industry-wide AI adoption has created compute shortages, reducing spare capacity. Spotify's regional failover system, designed to minimize customer impact during outages, faced new challenges. Earlier this year, failovers revealed previously trivial issues due to capacity constraints. The company responded by reviewing network edge approaches and doubling reserved edge capacity after a May incident. ## Quality-velocity balance Spotify's mobile app quality historically cycles between feature pushes and quality recovery periods. AI accelerated this cycle, requiring faster adaptation of quality measurement systems. The company broadened quality signals and added long-term trend analysis to detect accumulating regressions earlier, particularly for specific devices or edge cases not covered by existing metrics. ## Data-driven quality assessment Analyzing production incidents revealed AI-authored code wasn't a direct contributor, but increased change volume stressed verification controls. Monthly merged PRs more than doubled YoY to 17,000, with quality/optimization work rising to 31%. Rework metrics show no increase in quality debt, though code complexity and PR sizes are growing - potential leading indicators requiring monitoring. ## The verification bottleneck AI's greatest impact has been increasing change production capacity, shifting the constraint to verification capabilities. Spotify now focuses on aligning delivery systems - automated safeguards, rollback, observability, and quality measurement - with this accelerated pace. The key insight: maintaining quality at scale requires continuous evolution of controls to match development velocity. ## Sources covering this story - https://engineering.atspotify.com/2026/9/ai-changed-how-spotify-builds-what-we-learned-and-fixed-about-quality-at-higher-velocity/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._