# 3x Performance Boost by Removing One Log Line How an unnecessary log statement was degrading system performance - **Source:** [SoundCloud Backstage Blog](https://brifing.si/source/33500dd9-f9dd-4139-8ac3-4efdec37df99) - **Published:** 2026-08-11 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/f19824b7-2d3e-4110-87f7-4214c5f7fb31 ## Discovery of the Performance Issue SoundCloud engineers observed the 99th percentile (p99) response time of their ad server service drop from 48ms to 18ms simply by removing one log statement. CPU usage decreased proportionally. Before the change, p99 never went below 40ms; afterward, it never exceeded 40ms. The two performance distributions barely overlapped, showing a clear improvement. ## Critical System Details SoundCloud's ad system dynamically determines ad frequency for free-tier users. This decision-making now runs through an internal microservice that decides whether to show an ad pod or continue without ads. The service has two key characteristics: it requires low latency as it's user-facing, and it supports multiple policy variants through the same API endpoint. ## Root Cause Analysis The service API accepts statistical state data (posteriors) for Bayesian policies, but fixed-policy callers send none by design. The validation code was generating a warning log and metric increment for every missing action. Since fixed-policy callers had all actions 'missing', they produced 4 warnings per request. This unnecessary processing occurred on 100% of their traffic. ## Initial Misdiagnosis Engineers first suspected Python string formatting costs - building log messages with set serialization and string interpolation on every request. However, memory graphs disproved this theory. While memory usage did decrease (from ~170-175MB to ~150-160MB per pod), the improvement wasn't proportional to the 3x latency reduction. ## Identifying the True Cost The actual bottleneck was the log writing process itself. The structured logging system rendered, serialized, and wrote to stdout synchronously within each request. This operation held the GIL during serialization and parked requests mid-flight during writing. While negligible in average latency, this cost dominated the tail latency measurements. ## The Fix The solution required just 5 lines added and 2 lines removed. The new code only warns when some posteriors are provided but others are missing (a true anomaly). Fixed-policy callers no longer trigger any warnings. Most of the change consisted of comments explaining why this optimization was necessary. ## Importance of Measurement This incident highlighted the value of measurement over code reading. The warning system was visible on dashboards with no active alerts. The issue was discovered through performance metrics showing higher-than-expected resource consumption. Engineers emphasize trusting data when prioritizing optimizations. ## Architectural Lessons The experience demonstrated that logging decisions in hot paths require careful consideration. "Don't emit logs on 100% of hot path traffic" became a key architectural principle. It also revealed the impact of synchronous logging in Python on GIL contention and tail latency. ## Scalability Impact SoundCloud's ad decision system is expanding to cover all free-tier listening sessions. Without this change, logging costs would have scaled with traffic. The 3x latency improvement provides crucial headroom for future growth while maintaining service quality at scale. ## Performance Optimization Philosophy This case underscores the "measure first, optimize second" approach. While engineers initially pursued the wrong theory, data guided them to the true issue. It also serves as a reminder that systems can be more expensive to run than expected, even when nothing appears broken. ## Conclusion and Takeaways Removing a single log statement yielded a 3x performance improvement in SoundCloud's ad infrastructure. This incident revealed how invisible costs accumulate in systems and how small changes can have outsized impacts. Engineering teams are now reevaluating the cost of every log line in production. ## Sources covering this story - https://developers.soundcloud.com/blog//one-line-of-python-and-a-3x-drop-in-p99 — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._