# The New Approach to GPU Utilization: Diffusion Models A solution for enterprises' idle AI hardware investments - **Source:** [O'Reilly Radar](https://www.oreilly.com/radar) - **Published:** 2026-07-20 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/afb24d73-482b-4e99-9bbf-2a8c928bd003 ## Idling GPUs H100 GPUs sitting idle overnight at a Singapore bank reveal inefficient use of AI infrastructure. Finance teams question why machines costing more per hour than engineers operate at low capacity. ## The GPU Hangover Enterprises rushed to build private AI clusters but utilization fell short. This stems from mismatch between how standard models generate text and actual corporate needs. ## Limits of Autoregressive Models Models like Llama or GPT generate tokens sequentially, creating memory bandwidth bottlenecks. The full model weights reload for each token, starving GPU compute. ## Power of Batching Processing hundreds of requests simultaneously reduces cost 10-30x. But this fails for low-latency single requests like customer-facing systems where users wait. ## Diffusion Revolution Inspired by image generation, diffusion models process noisy token blocks in parallel, achieving arithmetic intensity in hundreds versus autoregressive models' near 1. ## Real-World Performance Inception Labs' Mercury hits 1,100 tokens/sec on H100s while Google pushes boundaries with Gemini Diffusion. Open-source LLaDA confirms similar scaling laws. ## Technical Superiority Diffusion models are 10x faster for low-latency single requests. But the advantage disappears in batch processing where both architectures reach compute limits. ## Speculative Decoding Rival Techniques like Medusa and EAGLE use small draft models for 2-4x speedups. Mixture-of-experts models activate fewer weights per token to reduce memory movement. ## Economic Equation The true metric is useful tokens per dollar. Diffusion's advantage applies only to specific workloads. Traditional methods suffice for batch processing. ## False Dichotomy There's no absolute winner between diffusion and autoregressive models. Comparisons ignoring speculative decoding mislead. Choose technology based on workload. ## Industry Impact These commercially available technologies by 2026 could revolutionize low-latency domains like finance and customer service. ## Efficiency Paradigm Beyond engineering metrics, consider business needs for AI infrastructure ROI. Diffusion models may breakthrough in GPU utilization for specific scenarios. ## Sources covering this story - https://www.oreilly.com/radar/the-tokens-you-cant-wait-for/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._