# Towards Free Normalization: Fusing Normalization into GEMM and Attention Kernels Novel Techniques for Significant Performance Improvements - **Source:** [PyTorch Blog](https://pytorch.org/blog) - **Published:** 2026-07-10 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/b757424d-d3b6-447a-b51f-539d42454984 ## The Importance of Normalization Normalization techniques play a critical role in stabilizing training and accelerating convergence in deep learning models. Techniques like LayerNorm and RMSNorm are widely used in large language models and recommendation systems. ## Performance Challenges Normalization operations are memory-bound, leading to performance bottlenecks. This prevents full utilization of hardware compute capabilities and extends training time. ## Kernel Fusion Techniques Meta researchers developed novel techniques to fuse normalization operations with GEMM (General Matrix Multiplication) and Attention kernels, reducing memory access costs. These techniques can hide up to 90% of normalization latency. ## Lazy Pre-Norm and Multi-CTA Norm Fusion Techniques like Lazy Pre-Norm and Multi-CTA Norm Fusion enable more efficient integration of normalization with matrix multiplications. These approaches make better use of hardware compute and memory resources. ## FlashNormAttention Algorithm The FlashNormAttention algorithm fuses multiple normalization operations around an Attention kernel. This method achieves up to 35% kernel speedup, providing significant performance improvements. ## TLX and Helion DSLs This work utilizes DSLs like TLX and Helion. TLX offers low-level support for GPU execution control, while Helion provides rapid development and portability. ## Hardware and Data Type Benchmark tests were conducted on NVIDIA B200 GPUs using the bfloat16 data type in Meta's data centers. The hardware operated under a 750 W power cap. ## Role of Normalization in Models In Meta's Kunlun architecture, normalization techniques like LayerNorm and RMSNorm are widely used in components such as Multi-Head Attention and GDPA. These techniques account for 20% of the model's training time. ## Tiling Strategies The fundamental challenge between normalization and GEMM operations stems from different tiling strategies. Normalization requires access to data along a dimension, while GEMM tiles in both dimensions. ## Naive Fusion Solution The naive fusion solution forces the GEMM algorithm to follow the same tiling strategy as normalization. While this works well for small N values, it becomes inefficient as N increases. ## Performance Results The developed techniques significantly reduce normalization latency and improve hardware compute efficiency. This provides a notable advantage, especially in large language models. ## Conclusion and Significance This work introduces a new approach to improving normalization performance. Techniques integrated with GEMM and Attention kernels shorten training time and enhance hardware efficiency in deep learning models. ## Sources covering this story - https://pytorch.org/blog/towards-free-normalization-fusing-normalization-into-gemm-and-attention-kernels/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._