Towards Free Normalization: Fusing Normalization into GEMM and Attention Kernels
Novel Techniques for Significant Performance Improvements
1The Importance of Normalization
Normalization techniques play a critical role in stabilizing training and accelerating convergence in deep learning models. Techniques like LayerNorm and RMSNorm are widely used in large language models and recommendation systems.
2Performance Challenges
Normalization operations are memory-bound, leading to performance bottlenecks. This prevents full utilization of hardware compute capabilities and extends training time.
3Kernel Fusion Techniques
Meta researchers developed novel techniques to fuse normalization operations with GEMM (General Matrix Multiplication) and Attention kernels, reducing memory access costs. These techniques can hide up to 90% of normalization latency.
4Lazy Pre-Norm and Multi-CTA Norm Fusion
Techniques like Lazy Pre-Norm and Multi-CTA Norm Fusion enable more efficient integration of normalization with matrix multiplications. These approaches make better use of hardware compute and memory resources.
5FlashNormAttention Algorithm
The FlashNormAttention algorithm fuses multiple normalization operations around an Attention kernel. This method achieves up to 35% kernel speedup, providing significant performance improvements.
6TLX and Helion DSLs
This work utilizes DSLs like TLX and Helion. TLX offers low-level support for GPU execution control, while Helion provides rapid development and portability.
7Hardware and Data Type
Benchmark tests were conducted on NVIDIA B200 GPUs using the bfloat16 data type in Meta's data centers. The hardware operated under a 750 W power cap.
8Role of Normalization in Models
In Meta's Kunlun architecture, normalization techniques like LayerNorm and RMSNorm are widely used in components such as Multi-Head Attention and GDPA. These techniques account for 20% of the model's training time.
9Tiling Strategies
The fundamental challenge between normalization and GEMM operations stems from different tiling strategies. Normalization requires access to data along a dimension, while GEMM tiles in both dimensions.
10Naive Fusion Solution
The naive fusion solution forces the GEMM algorithm to follow the same tiling strategy as normalization. While this works well for small N values, it becomes inefficient as N increases.
11Performance Results
The developed techniques significantly reduce normalization latency and improve hardware compute efficiency. This provides a notable advantage, especially in large language models.
12Conclusion and Significance
This work introduces a new approach to improving normalization performance. Techniques integrated with GEMM and Attention kernels shorten training time and enhance hardware efficiency in deep learning models.
Original article: PyTorch Blog — Normalizasyon İşlemlerinde Performans İyileştirmeleri