How Meta's GEM Model Doubled Ad Efficiency
The massive AI model trained on thousands of GPUs is revolutionizing ad recommendations.
1What is the GEM Model?
Meta's Generative Ads Recommendation Model (GEM) is a massive AI foundation powering ad recommendations across Instagram and Facebook. With trillions of sparse and billions of dense parameters, GEM features a hybrid architecture. It processes both sequential features (like user activity history) and non-sequential features (like user location) simultaneously, trained on ad content and user engagement data.
2Unique Engineering Challenges
Training GEM presents unique challenges at the intersection of recommendation systems and LLMs. Variable sequence lengths in user activity history lead to computational inefficiency. Asymmetric attention patterns (long queries/short keys) and the numerical sensitivity of ad optimization tasks (CTR/CVR prediction) make direct application of standard LLM training methods impossible, requiring novel solutions.
3Custom Kernel Design
Meta engineers developed specialized kernels like Jagged Flash Attention (JFA) and Generalized Dot-Product Attention (GDPA) specifically for GEM. These kernels leverage the latest GPU architectures. Ultra-low precision training methods like MXFP8 were optimized for ad workloads. These customizations maximize hardware utilization efficiency during training.
45D Parallelism Architecture
To scale efficiently across thousands of GPUs, Meta implemented topology-aware 5D parallelism. The system combines 2D FSDP + Expert Parallelism for dense parameters with Fully Sharded 2D Model Parallelism for sparse parameters. Co-designed with Meta's multi-tier network hierarchy, this approach significantly reduces communication overhead during distributed training.
5Compute Efficiency Breakthroughs
Local MFU (Model FLOPs Utilization), measuring single GPU compute unit usage, was improved through kernel design and numerical precision optimizations. Custom attention mechanisms better map the workload's compute patterns to GPU architecture (Tensor cores, memory hierarchy). This brings performance closer to the hardware's theoretical limits.
6Scaling Efficiency Solutions
In distributed training, step latency is determined by the maximum of compute and communication times across GPU ranks. GEM's trillion parameters make this balance challenging. Meta's solution hides communication behind computation and minimizes recomputation from memory pressure. These innovations dramatically improved the scaling ratio.
7Numerical Precision Challenge
Ad click-through (CTR) and conversion rates (CVR) predictions are extremely sensitive to numerical changes. Naive low-precision training caused quality regression. Meta developed sophisticated low-precision recipes - MXFP8 for attention layers and custom mixed precision approaches for MLPs. This maintained model quality while improving efficiency.
8Memory Optimization
Long sequences with large activations push memory usage to its limits, forcing recomputation that hurts efficiency. Meta introduced new normalization techniques and block-based attention mechanisms like BlockAttention to optimize memory consumption. These innovations reduced the need for activation recomputation during training.
9Load Balancing Innovation
Variable sequence lengths across samples created data-dependent load imbalance across GPU ranks. Meta developed topology-aware load balancing algorithms to ensure all GPUs complete work simultaneously. This minimized idle time where faster ranks waited for slower ones, improving overall training efficiency.
10Network Hierarchy Design
Meta's multi-tier network infrastructure was co-designed with the 5D parallelism approach. Streaming Multiprocessor (SM)-free collectives reduced communication overhead. Different parallelism strategies for dense and sparse parameters optimized network traffic. This design significantly alleviated communication bottlenecks across thousands of GPUs.
114x Growth in 12 Months
Over the past year, Meta successfully scaled GEM's training FLOPs by 4x while doubling end-to-end training efficiency to 20-25% Model FLOPs Utilization (MFU). This achievement resulted from hardware/software co-design and innovative parallelism approaches. The system continues to scale efficiently for even larger training runs.
12Industry-Wide Impact
GEM's success has significant implications beyond Meta for the entire ad industry. The training efficiency breakthroughs enable more accurate, personalized ad recommendations. This technology promises to improve advertiser ROI while enhancing user experience. Meta's innovation appears set to raise the industry standard for ad recommendation systems.
Original article: Engineering at Meta — Meta'nın GEM Modeli ile Reklam Verimliliğini İkiye Katlaması