How Meta's GEM Model Doubled Advertising Efficiency
Trained at LLM scale on thousands of GPUs, GEM is revolutionizing ad recommendation systems.
1What is the GEM Model?
Meta's Generative Ads Recommendation Model (GEM) is the AI foundation behind ad recommendations on Instagram and Facebook. The model features a hybrid architecture with trillions of sparse and billions of dense parameters. GEM trains on user engagement data and ad content, specifically optimized for advertising-domain data properties that differ from typical LLM workloads. This customization enables superior performance in ad recommendation tasks compared to generic large language models.
2Challenges of Large-Scale Training
Training GEM across thousands of GPUs presents unique engineering challenges. The model's hybrid architecture and advertising-specific data characteristics prevent direct application of standard LLM training methods. Particularly, variable-length input sequences (jagged inputs) and asymmetric interaction patterns make it difficult to fully utilize GPU compute units efficiently. These challenges necessitate co-design of specialized hardware and software solutions tailored to recommendation system workloads.
3Compute Efficiency Innovations
Meta engineers developed custom kernels to boost GEM's training efficiency. Techniques like Jagged Flash Attention (JFA) and Generalized Dot-Product Attention (GDPA) enable efficient processing of variable-length sequences. Additionally, ultra-low precision training methods like MXFP8 were specifically optimized for advertising workloads. These innovations maximize utilization of latest-generation GPU architectures while maintaining model quality for sensitive advertising prediction tasks.
4Scalability Solutions
For efficient parallel training across thousands of GPUs, Meta implemented a 5D parallelism approach. This topology-aware method combines 2D FSDP + Expert Parallelism (for dense parameters) with Fully Sharded 2D Model Parallelism (for sparse parameters). Co-designed with Meta's multi-tiered network hierarchy, this system significantly reduces communication overhead while enabling linear scaling. The solution addresses unique challenges posed by GEM's hybrid parameter distribution.
5Numerical Precision Challenges
Advertising optimization tasks (CTR/CVR prediction) show exceptional sensitivity to numerical changes. Naive low-precision training approaches risk model quality degradation. Meta engineers developed mixed-precision training recipes that maintain stability while improving computational efficiency. This innovation proves particularly crucial for MLPs with small embedding dimensions, where numerical precision significantly impacts model performance in advertising applications.
6Memory Management Obstacles
Long sequences and large activations push memory usage to its limits, forcing activation recomputation that erodes efficiency. During GEM training, variable sequence lengths across samples (jagged sequences) create data-driven load imbalance across GPU ranks. Meta implemented advanced memory management techniques to mitigate these issues, including optimized activation checkpointing and load balancing strategies tailored to recommendation system characteristics.
7Model FLOPs Utilization (MFU)
Meta successfully doubled GEM's end-to-end training efficiency to 20-25% MFU. MFU measures what percentage of a hardware's theoretical compute capacity is actually utilized. This improvement resulted from co-optimization of local MFU (compute efficiency) and scaling ratio (scaling efficiency). While achieving this, Meta also scaled total training FLOPs 4x over the past 12 months, demonstrating both efficiency and capacity gains in their advertising AI infrastructure.
8Custom Attention Mechanisms
GEM employs specialized attention mechanisms for two feature categories: sequence features (like user activity history) and non-sequence features (like user location). These mechanisms operate independently per group while enabling cross-feature learning. The interplay between GEM's hybrid architecture and advertising-domain data properties creates unique training challenges that demand architectural innovations beyond standard transformer models used in LLMs.
9Distributed Training Latency
In distributed training, end-to-end latency per training step is determined by the maximum across GPU ranks of either local compute time or communication time. Near-linear scaling requires four conditions: total compute time exceeding communication time, communication hidden behind compute without contention, minimal recomputation from memory pressure, and good load balancing across ranks. GEM's workload uniquely threatens all these requirements due to its hybrid parameter distribution and advertising-specific data patterns.
10Meta's Efficiency Framework
Meta engineers developed a framework that transforms complex co-design efforts into actionable technical levers. Training efficiency is measured through E2E MFU, which decomposes into two factors: local MFU (compute efficiency) and scaling ratio (scaling efficiency). Local MFU evaluates single GPU compute unit utilization against hardware rooflines, determined by kernel design and numerical precision. The scaling ratio reflects distributed system efficiency, addressing challenges unique to GEM's hybrid architecture at massive scale.
11Implementation and Results
Meta's innovative approaches delivered dramatic improvements in GEM's training efficiency. While scaling total training FLOPs 4x over 12 months, they achieved 20-25% end-to-end MFU. This success stems from synergistic combination of custom kernel designs, mixed-precision training, and advanced parallelism strategies. The work establishes new benchmarks for large-scale recommendation system training, demonstrating that advertising models can achieve LLM-scale efficiency through domain-specific optimizations.
12Industry Significance
GEM's success represents a milestone not just for Meta's advertising systems but for all companies developing large-scale recommendation models. These innovations beyond traditional LLM training methods open new possibilities for models with hybrid architectures and domain-specific data characteristics. Meta's work pushes the boundaries of AI infrastructure, inspiring the industry while delivering tangible business value through improved ad recommendation quality and efficiency.