Achieving Near-Linear Training Scalability for Pinterest's Foundation Models
How multi-node training efficiency was improved to 7.5x scale
1Pinterest's Scalability Challenge
Pinterest uses foundation models to power recommendations for 600M monthly active users. But adding a second machine in multi-node training slowed training by 5x. AWS EFA fixed the network layer but scaling remained poor: only 21% gain at 4 nodes.
2Core Distributed Training Issue
99% of model parameters reside in embedding tables that don't fit single-GPU memory. Sharding across GPUs with TorchRec created NCCL all-to-all communication bottlenecks. Profiling showed GPUs idle 46% of time waiting for network transfers.
3Quantized Communications Breakthrough
FBGEMM's FP8 quantization reduced communication payload by 75%, cutting the largest NCCL operation from 20% to 5% of forward pass. This improved 2-node scaling from 1.13x to 1.57x and 4-node from 1.21x to 2.3x with no quality loss.
4Batch Size Optimization
Increasing embedding layer batch size from 512 to 2048 improved communication efficiency, boosting 4-node scaling to 3.1x. Gaps between batches were minimized, optimizing GPU utilization while maintaining model convergence.
5Communication Overlapping
Restructuring communication ops to overlap with computation improved 4-node scaling to 3.5x. GPUs now spend less time idle by parallelizing computation and communication more effectively.
6NCCL Parameter Tuning
Optimizing NCCL_NSOCKS_PERTHREAD parameters improved socket utilization, achieving 3.9x at 4 nodes (97.5% of ideal) and 7.5x at 8 nodes. Resource addition now yields near-linear efficiency gains.
7Impact and Results
The optimizations drove significant engagement gains across Pinterest's recommendation systems. Larger models can now be trained efficiently, with 7.5x scaling at 8 nodes dramatically improving cost efficiency. These techniques are applicable to other large-scale ML applications.