# Achieving Near-Linear Training Scalability for Pinterest's Foundation Models How multi-node training efficiency was improved to 7.5x scale - **Source:** [Pinterest Engineering](https://medium.com/pinterest-engineering) - **Published:** 2026-06-25 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/cdc13074-a73e-4f17-8832-0c4beba52b29 ## Pinterest's Scalability Challenge Pinterest uses foundation models to power recommendations for 600M monthly active users. But adding a second machine in multi-node training slowed training by 5x. AWS EFA fixed the network layer but scaling remained poor: only 21% gain at 4 nodes. ## Core Distributed Training Issue 99% of model parameters reside in embedding tables that don't fit single-GPU memory. Sharding across GPUs with TorchRec created NCCL all-to-all communication bottlenecks. Profiling showed GPUs idle 46% of time waiting for network transfers. ## Quantized Communications Breakthrough FBGEMM's FP8 quantization reduced communication payload by 75%, cutting the largest NCCL operation from 20% to 5% of forward pass. This improved 2-node scaling from 1.13x to 1.57x and 4-node from 1.21x to 2.3x with no quality loss. ## Batch Size Optimization Increasing embedding layer batch size from 512 to 2048 improved communication efficiency, boosting 4-node scaling to 3.1x. Gaps between batches were minimized, optimizing GPU utilization while maintaining model convergence. ## Communication Overlapping Restructuring communication ops to overlap with computation improved 4-node scaling to 3.5x. GPUs now spend less time idle by parallelizing computation and communication more effectively. ## NCCL Parameter Tuning Optimizing NCCL_NSOCKS_PERTHREAD parameters improved socket utilization, achieving 3.9x at 4 nodes (97.5% of ideal) and 7.5x at 8 nodes. Resource addition now yields near-linear efficiency gains. ## Impact and Results The optimizations drove significant engagement gains across Pinterest's recommendation systems. Larger models can now be trained efficiently, with 7.5x scaling at 8 nodes dramatically improving cost efficiency. These techniques are applicable to other large-scale ML applications. ## Sources covering this story - https://medium.com/pinterest-engineering/achieving-near-linear-training-scalability-for-pinterests-foundation-models-14d4f59fe6f6?source=rss----4c5a5f6279b6---4 — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._