World Record for MoE Pre-Training Set on NVIDIA GB300 NVL72
DeepSeek-V3 671B model achieves 1,648 TFLOPs/GPU performance.
1The MoE Revolution
Mixture of Experts (MoE) architectures are revolutionizing efficiency in large-scale AI training. Models like DeepSeek-V3 activate only ~37B parameters per token despite having 671B total parameters, dramatically reducing computational costs. This approach enables researchers to train larger models on the same infrastructure.
2The Communication Challenge
In MoE models, experts reside on different GPUs, requiring token distribution at every layer. This creates all-to-all communication patterns during forward/backward passes. Even minor delays accumulate, making low-latency communication critical for maintaining performance at scale.
3NVLink Architecture
NVIDIA GB300 NVL72 features NVLink technology that connects 72 Blackwell Ultra GPUs as a single unit. Fifth-gen NVLink delivers 1.8 TB/s per GPU and 130 TB/s non-blocking bandwidth across the rack. Memory semantics allow direct peer HBM access, eliminating software overhead.
4Two-Tier Scaling
The system optimizes both intra-rack (scale-up) and inter-rack (scale-out) communication. ConnectX-8 SuperNICs with Quantum-X800 InfiniBand keep gradient traffic hidden behind computation. This design makes training massive models practically feasible.
5Software Breakthroughs
NVIDIA's Megatron Core achieved 1,648 TFLOPs/GPU on DeepSeek-V3 671B using 256 GPUs - 3x higher than previous generation. Software optimizations alone delivered 1.5x performance gains in six months, proving continuous post-silicon improvements.
6Open Source Contributions
NVIDIA engineers collaborate with PyTorch and JAX communities to optimize frameworks like TorchTitan. These enhancements enable full-speed operation on GB300 NVL72 systems. BlueField DPUs offload infrastructure processing for large-scale training workloads.
7The Future of AI Factories
This record demonstrates the power of co-design - from silicon to software. Every efficiency gain in MoE-based pre-training accelerates progress toward frontier capabilities, transforming how the AI industry develops foundation models.
Original article: Nvidia Developer Blog — NVIDIA GB300 NVL72 ile MoE Ön Eğitiminde Dünya Rekoru