# World Record for MoE Pre-Training Set on NVIDIA GB300 NVL72

DeepSeek-V3 671B model achieves 1,648 TFLOPs/GPU performance.

- **Source:** [Nvidia Developer Blog](https://developer.nvidia.com/blog)
- **Published:** 2026-07-21
- **Topic:** [Data & AI](https://brifing.si/c/data-ai)
- **Original article:** <https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/>
- **On Brifing:** https://brifing.si/s/9b1da55c-d509-4215-9b58-06301d14278a

## The MoE Revolution

Mixture of Experts (MoE) architectures are revolutionizing efficiency in large-scale AI training. Models like DeepSeek-V3 activate only ~37B parameters per token despite having 671B total parameters, dramatically reducing computational costs. This approach enables researchers to train larger models on the same infrastructure.

## The Communication Challenge

In MoE models, experts reside on different GPUs, requiring token distribution at every layer. This creates all-to-all communication patterns during forward/backward passes. Even minor delays accumulate, making low-latency communication critical for maintaining performance at scale.

## NVLink Architecture

NVIDIA GB300 NVL72 features NVLink technology that connects 72 Blackwell Ultra GPUs as a single unit. Fifth-gen NVLink delivers 1.8 TB/s per GPU and 130 TB/s non-blocking bandwidth across the rack. Memory semantics allow direct peer HBM access, eliminating software overhead.

## Two-Tier Scaling

The system optimizes both intra-rack (scale-up) and inter-rack (scale-out) communication. ConnectX-8 SuperNICs with Quantum-X800 InfiniBand keep gradient traffic hidden behind computation. This design makes training massive models practically feasible.

## Software Breakthroughs

NVIDIA's Megatron Core achieved 1,648 TFLOPs/GPU on DeepSeek-V3 671B using 256 GPUs - 3x higher than previous generation. Software optimizations alone delivered 1.5x performance gains in six months, proving continuous post-silicon improvements.

## Open Source Contributions

NVIDIA engineers collaborate with PyTorch and JAX communities to optimize frameworks like TorchTitan. These enhancements enable full-speed operation on GB300 NVL72 systems. BlueField DPUs offload infrastructure processing for large-scale training workloads.

## The Future of AI Factories

This record demonstrates the power of co-design - from silicon to software. Every efficiency gain in MoE-based pre-training accelerates progress toward frontier capabilities, transforming how the AI industry develops foundation models.

## Sources covering this story

- https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/ — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._