Scaling 1T-Parameter MoE Models on Public Supercomputing Infrastructure
EUROPA consortium achieves frontier-scale AI training on JUPITER using PyTorch and TorchTitan optimizations.
1Frontier AI on Public Infrastructure
The EUROPA consortium successfully trained a 1-trillion-parameter Mixture-of-Experts model on the JUPITER supercomputer. This project demonstrates that shared, publicly funded European infrastructure can support large-scale AI development. By overcoming significant architectural bottlenecks, the team proved that sovereign, open-source AI models can compete with private, exclusive-use clusters in performance and scale.
2JUPITER Supercomputing Architecture
JUPITER, hosted at Forschungszentrum Jülich, utilizes 384 NVIDIA GH200 Grace Hopper Superchips per replica. Each node features a 900 GB/s NVLink-C2C connection between the GPU and CPU. However, inter-node InfiniBand bandwidth is 36 times slower than internal links, necessitating aggressive overlap of communication with computation to maintain efficient training throughput.
3Optimizing Throughput with TorchTitan
The team utilized PyTorch and TorchTitan to increase throughput from 90 to approximately 1,300 tokens per second per GPU. This massive improvement was achieved by refining distributed training techniques specifically for the JUPITER environment. These optimizations allowed the researchers to bypass standard limitations associated with shared, non-exclusive public computing resources.
4FlexShard and Muon Optimizer
The team implemented FlexShard to enable the Muon optimizer without excessive communication overhead. By sharding experts by whole units, each rank processes its own matrices independently. This approach eliminated the need for costly Newton-Schulz iteration communication, significantly accelerating the optimizer step from 41.7 seconds down to just 2.2 seconds per iteration.
5Activation Checkpoint Offloading
To manage memory constraints, the team offloaded boundary activations to the Grace CPU. This strategy traded high-bandwidth memory for larger CPU memory capacity, enabling longer sequence lengths of 65,536 tokens. This modification was critical for fitting the massive model state within the 120 GB LPDDR5X memory configuration available on JUPITER nodes.
6Resolving Memory Fragmentation
The team identified that FSDP’s default stream management caused memory fragmentation, preventing the compute stream from accessing free memory held by side streams. By enabling the unshard_async_op flag, they unified memory allocation under the compute stream. This change reduced peak reserved memory from 88.1 GiB to 74.6 GiB, eliminating allocator retries.
7Model Configuration Details
The 1T MoE model features 951.9 billion total parameters with 32.8 billion active parameters per token. It utilizes 74 layers, a hidden size of 6,144, and 384 routed experts. The training process employs a sequence length of 65,536 tokens and uses a combination of Muon and AdamW optimizers for stability.
8Upstreaming Technical Contributions
The researchers contributed several fixes and features back to the PyTorch and TorchTitan repositories. Key contributions include the shared RoPE cache, configurable seed-offset mesh axes, and the unshard_async_op API. These upstreamed improvements ensure that other developers can reproduce these results and benefit from the optimizations developed during the challenge.
9Compliance and Language Support
The project adheres to the strict requirements of the EU AI Act while supporting all 24 official European Union languages. This focus on compliance and multilingualism highlights the consortium's commitment to creating a sovereign, transparent, and accessible AI ecosystem. The model serves as a benchmark for future European open-source AI initiatives.
10Future of Public Compute
This achievement validates that public supercomputing infrastructure is capable of sustaining frontier-scale AI workloads. The success of the EUROPA consortium provides a blueprint for future research projects. It is expected to motivate increased investment in public compute resources, fostering a more robust and independent European AI research and development landscape.