# Scaling 1T-Parameter MoE Models on Public Supercomputing Infrastructure EUROPA consortium achieves frontier-scale AI training on JUPITER using PyTorch and TorchTitan optimizations. - **Source:** [PyTorch Blog](https://pytorch.org/blog) - **Published:** 2026-11-12 - **Topic:** [technology](https://brifing.si/c/technology) - **Original article:** - **On Brifing:** https://brifing.si/s/10e1da93-6360-4f7b-9ec0-0fa430cf19e9 ## Frontier AI on Public Infrastructure The EUROPA consortium successfully trained a 1-trillion-parameter Mixture-of-Experts model on the JUPITER supercomputer. This project demonstrates that shared, publicly funded European infrastructure can support large-scale AI development. By overcoming significant architectural bottlenecks, the team proved that sovereign, open-source AI models can compete with private, exclusive-use clusters in performance and scale. ## JUPITER Supercomputing Architecture JUPITER, hosted at Forschungszentrum Jülich, utilizes 384 NVIDIA GH200 Grace Hopper Superchips per replica. Each node features a 900 GB/s NVLink-C2C connection between the GPU and CPU. However, inter-node InfiniBand bandwidth is 36 times slower than internal links, necessitating aggressive overlap of communication with computation to maintain efficient training throughput. ## Optimizing Throughput with TorchTitan The team utilized PyTorch and TorchTitan to increase throughput from 90 to approximately 1,300 tokens per second per GPU. This massive improvement was achieved by refining distributed training techniques specifically for the JUPITER environment. These optimizations allowed the researchers to bypass standard limitations associated with shared, non-exclusive public computing resources. ## FlexShard and Muon Optimizer The team implemented FlexShard to enable the Muon optimizer without excessive communication overhead. By sharding experts by whole units, each rank processes its own matrices independently. This approach eliminated the need for costly Newton-Schulz iteration communication, significantly accelerating the optimizer step from 41.7 seconds down to just 2.2 seconds per iteration. ## Activation Checkpoint Offloading To manage memory constraints, the team offloaded boundary activations to the Grace CPU. This strategy traded high-bandwidth memory for larger CPU memory capacity, enabling longer sequence lengths of 65,536 tokens. This modification was critical for fitting the massive model state within the 120 GB LPDDR5X memory configuration available on JUPITER nodes. ## Resolving Memory Fragmentation The team identified that FSDP’s default stream management caused memory fragmentation, preventing the compute stream from accessing free memory held by side streams. By enabling the unshard_async_op flag, they unified memory allocation under the compute stream. This change reduced peak reserved memory from 88.1 GiB to 74.6 GiB, eliminating allocator retries. ## Model Configuration Details The 1T MoE model features 951.9 billion total parameters with 32.8 billion active parameters per token. It utilizes 74 layers, a hidden size of 6,144, and 384 routed experts. The training process employs a sequence length of 65,536 tokens and uses a combination of Muon and AdamW optimizers for stability. ## Upstreaming Technical Contributions The researchers contributed several fixes and features back to the PyTorch and TorchTitan repositories. Key contributions include the shared RoPE cache, configurable seed-offset mesh axes, and the unshard_async_op API. These upstreamed improvements ensure that other developers can reproduce these results and benefit from the optimizations developed during the challenge. ## Compliance and Language Support The project adheres to the strict requirements of the EU AI Act while supporting all 24 official European Union languages. This focus on compliance and multilingualism highlights the consortium's commitment to creating a sovereign, transparent, and accessible AI ecosystem. The model serves as a benchmark for future European open-source AI initiatives. ## Future of Public Compute This achievement validates that public supercomputing infrastructure is capable of sustaining frontier-scale AI workloads. The success of the EUROPA consortium provides a blueprint for future research projects. It is expected to motivate increased investment in public compute resources, fostering a more robust and independent European AI research and development landscape. ## Sources covering this story - https://pytorch.org/blog/training-a-1t-parameter-moe-on-eu-public-infrastructure-with-pytorch-and-torchtitan/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._