Inside Prometheus: Meta’s Gigawatt-Scale AI Supercluster Infrastructure
Meta scales its New Albany supercluster to one million GPUs for frontier AI.
1Prometheus Infrastructure Overview
Meta is currently constructing Prometheus, a gigawatt-scale supercluster located in New Albany, Ohio. The facility is designed to support the training of frontier foundational models, including the Muse series. The project aims to scale to approximately one million distributed GPUs, functioning as a single, highly interconnected machine for massive AI workloads.
2Phase One Completion
The first phase of the Prometheus supercluster is already operational and has successfully trained the Muse Wonder models. These models demonstrate state-of-the-art performance on agentic tasks. By optimizing system stability, Meta achieved over 85% goodput for pre-training runs and exceeded 90% goodput for reinforcement learning tasks during the initial deployment phase.
3Hardware Evolution Strategy
Prometheus utilizes multiple generations of NVIDIA GPU platforms to maximize performance. The Catalina platform leverages the GB200 NVL72, while the upcoming Clemente platform incorporates the GB300 NVL72. Each generation provides significant leaps in HBM capacity and compute throughput, allowing Meta to handle larger models and longer context windows than previously possible.
4Catalina Platform Architecture
The Catalina platform uses the Ariel configuration, featuring 72 NVIDIA Blackwell GPUs per domain. It employs a 1:1 CPU-to-GPU ratio and dual 400G NICs to provide 800G of I/O per GPU. This design choice ensures uniformity across the region and addresses bandwidth imbalances that typically limit performance in large-scale training clusters.
5Clemente Platform Advancements
The Clemente platform upgrades to the GB300 NVL72, increasing HBM capacity per GPU to 288 GB. It features a 2:1 GPU-to-CPU ratio, enabling the entire 72-GPU domain to fit within a single rack. This design streamlines physical deployment and improves serviceability through reduced cabling and advanced liquid cooling options like AALC or FLC.
6Backend Network Design
Meta implemented a dedicated Backend network to facilitate direct GPU-to-GPU communication across data center buildings. This network architecture is hierarchical, matching the data-parallel model replicas used in training. By minimizing network latency and congestion, the design ensures that GPUs remain productive rather than waiting for data transfers across the fabric.
7Network Fabric Evolution
The cluster utilizes both Disaggregated Scheduled Fabric and Non-Scheduled Fabric designs to manage network traffic. By employing 51.2T Ethernet switches and adaptive routing, Meta mitigates congestion caused by large network flows. The FBOSS networking stack provides the necessary visibility and control to iterate on these fabrics as the cluster scales.
8Aggregation Network Scale
The Backend Aggregation network connects building-level fabrics across distances ranging from 2 km to over 100 km. Utilizing Arista 7800-series chassis and deep-buffered Broadcom Jericho3 silicon, it delivers multiple Pbps of bandwidth. This complex system integrates millions of miles of fiber and specialized optics to maintain high-speed connectivity across the region.
9Co-Engineering Reliability Model
Maintaining reliability at the scale of Prometheus requires deep collaboration with NVIDIA. Teams share real-time telemetry and conduct joint triage sessions to identify root causes for system halts. This attribution-driven model allows Meta to optimize scheduling while NVIDIA addresses firmware and memory handling, driving faster fixes across both organizations' infrastructure.
10Managing Failure Domains
The transition to NVL72 architecture increased the fault blast radius to 72 GPUs. To manage this, Meta uses a 64-of-72 slice geometry, keeping eight GPUs as in-rack spares. If a GPU fails, a spare immediately takes its place, allowing the job to continue without waiting for physical hardware repairs or full restarts.
11Fault Recovery Mechanisms
Meta employs Parallelism-Aware Fault Tolerance to isolate faults within data-parallel model replicas. When a failure occurs, the compromised slice restarts on spare hosts while the rest of the cluster continues training. For larger faults, the system uses asynchronous delta checkpointing to save model states, minimizing the amount of work lost during interruptions.
12Future Scaling Implications
The Prometheus supercluster represents a shift toward massive, unified infrastructure for AI development. By focusing on system-level reliability, high-bandwidth interconnects, and modular hardware design, Meta is establishing a blueprint for future exascale training. The ongoing improvements in MTBI ensure that the cluster can sustain the momentum required for next-generation intelligence.