# Inside Prometheus: Meta’s Gigawatt-Scale AI Supercluster Infrastructure Meta scales its New Albany supercluster to one million GPUs for frontier model training. - **Source:** [Facebook Engineering](https://engineering.fb.com) - **Published:** 2026-10-19 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/cd621135-75f9-4f49-a465-d9e8cf8571f9 ## Prometheus Infrastructure Overview Meta is constructing Prometheus, a gigawatt-scale supercluster in New Albany, Ohio, designed to train next-generation foundational models. The facility currently utilizes hundreds of thousands of GPUs across interconnected buildings. The project aims to scale to one million distributed GPUs, functioning as a single machine to support larger models and longer context windows. ## Phase One Success Metrics The first phase of the supercluster is complete and powered the training of Muse foundational models. Meta achieved over 85% goodput for pre-training runs and 90% for reinforcement learning. Initial mean time between interruptions reached 47,000 GPU hours, which has since been improved to over 54,000 hours through firmware updates and system optimizations. ## Catalina Platform Architecture The Catalina platform leverages NVIDIA GB200 NVL72 hardware within an Open Rack V3 structure. It utilizes a 1:1 CPU-to-GPU ratio, enabling 800G of I/O per GPU. This configuration provides 13.4 TB of HBM3e memory and 720 peak PFLOPS of FP8 training compute, significantly outperforming previous H100-based Grand Teton systems in raw capacity. ## Clemente Platform Advancements The Clemente platform upgrades to NVIDIA GB300 NVL72, increasing HBM capacity per GPU to 288 GB. It features a 2:1 GPU-to-CPU ratio, allowing a full 72-GPU domain to fit into a single rack. This design improves serviceability, reduces cabling complexity, and delivers over one exaFLOP of peak FP4 compute for training workloads. ## Dedicated Backend Networking Large-scale AI training requires a specialized network beyond standard data center traffic. Meta implemented a dedicated Ethernet-based Backend network to facilitate direct GPU-to-GPU communication. This architecture supports data-parallel model replicas, ensuring that latency-sensitive operations remain optimized while minimizing the time GPUs spend waiting for data transmission across the massive cluster fabric. ## Building-Level Network Fabrics Each building-level fabric connects up to 20,736 GPUs using a multi-layer topology. Meta transitioned from a Disaggregated Scheduled Fabric to a Non-Scheduled Fabric using shallow-buffer 51.2T Ethernet switches. These fabrics utilize Dynamic Load Balancing and adaptive routing, managed by Meta’s FBOSS stack, to prevent congestion during complex training collectives and large-scale flows. ## Aggregation Network Scale The Backend Aggregation network connects building fabrics across distances ranging from 2 km to over 100 km. It employs Arista 7800-series chassis with deep-buffered Broadcom Jericho3 silicon to deliver multiple Pbps of bandwidth. This infrastructure integrates millions of miles of fiber and 800G-ZR coherent optics to maintain performance despite speed-of-light constraints. ## Co-Engineering Reliability Model Meta maintains a deep co-engineering partnership with NVIDIA to ensure cluster reliability. Teams share real-time telemetry and conduct joint triage sessions to identify root causes of system halts. Meta focuses on infrastructure and scheduling, while NVIDIA addresses firmware, memory handling, and signal diagnostics, creating a transparent accountability model that accelerates system-wide fixes. ## Managing Failure Domains The shift to NVL72 architecture increased the fault blast radius by 9X compared to the H100 era. To mitigate this, Meta treats the 72-GPU rack as the primary unit of allocation. Each domain runs a 64-GPU slice, keeping eight GPUs as in-rack spares to replace failed units immediately without stopping the training job. ## Fault Recovery Mechanisms Meta employs Parallelism-Aware Fault Tolerance to isolate faults within data-parallel model replicas. If a fault occurs, the compromised slice restarts on spare hosts while the rest of the cluster continues training. For larger failures, the system uses asynchronous delta checkpointing to save model states, minimizing work loss and maintaining high training throughput. ## Physical Engineering Challenges Maintaining signal integrity across the dense NVL72 fabric is a significant engineering hurdle. Each domain incorporates 5,184 tuned differential-pair signaling links operating at 200 Gbps. These links traverse multiple points of failure, including GPU SerDes and cable cartridges, requiring rigorous monitoring to prevent hardware anomalies from halting large-scale pre-training jobs across the cluster. ## Future Scaling Trajectory Prometheus represents a fundamental shift in how Meta approaches infrastructure for frontier AI models. By integrating advanced liquid cooling, high-bandwidth networking, and robust fault-tolerance strategies, the company is building a foundation for future Superintelligence Labs projects. The ongoing expansion toward one million GPUs underscores the necessity of extreme-scale reliability and efficient hardware utilization. ## Sources covering this story - https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._