Meta Manages Massive GPU Failures in Prometheus Supercluster
Redundant hardware and advanced checkpointing strategies maintain AI training continuity at scale.
1Prometheus Cluster Scale
Meta is currently operating the massive Prometheus supercluster in New Albany, Ohio, utilizing Nvidia Grace Blackwell and Vera Rubin rack systems. The infrastructure is designed to eventually scale toward a gigawatt of capacity, incorporating approximately one million accelerators. Managing such a vast compute deployment requires addressing inevitable hardware failures across distributed datacenters.
2High Failure Rates
During the initial deployment phase, Meta experienced roughly 1,000 interruptions daily for every 8,000 GPUs. These interruptions were caused by crashes or timeouts, forcing training runs to roll back to previous checkpoints. The company has since reduced this rate by 99 percent, achieving approximately 11 interruptions per day per 8,000 GPUs.
3Hot Spare Strategy
Meta now utilizes a hot spare configuration within its NVLink domains to maintain continuity. Each domain contains 72 GPUs, but only 64 are active at any given time, leaving eight as reserves. When a GPU fails, a spare immediately replaces it, preventing the need to remove entire racks from the cluster.
4Data Parallel Resiliency
To mitigate hardware faults, Meta leverages data parallelism by maintaining multiple model copies within the cluster. When a fault occurs, the system isolates the faulty replica, allowing the remaining cluster to continue productive training. This approach converts a potential total system halt into a minor, temporary reduction in overall aggregate throughput.
5Distributed Checkpointing Architecture
Meta implemented a comprehensive checkpointing scheme that functions similarly to an auto-save feature. The architecture asynchronously saves model states, including gradients and optimizer data, to multi-tier storage. This ensures that if a fault exceeds the data-parallel domain, the team does not need to restart training runs from the very beginning.
6NVLink Signal Challenges
Scaling NVLink fabrics to connect 72 GPUs presented significant signal integrity issues compared to previous eight-GPU domains. With over 600,000 connections, marginal signal degradation triggered frequent recovery cycles. Meta addressed this by implementing continuous telemetry systems and conducting L11 diagnostic burns to isolate weak links before putting systems into active production.
7Power Transient Management
Ramping tens of thousands of GPUs for training runs creates massive power consumption spikes known as transients. These spikes can potentially overwhelm power supply equipment if not managed correctly. Meta focused on cluster stability to ensure that power demands remain within safe operational limits during the intensive pretraining of large models.
8Operational Efficiency Goals
Meta currently achieves over 85 percent goodput for pretraining and 90 percent for reinforcement learning on its Muse Wonder model. Looking forward, the company aims to further reduce interruption rates to just two per day per 8,000 GPUs, targeting 100,000 GPU-hours of mean time between interrupts as the cluster grows.