Meta's New RDMA Protocol: MetaRoCE Built for AI-Scale Ethernet
Meta announces a new network protocol specifically designed for million-GPU AI infrastructures.
1The Critical Role of Networking in AI Infrastructure
Training frontier AI models requires fast, reliable data transfer between GPUs. Even minor network latency can waste significant compute capacity. Meta aims to solve this by scaling clusters across hundreds of thousands of GPUs spread over multiple data centers. Collective operations synchronize thousands of accelerators during training, where the slowest transfer determines overall job speed. For inference, low-latency communication directly impacts response times for hundreds of millions of users.
2Limitations of Existing RoCE
Traditional RoCE expects in-order frame delivery and discourages packet spraying, causing performance issues in multiplane and large-scale networks. PFC usage increases operational complexity as scale grows. Meta's analysis shows standard RoCE is insufficient for million-GPU AI workloads. These limitations highlight the need for next-generation network protocols specifically optimized for AI infrastructure at unprecedented scale.
3Core Innovations of MetaRoCE
MetaRoCE moves intelligence to endpoints, departing from traditional architectures. Each packet carries its own destination, eliminating reorder buffers. Native multipathing allows per-packet spraying across first-class paths. Loss tolerance is built-in - Ethernet is treated as lossy by design without PFC. This approach enables simpler operations at massive scale while maintaining high throughput and low tail latency essential for AI workloads.
4Native Out-of-Order Delivery and Multipathing
MetaRoCE treats out-of-order arrival as normal operation. Destination information in every packet enables direct memory writes without reordering. Multipathing uses distinct UDP source ports as ECMP entropy, allowing NICs to steer traffic away from bad routes. Each path maintains independent window and RTT estimates, enabling explicit congestion detection and rebalancing without stalling entire connections.
5Loss Tolerance and Congestion Control
MetaRoCE's 256-bit selective acknowledgment bitvector distinguishes loss from reordering. Unlike traditional protocols, SACK triggers retransmission of exactly the missing packet on the path that lost it. Congestion control combines sender-driven ECN-based AIMD with receiver-driven fair-share rate hints. Receivers report allocated inbound bandwidth share, enabling senders to quickly converge to optimal rates and resolve incast in 1-2 RTTs.
6Topology Independence and Flexibility
MetaRoCE requires only ECN marking and ECMP from the network fabric. It doesn't need packet trimming, in-network telemetry, or switch-side spraying. The same transport works across fat-tree, multiplane, and shallow-buffer fabrics, including vendor clouds with uncontrolled configurations. No proprietary technologies are involved, keeping the fabric free for cost and cabling optimizations.
7Unified Connections at Scale
MetaRoCE decouples ordered message streams from bandwidth. While traditional RDMA requires dozens of queue pairs (QPs) per node pair, MetaRoCE supports multiple independent ordered streams per connection under a single congestion controller. Connection state doesn't grow with workload parallelism. The application layer remains largely unchanged - existing RDMA Verbs APIs and software stacks work without modification.
8Path to Industry-Wide Standard
Meta is releasing the MetaRoCE specification, reference implementation, and compliance test suite through Open Compute Project (OCP). This move aims to enable broad industry adoption and implementation. The company has been a strong advocate for Ethernet as the preferred fabric for AI infrastructure. MetaRoCE is positioned to shape the future of distributed AI training at million-GPU scale, addressing critical networking challenges for next-generation AI systems.