# Meta's New RDMA Protocol: MetaRoCE Built for AI-Scale Ethernet Meta announces a new network protocol specifically designed for million-GPU AI infrastructures. - **Source:** [Engineering at Meta](https://brifing.si/source/28383801-e365-4f15-8f20-49d8c0635946) - **Published:** 2026-08-24 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/79a6f48d-507a-4d4c-9992-d3c683fc1acd ## The Critical Role of Networking in AI Infrastructure Training frontier AI models requires fast, reliable data transfer between GPUs. Even minor network latency can waste significant compute capacity. Meta aims to solve this by scaling clusters across hundreds of thousands of GPUs spread over multiple data centers. Collective operations synchronize thousands of accelerators during training, where the slowest transfer determines overall job speed. For inference, low-latency communication directly impacts response times for hundreds of millions of users. ## Limitations of Existing RoCE Traditional RoCE expects in-order frame delivery and discourages packet spraying, causing performance issues in multiplane and large-scale networks. PFC usage increases operational complexity as scale grows. Meta's analysis shows standard RoCE is insufficient for million-GPU AI workloads. These limitations highlight the need for next-generation network protocols specifically optimized for AI infrastructure at unprecedented scale. ## Core Innovations of MetaRoCE MetaRoCE moves intelligence to endpoints, departing from traditional architectures. Each packet carries its own destination, eliminating reorder buffers. Native multipathing allows per-packet spraying across first-class paths. Loss tolerance is built-in - Ethernet is treated as lossy by design without PFC. This approach enables simpler operations at massive scale while maintaining high throughput and low tail latency essential for AI workloads. ## Native Out-of-Order Delivery and Multipathing MetaRoCE treats out-of-order arrival as normal operation. Destination information in every packet enables direct memory writes without reordering. Multipathing uses distinct UDP source ports as ECMP entropy, allowing NICs to steer traffic away from bad routes. Each path maintains independent window and RTT estimates, enabling explicit congestion detection and rebalancing without stalling entire connections. ## Loss Tolerance and Congestion Control MetaRoCE's 256-bit selective acknowledgment bitvector distinguishes loss from reordering. Unlike traditional protocols, SACK triggers retransmission of exactly the missing packet on the path that lost it. Congestion control combines sender-driven ECN-based AIMD with receiver-driven fair-share rate hints. Receivers report allocated inbound bandwidth share, enabling senders to quickly converge to optimal rates and resolve incast in 1-2 RTTs. ## Topology Independence and Flexibility MetaRoCE requires only ECN marking and ECMP from the network fabric. It doesn't need packet trimming, in-network telemetry, or switch-side spraying. The same transport works across fat-tree, multiplane, and shallow-buffer fabrics, including vendor clouds with uncontrolled configurations. No proprietary technologies are involved, keeping the fabric free for cost and cabling optimizations. ## Unified Connections at Scale MetaRoCE decouples ordered message streams from bandwidth. While traditional RDMA requires dozens of queue pairs (QPs) per node pair, MetaRoCE supports multiple independent ordered streams per connection under a single congestion controller. Connection state doesn't grow with workload parallelism. The application layer remains largely unchanged - existing RDMA Verbs APIs and software stacks work without modification. ## Path to Industry-Wide Standard Meta is releasing the MetaRoCE specification, reference implementation, and compliance test suite through Open Compute Project (OCP). This move aims to enable broad industry adoption and implementation. The company has been a strong advocate for Ethernet as the preferred fabric for AI infrastructure. MetaRoCE is positioned to shape the future of distributed AI training at million-GPU scale, addressing critical networking challenges for next-generation AI systems. ## Sources covering this story - https://engineering.fb.com/2026/08/24/networking-traffic/metaroce-rdma-transport-ai-ethernet/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._