# Bringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Training on ROCm A single-controller distributed training solution on ROCm. - **Source:** [PyTorch Blog](https://pytorch.org/blog) - **Published:** 2026-07-06 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/49c478b1-f52a-4062-b0f7-8bb0c6d5cb34 ## Challenges of Distributed Training Training large language models requires distributed systems across thousands of GPUs. At this scale, hardware failures are inevitable and can severely impact the training process. ## The Role of PyTorch Monarch PyTorch Monarch simplifies large-scale distributed training by enabling GPU cluster management through a single Python program, offering fault tolerance and flexibility. ## Porting to ROCm Bringing Monarch to AMD GPUs required adapting the GPU runtime and distributed communication stack to ROCm, a significant engineering effort. ## Collective Communications hipify_torch was used to transition from CUDA to HIP, and integration with RCCL was achieved, facilitating the adaptation of NCCL API to ROCm. ## GPU Memory Management CUDA driver API calls were routed through their HIP equivalents, enhancing cross-platform compatibility. ## RDMA Integration The RDMA path was preserved with GPU_PLATFORM=rocm, and GPU-side bindings were switched from CUDA to HIP, supporting GPU-direct transfers. ## Static Linking Issue ROCm lacks static linking for HIP runtime. This was resolved using dynamic linking, ensuring consistency between CUDA and ROCm. ## Rust Compatibility HIP symbols were re-exported under CUDA names, allowing Rust code to remain platform-agnostic. ## Tests and Community Contribution 1,171 tests passed successfully, ensuring full support for ROCm 7.0+. These contributions were shared with the open-source community. ## The Future of Monarch Monarch enables flexible, fault-tolerant distributed training on AMD GPUs, marking a significant step toward large-scale AI infrastructure. ## Sources covering this story - https://pytorch.org/blog/bringing-pytorch-monarch-to-amd-gpus-single-controller-distributed-training-on-rocm/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._