Making Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One
Streamlining Vera Rubin NVL72 cluster deployment using NVIDIA AI Cluster Runtime recipes.
1The Challenge of New Silicon
Configuring Kubernetes for NVIDIA Vera Rubin NVL72 hardware is inherently complex and prone to errors. Drivers, kernels, runtimes, and operators must be perfectly synchronized to ensure performance. Small mismatches between the orchestration layer and the underlying GPU architecture can prevent workloads from running or significantly limit the overall cluster throughput.
2Introducing NVIDIA AICR Platform
The NVIDIA AI Cluster Runtime (AICR), part of the DSX AI Factory Platform, addresses these integration hurdles. It captures tuned configurations as reusable, versioned recipes. These recipes generate deployment bundles for existing tools, allowing platform teams to validate cluster readiness for Vera Rubin compute trays on Day 1 with high reproducibility.
3Validation on Early Silicon
End-to-end validation was performed on Vera Rubin NVL72 clusters using GPU Operator v26.7.0 and DRA driver v0.5.0. Initial testing on early silicon identified an unmapped architecture value in go-nvlib that caused kubelet plugin crashes. This critical fix was propagated through the stack and included in the official DRA driver v0.5.0 release.
4Recipe Generation and Deployment
AICR generates recipes offline by resolving criteria against a catalog, taking only seconds to complete. The process layers five overlays into 14 components, including the GPU Operator and DRA driver. This automated approach ensures that platform-specific settings, such as NFD topology updater memory limits and exclusive RDMA netns modes, are consistently applied.
5Performance Gate Benchmarking
The training recipe includes a cross-node NCCL all-reduce performance gate over the NVLink fabric. This test ensures that GPU pods run cleanly and IMEX channels are correctly injected across multiple nodes. The validation achieved a bus bandwidth of 744.71 GB/s at 16 GiB, confirming the fabric's integrity under real workload traffic conditions.
6ComputeDomain Integration Details
The DRA/IMEX path utilizes ComputeDomains to manage resources across compute trays. A two-node test demonstrated that pods can join a single domain in approximately 12 seconds. Users must ensure the static nvidia-imex service is masked, as host-managed IMEX and DRA-managed ComputeDomains are mutually exclusive, requiring careful configuration during the initial cluster setup.
7Handling Hardware Realities
Bringing up new silicon involves physical and software challenges. Early validation rounds were temporarily blocked by a fabric issue unrelated to the software stack, which prevented a compute tray from joining the NVLink fabric. Once the hardware was reseated, all GPUs successfully joined a clean fabric clique, enabling full system validation.
8Inference Variant Limitations
While the inference variant passes deployment and conformance tests, performance benchmarking remains deferred. Current vLLM releases are not built against the required CUDA 13.4, causing JIT-compile crashes. A verified inference performance floor requires a runtime image built specifically against CUDA 13.4 to move beyond simple version checks on the Vera Rubin hardware.
9Component Version Pinning
The Vera Rubin recipe family currently ships as a Preview release, encoding decisions made during the initial bring-up phase. Components like NodeWright, KAI Scheduler, and NVSentinel are pinned to specific versions validated on Vera Rubin hardware. This ensures that the configuration remains explicit and verifiable across different cluster deployments during the pre-release period.
10Repeatability and Efficiency
The AICR workflow successfully deployed a complex stack—including arm64, Ubuntu 26.04, and a 64k-page kernel—in hours. This result was reproduced on a second cluster the following day, demonstrating the primary benefit of the recipe-based approach. The entire sequence, from recipe generation to performance validation, completes in approximately five minutes on live hardware.
11Strategic Importance of Recipes
AICR transforms the manual, error-prone process of hardware enablement into a reproducible software lifecycle. By providing signed validation evidence for training and inference variants, it allows organizations to plan software changes alongside upcoming hardware releases. This framework is essential for scaling efficient AI factories in the evolving Vera Rubin ecosystem.