# NVIDIA's Cluster Readiness Engine Validates GPU Infrastructure for AI Workloads Open-source Kubernetes controller proactively tests GPU clusters with real distributed workloads before production deployment. - **Source:** [Nvidia Developer Blog](https://developer.nvidia.com/blog) - **Published:** 2026-09-23 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/b883f105-e58d-4ae8-b718-8e63baa270b8 ## The hidden challenges of GPU clusters GPU clusters can pass all health checks yet still fail during actual AI workloads due to subtle issues like single slow GPUs, degraded network links under load, or misconfigured traffic routing. These problems often surface hours into training jobs or via customer complaints, leading to days of diagnostic work while expensive capacity remains unused. NVIDIA's solution addresses this gap between apparent health and operational readiness through proactive testing. ## How NVCRE transforms validation The NVIDIA Cluster Readiness Engine (NVCRE) is an open-source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production deployment. It precisely identifies failing nodes and categorizes failures (e.g., NCCL communication issues or NeMo pretraining problems) through a layered API of Certification, Workflow, and Job custom resources. This replaces manual bisecting of racks and reactive troubleshooting from customer tickets. ## Three-phase API architecture NVCRE's hierarchical API structure creates one workflow per test category, with each workflow spawning child jobs. Results propagate upward: jobs record node-specific failures, workflows aggregate test results, and certifications group outcomes by category. This structure precisely attributes failures to specific nodes and root causes, such as hardware faults during NCCL operations or missed bandwidth targets. ## Automated fault isolation For failures that can't be attributed to single nodes (like low bandwidth across 64-node groups), NVCRE implements topology-aware hierarchical testing. It automatically splits failing groups, reruns tests on subsets, and continues until reaching minimum group size. This process identifies suspect nodes within days instead of the weeks manual diagnosis might require, while preventing fabric saturation through concurrent job limits. ## WorkloadRun simplifies setup NVCRE's WorkloadRun API handles repetitive multi-node GPU workload configuration on Kubernetes, including platform detection, framework-specific runtime setup, and optional gang scheduling. It supports PyTorch (torchrun), MPI (NCCL tests), and arbitrary commands, generating appropriate Kubeflow TrainingRuntime configurations. The API automatically injects shared-memory volumes and sets NCCL/platform environment variables. ## Gang scheduling advantage Without gang scheduling, Kubernetes may partially place GPU pods that then deadlock waiting for peers. NVCRE optionally integrates with gang-aware schedulers like KAI Scheduler to ensure all pods in a workload start simultaneously. This prevents resource fragmentation where partially placed pods hold GPUs indefinitely while waiting for never-arriving peers on busy clusters. ## DSX OS layered approach NVIDIA's full-stack solution addresses cluster lifecycle through three layers: AI Cluster Runtime for validated configurations, NVCRE for active workload testing, and NVSentinel for passive telemetry monitoring. NVCRE specifically finds active failures that produce no telemetry signals, like single degraded GPUs that slow synchronous training jobs to their worst rank's performance. ## Integration with monitoring While NVCRE identifies and records failed nodes, it avoids direct remediation to prevent conflicts. Instead, the NVSentinel monitor translates certification failures into health events, triggering configured quarantine, drain, or remediation workflows. Successful subsequent certifications can clear failure signals, creating a closed-loop system between active testing and passive monitoring. ## Current test coverage NVCRE's built-in catalog includes NCCL communication variants (all-reduce, all-gather, etc.), DCGM level-4 diagnostics, and NeMo pretraining with Nemotron models. It automatically detects GPU architecture and cloud platforms to derive configurations like GPUs per node and platform-specific networking. Thresholds use Common Expression Language but ship without defaults, requiring team-specific targets. ## Implementation roadmap Requiring Kubernetes 1.29+, Helm 3.x, and NVIDIA GPU Operator, NVCRE installs via CLI with CRDs, controller, and Kubeflow Trainer. The open-source project welcomes contributions for new architectures, inference workloads, and lifecycle validation features. As part of NVIDIA's DSX AI Factory Platform, it represents a critical operational layer for enterprises scaling AI infrastructure with reliability. ## Sources covering this story - https://developer.nvidia.com/blog/validate-gpu-cluster-readiness-before-ai-workloads-land/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._