Running Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure
Create secure and efficient multi-tenant environments for your teams using KAI Scheduler and vCluster.
1Multi-Tenancy Challenge
Running multiple teams on Kubernetes clusters leads to CRD version conflicts, RBAC complexity, and GPU resource sharing issues. Traditional solutions offer either full isolation (inefficient) or full sharing (uncontrolled). GPU sharing has become a critical efficiency factor, especially for AI teams working on NLP, computer vision, and recommendation systems.
2KAI Scheduler's Role
KAI Scheduler is a Kubernetes scheduler optimized specifically for AI workloads. Its topology awareness enables hierarchical GPU resource management. Team-based quotas and dynamic allocation features support both shared and burst usage models. Compatible with NVIDIA GPU Operator, it delivers performance at scale across thousands of nodes.
3vCluster Advantages
vCluster provides each team with their own virtual Kubernetes control plane. It offers complete isolation with real API servers, RBAC, and CRDs while sharing underlying physical GPUs. Without lateral movement between tenants, each team operates as if they have dedicated clusters. Ideal for trusted internal engineering teams collaborating on AI projects.
4Architecture Details
The solution operates on a single control plane cluster. Each tenant gets separate API servers, controllers, data stores, and synchronization mechanisms. KAI Scheduler manages the shared GPU pool while vCluster transparently distributes these resources to virtual environments. In the example, NLP, Vision, and Recommender teams share the same physical GPU efficiently.
5Setup Process
The system was tested with three teams sharing one NVIDIA L40S GPU. Setup begins with kubectl and Helm installation on a MicroK8s environment. DNS and hostpath-storage addons must be enabled. Requires NVIDIA GPU Operator with Container Device Interface (CDI). After deploying KAI Scheduler and vCluster, teams can create their isolated tenant clusters.
6Scalability
While demonstrated at small scale, the solution works with hundreds of GPU nodes and dozens of teams. Node pools, queue hierarchies, and tenant cluster counts scale independently. Particularly valuable for large organizations seeking infrastructure cost optimization without compromising team autonomy.
7Security Model
Shared node model designed for trusted internal teams. For untrusted tenants, vCluster offers private nodes option with complete isolation at node, network, and storage levels. In both models, teams cannot view or impact each other's workloads, maintaining strict security boundaries while sharing physical resources.
8Use Cases
NLP team needs custom CRDs. Vision team requires cluster-admin privileges for debugging. Recommender team runs different Kubeflow version. Traditional approaches would require physical cluster separation. This architecture allows all teams to operate independently on shared infrastructure with full administrative control.
9Performance Optimization
KAI Scheduler allocates GPU resources with topology awareness, reducing data transfer costs. Dynamic allocation automatically redistributes idle resources to other teams. Handles workload spikes efficiently in burst mode. Custom queue CRDs enable complex sharing rules tailored to organizational needs and priorities.
10Integration Simplicity
The solution integrates seamlessly with existing Kubernetes environments. KAI Scheduler operates alongside default kube-scheduler - pods with schedulerName: kai-scheduler use KAI, others follow standard process. vCluster works across environments including bare metal, providing flexibility in deployment options.
11Industry Impact
This architecture can reduce infrastructure costs by up to 70% for AI/ML teams. By delivering full isolation without physical cluster separation, it accelerates enterprise adoption. Proven on platforms like NVIDIA Brev and Nebius, the solution offers production-ready reliability for mission-critical AI workloads.