NVIDIA Snapshot Accelerates GPU Workload Recovery on Kubernetes
New open-source tool reduces inference startup times by up to 36x using checkpointing.
1Reducing Inference Cold Starts
NVIDIA Snapshot is an open-source system designed to accelerate GPU workload recovery on Kubernetes. By capturing fully initialized CPU and GPU memory states, it allows inference replicas to come online in seconds. This approach bypasses the lengthy initialization phases that typically delay scaling operations when demand spikes or failures occur.
2Performance Gains and Benchmarks
Testing on NVIDIA B200 GPUs shows that Snapshot delivers 8x to 36x speedups compared to traditional cold starts. While cold starts can take several minutes, Snapshot restores inference capacity in just 3.6 to 42.7 seconds. These results were consistent across vLLM, SGLang, and NVIDIA TensorRT LLM inference engines.
3Handling Large Model Parallelism
The system supports complex multi-GPU workloads, including mixture-of-experts models with checkpoints up to 1 TB across eight GPUs. It utilizes advanced strategies like tensor, data, expert, and decode context parallelism. This ensures that even massive, highly distributed AI models can be restored efficiently without requiring a complete system re-initialization.
4Integration via Kubernetes Resources
Snapshot integrates into Kubernetes using custom resources for seamless workflow management. Users can checkpoint running pods via PodSnapshot or initialize new ones with SnapshotJob. Restoring a workload is as simple as adding the nvidia.com/restore-from annotation to a new pod, which triggers the recovery of the previously saved memory state.
5Checkpoint Creation and Costs
Creating a checkpoint takes between 15 seconds and 5.5 minutes, depending on the model size and configuration. This is a one-time cost per model, which is significantly offset by the rapid restore times. The combined duration of checkpointing and restoring is shorter than a single cold start in all tested scenarios.
6Technical Restore Breakdown
Restoration involves several distinct stages, including agent setup, CRIU-based CPU process restoration, and CUDA state recovery. Benchmarks on VAST NFS storage indicate that GPU state restoration is the dominant factor, taking 18 to 29 seconds. Meanwhile, the CRIU-based CPU process restore completes in a much faster 4.2 to 10.8 seconds.
7Compatibility and Ecosystem
Originally developed for NVIDIA Dynamo inference workers, Snapshot is designed for broad compatibility with various inference engines and schedulers. The repository provides ready-made examples for vLLM, SGLang, and TensorRT LLM. Platform teams can integrate these capabilities to improve the resilience and responsiveness of their AI infrastructure in production environments.
8Availability and Next Steps
The NVIDIA Snapshot project is now available as an open-source component of the NVIDIA DSX AI Factory Platform. Developers can access installation guides and benchmarking tools in the official repository. Feedback and integration discussions are encouraged through the NVIDIA Dynamo Slack workspace or by visiting the NVIDIA booth at KubeCon.