# Restore LLM Inference Capacity in Seconds with NVIDIA Dynamo's Shadow Engine Recovery Seamless service and rapid recovery with Shadow Engine Recovery. - **Source:** [Nvidia Developer Blog](https://developer.nvidia.com/blog) - **Published:** 2026-08-25 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/45a67a5d-3c71-4090-a885-bf04ac482f23 ## LLM Inference Recovery Processes When an LLM engine process fails, the standard recovery path involves a cold restart. This requires loading weights into HBM from storage, compiling kernels, and capturing NVIDIA CUDA graphs. For large models, initialization can take several minutes, during which surviving workers must absorb the displaced traffic. ## What is Shadow Engine Recovery? Shadow Engine Recovery, available as a preview feature in NVIDIA Dynamo, moves most of the recovery work off the serving path. It keeps a fully initialized shadow engine idle on the same GPUs as the active engine. GPU Memory Service (GMS) shares the existing weights between the engines without creating another copy in HBM. If the active process fails, the shadow takes over within seconds. ## Performance Measurements The impact was measured by deliberately terminating one worker in a two-worker GLM-5.2 deployment. Without Shadow Engine Recovery, the remaining worker served all incoming traffic during the 283-second cold restart, increasing TTFT. With Shadow Engine Recovery, a second worker resumed serving in 7.3 seconds, minimizing disruption. ## Why LLM Inference Recovery is Slow Production LLM engines commonly experience recoverable software faults. In these cases, the hardware, drivers, and node remain healthy, but only the process holding the corrupted state is lost. A new engine can typically start on the same GPUs, but the initialization cost cannot be skipped. ## Solutions Offered by Shadow Engine Recovery Shadow Engine Recovery solves these problems by decoupling weight lifetime from the engine process and completing non-transferable initialization before a failure. It uses persistent GPU memory, a pre-warmed standby engine, and worker-level coordination to recover without a cold restart. ## GPU Memory Service (GMS) GMS manages specific memory regions, such as weights, independently of the engine process. Even as engines are restarted, weights remain resident in memory. GMS owns physical GPU memory on behalf of inference engines and allows engines to map this memory in their own CUDA contexts. ## Benefits of GMS GMS ensures that weights persist beyond engine failure. It also allows weight sharing between concurrent engines on the same GPU, resulting in zero marginal weight cost for secondary engines. ## GMS Integration Integrating GMS into inference frameworks is straightforward. vLLM, SGLang, and NVIDIA TensorRT-LLM integrate GMS through a custom torch.cuda.CUDAPluggableAllocator bound to the weight memory pool. Inside the engine, weights remain ordinary torch.Tensors. ## Future of GMS GMS currently does not support the KV cache, but this capability is under active development. The goal is for a promoted shadow to map the outgoing engine’s cache instead of rebuilding it. ## Impact of Shadow Engine Recovery Shadow Engine Recovery ensures seamless service in LLM inference and significantly reduces recovery times. This is a critical feature for maintaining service quality, especially for large models. ## Conclusion and Importance Shadow Engine Recovery is a revolutionary development in LLM inference. Integrated with NVIDIA Dynamo, this feature minimizes service disruptions and enhances user experience. This technology could become a standard for future LLM deployments. ## Sources covering this story - https://developer.nvidia.com/blog/restore-llm-inference-capacity-in-seconds-with-shadow-engine-recovery-in-nvidia-dynamo/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._