ModelExpress: Distributing Model Artifacts at the Speed of Light
ModelExpress significantly reduces the transfer time of model weights.
1The Cost of Transferring Model Weights
As model checkpoints grow to hundreds of gigabytes or even terabytes, the cost of transferring these weights increases rapidly. Moving these weights within a cluster has become a frequent operation.
2The Core Idea of ModelExpress
ModelExpress introduces an approach that queries where compatible weights are located before loading a model. This ensures that each replica is not treated as an independent cold start but rather selects the fastest source and transfer path.
3Direct GPU-to-GPU Transfer
If a serving peer already holds compatible weights in GPU, ModelExpress transfers them directly from GPU to GPU over P2P RDMA. This eliminates redundant storage accesses.
4Fast Startup Time
ModelExpress transfers DeepSeek-V4 Pro weights and JIT Kernel cache artifacts from a serving replica into a fresh replica in under 10 seconds, reducing the total startup time from 8 minutes to 1 minute 44 seconds.
5Streaming from Remote Storage to GPU
ModelExpress streams checkpoints from remote storage directly to GPU, eliminating intermediate downloads and reloads on local disk. This significantly speeds up the data transfer process.
6Multithreaded Tensor Reading
Model Streamer uses a multithreaded tensor reader to fetch tensor ranges concurrently and place them in GPU. This keeps storage, network, and GPU copy paths continuously busy.
7Cluster Ingress: Single Download
ModelExpress ensures that a shared disk cache layer in the cluster is populated only once. This prevents the same data from being downloaded multiple times.
8Direct Access from Local Storage to GPU
In systems supporting GPUDirect Storage (GDS), ModelExpress reads checkpoints directly from local storage into GPU memory. This bypasses the intermediate copy required by conventional loaders.
9Local Reading with ModelStreamer
ModelExpress can also load local checkpoints through ModelStreamer. This overlaps disk I/O with GPU placement, providing a fast path when direct storage-to-GPU access is unavailable.
10Weight Transfer from Serving Peer
A key feature of ModelExpress is that if a replica is already serving the same model, the weights have mostly completed their journey. This replica is used as a live weight source.
11Conclusion and Importance
ModelExpress significantly accelerates the transfer process of model weights, enhancing the efficiency of machine learning workflows. This is particularly advantageous in scenarios involving large models.
Original article: Nvidia Developer Blog — ModelExpress: Model Ağırlıklarını Işık Hızında Dağıtmak