How to Build Your Own LLM Runtime from Scratch
A guide to creating a CUDA-based runtime for Qwen2.5-Coder-7B-Instruct on NVIDIA H100.
1What is an LLM Runtime?
An LLM runtime is a software layer that manages the inference processes of large language models. This article explains the steps to create a CUDA-based runtime for the Qwen2.5-Coder-7B-Instruct model on NVIDIA H100 GPU.
2Why Build Your Own Runtime?
While ready-made solutions are fast and effective, customizing your runtime may be necessary for specific needs. For example, transitioning to a new architecture or adding a custom sampler requires building your own runtime.
3The Importance of CUDA Graphs
CUDA graphs play a critical role in optimizing inference processes. The time per token in eager decode mode, which is 119 ms/token, can be reduced to 17 ms/token using CUDA graphs. This allows for more efficient operation with the same kernels.
4Bug Stories and Lessons Learned
Three significant bugs encountered during the runtime creation process shaped the learning curve. For instance, using __syncthreads() inside a warp-specialized branch can cause undefined behavior and corrupt softmax operations.
5Reference Implementation: llama.cpp
llama.cpp is a reference implementation for LLM runtimes. When run with Q4_K_M quantization on the same GPU class, it achieves 43 ms TTFT and 4.95 ms/token decode times. This serves as a benchmark to compare your own solution's performance.
6Current State of the Runtime
The current runtime achieves 128 ms TTFT for a 512-token prompt and 16.7 ms/token for steady-state decode. This represents higher latency compared to llama.cpp but offers customization potential.
7Model Dimensions and Structure
The Qwen2.5-Coder-7B-Instruct model has 28 decoder layers, 3584 hidden size, and a vocabulary of 152064. These dimensions determine the runtime's performance and optimization strategies.
8CUDA Kernels and Optimization
CUDA kernels are optimized to enhance the runtime's performance. For example, the prmt.b32 command is more efficient than __dp4a for gate/up/down GEMV shapes on the Hopper architecture.
9Github Repository and Resources
The runtime discussed in this article is available on Github in the annotated-llm-runtime repository. The repo contains roughly two dozen CUDA/C++ source files with dense comments.
10The Future of LLM Runtimes
LLM runtimes will gain more importance in the future due to their customization and optimization potential. Developing such solutions for new architectures and specific needs can provide a competitive edge.
11Conclusion and Recommendations
Building your own LLM runtime is a valuable step to meet specific needs and optimize inference processes. However, using ready-made solutions in production environments is also recommended.
12Lessons Learned and Next Steps
The lessons learned in this process form a foundation for future projects. Topics like CUDA graphs and warp specialization are critical for enhancing the performance of LLM runtimes.
Original article: Towards Data Science — Sıfırdan Kendi LLM Çalışma Zamanınızı Nasıl Oluşturursunuz?