vLLM Introduces Hardware-Agnostic Layers for Broader Compatibility
New implementation balances performance optimization with support for diverse hardware.
1vLLM's performance challenge
vLLM faces a dilemma between optimizing for cutting-edge GPU performance and maintaining compatibility with diverse hardware. The project's current architecture uses torch.compile for optimization, but this approach is becoming incompatible with newer model architectures. Performance-focused changes risk breaking support for older GPUs, out-of-tree accelerators, and exotic models, which represent a significant portion of vLLM's user base.
2Diverging model architectures
Recent open-weight models like DeepSeek V4 and Kimi K3 use completely different approaches for core mechanisms like attention. These models increasingly ship with bespoke layers and optimized kernels, making them difficult to fit into vLLM's existing abstractions. Each new model requires significant development effort to maintain fullgraph compilability and proper mutation annotations for torch.compile compatibility.
3Hardware-specific optimizations
New GPU architectures like NVIDIA Blackwell and rack-scale systems require specialized kernel engineering to exploit their features. At the same time, coding agents like Claude Code and OpenAI Codex enable easier optimization for specific hardware-model combinations. These trends push vLLM toward maintaining hardware-specific 'flat' model definitions that use custom fusions rather than torch.compile.
4Compatibility trade-offs
The move toward hardware-specific optimizations risks breaking two critical features: torch.compile support (essential for out-of-tree accelerators) and extensibility mechanisms like CustomOp. This could force OOT plugin developers to maintain their own model definitions and layers, creating significant maintenance overhead across the ecosystem.
5Transformers backend impact
vLLM increasingly relies on its transformers backend to support older or exotic models. Without torch-compilable layers, performance for these models on GPUs would regress significantly. The project's own usage statistics show many users still depend on older or consumer-grade hardware that benefits from these more portable implementations.
6Hardware-agnostic solution
vLLM is introducing a new set of hardware-agnostic layers that adhere to four design principles: compilable (supporting torch.compile), extensible (maintaining CustomOp/PluggableLayer), isolated (separate from hardware-specific paths), and portable (using PyTorch or DSLs like Triton). These layers reside in model_executor/hw_agnostic.
7Implementation approach
The hardware-agnostic layers will support both the transformers backend (through modified rewiring) and new flat model implementations. Shared layers across flat models will live in hw_agnostic, while model-specific layers will reside with their model.py but follow the same design principles. Early examples include the hardware-agnostic DeepSeek V4 implementation currently under review.
8Performance validation
Initial tests on NVIDIA H100 GPUs show hardware-agnostic models achieve total token throughput within 3.4% of native implementations (geometric mean across three recent models). Some cases even show slightly better performance despite using only portable implementations rather than CUDA-optimized libraries like FlashAttention.
9Development status
The hardware-agnostic layers are already partially implemented in vLLM's main branch, enabled by setting USE_HW_AGNOSTIC=1 with the transformers backend. The project plans to gradually switch to this as the default pathway for serving models on systems like IBM Spyre, while continuing to add support for more layers and models.
10Balancing innovation and compatibility
vLLM's hardware-agnostic layers represent an important compromise between pushing performance boundaries and maintaining ecosystem support. This approach allows the project to continue serving diverse hardware users without slowing down frontier GPU optimization, preserving its position as a versatile abstraction layer for AI inference.
Original article: PyTorch Blog — vLLM Introduces Hardware-Agnostic Layers for Broader Compatibility