NVIDIA's TensorRT Edge-LLM Achieves 6.4x Speed Boost in MLPerf Edge Benchmark
Jetson AGX Thor runs agentic AI workloads 6.4x faster with optimized KV cache and tree-based prediction
1Breakthrough in edge AI performance
NVIDIA's TensorRT Edge-LLM completed the MLPerf Edge Agentic benchmark with Qwen3.6-27B model at 52.33 tokens per second on a single Jetson AGX Thor kit. The system finished 1,007 agent turns in 24 minutes 36 seconds, 6.4x faster than the llama.cpp reference implementation. This demonstrates significant progress in running complex agentic workflows on edge devices with constrained resources.
2Key optimization techniques
The performance gain comes from three innovations: NVFP4 quantization for weights/activations, FP8 KV cache format, and tree-based multi-token prediction. Together these reduce memory bandwidth pressure while maintaining accuracy. The 4-bit NVFP4 format is specifically supported by Blackwell GPUs in Jetson AGX Thor, allowing smaller model representations without compromising precision requirements of MLPerf benchmarks.
3KV cache reuse efficiency
The system achieves 96% prompt token cache hit rate by reusing KV cache and recurrent state across agent turns. This avoids recomputing shared conversation history, reducing total prefill tokens from 13.6M to just 0.5M. The hybrid architecture of Qwen3.6 enables both KV page restoration and recurrent state maintenance, crucial for correct execution continuation in long conversations.
4Tree-based prediction advantage
Unlike linear multi-token prediction, the tree-based approach organizes high-probability candidates into an 8-step, top-2, 16-node verification structure. This proves particularly effective for function-calling workloads where tool names and JSON structures are predictable. The method delivered 40% better decoding performance compared to traditional linear MTP approaches.
5Benchmark specifics
MLPerf Edge Agentic evaluates OpenAI-compatible endpoints through performance and accuracy phases. The performance test replays 20 software-engineering agent trajectories totaling 1,007 turns, with context growing to 23.5K tokens. The accuracy phase uses Berkeley Function Calling Leaderboard prompts to validate correct tool selection and argument generation without unnecessary calls.
6Hardware configuration
The submission used one Jetson AGX Thor Developer Kit with 128GB unified memory in MAXN power mode. The substantial memory capacity accommodates both the quantized model and long conversation contexts. Developers can reproduce results using published NVFP4 checkpoints and TensorRT Edge-LLM's release/0.9.1-mlpinf branch with specific engine build commands.
7Implementation details
The solution combines TensorRT-optimized kernels for quantized models with careful state management. The runtime identifies reusable prompt prefixes and restores necessary attention states while only prefilling new conversation suffixes. This approach complements tree-based MTP by reducing pre-generation costs while MTP accelerates the generation phase itself.
8Industry implications
This advancement enables more sophisticated AI agents on edge devices like vehicles and robots, where power and memory constraints previously limited functionality. The 6.4x speedup makes practical what was previously theoretical - multi-step agentic workflows with tool selection and iterative reasoning in resource-constrained environments.