# NVIDIA AIPerf: Benchmarking LLM Inference at Scale

A multiprocess tool for accurate high-concurrency LLM performance testing.

- **Source:** [Nvidia Developer Blog](https://developer.nvidia.com/blog)
- **Published:** 2026-09-18
- **Topic:** [Data & AI](https://brifing.si/c/data-ai)
- **Original article:** <https://developer.nvidia.com/blog/benchmarking-llm-inference-at-scale-with-aiperf/>
- **On Brifing:** https://brifing.si/s/e9e99e8b-b2e3-44c6-82fd-5ffc17b7759d

## Bottleneck-free benchmarking

AIPerf replaces GenAI-Perf with a multiprocess architecture that prevents client-side bottlenecks during high-concurrency LLM testing. Unlike single-process tools limited by Python's GIL, it can fully saturate servers while maintaining measurement accuracy. The system supports over 15 endpoint types and integrates with public datasets like ShareGPT for realistic workload simulation.

## Configurable traffic patterns

Engineers can model production traffic characteristics using tunable arrival patterns including constant, Poisson, and gamma distributions with adjustable burstiness. This allows testing under conditions that mimic real-world usage spikes and lulls rather than idealized constant loads. The tool maintains reproducibility through random seed controls for consistent test scenarios.

## Comprehensive metric reporting

AIPerf measures four core LLM performance metrics: Time To First Token (TTFT), Inter-Token Latency (ITL), end-to-end request latency, and output token throughput. Each metric includes detailed percentile breakdowns (p25-p99) to identify long-tail performance issues. When DCGM or pynvml is available, it correlates these with GPU telemetry like power draw and memory utilization.

## Static vs dynamic benchmarking

The tool supports both fixed benchmarks (128 input/output tokens) and dynamic scenarios with variable prompt lengths (512±128 tokens) and output sizes (128±32 tokens). Static tests establish baselines while Poisson arrival patterns better simulate production traffic with request queuing effects. Variance in prefill lengths creates more realistic GPU contention patterns.

## Streaming requirement

Accurate TTFT and ITL measurement requires enabling streaming mode, as non-streaming endpoints batch complete responses before transmission. Without streaming, servers mask first-token delays and decode performance characteristics. AIPerf enforces this requirement to prevent misleading throughput measurements from early-terminated sequences.

## GPU telemetry integration

Beyond standard latency metrics, AIPerf captures GPU power consumption, utilization, and memory pressure during test runs. This enables direct correlation between performance degradation events and hardware resource constraints without separate profiling sessions. The combined dataset helps identify thermal throttling or memory bandwidth bottlenecks.

## Reproducible testing

Configurable random seeds ensure identical request sequences across test runs for reliable comparison. The tool prevents common benchmarking pitfalls like variable output lengths (via ignore_eos flag) and early termination that would otherwise skew throughput measurements. Synthetic inputs maintain consistency while allowing parameter variation.

## Production-scale features

AIPerf handles complex scenarios including multi-node Kubernetes deployments, KV cache warmup mechanics, and production trace replay from platforms like Baseten and WEKA AgentX. It supports distributed inference benchmarking when used with NVIDIA Dynamo 1.0, providing insights into scaling challenges across node clusters.

## Industry collaboration

Developed jointly by NVIDIA and cloud partners including AWS and Coreweave, AIPerf incorporates real-world testing requirements from major AI infrastructure providers. Contributions include Baseten trace replay support, Weights & Biases export integration, and reliability improvements for high-concurrency credit dispatch scenarios.

## Sources covering this story

- https://developer.nvidia.com/blog/benchmarking-llm-inference-at-scale-with-aiperf/ — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._