Grafana Tempo 3.1 Enhances Trace Redaction and TraceQL Metrics
New release introduces query-based redaction, sampling-aware metrics, and expanded Kafka client security features.
1Tempo 3.1 Release Overview
Grafana Labs released Tempo 3.1, the first minor update following the 3.0 architecture shift. This version introduces TLS and expanded SASL mechanisms for Kafka clients, alongside experimental query-time hints for TraceQL metrics. Additionally, vParquet5 is now the default block format, which Grafana reports enables simple metrics queries to execute nearly twice as fast.
2Kafka Client Security Upgrades
The 3.1 update addresses critical limitations in the 3.0 Kafka client by adding support for TLS, mTLS, and multiple SASL mechanisms including SCRAM-SHA-256 and OAUTHBEARER. These enhancements, combined with rack-aware fetching via KIP-392, allow operators to reduce cross-zone transfer charges. These changes specifically enable the use of Azure Event Hubs as a backend.
3Query-Based Trace Redaction
Tempo 3.1 evolves trace redaction by allowing operators to use TraceQL queries instead of manual trace ID enumeration. This feature supports dry-run modes to count matches before modification. Because redaction pauses compaction for a tenant, Grafana advises ensuring all workers are updated to 3.1 to prevent errors during the sensitive data removal process.
4Sampling-Aware TraceQL Metrics
The release introduces an experimental hint, extrapolate=true, which scales metrics by the sampling rate recorded in tracestate. This follows OpenTelemetry probability sampling rules, ensuring that partial adoption does not distort metrics. It applies to functions like rate and sum_over_time, requiring vParquet4 or later to ensure consistency between ad-hoc queries and pre-aggregated series.
5Performance and Query Improvements
The span-only fetch path is now enabled by default for vParquet5 blocks, significantly improving query speed. Furthermore, users can now perform arithmetic operations between metrics queries and scalars, simplifying error rate calculations. These updates, combined with the default vParquet5 format, contribute to the observed performance gains in metrics-heavy observability workloads for microservices.
6Advanced Trace Analysis Tools
New experimental features include span pruning on the trace-by-ID v2 endpoint, which collapses similar leaf spans into summary statistics. Additionally, a new POST /api/v2/traces/diff endpoint allows for direct trace comparisons. These smaller payloads are specifically designed to assist LLM-driven investigations, providing more concise data for automated analysis and debugging workflows.
7Metrics Generator Optimizations
The metrics-generator received several refinements, including an unmatched_span_kind label for expired service-graph edges and support for the renamed OpenTelemetry db.system.name attribute. Grafana also implemented pooled label buffers, which the company states reduces per-span allocations to zero in service-graph benchmarks, further optimizing memory usage during high-throughput trace processing and metrics generation tasks.
8Architectural Evolution Strategy
Tempo 3.1 builds upon the Project Rhythm architecture introduced in 3.0, which replaced ingesters and compactors with a Kafka-compatible design. By moving away from replication factor 3, the system reduces storage requirements significantly. These iterative updates demonstrate a focus on operational efficiency, security, and query performance for large-scale distributed tracing environments.
Original article: InfoQ — Grafana Tempo 3.1 Enhances Trace Redaction and TraceQL Metrics