Wrangling Telemetry at Scale: A Guide to Self-Hosted Observability
Expert strategies for managing complex observability stacks while avoiding vendor lock-in and high costs.
1The Inevitability of Complexity
Modern distributed systems inherently accumulate complexity through organic growth, necessity, or technical neglect. Engineers often attempt to escape this by rewriting code, ignoring the problem, or delegating it to new hires. However, as systems scale, these avoidance strategies fail, forcing teams to confront the underlying architectural challenges directly through robust observability.
2The Case Against Self-Hosting
While SaaS bills from providers like Datadog or Splunk are painful, building a custom observability stack is rarely the optimal solution. It requires significant engineering overhead, often demanding at least two to three dedicated engineers. The hidden costs of maintenance and debugging often exceed the financial burden of paying for managed services.
3When to Build Internally
Self-hosting becomes necessary when telemetry volumes outgrow provider limits, or when auditors demand data sovereignty. It is also required when internal business intelligence teams need features unavailable in standard SaaS offerings. If the complexity of managing your own stack is lower than the operational pain of scaling a vendor, transition is justified.
4Defining the Observability Canvas
Observability is not merely storing data; it is the ability to navigate between metrics, logs, and traces to identify root causes. The primary challenge in self-hosting is unifying these disparate telemetry types. Most open-source tools lack native correlation, requiring engineers to manually weave them together into a single, functional debugging canvas.
5Metrics Storage Trade-offs
Prometheus remains the industry standard for metrics, despite the superior efficiency of VictoriaMetrics. While VictoriaMetrics offers better scalability and performance, Prometheus supports exemplars. These are critical for stitching metrics to other telemetry types, allowing engineers to jump from a graph anomaly directly to the specific trace that triggered the error.
6The Role of Exemplars
Exemplars allow attaching a secondary labelset to metrics, enabling correlation between a specific error spike and a unique request trace. Currently, this requires enabling a feature flag in Prometheus. As part of the OpenMetrics specification, this functionality is essential for moving beyond simple monitoring to true, actionable system observability.
7Scaling Prometheus Limitations
Prometheus does not natively support horizontal scaling, relying instead on vertical scaling or manual sharding. Sharding metrics across different servers creates significant operational overhead without natural boundaries. For teams on cloud providers, these limitations become apparent quickly as telemetry volumes increase, necessitating a shift toward more robust long-term storage solutions.
8Managing Active Time Series
Capacity planning for Prometheus requires accounting for active time series, which are buffered in memory for two hours within the head chunk. Each series consumes approximately 4 kilobytes of memory. Understanding this memory footprint is vital for preventing crashes and ensuring the stability of the monitoring infrastructure during high-traffic periods.
9Long-Term Storage Architectures
Systems like Mimir, Cortex, and Thanos solve retention issues by offloading data to blob storage. Mimir and Cortex use remote write to bypass the head chunk entirely. Thanos, conversely, pulls data from sidecars, which can complicate highly distributed environments unless specific architectural optimizations are applied to the query engine.
10Optimizing Distributed Queries
When using Thanos across multiple data centers, centralizing data for queries is disastrous for performance. Enabling the distributed query engine pushes computation down to the sidecars, returning only partial results. This architecture allows for horizontal scaling of the query layer, providing a significant performance boost for large-scale, geographically distributed infrastructure.
11Parquet and Future Performance
Thanos is experimenting with rewriting Prometheus blocks into Parquet format to improve storage efficiency. Early benchmarks suggest this approach achieves performance levels comparable to VictoriaMetrics. This development represents a significant shift in the open-source ecosystem, potentially closing the gap between standard Prometheus deployments and high-performance, specialized time-series database engines.
12Strategic Operational Mindset
Building a self-hosted observability stack is a commitment to long-term maintenance and deep technical involvement. It requires balancing the need for control against the reality of operational burden. Success depends on selecting tools that support correlation, scaling efficiently, and accepting that the complexity demon is a permanent, manageable part of engineering.
Original article: InfoQ — Wrangling Telemetry at Scale: A Guide to Self-Hosted Observability