Uber Automates Dependency Analysis Across Its Service Mesh
Uber implements middleware-based dependency mapping to classify failure propagation across its microservices architecture.
1Scaling Dependency Mapping Challenges
Uber manages thousands of microservices where requests traverse deeply nested and asynchronous paths. Traditional distributed tracing fails to capture rare errors effectively due to low sampling rates. With 99.9% availability, capturing a single failed request at a 0.01% sampling rate could take over five hours, necessitating a more efficient automated approach.
2Middleware-Based Failure Correlation
Engineers Deepanshu Mehndiratta, Alok Srivastava, and Shivam Jindal developed a solution using RPC middleware within the YARPC framework for Go and Java. By utilizing inbound and outbound middleware, the system records caller and callee endpoints in an in-memory RequestTracker, enabling precise correlation of failure states without relying on expensive distributed traces.
3Defining Dependency Failure Classifications
The system categorizes dependencies into three distinct types based on failure propagation behavior. Fail-close dependencies cause the caller to fail consistently, while fail-open dependencies allow the caller to succeed despite downstream issues. Unknown classifications are assigned when insufficient evidence exists, providing a clear framework for engineers to assess service reliability and risk.
4Leveraging Muttley Proxy Infrastructure
Uber routes service-to-service communication through Muttley, an on-host proxy responsible for routing and telemetry. Because Muttley lacks sufficient caller endpoint context for granular dependency mapping, the team integrated analysis directly into the RPC middleware. This allows for the capture of failure metrics at the request level while maintaining high performance.
5Statistical Probability for Classification
The system calculates a failure correlation probability for each dependency based on observed outcomes. Values above 0.8 are classified as fail-close, while values below 0.2 are marked as fail-open. Intermediate values remain unknown, and the logic explicitly excludes retries to prevent them from being incorrectly identified as independent dependencies within the mesh.
6Trade-offs in Observability Design
Unlike OpenTelemetry-based pipelines that utilize tail sampling to retain specific traces, Uber prioritizes broader coverage over individual request-level detail. By emitting metrics through middleware, the team avoids the significant overhead associated with storing and evaluating massive volumes of trace data, ensuring faster feedback loops for identifying critical service dependency failures.
7Operational Impact on Reliability
This automated approach has successfully cataloged dependencies for critical APIs and indexed most fail-close anomalies across the service mesh. By mapping how failures propagate toward originating APIs, Uber engineers can now identify systemic risks more rapidly. This shift from manual tracing to automated metric-based analysis significantly improves the reliability of complex microservices.
8Future of Mesh Observability
Uber’s implementation demonstrates the necessity of custom telemetry solutions when standard distributed tracing tools fail to meet specific scale requirements. By embedding dependency analysis into the RPC layer, organizations can achieve deeper visibility into failure propagation. This methodology serves as a blueprint for managing high-scale microservice architectures where traditional observability methods fall short.
Original article: InfoQ — Uber Automates Dependency Analysis Across Its Service Mesh