Implementing Distributed Tracing for Mobile Business Operations
Moving beyond HTTP calls to trace actual business operations in mobile applications.
1Defining the Business Operation
The core challenge was shifting focus from individual HTTP calls to business operations. By using a pipeline framework, each operation like placing a bid is treated as a single trace root. This approach captures the entire lifecycle from the initial user action to the final backend service response, providing a complete story.
2The Pipeline Framework Integration
The tracing system leverages existing pipeline structures where operations end in build or buildOnce methods. By naming operations in a central catalogue, developers avoid inconsistent attribute naming. This abstraction ensures that tracing remains intuitive, requiring only a single argument to instrument complex flows without cluttering the underlying business logic code.
3Choosing OpenTelemetry and Grafana
The stack relies on OpenTelemetry for instrumentation and Grafana Tempo for storage. Backend teams were already using Grafana, allowing mobile traces to join backend spans via W3C traceparent headers. This end-to-end visibility is critical, as mobile traces are only partially useful if they stop at the application edge without backend context.
4Handling Retries and Errors
A mid-project pivot introduced an Apollo interceptor to handle retries. This ensures that each physical request is captured as a distinct child span under the main operation root. Furthermore, the pipeline automatically converts exceptions into typed failures, allowing the root span to record specific error types and codes for accurate diagnostics.
5Managing Live Stream Traces
Live shopping rooms involve long-running subscriptions rather than one-shot operations. To avoid the session-trace anti-pattern, the system uses bounded roots that close once the screen displays real data. This setup tracing focuses on the time taken to load initial auction states, intentionally deferring per-emission tracing to prevent flooding the system.
6Kotlin Multiplatform SDK Strategy
Using the alpha opentelemetry-kotlin SDK enabled a single implementation for both Android and iOS. To manage the volatility of an alpha library, a thin wrapper was created. This isolation layer allowed the team to implement workarounds for SDK defects, such as thread-safety issues and crashes, without refactoring the entire codebase.
7Solving Clock Synchronization Issues
Mobile devices often have unreliable system clocks, causing backend spans to appear before client spans. The solution involved passing a server-synced clock to the SDK instead of relying on the device clock. This synchronization ensures that the waterfall visualization in Grafana accurately reflects the true sequence of events across distributed systems.
8Production Rollout and Sampling
Tracing is controlled via feature flags, allowing for a gradual rollout by cohort. There is no sampling on the device; instead, volume is managed by enabling the flag for specific user groups. This strategy ensures that all spans for a targeted user are captured, which is essential for deep-dive debugging on the collector.
9Distinguishing Failure from Recovery
The system distinguishes between recovered retries and actual failures. A failed attempt followed by a successful retry is marked as a child span with an error, but the root operation status remains successful. This prevents false alarms in alerting systems, ensuring that only true failures trigger notifications for the engineering team.
10Metrics and Data Derivation
Since the SDK lacks native metric exporters, the team derives performance data directly from spans. Using the spanmetrics connector in the OpenTelemetry Collector, they calculate rates, errors, and durations. This approach avoids faking metrics with extra spans, keeping the trace data clean while providing necessary insights into application performance and health.
11Self-Serve Developer Experience
The project reached maturity when other developers could instrument their own features independently. An interactive agent skill assists with selecting the correct catalogue, proposing attributes, and generating TraceQL queries. This self-serve model reduces the burden on the original designer and ensures consistent tracing practices across the entire mobile engineering team.
12Strategic Tracing Implementation
Effective tracing requires deliberate implementation rather than blanket coverage. By focusing on business operations and maintaining a clean abstraction, the team transformed guesswork into actionable data. The success of this system demonstrates that infrastructure choices, like using a unified pipeline, are just as important as the observability tools themselves.