Building Scalable Observability for eBay's Monetization Platform
Minerva decouples fee processing from monitoring to ensure system reliability and data visibility.
1The Observability Gap
eBay faced challenges monitoring complex fee structures across 2.6 billion active listings. Previous systems relied on delayed data warehouse reports, forcing teams to manually correlate events after customer complaints. This lack of real-time visibility made it difficult to distinguish between legitimate promotional shifts and actual pricing defects within the monetization platform.
2Decoupling Critical Paths
Minerva was engineered to provide observability without impacting core fee-processing services. By using a local queue and asynchronous enrichment, the system ensures that monitoring tasks never introduce latency or availability risks to transaction paths. This design prioritizes the stability of the primary fee-calculation engine over lossless data delivery for monitoring.
3Asynchronous Data Enrichment
The fee-processing service records only essential identifiers locally, which are then processed by a background producer. This consumer retrieves detailed attributes from transactional databases to enrich the records. This approach keeps the analytical workload isolated from production traffic, ensuring that monitoring data remains eventually consistent without slowing down live user transactions.
4Managing Backpressure Risks
To prevent queue backlogs from affecting service performance, Minerva implements a bounded capacity strategy. If the producer cannot drain the queue due to downstream delays, the system uses a ring-buffer-style policy to overwrite or flush records. This mechanism explicitly favors fee-processing availability, accepting potential gaps in monitoring data during prolonged outages.
5Analytical Storage Strategy
The team selected ClickHouse as the analytical store to handle millions of records with nearly 250 columns each. Its columnar storage format allows for efficient historical scans, which are impractical in transactional databases. Early proof-of-concept tests demonstrated that aggregations on 1.1 billion rows could complete in under one second on standard hardware.
6Contextual Data Modeling
Minerva organizes data around listing-session and account-event identifiers to simplify complex investigations. By flattening values and consuming upstream decisions, the system avoids re-implementing fee logic. This model allows engineers to query trends by site, category, or promotion type without performing expensive joins across multiple operational stores during every single investigation.
7Actionable Signal Processing
The system processes historical data through three distinct stages: query execution, use-case evaluation, and notification delivery. Processors calculate deviations or spikes, assign severity levels, and route alerts to specific channels like PagerDuty or Slack. This modular architecture allows teams to add new monitoring rules without modifying the underlying fee-processing service code.
8Architectural Evolution Impact
Minerva provides a scalable foundation for eBay’s monetization platform by separating decision-making from behavioral explanation. This decoupling allows the platform to evolve alongside changing marketplace dynamics and promotional strategies. By maintaining high-volume history and configurable alerts, the team can now identify localized pricing issues in real-time without compromising core service performance.
Original article: eBay Tech Blog — Building Scalable Observability for eBay's Monetization Platform