Data Reliability at Grab: Operationalizing with Automated DPIs
How continuous data reliability is maintained in production
1Data Mesh and Certification's Role
Grab implements data mesh through the Signals Marketplace, enabling teams to publish, discover, and reuse trusted data products across domains. Certification provides confidence about ownership, documentation, lineage, and quality controls. However, certification alone doesn't prevent runtime failures, necessitating additional operational measures.
2The DPI Lifecycle
Data Production Issues (DPIs) transform data quality signals into operational workflows. When Kinabalu (the incident orchestrator) detects a data asset no longer meeting its contract, the DPI lifecycle begins. Contracts encapsulate reliability expectations and health checks while keeping the orchestrator decoupled from platform internals.
3Triage: From Alerts to Confirmed Breaches
Not all platform alerts become DPIs. The triage process determines whether an alert represents an actual contract breach. Data contracts codify schema, freshness, completeness guarantees through Genchi tests. The orchestrator evaluates test health holistically, considering historical patterns rather than single test runs.
4DPI Grouping and Context Gathering
The orchestrator groups DPIs sharing root causes to focus responders. During triage, it collects context: affected asset, breached contract, unhealthy tests, data interval, and dependencies. This filtering prevents operational noise by elevating only meaningful contract breaches into production issues.
5Diagnosis: Root Cause Analysis
Upon DPI creation, the system identifies why data is unhealthy, who should fix it, and how. The Data Health API classifies failures into four categories: UPSTREAM_ERROR (dependency issues), PLATFORM_ERROR (infrastructure problems), JOB_ERROR (pipeline failures), and DATA_ERROR (quality violations).
6Platform-Agnostic Approach
The Data Health API doesn't replace platform-specific diagnostics. Ingestion, streaming, metrics, and ML platforms maintain detailed error catalogs internally. The orchestrator consumes a stable API contract, enabling DPI workflows to interoperate across heterogeneous systems while allowing platforms to evolve independently.
7Automated DPI Resolution
Grab's DPI system minimizes human intervention. Issues automatically close when underlying conditions recover through continuous verification. This automation reduces human errors and delays in data operations, significantly improving overall data reliability in production environments.
8Data Product Ownership Assignment
Not all issues auto-assign to data asset owners. RCA determines responsible teams: platform engineers for infrastructure issues, job owners for pipeline failures, or upstream teams for dependency problems. This targeted assignment strategy eliminates unnecessary handoffs, accelerating resolution.
9Importance of Operational Data Reliability
Grab's automated DPI system goes beyond certification to maintain continuous reliability of data products in production. This operational approach preserves data-driven decision integrity and supports the data mesh vision. As data ecosystems grow more complex, such systems become increasingly critical for sustainable data governance.
Original article: Grab Tech — Grab'da Veri Güvenilirliği: Otomatik DPI'lar ile Operasyonel Süreçler