Scaling Grab's Data Lake: The Apache Iceberg Transition
Adopting modern table formats for petabyte-scale data management.
1Introduction: Grab's Data Lake Evolution
Grab is transitioning from Hive Parquet to modern table formats to manage petabytes of data. Leveraging S3 as an elastic foundation, the goal is consistency and performance for both near-real-time metrics and large-scale batch transformations. Slide and Hugo platforms are key components of this shift.
2The Catalyst: Hive Parquet Challenges
Hive Parquet's directory-based architecture created bottlenecks as data grew. Catalog latency, small file problems, operational toil, and metadata disconnect were major issues. ML datasets with sub-1MB files significantly increased S3 API costs.
3Why Iceberg? Strategic Fit
Apache Iceberg was chosen for its community governance, engine compatibility, and long-term flexibility. Growing cloud-native support validated this decision. While Grab aims for format-agnostic architecture, Iceberg offers the most mature solution currently.
4Large-Scale Iceberg Adoption
Instead of full migration, high-value tables were prioritized first. Z-ordering reduced query time from 70 to 6 seconds for navigation data. S3 API costs dropped by 95%, and ML feature freshness improved significantly.
5UnifiedSparkCatalog: Mixed-Format Solution
The UnifiedSparkCatalog allows seamless operation across Iceberg, Delta, and Hudi tables without requiring format awareness. Inspired by Trino's Table Redirection, it routes queries to the correct format transparently.
6Table Detection & Routing
The catalog loads metadata from Hive Metastore to identify table formats via provider fields or path inference. This prevents query breaks during format migrations by automatically routing operations.
7Performance Gains
Z-ordering enables data skipping and min/max statistics for optimized query planning. Larger file sizes and reduced object listing overhead cut resource usage by half in some pipelines.
8Future Implications
Grab's experience highlights the importance of table-centric architectures for modern data lakes. Formats like Iceberg deliver cost and performance benefits at scale. Open-source contributions will accelerate ecosystem growth.
Original article: Grab Tech — Grab'ın Veri Gölünde Apache Iceberg'e Geçiş Süreci