# Pinterest Implements Event Time Watermarks for Petabyte-Scale Data Ingestion New streaming architecture ensures data completeness using Apache Flink and Iceberg metadata. - **Source:** [InfoQ](https://brifing.si/source/68ac18af-a4f3-4f7c-9c2c-44d4af8f89dd) - **Published:** 2026-10-19 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/77a0177f-9f2e-482b-91a7-10b5f12687c9 ## Solving Streaming Ingestion Challenges Pinterest transitioned from batch-based ingestion to a continuous streaming model for petabyte-scale data. Unlike batch processing, where job completion signals partition readiness, streaming lacks a clear finalization point. The new architecture uses event time-based watermarks to determine when data is sufficiently complete for downstream consumption, effectively managing late-arriving records in CDC pipelines. ## Technical Stack and Data Flow The ingestion framework integrates Kafka, Flink, Spark, and Iceberg to process database changes from MySQL, TiDB, and KVStore. By leveraging CDC events, the platform maintains analytical tables with latency typically between fifteen minutes and one hour. This setup supports thousands of pipelines, ensuring high-throughput data movement across the entire infrastructure. ## Mechanism of Watermark Generation The system operates within the Flink-to-Iceberg sink, extracting event times from each CDC record. During Flink checkpoints, the framework collects statistics including minimum, maximum, and percentile event times. These metrics are stored in Iceberg snapshot summaries using a t-digest, which represents the distribution in only a few hundred bytes of data. ## Ensuring Monotonic Finalization After each Iceberg commit, the system calculates a partition finalization watermark based on recent checkpoint statistics. The default policy rounds the minimum event time down to the nearest hour. This watermark is strictly monotonic, preventing late-arriving records from causing the published value to move backward, which ensures consistency for all downstream consumers. ## Metadata Storage and Accessibility Pinterest stores the finalization watermark directly as an Iceberg table property. This design choice allows downstream sensors to access completeness signals without requiring an external coordination service. Spark further propagates this metadata to the base table during CDC upsert commits, decoupling finalization from the lifecycle of individual batch jobs or processes. ## Balancing Freshness and Completeness The implementation allows for flexible policies, enabling engineers to trade off some completeness for increased data freshness. By adjusting the percentile-based policies, teams can tune the system to meet specific latency requirements. This capability addresses the core challenge of maintaining the reliability guarantees of batch processing within a modern streaming environment. ## Leveraging Existing Framework Capabilities The design builds upon native Flink and Iceberg features to derive table-specific completeness signals. Flink watermarks track event time progress in out-of-order streams, while Iceberg snapshots provide persistent table metadata. By combining these, Pinterest creates a robust mechanism that stores the finalization status alongside the actual data, simplifying the overall architecture. ## Impact on Data Architecture This approach represents a shift where finalization is no longer a byproduct of a job finishing. By embedding metadata into the storage layer, Pinterest enables more reliable downstream data consumption at petabyte scale. This innovation highlights the importance of integrating event-time awareness directly into the sink layer for complex streaming data pipelines. ## Sources covering this story - https://www.infoq.com/news/2026/10/pinterest-event-time-partition/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._