Pinterest's Event-Time Partition Finalization for Stream-Based Database Ingestion
How Pinterest ensures data completeness in their next-gen streaming DB ingestion framework.
1The data completeness challenge
Pinterest's migration from batch to stream-based database ingestion created a new challenge: determining when a time-partitioned dataset is complete enough for downstream consumption. Unlike batch systems where job completion signals finality, streaming pipelines continuously write data with event-time lags, requiring an explicit mechanism to mark partition completeness without blocking freshness.
2Time partitions explained
Time partitions group table rows by temporal windows (e.g., hourly buckets), enabling efficient partial reads. While Hive uses explicit directory-based partitioning, Iceberg implements hidden partitioning through metadata, abstracting physical file layout from consumers. This architectural difference impacts how finalization signals must be propagated across Pinterest's stack.
3Why finalization matters
Downstream batch jobs processing hourly partitions need to run exactly once per window after data stabilizes. Premature execution risks missing late-arriving updates due to event-time vs processing-time discrepancies. The solution requires producers to emit machine-readable completeness signals that consumers can reliably poll before processing.
4Streaming vs batch finalization
Batch systems implicitly finalize partitions when the export job completes. Streaming pipelines using Flink continuously write small batches, with no natural completion point. Records may arrive out-of-order with event-time lags, making partition completeness a probabilistic determination rather than a binary state.
5Event-time statistics collection
Pinterest's solution extracts event-time timestamps from each CDC record during Flink processing. Between checkpoints (every few minutes), the system tracks minimum, maximum, and percentile distributions of event times using t-digest sketches. These compact statistical representations enable efficient storage in Iceberg snapshot metadata.
6Watermark generation algorithm
The framework computes partition finalization watermarks by analyzing event-time statistics across recent checkpoints. The default policy takes the minimum event time from the last X commits, rounded to hour boundaries. Watermarks are kept monotonically increasing - late data triggers alerts rather than reversing finalization decisions.
7Metadata propagation strategy
Finalization watermarks originate in CDC tables but must propagate to base tables consumed by downstream jobs. Pinterest's Spark upsert jobs copy these Iceberg table properties during commits, ensuring signals follow data flows without requiring side channels. Consumers access finalization status through ordinary metadata lookups.
8Extensible sink architecture
The solution builds on Flink's Iceberg sink via two extension points: Accumulators collect per-record metrics (like event times) that merge at commit time, while CommitProcessors execute custom logic pre/post commit. This design avoids hardcoding finalization while maintaining zero overhead for pipelines not using these features.
9Accumulator implementation
Each writer subtask maintains lightweight aggregators that build statistics between checkpoints. At commit time, per-task snapshots merge into global summaries written to Iceberg metadata. The merge operation must be order-independent to handle parallel subtask completion. Statistics include both extremes (min/max) and distribution shapes (percentiles).
10Commit processor workflow
Post-commit hooks read newly written statistics to update partition watermarks. The processor considers both current snapshot data and historical trends to make finalization decisions. All operations occur within the commit cycle without external services, ensuring atomicity with the data write itself.
11Configurable tradeoffs
Different tables can adjust the completeness-latency balance by choosing watermark policies. Conservative defaults minimize late data risks, while freshness-sensitive tables may use percentile-based policies accepting occasional late arrivals. The distributed event-time statistics enable these tunable guarantees without custom coding per pipeline.
12Broader applicability
While built for DB ingestion, Pinterest's partition finalization mechanism generalizes to any Flink-to-Iceberg pipeline needing completeness signals. The team is extending it to their Stream Ingestion framework, demonstrating how event-time watermarking solves a fundamental streaming challenge across multiple data platform components.