Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework
How schema changes are made safe in distributed CDC systems?
1Introduction
Pinterest developed a next-generation CDC-based data ingestion platform built on Kafka, Flink, Spark, and Iceberg. Schema changes play a critical role in ensuring consistency in distributed systems.
2Challenges of Schema Evolution
Schema changes can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline data representations. Therefore, schema evolution must be handled carefully.
3Automated Schema Evolution Framework
Pinterest developed a framework that automates schema change propagation across Kafka, Flink, Spark, and Iceberg. This framework makes schema changes safe, repeatable, and scalable.
4Schema Change Propagation
Schema changes are treated as a multi-stage convergence process across the control plane and data plane. This approach preserves pipeline availability while gradually restoring schema and data correctness.
5Onboarding Model
Automated schema evolution relies on a reliable onboarding model. Each column is defined with a name, type, and a stable numeric identifier that persists across revisions.
6Supported Schema Changes
The system limits automated schema evolution to additive changes only. This preserves backward compatibility while avoiding historical replay complexity.
7Type Changes
Type changes are particularly risky as they may require reinterpretation of existing data. Therefore, only a narrow subset where semantic meaning remains unchanged is allowed.
8Unsupported Changes
Unsupported changes like new primary keys or columns with default values are handled through migration or re-onboarding.
9Automated PR Checks
To prevent unsupported changes from accidentally entering the pipeline, automated schema change PR checks are run.
10Monitoring and Recovery
Clear paths are provided to monitor and recover from issues that may arise during schema changes. This enhances system reliability.
11Next Steps
Pinterest is moving towards the goal of zero-gap schema evolution. This will enable faster and safer implementation of schema changes.
12Conclusion
The automated schema evolution framework makes schema changes safe and scalable in Pinterest's data ingestion systems. This improves data consistency and system reliability.