# Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

How schema changes are made safe in distributed CDC systems?

- **Source:** [Pinterest Engineering](https://medium.com/pinterest-engineering)
- **Published:** 2026-06-24
- **Topic:** [Software & Engineering](https://brifing.si/c/engineering)
- **Original article:** <https://medium.com/pinterest-engineering/automated-schema-evolution-in-pinterests-next-generation-db-ingestion-framework-36c5c07070de?source=rss----4c5a5f6279b6---4>
- **On Brifing:** https://brifing.si/s/2ac75e2d-d107-474e-8242-0cbd9a5b4e3a

## Introduction

Pinterest developed a next-generation CDC-based data ingestion platform built on Kafka, Flink, Spark, and Iceberg. Schema changes play a critical role in ensuring consistency in distributed systems.

## Challenges of Schema Evolution

Schema changes can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline data representations. Therefore, schema evolution must be handled carefully.

## Automated Schema Evolution Framework

Pinterest developed a framework that automates schema change propagation across Kafka, Flink, Spark, and Iceberg. This framework makes schema changes safe, repeatable, and scalable.

## Schema Change Propagation

Schema changes are treated as a multi-stage convergence process across the control plane and data plane. This approach preserves pipeline availability while gradually restoring schema and data correctness.

## Onboarding Model

Automated schema evolution relies on a reliable onboarding model. Each column is defined with a name, type, and a stable numeric identifier that persists across revisions.

## Supported Schema Changes

The system limits automated schema evolution to additive changes only. This preserves backward compatibility while avoiding historical replay complexity.

## Type Changes

Type changes are particularly risky as they may require reinterpretation of existing data. Therefore, only a narrow subset where semantic meaning remains unchanged is allowed.

## Unsupported Changes

Unsupported changes like new primary keys or columns with default values are handled through migration or re-onboarding.

## Automated PR Checks

To prevent unsupported changes from accidentally entering the pipeline, automated schema change PR checks are run.

## Monitoring and Recovery

Clear paths are provided to monitor and recover from issues that may arise during schema changes. This enhances system reliability.

## Next Steps

Pinterest is moving towards the goal of zero-gap schema evolution. This will enable faster and safer implementation of schema changes.

## Conclusion

The automated schema evolution framework makes schema changes safe and scalable in Pinterest's data ingestion systems. This improves data consistency and system reliability.

## Sources covering this story

- https://medium.com/pinterest-engineering/automated-schema-evolution-in-pinterests-next-generation-db-ingestion-framework-36c5c07070de?source=rss----4c5a5f6279b6---4 — 

---

_Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._