Making User Sequence Data More Cost-Efficient and Usable
A method to create fast, cost-effective, and consistent user sequences for ML models.
1What is a User Sequence?
User sequences are ordered lists of a user's recent actions with enriched signals. These structures capture temporal behavior for ML training and inference. For example, a user's last 500 Pinterest interactions are recorded with timestamps and content features.
2Challenges of Current Systems
User sequences are often the most expensive and fragile part of the ML data stack. They need to be fresh and consistent while supporting low-latency production workloads. Meeting the needs of different teams and models adds complexity to the system.
3Use Cases
Sequences are used across training datasets, offline analysis, and online inference. At Pinterest, they power surfaces like home feed, related pins, and search results. Quality issues immediately impact user experience and revenue.
4Quality Dimensions
Sequence quality is multidimensional: freshness (how quickly new events appear), completeness (handling late-arriving data), consistent enrichment (alignment between streams and batches), and schema stability (predictable structure) are all critical.
5Redesign Goals
The new platform aims to provide a consistent data contract, improve cost efficiency at scale, simplify adding new event types, and support both real-time and batch processing. It intentionally avoids changing existing model architectures.
6Core Idea: One Definition, Multiple Runtimes
The redesign centers on defining a signal or event type once and using it across different runtimes (real-time, batch, etc.). This approach increases consistency while allowing configuration-based expansion.
7Architectural Benefits
The new system separates event filtering, enrichment, and sequence assembly for modularity. Each component scales independently and can be customized for different teams' needs, reducing maintenance costs.
8Data Freshness and Consistency
The platform offers a clear policy merging real-time updates with batch corrections. User interactions appear in sequences within minutes while batch processes maintain historical coverage.
9Cost Optimization
New storage strategies and network optimizations deliver significant savings. Data compression and intelligent caching techniques reduce resource usage compared to the previous version.
10Operational Ease
Observability tools make it easier to monitor sequence quality and system health. Debugging processes are simplified, reducing mean time to resolution.
11Multi-Tenant Support
The platform offers customizable configurations to meet different teams' needs. Each model can define its event filtering and enrichment logic while sharing infrastructure.
12Impact and Results
The redesign made Pinterest's user sequences more efficient, flexible, and sustainable. Teams iterate faster, costs are reduced, and system performance and reliability have improved.