Pinterest Introduces 3-Tower Co-Train Model for Ad Engagement
Balancing latency and performance in ad ranking at scale
1Beyond Two Towers
Pinterest's Nexus system decouples candidate generation from scoring, enabling richer model architectures. The new 3-tower co-train model predicts three probabilities: pCTR, pGCTR30, and pOCTR. This approach allows for dedicated task representations while sharing the overall model, addressing the challenge of scoring hundreds of thousands of candidates per request.
2Model Architecture
The 3-tower model extends the existing two-tower architecture with a cross tower and an inter module. The cross tower performs cross-attention between user sequences and candidate features, capturing high-order interactions. The inter module processes query, Pin, and cross embeddings to produce a shared representation for all engagement tasks, balancing complexity and latency.
3Cross Tower Design
The cross tower uses reduced-query cross-attention to model interactions between user activity and candidate ads. It leverages user sequence features like organic engagement and search history, and candidate features such as advertiser ID and GraphSAGE embeddings. This design optimizes offline performance while maintaining a predictable compute profile.
4Inter Module Functionality
The inter module combines query, Pin, and cross embeddings to produce task-specific logits. It uses a shared-bottom MLP design, adding task-specific biases for gCTR30 and oCTR. This structure captures shared patterns across tasks while preserving task-specific flexibility, achieving performance gains with minimal latency impact.
5Loss Function
The model is trained using a multi-task loss combining main three-tower losses and auxiliary two-tower co-train losses. The loss function emphasizes CTR prediction while balancing gCTR30 and oCTR tasks. This configuration ensures the three-tower predictions remain the primary optimization target without sacrificing two-tower performance.
6Offline Performance
Offline evaluations showed a 30% reduction in loss for three-tower predictions compared to the production two-tower model. The co-train task maintained performance comparable to the existing model, enabling the deprecation of the standalone two-tower model without quality loss.
7Serving Optimizations
To reduce latency, several optimizations were implemented. Pin preprocessing was moved into the Pin tower, reducing per-request compute. Reduced-precision inference (BF16) lowered latency significantly. Late expansion of user features and pre-layer normalization in attention further reduced latency, making the three-tower model practical at scale.
8Two-Stage Scoring
The serving flow was redesigned as a two-stage pipeline. Stage 1 uses fast two-tower scoring for all candidates. Stage 2 applies the richer three-tower model to a selected subset of candidates. This approach balances latency and performance, ensuring efficient ad ranking at high candidate volumes.
9Latency Reduction
Model-level optimizations reduced P99 inference latency from over 200 ms to about 30 ms. Techniques like Pin preprocessing, reduced-precision inference, and late feature expansion contributed to this improvement, making the three-tower model feasible for large-scale ad ranking.
10Operational Impact
The 3-tower co-train model allows Pinterest to deprecate the standalone two-tower model without sacrificing quality. The co-train head provides fast scoring, while the three-tower predictions deliver substantial improvements in CTR and gCTR30, reducing cost per click with a modest increase in infrastructure cost.
11Future Directions
The success of the 3-tower co-train model opens opportunities for further architectural innovations. Cross-attention, sequence modeling, and more expressive interaction modules could be explored within the Nexus framework, pushing the boundaries of ad ranking performance while maintaining latency and cost constraints.
12Conclusion
Pinterest's 3-tower co-train model represents a significant advancement in ad ranking, balancing latency and performance at scale. By leveraging cross-attention and inter module designs, the model achieves substantial improvements in engagement metrics while maintaining operational efficiency, setting a new standard for large-scale ad systems.