Scaling Cross-Store Product Clustering for Millions of Listings
A tiered machine learning pipeline standardizes product identity across global e-commerce catalogs.
1Unified Catalog Intelligence
Shopify hosts billions of product listings across millions of independent stores. Merchants organize data differently, creating a fragmented landscape where identical items appear as distinct products. The Catalog API provides a unified intelligence layer, assigning a Universal Product Identifier (UPID) to standardize these items for developers and AI agents across the entire platform.
2The Cross-Store Challenge
Intra-store clustering relies on a single merchant's naming conventions, which fail when applied globally. Across stores, titles, descriptions, and image quality vary significantly. Barcodes are often missing or unreliable, forcing the system to compare products based on metadata and visual features rather than relying on consistent, platform-wide identifiers for every single listing.
3Candidate Retrieval Strategy
To manage trillions of potential pairs, the system uses approximate nearest-neighbor search with product embeddings. This retrieval step narrows the search space significantly, though it risks missing matches if the shortlist is too restrictive. The goal is to balance retrieval quality, index size, and throughput to ensure the most plausible pairs reach the scorers.
4Tiered Matching Models
The system employs three scorers with varying inference costs to evaluate candidate pairs. L1 uses a GBDT classifier for fast, inexpensive filtering. L2 utilizes a fine-tuned multilingual E5 cross-encoder for joint text encoding. L3, an open model reranker, incorporates image data to distinguish products that appear identical in text but differ in physical design.
5LLM-Driven Training Data
Rather than using expensive LLMs for every comparison, the team uses them as judges for sampled examples. These judgments validate human annotations and generate training signals for the specialized matching models. This approach allows the system to leverage high-level reasoning for training data creation while keeping real-time inference costs manageable and efficient.
6Graph-Based Clustering Logic
Products are represented as nodes and scored relationships as edges. To prevent a single incorrect connection from merging unrelated groups, the graph is partitioned into bounded-size components. These components serve as independent work units, allowing the UPI builder to evaluate cluster merges using a trained model that weighs evidence for and against each union.
7Efficient Graph Partitioning
Processing the entire edge set on one machine is a bottleneck. The system divides edges among parallel workers, which use union-find to remove redundant connections and produce a spanning forest. This reduced set is then processed using a bounded merge algorithm, ensuring that no component exceeds the memory limits of a single machine.
8Hierarchical Agglomerative Clustering
Within each component, the system applies hierarchical agglomerative clustering (HAC) with a learned merge rule. A GBDT model scores each proposed merge, prioritizing high precision to avoid mixing distinct product groups. This process progressively builds larger clusters, ensuring that only high-confidence matches are grouped under a single, global Universal Product Identifier.
9Joint Encoding Advantages
The L2 model uses joint encoding to compare two listings simultaneously. This allows the model to detect subtle differences, such as compatibility suffixes like 'for iPhone 15 Pro' versus 'for iPhone 15'. By attending to these specific details, the model avoids the common pitfall of letting overall text similarity overwhelm critical identity-defining information.
10Visual Evidence Integration
The L3 reranker processes product images to identify physical differences like button layouts or case cutouts. Because merchants photograph items from different angles, the model interprets images in context rather than performing pixel-level comparisons. This provides a crucial high-precision signal for pairs where text metadata remains ambiguous or lacks sufficient distinguishing detail.
11Iterative Model Refinement
Building the training set requires multiple iterations because each new model changes the accepted merges and intermediate clusters. The team follows a cycle of running the builder, sampling merge proposals, labeling them with the LLM judge, and retraining. This ensures the system adapts to the evolving complexity of the global product catalog.
12Systemic Impact and Future
By transforming a massive, catalog-wide problem into smaller, parallelizable tasks, the architecture maintains high precision at scale. This tiered approach enables Shopify to provide a unified view of products across millions of stores. The resulting UPIs empower developers and AI agents to interact with a standardized, reliable representation of global e-commerce inventory.