Amortizing Maximum Inner Product Search for Faster Retrieval
Researchers introduce neural networks to optimize vector database retrieval for RAG systems.
1Core Retrieval Challenges
Maximum inner product search is essential for recommendation systems and retrieval-augmented generation. As databases scale to millions of vectors, exhaustive comparisons become computationally expensive. Current methods often rely on geometric clustering, which assumes query distributions match key distributions, a premise that frequently fails in real-world production environments like RAG.
2Amortized MIPS Concept
Apple researchers presented a method at ICML 2026 to amortize search costs by training small neural networks offline. By learning the specific query distribution of a fixed database, these models route queries more efficiently. This approach trades upfront training time for significantly faster and more accurate inference during live production requests.
3Mathematical Foundations
The research leverages the support function of a set, a concept from convex analysis. Since the function mapping a query to its best inner product is convex, it can be approximated using Input Convex Neural Networks. The gradient of this function at any query point directly identifies the optimal matching key vector.
4Regression-Based Training
The team uses regression to train neural networks to mimic the ground-truth support function. By sampling queries from a known distribution and pairing them with optimal MIPS outputs, the models learn to replicate the solution landscape. This supervised learning process ensures the networks are grounded in the actual data distribution.
5SupportNet Architecture
SupportNet models learn the support function directly using constrained ICNN architectures. This method is mathematically principled and aligns with the convex structure of the MIPS problem. It is particularly effective when computing one support function per data cluster, which significantly improves the quality of query routing compared to simple centroid methods.
6KeyNet Query Rewriting
KeyNet models are designed to output the best-matching key vector directly when provided with a query. They act as a query rewriter that sits in front of standard approximate search indices. By predicting the likely location of the best match in embedding space, they allow existing systems to search more effectively.
7Cluster Routing Improvements
Traditional centroid routing often fails when a query is near an outlier key far from the cluster center. By using learned support functions, the system can quantify exactly how well a cluster matches a query. This enables more precise routing, ensuring that clusters containing relevant keys are not erroneously skipped.
8Performance on Benchmarks
Testing on the Natural Questions dataset with 128 clusters showed significant gains. Small SupportNet models achieved 95% routing recall 1.9 times faster than centroid-based baselines. Larger models reached 97% recall in just 1.74 milliseconds, demonstrating that learned routers provide superior performance even when accounting for the additional neural network compute overhead.
9Efficiency Gains
KeyNet-mapped queries consistently outperform raw questions across various search budgets. Scanning only 0.001% of the database, KeyNet-mapped queries retrieved the true key up to 96.7% of the time. This represents an 11-fold reduction in the number of keys scanned compared to raw queries, maintaining high recall with significantly lower computational costs.
10Model Selection Criteria
Choosing between architectures depends on specific deployment needs. KeyNet is ideal when inference speed is the primary bottleneck. SupportNet is preferred when calibrated scores are required for re-ranking or thresholding. Both models are robust to mild distribution shifts, making them practical for real-world production systems with stable, well-characterized query patterns.
11Scaling Large Systems
For massive-scale MIPS systems, a hybrid approach is recommended. Developers can train a SupportNet router to manage all clusters, while deploying individual KeyNet models within each cluster. This layered strategy allows the system to zero in on the best keys rapidly, combining high-level routing with precise, low-level vector retrieval optimizations.
12Future of Learned Retrieval
Amortized MIPS marks a shift toward retrieval systems explicitly tailored to their specific input distributions. By encoding MIPS solutions into network weights, developers can create lightweight, high-performance search engines. This research provides a practical path for integrating advanced machine learning into existing infrastructure, enhancing efficiency for modern AI-driven search applications.