Comparing BM25, Dense Retrieval, and SPLADE
How I tested information retrieval methods on a 16GB MacBook
1The Core Challenge in Retrieval Systems
Retrieval-Augmented Generation (RAG) systems often compare different search methods, but few actually test these comparisons themselves. Understanding the performance of BM25, dense retrieval, and SPLADE on real data is foundational for information retrieval systems.
2BM25: The Classic Keyword-Based Method
BM25 is a keyword scoring method used for decades. It counts shared words between documents and queries, weighting them by rarity and frequency. It requires no training data and uses a fixed formula, producing a sparse vector mostly filled with zeros.
3Dense Retrieval and Semantic Proximity
Dense retrieval uses a trained neural network to convert text into fixed-length embeddings, capturing semantic meaning rather than exact wording. Query and document embeddings are compared via dot product, ranking documents by similarity.
4SPLADE: Learned Sparse Retrieval
SPLADE bridges BM25 and dense retrieval. Like BM25, it produces sparse vectors, but weights are determined by a trained model rather than a fixed formula. This allows for more flexible and context-aware retrieval.
5MS MARCO and NFCorpus Datasets
Two datasets were used: MS MARCO, a large-scale benchmark with 8.8 million passages, and NFCorpus, a smaller medical dataset with 3,633 documents. NFCorpus enables direct comparison of methods on the same data.
6Setup Challenges and Solutions
Cloning Anserini and Pyserini with full git history caused timeouts. A shallow clone resolved this. Other hurdles included native library crashes and API key requirements not mentioned in official docs.
7Strengths and Weaknesses of BM25
BM25 is robust due to its simplicity and no training needs but fails at semantic proximity. For example, 'cars' and 'automobiles' may yield different results. Dense retrieval handles such semantic similarities better.
8Advantages and Costs of Dense Retrieval
Dense retrieval excels at semantic queries but requires vast training data and computational resources. Updating the model may also necessitate re-embedding all documents.
9SPLADE's Balancing Act
SPLADE combines BM25's simplicity with dense retrieval's semantic power. It produces sparse vectors via a trained model, enabling efficient search. However, model training and maintenance add overhead.
10The Value of Local Testing
Testing these methods locally allows comparing theoretical claims with practical results. Small datasets like NFCorpus are ideal for understanding method behaviors without heavy computational demands.
11Implications for RAG Systems
When building RAG systems, verifying retrieval quality is the first step. Understanding the performance of BM25, dense retrieval, and SPLADE strengthens the system's foundation.
12Conclusion: Establishing a Baseline
This comparison provides a reference point for evaluating retrieval methods. Local testing is crucial for validating claims and overcoming practical hurdles in real-world applications.
Original article: Towards Data Science — BM25, Dense Retrieval ve SPLADE Karşılaştırması