New Approaches for Listing Questions in RAG Systems
The answer is not in a single passage, but scattered across the entire document.
1The Challenge of Listing Questions
RAG systems are typically optimized for answers found in a single passage. However, questions like 'list all exclusions in this policy' fail when answers are distributed across different parts of the document. These questions break the assumption that the answer is in one passage, leading systems to produce incomplete answers.
2Limitations of Traditional Approaches
Traditional RAG systems operate using top-k retrieval. This method retrieves the k most similar passages and expects the LLM to extract the answer from them. However, in listing questions, this approach fails to capture all items and produces incomplete answers. For example, only five out of nine exclusions in a policy might be listed.
3The NIST Example
In the NIST Cybersecurity Framework, the question 'list all categories under the GOVERN function' includes six categories. However, top-k retrieval often captures only four or five of these categories and remains silent about the missing ones. This leaves users with incomplete information.
4Limitations of Long-Context Models
Long-context models excel at finding a single needle in a haystack. However, listing questions require finding six needles scattered in different places. These models miss items expressed differently and present incomplete lists. For example, they might fail to capture a GOVERN category phrased differently in a passage.
5Structural Issues with Top-k Retrieval
In listing questions, top-k retrieval operates with a fixed k value. This either captures too few items or dilutes the LLM's attention with irrelevant passages. When items are similar, their similarity scores are also similar, making it hard for the LLM to distinguish which passages are actual definitions.
6Proposed Structural Solution
It is recommended not to use top-k retrieval for listing questions. Instead, an approach that scans the entire document and captures all items should be adopted. This prevents incomplete answers and ensures users receive a full list.
7Completion Signal
For listing questions, the system needs to generate a completion signal. This signal indicates that the list is complete and prevents users from encountering incomplete information. For example, a signal confirming all categories in NIST CSF have been listed can be used.
8The Role of Expert Knowledge
Expert knowledge is crucial in listing questions. The expert knows that bounded lists exist in their domain and what a complete list looks like. The system uses this knowledge to generate completion signals and allows the expert to ratify the result.
9Code Examples
The article provides runnable code examples for three different strategies to handle listing questions. These codes can be used to pull GOVERN categories, scan GV.XX-NN codes, and capture missing items. They are accessible via GitHub.
10Lack of Benchmarks
Current RAG benchmarks typically measure questions with a single answer. Listing questions have a shape that these benchmarks do not measure. Therefore, new benchmarks should be developed to measure the performance of listing questions.
11Next Steps
New retrieval methods need to be developed to handle listing questions. These methods should scan the entire document and capture all items. Additionally, completion signals and expert knowledge should be integrated to prevent incomplete answers.
12Conclusion
Listing questions pose a significant challenge for RAG systems. Traditional top-k retrieval methods fall short in handling these questions. New approaches and benchmarks should be developed to ensure users receive complete and accurate answers.
Original article: Towards Data Science — Listeleme Soruları için RAG Sistemlerinde Yeni Yaklaşımlar