Detecting Numerical Errors in AI Systems
Discover how numerical errors slip through RAG systems.
1Numerical Errors in RAG Systems
A bank's customer service chat using a RAG system provided incorrect information about an invoice. While the source document stated $10,000, the system answered $1,000. Such errors highlight the challenges AI systems face in processing numerical data accurately. While humans can easily spot these mistakes, current AI detectors fail to catch them.
2Limitations of Current Detectors
Tests on five different detector families revealed that four failed to detect the numerical error. Only MinCheck showed a modest signal. Current detectors compress answers into vectors and measure their distance to the context. However, this method fails to capture errors caused by a single digit change. This exposes the detectors' weakness in numerical accuracy.
3The Failure of the LLM Judge
The LLM judge demonstrated that the model checking the answer shares the same blind spot as the model making the mistake. The model evaluates answers for plausibility. For example, 'The total is $1,000' seems plausible for an invoice. However, the prompt does not force a digit comparison. This highlights the LLM judge's inadequacy in detecting numerical errors.
4The Problem with Trained Detectors
Trained detectors operate within the limits set by their training distributions. They learn the hallucinations labeled by annotators, but single-digit numerical changes are rare in these labels. This explains the detectors' failure to catch such errors. Detectors miss numerical errors due to gaps in their training data.
5Effectiveness of Traditional Methods
The solution to this problem might be one of the oldest ideas in computer science: manually checking the answer against the document. In this method, each part of the answer is searched in the document. For example, 'Invoice' and 'Total' are found in the document, but '1,000' is not. This comparison happens at the resolution of the error, ensuring the wrong token is not averaged with its neighbors.
6Token Similarity Metrics
Token similarity metrics aggregate by the mean, and single-token errors disappear precisely here. Thus, the answer-level figure is the floor: the weakest anchor or the mean of the few weakest. This explains why current detectors fail to catch numerical errors. The mean causes single-token errors to be overlooked.
7The Groundlens Library
To address this issue, a small open-source library called groundlens was developed. This library requires no training, labels, or API calls. It uses one forward pass of an existing encoder and a regular expression. It takes about seventy milliseconds per answer on a CPU. This method offers an effective solution for detecting numerical errors.
8Test Results
Tests on five different datasets showed that each method succeeded in the area its construction pointed to. For example, the span detector won its home benchmark. MiniCheck won the paraphrase set. The LLM judge correctly detected errors in the medical set. The proposed method succeeded in the wrong-numbers set. These results show that there is no universal hallucination detector.
9The AUROC Metric
All experiments used AUROC as the ranking metric. AUROC measures how often the detector ranks a bad answer worse than a good one. However, production systems do not run on rankings. They operate at a point like catching at least 95% of bad answers. This requires quantifying the risk.
10Conclusion and Importance
Detecting numerical errors in AI systems is crucial, especially in critical sectors like finance and healthcare. Current detectors fall short in catching these errors. New methods like groundlens show promise in addressing this issue. However, a universal solution has yet to be found. More research and development are needed in this field.
Original article: Towards Data Science — Yapay Zeka Sistemlerinde Sayısal Hataların Tespiti