Stress Testing RAG Pipelines with Deliberate Fault Injection
How to identify and fix common retrieval failures before they impact users
1RAG's vulnerability to real-world noise
Retrieval-augmented generation systems often fail when exposed to common document issues like outdated policies (60 vs 30 day return windows), OCR errors (SS0 instead of SSO), user typos (warehuse), and divided tables across page breaks. These problems cause retrievers to select wrong passages, leading language models to generate incorrect answers despite proper phrasing. The article demonstrates how each failure mode manifests in a simplified scoring system using token overlap.
2Document versioning pitfalls
When a returns policy changed from 60 to 30 days in 2026, both versions remained in the corpus. Without recency rules, the retriever equally scored both documents based on token matches (returns, days) and defaulted to the older version due to list order. This demonstrates how policy updates require explicit version control mechanisms like timestamp-based tiebreakers or status flags to prevent serving outdated information to users.
3OCR normalization challenges
Character recognition errors like Enterpr1se (Enterprise) and SS0 (SSO) caused complete retrieval failures as the altered tokens didn't match query terms. The solution implemented OCR-specific text normalization (0→o, 1→i) before tokenization. However, such substitutions must be carefully scoped to avoid corrupting legitimate alphanumeric strings like product codes where such conversions would be harmful.
4Fuzzy matching for typos
The warehuse (warehouse) typo case showed how exact token matching fails with minor spelling variations. Enabling fuzzy matching provided partial credit for close spellings, allowing the system to correctly identify warehouse-related documents. The implementation requires careful threshold tuning - too lenient causes false matches, too strict misses legitimate typos - best calibrated against real user query logs.
5Table fragmentation issues
PDF extraction tools that chunk by page boundaries can split tables mid-row (e.g., Basic | on one page, 10 GB on the next). This fragmentation causes retrievers to return incomplete context. The fix involves pre-processing to detect and reassemble table fragments during ingestion, though care must be taken to avoid creating oversized chunks by merging unrelated content.
6Evidence-based validation
The solution implements two-tier verification: checking if the correct document was retrieved (ID validation) and confirming the chunk contains required evidence (e.g., both policy duration and value). This distinguishes ranking failures (wrong ID) from content fragmentation issues (correct ID but missing evidence). The validation framework supports easier diagnosis of root causes in production systems.
7Fault injection methodology
The testing approach deliberately introduces four failure modes (outdated docs, OCR errors, typos, fragmented tables) to verify correction effectiveness. These complement standard relevance tests by evaluating robustness against real-world imperfections. Each test case becomes a regression test - when production issues emerge, corresponding tests ensure fixes remain effective through subsequent changes.
8Production considerations
While the demo uses simple token matching for clarity, the same failure modes affect complex retrieval systems. Production implementations require balancing normalization aggressiveness (avoiding over-correction), fuzzy matching thresholds, and chunking strategies. The article emphasizes building correction rules from actual observed errors in each system's documents and queries rather than hypothetical cases.
9Proactive quality assurance
The fault injection approach shifts RAG evaluation beyond perfect-input scenarios to anticipate real-world usage conditions. By systematically testing against document decay, extraction errors, and user mistakes during development, teams can prevent incorrect responses before they reach users. This methodology is particularly critical for policy-sensitive domains where outdated or fragmented information carries legal or financial consequences.