Most RAG Errors Are Extraction Issues: Seven Patterns for Typed Generation Contracts
Reduce RAG hallucinations with structured generation contracts.
1Root Cause of RAG Errors
Incorrect answers in RAG systems often stem from extraction chain failures rather than model hallucinations. This article explains seven patterns to prevent these errors. The core issue emphasizes returning structured data instead of free text, fundamentally changing how we approach RAG implementations.
2Problems with Naive Approach
Traditional RAG systems simply take raw text answers from LLMs without validation. This approach lacks reliability metrics, fails to document evidence, and can't handle 'not found' cases properly. The article demonstrates how typed contracts solve these fundamental issues in retrieval-augmented generation.
3Model as Function, Not Oracle
The first pattern treats LLMs as structured data producers rather than oracles. For 'what's the premium amount', it returns typed data (value, currency, evidence) instead of prose. This structured approach enables better data utilization and auditability throughout the pipeline.
4Extract, Don't Compute
The second pattern mandates that LLMs should only extract raw data, leaving computations to external systems. Currency conversions happen in Python with logged rates, not inside the model. This ensures every calculation step remains verifiable and reproducible during audits.
5Structural Completeness Checks
The third pattern prevents models from guessing completeness about unseen text. Instead, it analyzes document structure to determine section boundaries objectively. When listing 'all exclusions', the system fetches pages until the next section begins, creating deterministic completeness.
6Two Booleans Beat One Confidence Score
The fourth pattern replaces fuzzy confidence scores with two clear boolean flags: 'answer_found' and 'is_complete'. This binary approach enables more precise routing for different scenarios (off-corpus, partial, complete) compared to interpreting ambiguous confidence values.
7Optimizing for Small Models
The seventh pattern shows how to optimize schema design for smaller models. By decomposing complex schemas into simpler components, smaller models can approach the performance of frontier models. This maintains accuracy while significantly reducing computational costs.
8The Validator Layer
All patterns share the critical requirement of a validator layer. This pre-delivery check verifies every contract field before the user sees any response. It catches type mismatches, missing evidence, or inconsistent confidence scores, ensuring only validated responses proceed downstream.
9Power of Pydantic Schemas
The article demonstrates how Pydantic schemas can revolutionize RAG systems. These schemas automate data validation, improve documentation, and simplify integrations. Each field becomes part of an explicit contract between system components, reducing ambiguity throughout the pipeline.
10Code Examples and Implementation
Accompanying Jupyter notebooks provide step-by-step implementations of all seven patterns. They particularly demonstrate filling the AnswerWithEvidence schema, how self-assessment flags work, and schema decomposition techniques. Practical examples bridge the gap between theory and production deployment.
11New Standard for RAG Systems
These seven patterns propose a new standard for accuracy and reliability in RAG systems. Using typed contracts instead of free text improves both developer experience and end-user outcomes. The approach represents a paradigm shift for enterprise document intelligence applications.
Original article: Towards Data Science — RAG Sistemlerinde Çıkarım Hataları ve Yedi Çözüm Modeli