Failure PatternDecision layer
The Retrieval Drift Trap: Why RAG Tooling Stalls After the Demo Works
Symptom: Pilot answers look accurate on the 20 documents used in the sales demo, then client-facing accuracy complaints start arriving once the corpus passes a few thousand files. Root cause: Retrieval quality is treated as a launch milestone rather than a monitored metric, so degradation from new document types, OCR failures, or stale indexes goes undetected until a client notices.
By InnovaAI ResearchPublished Updated
How do you recognize it?
- •Pilot answers look accurate on the 20 documents used in the sales demo, then client-facing accuracy complaints start arriving once the corpus passes a few thousand files.
- •Nobody on the delivery team can state the current retrieval accuracy number, because no one has re-run an evaluation set since the build sprint ended.
- •Chunking parameters, embedding model, and reranking settings were chosen once during setup and have not been revisited in the last two quarters.
- •Client asks why a specific source was cited and the account lead cannot reproduce the retrieval path without opening the vendor dashboard.
- •Invoice line items for the RAG API grow month over month while the number of shipped client features stays flat.
Why does it happen?
- •Retrieval quality is treated as a launch milestone rather than a monitored metric, so degradation from new document types, OCR failures, or stale indexes goes undetected until a client notices.
- •The retrieval layer is wired directly into application code, which makes swapping providers expensive and turns an accuracy regression into a multi-week re-platforming project instead of a config change.
- •Evaluation sets are built from the same documents used during development, so they measure memorization of the pilot corpus rather than performance on the client's live content.
- •Ingestion pipelines accept whatever the client uploads, including scanned PDFs and image-heavy decks, without a parsing quality gate before indexing.
How do you fix it?
- •Freeze a 50-question evaluation set per client account with known correct source documents, and score retrieval hit rate weekly rather than at milestones.
- •Insert an abstraction layer between the application and the retrieval vendor so the context engine API can be swapped without touching prompt logic or UX code.
- •Log every retrieval call with the query, returned chunk IDs, and whether the answer was accepted, then review the rejection log at each retainer check-in.
- •Run a parsing audit on the client's five most common document types and quarantine files that fail extraction instead of indexing them silently.