RAG Fails at the Edges, Not the Demo

Every retrieval-augmented generation demo works. It works because the demo corpus is twenty clean documents, the questions were written by the person who built the system, and the index was built this morning. Production is none of those things. Production is forty thousand documents accumulated over a decade — duplicates, superseded versions, scanned PDFs, tables that matter — queried by people who phrase things their own way and who are only allowed to see some of the corpus.

RAG does not usually fail in the middle of that picture. It fails at the edges: in the ingestion nobody audited, the chunk boundary nobody chose deliberately, the permission check somebody left to the prompt. These are the six failures that matter, and each one has a specific fix.

The corpus is dirtier than the demo's

Failure 1: the corpus disagrees with itself. Most enterprise corpora contain several versions of the same policy, deck, or procedure. Embeddings rank by similarity, not by truth or recency, so the model retrieves the 2023 policy because it phrases the answer slightly better than the 2026 one. The answer is fluent, cited, and wrong. The fix is unglamorous data work, done before any modelling: deduplicate, stamp every document with version and effective-date metadata, define supersession rules (a newer version of the same document wins, deterministically), and keep a source-of-truth registry so "which document governs" is a lookup, not a similarity contest.

Failure 2: ingestion mangles the content. The highest-value content in an enterprise corpus is often the hardest to parse: tables, scanned forms, multi-column PDFs, spreadsheets with merged cells. A naive pipeline flattens a pricing table into a stream of numbers with the headers detached, and no retrieval quality downstream can recover meaning that was destroyed upstream. Fix: layout-aware parsing that keeps tables and sections intact as single retrievable units, OCR quality gates that reject garbage instead of indexing it, and a standing habit of sampling parsed output by eye. If a human cannot read the parsed chunk, the model cannot either.

Retrieval fails quietly

Failure 3: chunking severs the context. Split a document into fixed-size pieces and you will eventually cut the sentence that names the subject away from the sentence that states the fact — leaving a chunk that reads "coverage renews automatically at the end of each term" with no policy, product, or customer attached, retrievable for a dozen different questions and correct for none of them. Anthropic's engineering team measured how much this costs and how much of it is recoverable: in their published Contextual Retrieval experiments, prepending a short context note to each chunk before embedding cut failed retrievals by 35%, 49% combined with BM25, and 67% with a reranker added. Fix: chunk on document structure (headings, sections, whole tables) rather than character counts, carry document and section context into every chunk, and use overlap where prose genuinely flows across a boundary.

Failure 4: the user's words are not the document's words. People ask about "refunds"; the manual says "reimbursements." They paste an error code; the answer lives under a product name. Pure vector search smooths over exactly the exact-match cases — codes, SKUs, names, figures — where precision matters most, and a top-k list can quietly fill with near-identical chunks of the same document, crowding out the one different document that held the answer. Fix: hybrid retrieval (BM25 for exact terms alongside embeddings for meaning), a reranking pass over a wider candidate set, and diversity rules so one document cannot occupy the whole context window.

The two failures that are not about relevance

Failure 5: permissions enforced by politeness. If retrieval pulls from the whole index and the plan is for the model to "only use what the user is allowed to see," there is no access control — there is a hope. Retrieval must be filtered by the asking user's permissions at query time, using the same access-control lists as the source systems, with the filters applied by the retrieval layer rather than described in a prompt. Where permission boundaries are hard, partition the index. And log what was retrieved for whom: when something leaks, the audit trail is the difference between an incident review and a shrug.

Failure 6: the system cannot say "not in here." Ask a question the corpus does not answer and a typical RAG system answers anyway — from the nearest vaguely-related chunks, with citations that do not support the claim. This is the failure that destroys trust fastest, because it looks exactly like the system working. Fix: a genuine abstention path. Set retrieval-score thresholds below which the system declines or escalates to a human, verify that cited chunks actually contain the claims made (citation checking is a separate, cheap pass), and treat the no-answer rate as a first-class metric to monitor rather than a defect to hide.

Why your tests did not catch any of this

Demo questions test the demo. The eval set that predicts production behaviour is built from real user queries — pulled from logs and support tickets, including the misspelled, the ambiguous, the unanswerable, and the ones the asker had no permission to have answered. Measure the two halves of the system separately: retrieval quality (did the right chunks come back?) and answer faithfulness (does the answer stick to what came back?). A system can have excellent retrieval and a hallucinating generator, or perfect faithfulness to the wrong documents; a single end-to-end score hides which. Re-run the set on every index rebuild, embedding-model change, or corpus refresh — the edges move every time the corpus does.

When RAG is the wrong tool entirely

Sometimes the honest answer is not better retrieval — it is admitting the problem never needed retrieval. If the whole document set is small enough to fit in the model's context window, put it there and delete the index: one less system to build, refresh, and debug. If the question is really about records — what a customer ordered, what an account was charged, whether a ticket is still open — it belongs to a database query or an API call, where the answer is exact, not to a similarity search, where the answer is approximately some documents. And if the underlying data changes faster than an index can be rebuilt, the system should fetch it live at question time instead of retrieving yesterday's copy. Reaching for RAG because it is the familiar shape, for a problem shaped differently, is the zeroth failure — the one that happens before the pipeline exists.

Bring us a problem

If your organisation has a knowledge base your people cannot actually interrogate — or a chatbot that demos beautifully and disappoints in production — that is a deployment problem with known fixes, and it is the kind of problem Trida AI exists for. Start with Trida AI's intake — a few questions, then talk with founders and experts from Trida AI.