Why ambitious RAG projects fail — and how we turn them around
The RAG builds we are called in to rescue rarely fail because of the model. They fail on foundations: poor data quality, no information architecture, no governance, and the wrong chunking. Here is how we fix them and deliver the value.
Retrieval-Augmented Generation (RAG) is one of the easiest things to demo and one of the hardest to make reliably valuable. A weekend prototype answers questions impressively; the same pattern pointed at an entire organisation, at scale, confidently returns answers that are wrong, stale or unsafe — and adoption collapses.
By the time we are called, the pattern is familiar: an ambitious RAG programme has stalled, trust has evaporated, and everyone is blaming the model. The model is almost never the problem. The foundations are. Here is what actually goes wrong, and how we turn these projects around.
The ambition trap
Most failed RAG projects share an origin story: point it at everything and let the AI figure it out. Ingest every document, wire up a vector store, add a chat box, expect magic. It is an understandable instinct and a reliable way to fail, because RAG is only ever as good as what it retrieves — and “everything” is not a corpus, it is a mess.
The generation step is rarely the weak link. The retrieval step, starved of clean, well-structured, well-governed material, is. Four culprits do most of the damage.
Culprit 1 — Poor data quality
Garbage in, confident garbage out. The knowledge base is full of duplicates, near-duplicates, superseded versions sitting alongside current ones, contradictory answers, and documents mangled by bad PDF and OCR extraction. Retrieval does its job faithfully and surfaces the wrong or outdated content — and the model states it with total assurance.
No amount of prompt engineering fixes a corpus that cannot agree with itself. Data quality is the single biggest determinant of RAG quality, and it is the first thing we measure.
Culprit 2 — No information architecture
The second failure is structural: everything is dumped into one undifferentiated index with no metadata, no taxonomy, no notion of product, region, document type, recency or audience. With no structure, retrieval cannot be scoped or ranked intelligently — it cannot prefer the current policy over last year’s, or the EU document for an EU user, because it does not know which is which.
Good RAG is not one giant haystack; it is a well-labelled library. Without an information architecture, you have a haystack and a magnet, and you are surprised when you keep pulling out the wrong needle.
Culprit 3 — No information governance
The third is the one that ends programmes: no ownership of the content, no lifecycle, no access control. Nobody owns whether a document is still true. Sensitive material that should never have been in the index gets retrieved and shown to the wrong person. Stale content never expires. There is no answer to the auditor’s question of who can see what, and why.
Without information governance, a RAG system is both untrustworthy and a security and compliance liability. In regulated environments — where we do most of our work — this is not optional, and it is usually the gap that quietly sank the project.
Culprit 4 — The wrong chunking
The fourth is the most technical and the most overlooked. Documents are split into “chunks” before embedding, and get this wrong and retrieval degrades no matter how good everything else is. It is a Goldilocks problem:
- Chunks too small lose context. A fragment that cannot stand on its own retrieves cleanly but cannot answer the question, because the surrounding meaning was cut away.
- Chunks too large produce poor embeddings. Cram several topics into one chunk and its vector becomes an average of all of them — diluted, unfocused, and a weaker match for any specific query.
The fix is rarely a single magic number. It is structure-aware chunking — respecting the document’s own sections and semantics, sizing chunks to hold a complete idea, adding sensible overlap, and enriching each chunk with the metadata from the information architecture above. Chunking is where data quality, architecture and retrieval quality all meet.
How we turn them around
When we take on a stalled RAG programme, we do not start by swapping the model. We start with the foundations, and we work in a deliberate order:
- Diagnose honestly. We build an evaluation harness and measure retrieval quality with real questions and real ground truth — no vibes. You cannot fix what you refuse to measure, and most failed projects were never measured at all.
- Fix the data. Deduplicate, retire superseded content, establish a single source of truth, and extract structure cleanly from the documents that carry it.
- Build the information architecture. Metadata, taxonomy, document types, recency and the filters and permissions that let retrieval be scoped and ranked.
- Establish governance. Ownership, lifecycle, freshness and access control — so the system is trustworthy and defensible, not just clever.
- Re-chunk properly. Structure-aware, right-sized, metadata-enriched — tuned against the evaluation harness rather than guessed.
- Right-size the ambition. We narrow to a high-value, well-bounded use case, prove it works and is trusted, then expand from a foundation that holds — instead of boiling the ocean again.
Throughout, we measure both retrieval quality and the business outcome the project existed to deliver. A RAG system that is not measurably better is not fixed.
The point is value, not novelty
This is project recovery applied to AI: we identify why a programme failed, remediate the foundations that caused it, and rebuild toward something that actually delivers. The technology is rarely the hard part — the discipline around the data, the architecture and the governance is. That is exactly the work most teams skip on the way to the demo, and exactly the work that separates an impressive prototype from a system an organisation can trust.
We are not here to ship a cleverer model. We are here to make the technology deliver value to your business — even, and especially, when the first attempt did not.
Have a RAG or AI project that stalled, or one you want to get right the first time? Get in touch.