Most AI projects fail in production because of RAG retrieval problems. When an assistant gives a wrong answer, the usual cause is that the system fed it the wrong context: a stale policy document, a half-parsed PDF, or nothing at all. Given thin context, the model fills the gap confidently.
The model is the last thing we worry about on an AI engagement, and it is usually the first thing clients ask about.
Across the production systems we run, the failures cluster in the same places. Documents embedded once and never re-indexed when the source changed. Chunking that splits a policy table in half so neither half means anything. Knowledge bases that grew by accumulation instead of curation, where three versions of the same procedure disagree. Permissions bolted on after the fact, so the assistant either leaks or goes silent (the fix is covered in permission-aware RAG).
These are engineering problems, and ordinary discipline fixes them: curate what goes into the knowledge base, measure retrieval quality separately from answer quality, and re-index when the source changes instead of on a schedule.
The systems that hold up in production answer from a curated, embedded knowledge base via semantic search, so responses stay grounded in the client's own material. Where the answer needs live data, retrieval extends past documents into the platform itself. On recent engagements we have used MCP for that, so an assistant can cite the current state of an account instead of a description written months ago.
Structured output matters as much as grounded input. A risk score returned as JSON with a score and a reason can be audited, filtered and fed to a dashboard, while the same judgement as free text needs a person to read it first.
We route each task to the model that fits it on quality, latency and cost, with fallbacks, so no single provider dictates the system. Models will keep leapfrogging each other every quarter. The retrieval layer, the data quality and the evaluation harness keep their value through every model change, so the first month of an engagement with our AI, data and analytics practice goes there.
There is a working example in production in our Grapple case study.
In production, wrong answers usually trace to the context the model was given. Stale indexes, broken chunking, conflicting documents and missing permissions cause more failures than model capability.
Retrieval-augmented generation finds relevant passages in a knowledge base, typically by semantic search over embedded documents, and passes them to the model so it answers from your own material.
Keep tables and sections intact when chunking, remove superseded versions of documents, and test retrieval on its own so you know whether a wrong answer started there or in generation.
It matters per task for quality, latency and cost. Because models change so often, it is worth designing the system so any one model can be replaced.
Duke Le works in Coder Trove's AI, data and analytics practice, building production LLM systems and the data platforms under them.