Most AI Projects Fail on Retrieval, Not the Model

The model is the last thing we worry about on an AI engagement. It is usually the first thing clients ask about.

When an AI assistant gives a wrong answer in production, the cause is almost never that the model was not clever enough. The system fed it the wrong context. Somewhere between the question and the answer, the retrieval layer served up a stale policy document, a half-parsed PDF, or nothing at all, and the model did what models do with thin context: it filled the gap confidently.

Where the failures actually happen

Across production systems we run, the failure modes cluster in the same places. Documents that were embedded once and never re-indexed when the source changed. Chunking that splits a policy table in half so neither half means anything. Knowledge bases that grew by accumulation rather than curation, where three versions of the same procedure disagree. Permissions bolted on after the fact, so the assistant either leaks or goes silent.

None of these are model problems. All of them are engineering problems, and they are all fixable with ordinary discipline: curate what goes into the knowledge base, measure retrieval quality separately from answer quality, and re-index on change rather than on schedule.

Grounding beats guessing

The systems that hold up in production answer from a curated, embedded knowledge base via semantic search, so responses stay grounded in the client's own material rather than model guesswork. Where the answer needs live data, retrieval extends past documents into the platform itself. On recent engagements we have used MCP for exactly that reach, so an assistant can cite the actual state of an account rather than a description of it written months ago.

Structured output matters as much as grounded input. A risk score returned as JSON with a score and a reason can be audited, filtered and fed to a dashboard. The same judgement as free text is a paragraph someone has to read.

Models are interchangeable. The estate is not.

We route each task to the model that fits it on quality, latency and cost, with fallbacks, so no single provider dictates the system. Models will keep leapfrogging each other every quarter. The retrieval layer, the data quality, and the evaluation harness are the parts that compound. That is where the first month of an AI engagement should go, and it is where ours does.

There is a working example of all of this in production in our Grapple case study.

Duke Le works in Coder Trove's AI, data and analytics practice, building production LLM systems and the data platforms under them.