There is a meeting that happens on every AI project. Someone presents a comparison of models, with benchmark scores and pricing, and the room debates which to choose as if picking a database. Then the system ships, a user reports a wrong answer three weeks later, and the team's method for assessing the report is to try the question themselves and squint at the output.
That team spent its rigour in the wrong place. Models are the most replaceable component in the system; we swap them routinely as pricing and quality shift. The thing that cannot be swapped in, because it has to be built from your own cases, is the ability to know whether the system is good. That ability has a name: the evaluation harness.
Strip the jargon and it is three things. A test set: real questions from your domain, paired with what a correct response looks like, including the questions that should be refused. Scoring: automated judgements of each response for correctness, groundedness, whether the answer is supported by what was retrieved, and refusal behaviour. And a habit: the suite runs before every release and continuously against production, so quality is a number with a trend rather than an impression with a mood.
Building the first version takes a few weeks, most of it spent extracting a few hundred genuine cases and arguing about what "correct" means for each. That argument is not overhead. It is the moment the organisation discovers it never actually defined what it wanted the system to do, and settling it improves everything downstream.
Model choice becomes an experiment instead of a debate. Run the candidates against the harness, read the scores against the prices, decide in an afternoon. When a new model releases next quarter, the same afternoon repeats. The meeting from the first paragraph never happens again.
Regressions become visible before users see them. Prompt changes, retrieval tweaks and index updates all look harmless in spot checks; the harness catches the one that quietly dropped groundedness six points. This is the same discipline as regression testing in ordinary software, applied to a component that fails probabilistically instead of loudly.
And the wrong-answer report gets a real process. The case goes into the test set, the failure is diagnosed, retrieval, prompt, or genuinely the model, and the fix is verified by the suite rather than by trying it a few times. The test set grows from production reality, which means the harness gets more representative with age. It is the only part of an AI system that improves by default.
If a vendor or an internal team cannot show you their evaluation results, they do not know how good their system is. They may still be right that it is good. But they are guessing, and you would be buying the guess. The question "what is your accuracy on your test set, and what is in the test set" separates production AI engineering from demo engineering in one sentence.
Every system our AI practice ships includes the harness, and the number it reports is in the operating reviews, next to cost and latency, where it belongs.
Duke Le works in Coder Trove's AI, data and analytics practice.