An LLM evaluation harness tells you, with a number, whether your AI system gives correct answers. It matters more than model choice because models are the most replaceable part of the system, while the harness has to be built from your own cases and keeps its value through every model swap.
Most AI projects put their rigour in the other place. Someone presents a comparison of models, with benchmark scores and pricing, and the room debates which to choose as if picking a database. Then the system ships, a user reports a wrong answer three weeks later, and the team's method for assessing the report is to try the question themselves and squint at the output.
It has three parts. A test set: real questions from your domain, paired with what a correct response looks like, including the questions that should be refused. Scoring: automated judgements of each response for correctness, groundedness (whether the answer is supported by what was retrieved) and refusal behaviour. And a routine: the suite runs before every release and continuously against production, so quality is a number with a trend.
Building the first version takes a few weeks, most of it spent extracting a few hundred real cases and arguing about what "correct" means for each. That argument is useful work. It is when the organisation finds out it never defined what it wanted the system to do, and settling it improves everything downstream.
Model choice becomes an experiment. Run the candidates against the harness, read the scores against the prices, decide in an afternoon. When a new model releases next quarter, repeat the afternoon, and the model debate described above stops recurring.
Regressions show up before users see them. Prompt changes, retrieval tweaks and index updates all look harmless in spot checks; the harness catches the one that quietly dropped groundedness six points. This is the regression testing of ordinary software, applied to a component that fails probabilistically instead of loudly.
Wrong-answer reports get a real process. The case goes into the test set, the failure is diagnosed (retrieval, prompt, or the model itself) and the suite verifies the fix. Because the test set grows from production cases, the harness becomes more representative over time.
If a vendor or an internal team cannot show you their evaluation results, they do not know how good their system is. They may still be right that it is good, but they are guessing, and you would be buying the guess. Ask what their accuracy is on their test set, and what is in the test set. The answers show whether you are looking at production engineering or a demo.
Every system our AI practice ships includes the harness, and its scores sit in the operating reviews next to cost and latency. For the retrieval side of the same systems, see why AI projects fail on retrieval and permission-aware RAG.
A first version usually holds a few hundred real cases from your domain. It grows as wrong answers from production are added.
Correctness, groundedness in the retrieved material, and refusal behaviour on questions the system should decline. Track the scores alongside cost and latency.
Before every release, including prompt, retrieval and index changes, and continuously against production traffic.
Yes. A permission-filtered test set checks that an unauthorised user gets a clean refusal, and it runs with the rest of the suite before each release.
Duke Le works in Coder Trove's AI, data and analytics practice.