A language model feature can look convincing in a demonstration and still be unready for production. The gap is not explained by one bad answer. It is the long tail: a request that changes direction twice, a source document that contradicts another, a customer who needs a human, or a tool result that arrives empty while the model still has to decide what to say. Reliability starts when those situations become test cases rather than surprises.
What evaluation is meant to establish
Evaluation is evidence for a release decision. It should tell a delivery team what was tested, against which version of the corpus and configuration, with what expected behaviour, and where the system still needs human oversight. That is different from asking whether a model is generally good. The useful unit is the task the system performs for a particular audience, on a particular channel, with a particular consequence if it is wrong.
Build the evaluation set from the work
- Start with real intents and journeys. Sample the questions, handovers, policy boundaries and workflow steps the service actually has to support.
- Keep the difficult cases. Preserve ambiguity, missing information, conflicting sources, multilingual input, prompt-injection attempts and requests that must be escalated.
- Version the set. A test case has an owner, a source, an expected outcome and a history. When the product changes, add a new version rather than silently rewriting the evidence.
- Separate acceptance from exploration. A small stable release gate can sit beside a larger research suite. Mixing them makes the gate too slow or the research too shallow.
Choose the grader that matches the claim
Use deterministic checks when the answer has a deterministic shape: schema validation, exact fields, totals, dates, permissions and tool-call order. Use a rubric when quality depends on several observable properties, such as groundedness, completeness or an appropriate handover. A model-as-judge can help with open-ended comparisons, but it needs calibration against human labels and should not be treated as an authority simply because it produces a score. The grader is part of the system under test and needs its own version, threshold and review path.
Test safety and usefulness together
A useful answer that crosses a permission boundary is a failure. A perfectly safe assistant that escalates every request is also a failure if the service was meant to help people complete ordinary work. Test both sides of the contract: whether the system completes permitted tasks and whether it refuses, redirects or escalates the tasks it must not complete. For tool-using agents, include indirect instructions in retrieved content, over-broad requests, missing tool results and repeated attempts to obtain a consequential action.
Make regression part of delivery
A prompt change is a behaviour change. So is a model version change, a retrieval-index rebuild, a new policy document, a tool-schema edit or a change to the handover path. Run the stable suite in continuous integration or the release pipeline, retain the run manifest, and compare results by scenario class rather than relying on one aggregate score. A small drop in an overall average can hide a severe regression in the exact class of request that carries the most risk.
const result = await evaluate({ manifest, cases: releaseSuite })
assert(result.policyViolations === 0)
assert(result.criticalEscalations <= allowedEscalations)
assert(result.groundedness >= minimumGroundedness)
await storeRun({ manifest, result })What a release record should contain
Keep the model identifier, prompt or policy version, corpus and index version, tool definitions, evaluator versions, thresholds, run timestamp and scenario results together. The record should make a later comparison possible without relying on somebody remembering which settings were used. It should also show the cases that were excluded and why. Traceability is not paperwork added after the build; it is what turns a passing test into evidence another person can inspect.
Keep the evidence legible
A result should be understandable to the people who did not run the test. Name the scenario class, show the expected behaviour, state the observed failure and link the run to the versions that produced it. Avoid hiding a material limitation inside an average or a traffic-light label. A concise failure record is more useful than a polished chart when an incident review asks what the team knew before release.
A useful evaluation readout
Critical policy breaches
A release floor for explicitly prohibited behaviour.
Quality reporting
Separate results for ordinary, edge and high-consequence scenarios.
Reproducible run
The versions and thresholds needed to explain the result.
Where human review remains essential
Automation should reduce repetitive checking, not hide the decisions that require judgement. People still need to review new scenario classes, calibrate rubrics, inspect surprising failures, approve changes to escalation policy and decide whether a known limitation is acceptable for the service. The point of the harness is to make those decisions earlier, with a smaller and more legible body of evidence in front of the team.
A practical sequence
Start with a narrow task and a reviewed set. Add deterministic assertions first, then a rubric for the qualities that cannot be reduced to exact matching. Run adversarial cases before a public pilot. Put the suite behind the release process. After launch, promote real failures into the corpus and retire cases only with a recorded reason. That sequence is deliberately ordinary: the value comes from repeating it as the system, the source material and the risk change.
Common failure modes
Teams often begin by measuring what is easiest to count: latency, token cost or a broad average score. Those measures matter, but they do not tell you whether a customer was handed to the right person, whether a private source was exposed or whether a required explanation was omitted. Another common failure is changing the prompt, model and retrieval index at the same time, then calling the result an evaluation. If the manifest cannot tell you what moved, the run can describe a difference but cannot explain it. Finally, a suite that never gains new cases is not stable; it is simply becoming less representative as the service changes.
How to read the result
A release report is not a single green number. Read it by scenario class, failure severity and change from the previous run. A small overall improvement can still hide a regression in escalation or source grounding. A lower aggregate score can be acceptable when it reflects a deliberate change that improves the high-consequence cases and the release owner has recorded the trade-off. The decision should name what passed, what failed, who accepted the residual risk and what will be watched after launch. That makes evaluation useful to engineering, operations and governance at the same time.