Building an automated regression harness for retrieval-grounded agents

A CI-integrated regression harness that runs a fixed question suite against retrieval and generation independently, classifies every failure into one of four categories, and blocks a merge on any regression in the no-retrieve or hallucinated-citation classes.

The work established a small, versioned evaluation service around the application rather than a separate research notebook. Cases describe the user intent, expected evidence, prohibited behaviour and acceptable variation. Runs retain the model and prompt configuration, retrieved sources, tool calls and assertions. Reviewers can label the failure class, attach a decision and compare the result with the previous release. That makes the harness useful to delivery teams because it connects a technical regression to the behaviour it could change in production.

The operating pattern gives the team a practical release rhythm: define the claim, select the cases that exercise it, run the suite, inspect material failures and record the decision. That sequence is small enough to use during delivery and explicit enough to support a later review. It also creates a reusable foundation for new agents rather than a one-off report for one launch.

The implementation is also useful because it makes ownership visible. Product defines the journey claim, engineering owns the executable case, and risk or operations decides which failures require escalation. That division prevents evaluation from becoming either an engineering-only benchmark or a vague stakeholder review. The evidence can be carried into release notes, incident review and future change analysis, giving the team a durable record of why a behaviour was considered acceptable at a particular point in time.

The pattern also makes future change safer. When a new model, prompt or retrieval strategy is introduced, the team can compare the same behavioural claims before deciding whether the change is ready. When a case is retired, the reason is recorded rather than silently removing a difficult test. This creates a living quality model that follows the system as its users, tools and operating boundaries change.

A further benefit is that the evidence model can be reused across teams. A customer-service agent, an internal knowledge assistant and a tool-using workflow can each define different acceptance criteria while sharing the same run record, review states and escalation conventions. This avoids building a new evaluation process for every workload. It also helps an organisation compare readiness without reducing every system to one score: the important comparison is whether the right claims have been tested, whether failures have owners and whether the release decision is supported by evidence. In practice, that makes the harness a piece of delivery infrastructure rather than a demonstration asset.

This gives the team a practical way to explain quality to a buyer or operating sponsor: the system is not presented as magically reliable, but as a workload with defined claims, visible controls and a repeatable release decision. That is the useful story for enterprise delivery.

The operational value is that the team can make a decision with the evidence in front of it. A reviewer can see the user journey, the expected behaviour, the system response, the retrieved context and the disposition in one chain. That reduces ambiguity between product, engineering and risk, and it leaves a record that can be revisited when the workload changes. It also gives future delivery teams a starting point for their own cases, because the questions around ownership, severity and release readiness have already been made explicit.