Start with the suite that stopped being run. It existed, it was good, and for a quarter it caught real regressions. Then the release notes stopped mentioning it. Nobody decided to abandon it. It took most of an afternoon, it had to be started by hand by the one engineer who knew which credentials it wanted, and by the time anyone wanted to compare a result against the one from six weeks earlier, the corpus had been edited in the meantime and the two numbers were not measuring the same thing. Every one of those is survivable alone. Together they mean the suite had become a document rather than a control.
That is the characteristic failure of evaluation work, and it is worth naming precisely because it is not the failure people design against. Teams design against scoring the wrong thing. What actually happens is that the scoring is fine and the harness decays until nobody runs it, at which point the quality of its metrics is irrelevant.
The companion research note on the evaluation arena is about scoring: what a scenario contains, why deterministic assertions carry the spine of a score, and why a reliability floor beats an average. This piece is about everything underneath that, which is the part that decides whether a score from March can be compared with a score from July. Versioned corpora, a run identified by a manifest rather than a date, scoring split into stages that fail and re-run independently, results that accumulate into a queryable history, and a gate wired into the release path.
The design target throughout is handover. Not a notebook that one person can run, but infrastructure that an operations engineer who did not build it can operate, and whose corpus a domain specialist who does not write code can change. That target is what forces most of the decisions below, and it is worth stating up front because it is easy to build an evaluation system that is technically excellent and can only ever be run by its author.
Three ways a suite dies
Three deaths, and the order matters, because the first one causes the other two.
It becomes unreproducible. Someone fixes a typo in a scenario, adds twelve new cases, or removes one that was arguing with a change of policy. All of those are correct things to do. None of them is versioned, so the corpus is now a moving reference frame and every historical result silently changes meaning. The number went up. Nobody can say whether the system improved or the questions got easier.
It becomes unaffordable. Every run is a multiple of corpus size, episode length and judge calls, and all three grow monotonically because nobody ever deletes a scenario. A suite that cost little enough to run on a pull request becomes a suite that runs nightly, then weekly, then before a release, then before a big release. Each step is defensible on its own and the cumulative effect is that the feedback arrives after the decision it was supposed to inform.
It becomes unowned. Reproducibility problems and cost problems both push the run towards the one person who understands its quirks. Once running the suite requires knowing which environment it needs and which two flaky scenarios to ignore, it has an owner rather than an operator, and it will go quiet the week that person is on leave.
None of these are scoring problems, so no amount of work on the rubric fixes them. They are infrastructure problems, and the rest of this is the infrastructure.
The pipeline, end to end
The reference architecture behind the rest of this piece is a six-band pipeline. Three pinned inputs feed a run controller: a scenario corpus held in Amazon S3 with versioning on, so one immutable version is used per run; the system under test, pinned by image digest, prompt set and retriever index build; and the judge, pinned by model identifier and rubric, separately from the system under test. The run controller resolves a run manifest and pins every input, so a run is identified by that manifest rather than by a timestamp. The controller starts the execution plane, an AWS Batch array job whose concurrency is capped to the inference endpoint rather than to the compute fleet, fanning out into shards that each take a slice of the corpus. The shards feed two separate scoring stages: a deterministic assertion stage that runs with no model in the loop and passes or fails against recorded state, and a rubric stage that runs a judge model over stored transcripts and can be re-run without re-running the system under test. Both stages write to Amazon Aurora PostgreSQL, holding one row per episode per assertion per stage, each row carrying every pinned version above, with re-grading able to run from stored transcripts alone. The results store feeds two consumers: a release gate that blocks a release when a per-class floor is not met or when corpus coverage of a class has been lost, and a queryable history that answers when a regression started rather than only that it exists.
The corpus is an artefact
Begin with the corpus, because everything else pins to it. A scenario corpus is not a folder of files that engineers edit in place. It is a versioned artefact with a publication step, and treating it as anything less is what makes results incomparable.
Corpora live in Amazon S3 with versioning switched on, and the corpus a run used is recorded as the object version identifier, not as a path or a tag. This distinction does more work than it appears to. A tag is a pointer somebody can move, and the person who moves it is never the person reading the six-month-old result that depended on it. A version identifier is immutable by construction, so a run that recorded one can be replayed exactly, including on the day somebody has decided the corpus was wrong all along.
Publication is a step, not a save. A corpus version is assembled, validated against its schema, checked for duplicate and contradictory cases, and written once. Editing means publishing a new version. That sounds heavy until you notice it is the only mechanism that lets a domain specialist change what the system is tested against without an engineer present, which is the handover property you actually wanted.
Lifecycle rules then split three kinds of object that people tend to treat as one. Corpus versions are small and are kept, because a result that cannot be replayed is a result you will eventually stop trusting. Transcripts are large, are read intensively for about a fortnight while somebody investigates a failure, and are read almost never after that, so they transition to infrequent access early and expire on a stated schedule. Result rows are neither, and they do not belong in object storage at all.
A run is a manifest
Now the piece that ties the corpus to everything else. A run is not identified by when it started. It is identified by a manifest that pins every input by version, and the manifest is written before the run rather than reconstructed after it.
# A run is this document. Two runs are comparable when their manifests differ
# in exactly one field, and in no other way.
run_id: r-2026-07-27-4f2a # assigned, never derived from the clock alone
corpus:
bucket: eval-corpora
key: contact-handling/suite.json
version_id: 3sL4dQ9pRk1x # the S3 object version, not a tag anyone can move
episodes: 412
system_under_test:
image: agent-runtime@sha256:9c2f... # digest, not a mutable tag
prompt_set: prompts/v37
# The index build, not the retriever code. Rebuilding an index from the same
# documents with the same code does not reliably give you the same index.
retriever_index: idx-2026-07-19
inference_profile: eval-throughput
judge:
model_id: judge-v4 # pinned apart from the system under test
rubric: rubrics/v6
calibrated_against: human-sample-2026q3
agreement_rate: recorded_at_calibration
execution:
shards: 8
max_concurrency: 12 # capped to the endpoint, not to the fleet
# Fix the seed on the environment and the simulated counterpart. Never on the
# system under test, or the run measures your ability to replay a recording.
seed_policy: environment_onlyThe test this exists to satisfy is comparability. Two runs are comparable when their manifests differ in exactly one field and in no other way. If the corpus version, the image digest, the prompt set, the retriever index and the judge model are all pinned, then a change in the result is attributable, and if two of them moved you know before you start arguing that the comparison cannot settle anything. Most evaluation disagreements are actually this, discovered late.
Seed policy belongs in the manifest too, and it carries the discipline the arena note sets out: fix the seed on the environment and the simulated counterpart, never on the system under test. A run where the system is seeded is a run that measures your ability to replay a recording.
The retriever line is the one people leave out. A retrieval-grounded system has an index build as an input, and an index rebuilt against the same source documents with the same code is not necessarily the same index. Pin the build, not the builder.
Stages that fail independently
Split the work into four stages that can fail independently, be retried independently and be re-run independently. Materialise the run, execute the episodes, evaluate the assertions, grade the rubric.
The important split is the last two. Deterministic assertions and rubric grading are different stages, not two functions in one loop. Assertions are checkable against recorded state, they involve no model, they are cheap and they are exactly reproducible. Rubric grading involves a judge model, costs real money per episode, and is only as stable as the judge.
Fusing them costs you three things. You cannot re-grade without re-running, so improving a rubric means paying for every episode again. You cannot tell an infrastructure failure from a quality failure, because a judge timeout and a genuine rubric failure arrive as the same row. And you cannot run the cheap half on every commit and the expensive half nightly, which is the single most effective cost control available to you.
Separated, the transcript becomes the interface between execution and grading. Execution writes transcripts and assertion outcomes. Grading reads stored transcripts and writes scores. Re-grading a whole quarter against a new rubric becomes a batch job over storage rather than a re-run of the system under test, and that one property is what makes rubric iteration affordable enough to actually happen.
There is a sequencing rule inside this that is easy to get wrong. Assertions run first and their outcome is recorded whether or not grading succeeds. A run whose judge tier fell over is a run with complete deterministic results and missing scores, which is a legible state. A run that discarded both because one failed is a wasted afternoon.
Where the work actually runs
Three planes are worth considering and the constraint that chooses between them is not the one people reach for first.
AWS Batch with an array job is the default for this shape of work. The unit is a shard of the corpus, the job is long-running by container standards, the concurrency is explicit, and a failed shard is retried without touching its siblings. Amazon SageMaker processing jobs suit a team that already lives in that toolchain and wants the run recorded alongside its other jobs. A Lambda fan-out looks attractive until an episode with a slow tool call and a retry meets the timeout, and the failure mode is a truncated episode recorded as a result.
The binding constraint is almost never fleet capacity. It is the inference endpoint. Your model provider has a token throughput quota and a request rate, and a harness that scales its workers freely will discover that quota by exhausting it, usually while somebody else is trying to serve production traffic on the same account. So concurrency is capped to the endpoint rather than to the compute, throttling responses are retried with backoff and jitter, and the retry is counted as an infrastructure event rather than an episode failure.
That last distinction earns its place. If a throttled call is recorded as a failed episode, your reliability floor now moves with how busy the account was, and the first time it fails a release for that reason you will lose an afternoon and some of the trust the gate depends on.
Shard by episode rather than by scenario. Scenarios have wildly different episode counts, so a scenario-sharded run finishes when its largest scenario finishes and the rest of the fleet sits idle waiting.
The judge is a pinned dependency
The judge tier deserves to be treated as a dependency with a version, an owner and a change process, because that is what it is. It is a model with a prompt, it moves when the provider moves it, and its drift arrives in your results looking exactly like a regression in the system you were testing.
So it is pinned apart from the system under test. Different model identifier, different version, different change window. When the provider deprecates a version, the judge migration is its own piece of work with its own before-and-after comparison on a held-out sample, not a line in the release that also changed the prompt.
Amazon Bedrock model evaluation is a reasonable home for this tier, because it gives you a managed way to run a judge over a dataset with the job recorded and the outputs landed in storage, rather than a bespoke scoring service you now own. The part it does not give you, and nobody can, is calibration. That is yours.
Calibration means a held-out sample graded by people, a recorded agreement rate between the judge and those gradings, and a stated re-baseline interval. The agreement figure is not decoration. It is what tells you whether a rubric score of eight means anything, and it is the first thing to look at when scores move without a corresponding change in assertion outcomes.
A history, not a dashboard
Results go to Amazon Aurora PostgreSQL, and the reason is the question you will actually be asked. Not what is the score, which any dashboard answers. The question is when did this start, and that is a query over history.
-- One row per episode, per assertion, per stage. The grain is deliberate:
-- a run-level score cannot say which assertion regressed, and an
-- episode-level pass or fail cannot say when it started failing.
create table eval_result (
run_id text not null,
corpus_version text not null, -- the S3 object version, never "latest"
episode_id text not null,
assertion_id text not null,
stage text not null, -- assertion | rubric
outcome text not null, -- pass | fail | error
failure_class text, -- from the taxonomy, null on a pass
score numeric, -- rubric stage only
judge_model_id text, -- rubric stage only
judge_rubric text,
sut_image_digest text not null, -- denormalised on purpose, see below
sut_prompt_set text not null,
retriever_index text not null,
latency_ms integer,
input_tokens integer,
output_tokens integer,
transcript_uri text not null, -- s3://, so the row stays small
recorded_at timestamptz not null default now(),
primary key (run_id, episode_id, assertion_id, stage)
);
-- The two queries that get run in anger: has this check been failing for
-- weeks, and which corpus versions does a failure class appear in.
create index on eval_result (assertion_id, recorded_at desc);
create index on eval_result (failure_class, corpus_version);The grain decides everything downstream. One row per episode per assertion per stage. A run-level score cannot tell you which assertion regressed. An episode-level pass or fail cannot tell you whether the same episode has been failing the same check for three weeks. Both of those are the question, and a coarser grain simply cannot answer them, no matter what you put on top of it.
Every row carries the pinned versions from the manifest. That looks like duplication and it is the point: a result row that does not carry its own corpus version and image digest is a number that has to be joined back to a run record somebody may have tidied up. Denormalise deliberately here, because the row has to survive being read alone.
Transcripts stay in object storage and the row carries a pointer. A results table that holds full conversations will be the reason somebody eventually decides the evaluation database is too expensive to keep.
The gate that blocks a release
The gate is where all of this either becomes a control or stays a report. Wire it into the release path, give it the authority to block, and give it a break-glass that writes a record.
The mistake is to gate on the aggregate. A single overall score going down is a weak signal, and worse, it is a signal that can be paid off: a system that got noticeably safer and slightly less helpful reads as a regression, and a system that got much more helpful and quietly less safe reads as an improvement. Gate on a floor per failure class instead, using the taxonomy the harness already records. The classes that carry business risk block outright. The classes that carry quality alone report and do not block.
// The gate is not "did the score go down". It is a floor per failure class,
// evaluated against one run manifest, with the comparison run named in the
// decision so nobody has to reconstruct it from timestamps afterwards.
export async function evaluateGate(run: RunId): Promise<GateDecision> {
// Floors are versioned alongside the corpus rather than with the application
// code, because the people who set them are the people who own the corpus.
const floors = await loadFloors(run);
const observed = await summariseByFailureClass(run);
const breaches = floors.flatMap((floor) => {
const seen = observed[floor.failureClass];
if (!seen) return [];
return seen.failureRate > floor.maxFailureRate
? [{ ...floor, failureRate: seen.failureRate, episodes: seen.episodes }]
: [];
});
// Coverage loss fails. If the corpus no longer exercises a class that has a
// floor against it, every remaining assertion passing is exactly what the
// problem looks like from the inside, which is why it cannot be a pass.
const uncovered = floors.filter((floor) => !observed[floor.failureClass]);
return {
blocking: breaches.some((b) => b.blocking) || uncovered.length > 0,
breaches,
uncovered,
comparedWith: await lastPassingRun(run),
// A break-glass override is permitted and is recorded here, with an owner.
// An override that leaves no row is indistinguishable from a green run.
override: await pendingOverrideFor(run),
};
}Two rules make the gate honest rather than decorative.
The first is that coverage loss fails. If the corpus no longer contains any episode exercising a class with a floor against it, that is not a pass. Every remaining assertion still passing is exactly what coverage loss looks like from the inside, and it is the failure mode most likely to survive a year unnoticed.
The second is that the comparison run is named in the decision. Compared against what is the first question anyone asks about a blocked release, and if the answer has to be reconstructed from timestamps, the argument that follows is about the harness rather than about the change.
What a full run costs
An evaluation suite nobody can afford to run is an evaluation suite nobody runs, so the cost of a full regression pass is a design input rather than a line on an invoice you look at in arrears.
The cost of a run resolves to four terms. Inference for the system under test, which is episodes multiplied by turns multiplied by tokens per turn, and it dominates. Inference for the judge, which is episodes multiplied by graded dimensions, and which is the term that grows fastest when somebody adds a rubric criterion. Compute for the workers, which is usually small and is the term people try to optimise first because it is the one that looks like infrastructure. Storage and egress, which is small until transcripts accumulate without a lifecycle rule.
Instrument all four per run and store them on the run record. The point is not accounting. It is that a cost per run trend line is the earliest possible warning that the suite is heading for the second death, and it arrives months before anyone starts quietly skipping it.
Four levers, in the order they are worth pulling. Tier the corpus, so a fast subset that covers every blocking failure class runs on every commit and the full corpus runs nightly. Sample the judge rather than grading everything, because rubric scores are a distribution and a well-chosen sample answers the same question as a census at a fraction of the cost, while assertions stay at full coverage because they are nearly free. Cache aggressively on the retrieval side, where an unchanged index against an unchanged query is genuinely the same work. And prune, which is the lever nobody pulls: scenarios that have never failed and never will are paying rent.
Built to be handed over
Everything above is in service of one property, which is that somebody else can run this. It is worth being concrete about what that requires, because it is a higher bar than it sounds and it is not met by writing a README.
The infrastructure is defined as code and stands up from nothing.
Every input is pinned by version and none of them is a local path.
Credentials resolve from a role rather than from a file on somebody's machine.
The corpus is editable by a domain specialist through a publication step that validates, rather than through a pull request against a repository they have no reason to have access to.
The gate has an owner who is not the author.
The failure taxonomy is written down, because a result classified against a taxonomy only one person holds is a result only that person can act on.
The strongest test of it is unglamorous: can the suite produce a comparable result on a day when nobody who built it is available. If the honest answer is no, the harness is a notebook with better packaging, and it will die the first death within two quarters.
Settle these first
Six decisions, in the order they have to be made.
- Version the corpus before you write the runner. Retrofitting version identity onto a body of historical results is not possible, and every result recorded before you do it is a number with no frame of reference.
- Write the manifest first and make it the identity of a run. Comparability is a property you build in or do without.
- Separate assertions from rubric grading at the stage boundary not inside a function. Re-grading without re-running is what makes rubric work affordable, and cost is what decides whether the suite survives.
- Cap concurrency to the inference endpoint and record throttling as an infrastructure event. A reliability floor that moves with account load is not a floor.
- Pin the judge separately calibrate it against human gradings, and record the agreement rate next to the scores it produced.
- Gate on a floor per failure class fail on coverage loss, and name the comparison run in the decision.
The framing to resist is the one where evaluation is a phase. A harness is not something you run before a release; it is a piece of production infrastructure that happens to have engineers as its users rather than customers. Judge it the way you would judge anything else in production. Can it be operated by someone who did not build it, can its output be reproduced, does anyone find out when it stops, and does it cost what you expected. A harness that fails those questions will keep producing scores long after it has stopped telling you anything.
References and acknowledgements
Inside the agent evaluation arena, the companion note on scoring, scenario authoring, reliability floors and seed discipline. This piece assumes it rather than repeating it.
Building an automated regression harness for retrieval-grounded agents, the capability exhibit for the running system, including its failure taxonomy and its own measured behaviour.
Amazon S3 versioning and object lifecycle management, for version identifiers and the storage-class transitions the corpus and transcript rules above depend on.
AWS Batch array jobs, for the sharding, retry and concurrency semantics used in the execution plane.
Amazon Bedrock model evaluation, for running a judge tier as a recorded job rather than as a bespoke service.
ISO/IEC 42001, the AI management system standard, for the obligation to keep records of testing and oversight rather than assert that they happened.
With thanks to the engineers who kept insisting that a result with no recorded corpus version is not a result, and to the operations specialists who authored the awkward cases that make a corpus worth versioning in the first place.