Automated regression assurance for retrieval-grounded agents
Humint Labs delivery that turns retrieval and generation evidence into a fast, repeatable release decision.
- Delivery
- Designed, built and operated by Humint Labs as an internal evaluation and release assurance service.
Measured evaluation performance
Full question suite on CI hardware, sharded across 8 workers, July 2026 build.
Curated gold set spanning four internal retrieval corpora.
Agreement between the LLM-as-judge scorer and a held-out human-graded sample, quarterly re-baseline.
Per-question judge latency, single-region inference.
Measures describe the operated evaluation service and its release-path performance.
What your enterprise receives
A release assurance capability that makes the behaviour a workload promises testable, reviewable and repeatable as the system changes.
Evaluation baseline
Defined workload claims, failure classes and acceptance criteria tied to the behaviour the enterprise intends to operate.
Versioned evaluation suite
Cases and expected evidence held alongside the application so release decisions remain comparable over time.
Release gate
Automated checks that distinguish retrieval from generation failure and escalate regressions that need a decision.
Replayable evidence record
Run-level prompt, model, retrieval, tool and assertion context for review, release notes and future investigation.
Reviewer workflow
Clear ownership for deciding whether a failure is accepted, fixed, escalated or converted into a new case.
How Humint Labs gets it live
- Step 01
Define the workload claims
Identify the journeys, evidence and failure conditions that matter to product, engineering, risk and operations.
- Step 02
Build the suite and gate
Implement versioned cases, independent scoring and release-path checks around the application.
- Step 03
Operate and re-baseline
Use the evidence record to govern change, review material failures and maintain confidence in the evaluation system itself.
The operating challenge
Every retrieval-grounded agent needs a reliable release decision when a prompt, retriever, base model or surrounding configuration changes. Manual regression review consumed senior engineering time and made the evidence available only near release.
More importantly, retrieval and generation failures were mixed together, so every result had to be diagnosed from first principles before an owner could decide whether it affected a real journey or blocked release.
What we found
- The manual process wasn't slow because grading was hard. It was slow because nobody had separated 'did retrieval find the right passage' from 'did the model answer correctly given that passage', so every failure required re-diagnosing both at once.
- Once the two were split apart, most regressions traced back to retrieval, not generation.
Where the bottleneck sat
The bottleneck was a missing separation between retrieval quality and generation quality in how we measured failures, not a lack of compute or tooling.
Design rationale
The honest first move wasn't a new tool. It was a taxonomy: we defined four failure classes, no-retrieve, wrong-passage, right-passage-wrong-answer, and hallucinated-citation, before writing a line of harness code. A faster wrong measurement is worse than a slow right one.
The boundary of the harness is deliberate. It measures the claims the team has chosen to make about a workload; it does not certify a model in the abstract. New journeys are added when the product changes, and high-consequence cases are reviewed with the people who own the operation rather than left to an engineering-only score.
Our Solution
Humint Labs built a CI-integrated regression harness that evaluates retrieval and generation independently, classifies failures into defined categories and blocks a merge when a critical regression appears. Versioned cases retain the user intent, expected evidence, prohibited behaviour and acceptable variation.
Each run keeps the model and prompt configuration, retrieved sources, tool calls, assertions and reviewer decision together, turning a disputed result into a replayable release record.
Scroll diagram horizontally
Retrieval scorer
Grades retrieved passages against a labelled gold set independently of generation, using recall at k against curated relevance judgements rather than a proxy metric.
Generation scorer
An LLM-as-judge step constrained to the passages actually retrieved, so it can only fail generation, not retrieval, when a question misses.
Suite runner
Runs the full question suite on every pull request through a queued worker pool, sharded so a full evaluation run completes inside a single CI timeout window.
Making the LLM-as-judge scorer itself stable release to release. We pinned its model version and prompt separately from the system under test, and re-baseline it against a held-out human-graded sample every quarter, otherwise the judge's own drift shows up as a false regression in the system being tested.
What changed in the release operating model
- A regression that used to surface in user acceptance testing now surfaces on the pull request that caused it.
- Retrieval and generation failures are diagnosed separately by default, rather than re-litigated together every time.
- The eval suite is versioned alongside the code it tests, rather than living in a spreadsheet outside source control.
- A release review can trace a failure from user intent to retrieved evidence and assertion.
- New cases can be added without changing the application under test.
- The team has a repeatable way to decide whether a regression is fixed, accepted or escalated.
Design principles
- Versioned
- Automated
- Classified
- Replayable
Continue into the evidence
More case studies
Governed agent tooling for enterprise AI operations
One governed toolset for an enterprise platform’s management surface, designed and operated by Humint Labs to make authority, tenant isolation and auditability executable.
Multi-channel AI guardrails that hold
Humint Labs delivery for one accountable policy across voice, web chat and messaging, with every intervention traceable, tested and owned.
Industry: Insurance
Answering insurance enquiries, escalating everything else
Humint Labs designed and delivered a grounded service layer that resolves routine insurance enquiries, equips agents with usable evidence and escalates consequential matters with full context.