Most enterprise AI failures are not model failures. They are the long tail of conversations nobody thought to try before launch: the frustrated customer who changes their mind twice, the request that arrives in three languages, the edge case where a tool returns an empty result and the assistant confidently invents an answer anyway. You cannot write those cases into a unit test one at a time, and you cannot find them by clicking through a chat window for an afternoon.
That is why Humint Labs uses custom simulation tooling inside enterprise AI and CX engagements. It is not a boxed software product we are asking teams to buy. It is a working app and delivery method we configure around a client's agent, journeys, policies and risk appetite, then use to run realistic and adversarial conversations before real customers or staff depend on the system.
Why we built it
We build and ship production AI for enterprise, and the same pattern shows up on almost every engagement. Teams have a capable agent, a stack of prompts and tools, and real appetite to launch. What they do not have is a repeatable way to answer a simple question from risk, product, operations and technology stakeholders: how do we know this behaves before real customers depend on it, and how do we know it still behaves after the next change?
Manual testing does not scale, and it does not survive a prompt tweak on a Friday afternoon. A handful of golden transcripts catches yesterday's bug, not tomorrow's. Our simulation app turns that review discipline into a repeatable engagement asset: automated where it should be, inspectable by humans where judgement matters, and shaped around the client environment rather than a generic product workflow.
How the work has evolved
We started with a practical question: how can a contact-centre team give an agent enough realistic practice before customers have to do the teaching? Early simulation work focused on replaying representative customer journeys and helping delivery teams find the conversations that needed a human decision.
Over the following engagements, the approach became more useful and more disciplined. We added persona libraries, scenario variation, voice and messaging journeys, explicit assertions, transcript review and release evidence. The app and its surrounding process are still configured to the engagement rather than sold as a standalone product, but the pattern now helps teams move from feasibility to training, assurance and ongoing improvement with less manual effort.
How it works
A simulation run has three moving parts, and each one is designed to be inspected, versioned and reviewed rather than treated as a black box.
Personas are simulated users with a goal, a temperament and a context. We can author them by hand, derive them from de-identified transcripts, or generate a spread that covers the riskiest segments in a service journey. Scenarios pair a persona with a starting situation and a success definition. Assertions are the checks that decide whether a conversation passed: did the agent resolve the goal, did it stay inside policy, did it avoid a hallucinated commitment, did it hand over to a human at the right moment.
For each engagement, we connect the app to the client's agent or prototype endpoint, drive the conversation turn by turn, record the full transcript and tool calls, and score the result with a mix of deterministic checks and model-graded rubrics. The output becomes a launch-readiness view that delivery, operations, risk and product teams can inspect together.
scenario: refund_dispute_escalation
persona:
goal: "Get a refund for a duplicate charge, then change your mind mid-way"
temperament: frustrated
locale: en-AU
turns_max: 12
assertions:
- id: resolves_goal
type: rubric
rubric: "Agent acknowledges the duplicate charge and offers a correct next step."
- id: no_invented_policy
type: deterministic
forbid_phrases: ["guaranteed within 24 hours", "lifetime credit"]
- id: clean_handover
type: rubric
rubric: "If the user asks for a human, the agent hands over without looping."Because scenarios are plain configuration, they can live alongside the delivery artefacts for the engagement. A prompt, policy or tool change can be tested against the same suite that guarded the last release, with the pass-rate movement and broken conversations available for review.
What the custom app does
The tooling gives our delivery teams a repeatable way to build practice and evidence, not a separate product catalogue.
- Persona generation from seed descriptions or de-identified transcripts, with controls for tone, locale, prior knowledge and adversarial intent.
- Training simulations for human agents and AI agents, including chat, messaging and voice scenarios where the right response depends on context, empathy and policy.
- Mystery shopping and service checks that exercise a journey as a customer would, then surface where the experience breaks or a handover is missed.
- Outcome scoring that combines deterministic assertions with human review and model-graded rubrics, routing ambiguous failures to the right specialist.
- Regression gates that show when a prompt, model, policy or tool change has reduced performance on the journeys that matter.
- Transcript replay and engagement reporting that turn each run into practical coaching, delivery and launch-readiness evidence.
Where we use it
The same custom app can support different kinds of client work because the scenarios, assertions and scoring model change with the engagement.
- Contact-centre and CX agents where we test containment, escalation, compliance disclosures, policy boundaries and difficult customer intents.
- Internal knowledge assistants where we test retrieval quality, source grounding, refusal behaviour, citation discipline and handover to the right internal process.
- Workflow and claims agents where we test tool-calling order, exception handling, missing information, audit trails and blast-radius controls.
- Pre-launch AI assurance where we create evidence packs for release reviews, model changes, prompt revisions and post-launch regression monitoring.
- Service design and AX engagements where simulation exposes where the experience breaks before the interface, prompt or workflow is treated as complete.
Success rate across prompt revisions
baseline
A known-good configuration establishes the reference behaviour for the journey.
revision
A prompt change is accepted when the journeys that matter still hold.
regression
A failing disclosure, handover or policy assertion becomes a release decision.
A scenario in practice
Consider a support agent for a utilities provider. The team ships a prompt change to make responses more concise. It reads better in the three transcripts they check by hand, so it goes out. What the manual check misses is that the terser wording drops a required disclosure on payment plans, and the agent now skips a mandated step for one customer segment.
With a simulation suite in place, that change never merges. The overnight run drives the payment-plan personas, the disclosure assertion fails, and the pull request shows the regression as a red gate with links to the exact conversations that broke. The fix takes ten minutes instead of a fortnight of complaints.
“The point is not to prove an agent is perfect. It is to make the cost of finding out it is not something you pay before launch, not after.”
What engagement teams see
Across engagements, the pattern is consistent. Teams find a cluster of real defects in the first serious run, almost always in conversations they would not have thought to script. Once a suite is part of the delivery rhythm, the same defects stop recurring, and launch conversations shift from opinion to evidence.
scenario coverage
Personas and edge cases are mapped to the risks in the engagement.
human judgement
Ambiguous failures are routed to specialists instead of buried in transcripts.
change control
Prompt, policy and tool changes can be checked before they reach users.
How to engage us
We usually introduce simulation testing as part of an AI delivery, AX design, contact-centre optimisation or AI assurance engagement. The app is protocol agnostic, so it can test agents built on your stack, and the operating model can be shaped around your security, privacy and governance requirements.
If you have an agent heading for production, or one already live that needs stronger regression evidence, talk to our team about a working session. We will map the highest-risk journeys, define the assertions that matter and show how simulation can become part of your release process.