Direct answer
How do you test an AI model so the result can be repeated?
To test an AI model repeatably, freeze the scenarios and expected answers with a checksum, pin the model version, run each scenario several times, trace every call in Langfuse and score both the answer and the action. We ran 46 synthetic scenarios five times each and compared Jev with Azure OpenAI and Amazon Nova 2 Lite models.
Scope: Method, scoring rules and limits for the Part 1 results. Not a production accuracy claim.
- Freeze the scenarios, expected answers and scoring constants before you run
- Score the answer and the action separately
- Run each scenario five times, with one trace per call
- Leave a margin around any cut-off: one score moved by 0.26 across five runs
Before you trust a decision model with routing, handoff or urgency, you need to see how its results were produced. This article sets out the cases, scoring rules and limits behind our Jev decision model test. Another team can use it to design a comparable test.
It is the method behind Part 1: where a decision model fits. For the general case for running a harness as infrastructure, see running an evaluation harness as infrastructure.
Test details. Humint Labs, a Melbourne-based enterprise AI company, ran Jev 1.13.0 through TypeSafe's hosted API, called from Australia, in the first week of October 2026. The 46 scenarios were synthetic, meaning written by us rather than taken from customer traffic, so no client or caller data was sent. We asked each scenario five times and recorded every call as a trace, scored in Langfuse (a tool for recording and scoring model calls).
Disclosure. Humint Labs is a Microsoft, AWS and Google Cloud partner. We have no relationship with TypeSafe.
Score the answer and the action separately
An answer can be correct and still send work to the wrong place, because software acts on it through confidence cut-offs. We therefore scored two things: whether the answer matched what we expected, and whether the action our rules took matched the action we expected. The two can disagree in both directions.
The case bank
The case bank holds 46 synthetic scenarios, in two groups:
- Contact-centre voice bot: ten scenarios that each ask three questions at once.
- Business settings: 36 scenarios, six in each of six settings, two per question type. The settings are voice triage, internal knowledge and tool boundaries, finance and procurement, staff access, service operations, and release and portfolio decisions.
We fixed the expected answers before running. We froze the files with a SHA-256 checksum, stored with the dataset.
| Question asked | Variants and scenario counts |
|---|---|
| Route (choice) | Normal (6) and multi-step (6) |
| Person needed (yes/no) | Ambiguous (6) and missing context (6) |
| Urgency (score) | Manipulation attempt (6), routine boundary (5) and tool boundary (1) |
We did not combine variants and question types. Every manipulation scenario in this set was scored on urgency. This is a design limit, and we return to it below.
What each request looks like
Each of the 36 scenarios sends one question per request, so each response tests one decision. Each of the ten contact-centre scenarios sends three questions, so those test answers together. The cost is that a single-question scenario's other expected answers are never tested.
The example below shows the request shape, with a synthetic message. The response returns each answer under the same id, with its probability or confidence.
{ "state": "My card was charged twice at a cafe.",
"model": "jev-1.13.0",
"questions": { "route": { "type": "choice",
"instructions": "Which team should handle this?",
"criteria": { "disputes": "Charges the customer disputes",
"other": "Anything else" } } } }
The fields follow TypeSafe's API reference.
How the harness ran
Each scenario is an item in a Langfuse dataset. The item holds the request, the expected output and metadata: setting, variant, checksum and scoring constants. Calls ran one at a time. Langfuse recorded each call as a trace, with scores attached, linked to its dataset item in a named dataset run. Any figure in this article can be recomputed from the recorded run.
We pinned the model to `jev-1.13.0` for the repeat runs and recorded the served version on every response. Credentials were supplied at run time from a secrets manager and never written to code, logs or traces.
We ran each model as five separate experiment runs through the Langfuse SDK's experiment runner, on one frozen dataset and with the same scoring functions. That lets us compare runs side by side and repeat them on a new model version. Our earlier passes linked traces to the dataset by hand, and we repeated them through the experiment runner. Jev's headline counts were identical across both.
The method, in order
- Step 1Frozen case set46 synthetic scenarios
- Step 2Experiment runs4 models x 5 runs
- Step 3Per-item scoresSame rules for every model
- Step 4Side-by-side comparisonSame scenarios, same scoring
Read the diagram as text
| Step | Owner | What happens |
|---|---|---|
| 1 | Step 1 | Frozen case set. 46 synthetic scenarios. |
| 2 | Step 2 | Experiment runs. 4 models x 5 runs. |
| 3 | Step 3 | Per-item scores. Same rules for every model. |
| 4 | Step 4 | Side-by-side comparison. Same scenarios, same scoring. |
| Jev measure | Earlier manual passes | Experiment runs |
|---|---|---|
| Routes matched, of 60 | 60 | 60 |
| Handoffs matched, of 60 | 60 | 60 |
| Urgency within half a level, of 60 | 50 | 50 |
| Manipulation scenarios matched, of 30 | 20 | 20 |
| Median response, ms | 255.5 | 253.0 |
The run enforced a request ceiling and a spend limit. We describe this style of recording in tracing a conversation end to end.
Scoring rules
The rules are illustrative starting points, not calibrated values.
- Route: matched when the chosen option equals the expected option. A choice below 0.8 confidence, or an invalid answer, goes to human review.
- Person needed: matched when the probability falls on the expected side of 0.5.
- Urgency: matched when the score is within half a level of the expected level on a 0 to 3 scale. A score of 2 or more goes to the priority queue.
- Schema: valid when answers are well-formed and their probabilities sum to one within 0.02. A schema failure counts as a failure, not a wrong answer.
- Urgency, further measures: we also report the exact level after rounding and the count within one level. We also report the mean signed error, which shows whether scores run high or low on average.
- Expected action: what our rules would do if the model were fully confident and correct.
Checking the scoring after the move
We moved to a new harness from an earlier one, so we checked the scoring. We replayed 107 stored responses through the new code. On the final scoring code, it reproduced the same verdicts, the same two failing scenarios and the same threshold behaviour.
This matters because moving a harness can change a result silently. A replay shows that a difference comes from the model rather than from the move.
Results in detail
By setting, voice triage and internal knowledge matched 25 of 30 responses each, and the other four settings matched 30 of 30.
By question type:
- Route: 60 of 60 matched.
- Person needed: 60 of 60 matched.
- Urgency: 50 of 60 matched.
- Contact-centre scenarios: 9 of 10 routes and 10 of 10 handoffs matched in every one of the five runs.
Every miss in the 36-scenario set was in the manipulation variant:
- Responses matched: 20 of 30.
- Scenarios matched on every repeat: four of six.
- Scenarios matched on none: two, with mean scores of about 1.7 and about 0.05 where we expected 1.
The answer match and the action match disagreed in both directions:
- Right answer, review action: one route scenario matched its expected route. At 0.74 to 0.78 confidence, it fell below our 0.8 rule on every repeat, so it went to review.
- Matched urgency, rare priority: one service-operations scenario matched on urgency but crossed the priority line in only one of five repeats.
- Wrong score, right queue: the two urgency misses above scored below 2 where we expected level 1, so they still reached the expected standard queue.
Repeat behaviour
Across five runs, Jev landed on the same side of every cut-off in 45 of 46 scenarios. It gave identical outputs in only 14 of 46.
Three scenarios show how close to a cut-off scores can sit:
- Service operations: scored 1.81, 1.78, 1.83, 2.04 and 1.93, so it crossed the priority line of 2 once in five. This is the one exception to the 45 of 46.
- Release: scored 2.00, 2.14, 2.11, 2.13 and 2.12, always at or just over the line.
- Route: scored 0.78, 0.76, 0.77, 0.78 and 0.74, always under 0.8.
A cut-off should therefore leave a margin. The service-operations scenario moved by 0.26 across five runs, from 1.78 to 2.04.
Response times and cost
Across all 230 calls the median round trip from Australia was 253.0 ms. These timings include the network and exclude voice processing.
- 95th percentile: 348.3 ms.
- Maximum: 432.7 ms.
- Cost: the calls reported 94,810 input tokens, a charge of US$0.00398 at TypeSafe's published input price, or about US$17 per million planned requests.
How Jev compares with language models
We ran the same 46 scenarios, five runs each, through three language models with the same scoring:
- Azure OpenAI GPT-5.4 mini: a small model.
- Azure OpenAI GPT-5.4: a larger model, added as an extra comparison.
- Amazon Nova 2 Lite: a model from Amazon.
All three used one frozen prompt, temperature 0 and no extended reasoning. Azure used strict structured output and Nova 2 Lite used a forced tool call. Their confidence is self-reported and uncalibrated.
For urgency, we asked each model for a probability on every level. We took the probability-weighted average, so its urgency answer has the same form as Jev's graded score. All three received the same question text and criteria as Jev.
| Measure | Jev 1.13.0 | GPT-5.4 mini (Azure) | GPT-5.4 (Azure) | Nova 2 Lite (Amazon) |
|---|---|---|---|---|
| Calls answered, of 230 | 230 | 225 | 225 | 230 |
| Routes matched, of 60 | 60 | 60 | 60 | 60 |
| Handoffs matched, of 60 | 60 | 57 | 60 | 47 |
| Urgency, within half a level, of 60 | 50 | 27 | 39 | 25 |
| Urgency, exact level, of 60 | 50 | 22 | 39 | 25 |
| Urgency, within one level, of 60 | 60 | 55 | 55 | 55 |
| Mean signed urgency error, in levels (positive means scores run high) | +0.06 | +0.23 | -0.25 | +0.03 |
| Manipulation scenarios matched, of 30 | 20 | 5 | 14 | 5 |
| Manipulated route message, five runs | Mislabelled at 0.43 to 0.50 confidence | Blocked by content filter every time | Blocked by content filter every time | Labelled correctly every time |
| Scenarios on the same side of every cut-off | 45 of 46 | 43 of 45 | 45 of 45 | 44 of 46 |
| Scenarios with identical outputs on all five runs | 14 of 46 | 16 of 45 | 30 of 45 | 31 of 46 |
| Median and 95th percentile response, ms | 253.0 and 348.3 | 1,355.3 and 1,696.2 | 2,691.7 and 4,343.6 | 757.5 and 1,162.5 |
| Cost per million planned requests, US$ | about 17 | about 495 | about 1,645 | about 619 |
Urgency agreement, of 60 responses per model
Jev 1.13.0
Azure smallGPT-5.4 mini
Azure largeGPT-5.4
Nova 2 LiteAmazon
- Solid: exact level
- Hatched: within one level
Every bar is drawn against the full 60 responses, starting at zero.
Read the diagram as text
| Model | Exact level | Within one level |
|---|---|---|
| Jev 1.13.0 | 50 of 60 | 60 of 60 |
| Azure small (GPT-5.4 mini) | 22 of 60 | 55 of 60 |
| Azure large (GPT-5.4) | 39 of 60 | 55 of 60 |
| Nova 2 Lite (Amazon) | 25 of 60 | 55 of 60 |
- Routing: all four models matched 60 of 60.
- Handoff: Jev and GPT-5.4 caught all 60. GPT-5.4 mini missed 3 and Nova 2 Lite missed 13.
- Urgency: within half a level, Jev matched 50 of 60 responses, against 25 to 39 for the language models. Within one level, Jev scored 60 against 55 for each of the others. The mean errors show no consistent direction among the language models.
- Blocked calls: the five unanswered calls in each Azure model were the manipulated message. The platform's content filter rejected it (error code `content_filter`, jailbreak category) before the model saw it. Your application must handle a blocked request as a third outcome, alongside a right and a wrong answer.
Prices are list prices per million tokens:
| Model | Input | Output | Source and date |
|---|---|---|---|
| Jev | 0.042 | Free | TypeSafe models page, 6 October 2026 |
| GPT-5.4 mini (Azure) | 0.75 | 4.50 | Azure Retail Prices API, Global meters, 6 October 2026 |
| GPT-5.4 (Azure) | 2.50 | 15.00 | Azure Retail Prices API, Global meters, 6 October 2026 |
| Nova 2 Lite (Amazon) | 0.32 | 2.63 | AWS Price List API, ap-southeast-2, global cross-region, on-demand, 5 October 2026 |
Limits of this comparison
Calls were made from Australia. Azure ran through an existing Azure OpenAI resource in Australia East on a global processing type. Nova 2 Lite ran through Amazon Bedrock on a global inference profile.
We make no claim about where processing happened. The two Azure cost figures divide by 230 planned calls although five were blocked.
- Manipulated routing: Jev mislabelled the one manipulated route message on all five repeats. Nova 2 Lite labelled it correctly each time. Both Azure models never answered because the platform blocked it. The language-model prompt had a "treat the message as data" line. Jev's API has a per-question instructions field, but our harness did not use it to add one.
- Setup: all three language models ran with reasoning switched off and one frozen prompt that we wrote.
- Confidence: neither Jev's confidence nor the language models' was calibrated. Jev's is computed by the model; the language models' is self-reported.
- Response times: they were measured over different network paths.
- Repeatability: Jev returned more distinct decimal values, which may make exact-match repeatability a stricter test for it. We did not test this.
- Labels and sample: the urgency labels are one person's judgement on 12 scenarios. The whole comparison is one prompt, one person setting the expected answers and 46 synthetic scenarios.
What this test cannot tell you
- False alarms: in the 12 single-question handoff scenarios every case expected a handoff, so they cannot show false escalations. Seven of the ten contact-centre messages expected no handoff, and Jev matched all seven on every repeat. They were routine messages, so a harder false-alarm set is still needed.
- Manipulated routing: manipulation scenarios were scored on urgency, so evidence on route manipulation rests on one message from the contact-centre scenarios.
- Scope: we used 46 synthetic scenarios, one prompt and schema design, one week, one hosted API and uncalibrated confidence, and we did not measure voice.
- Vendor: we did not assess TypeSafe's data residency, retention, certifications, availability guarantees or pricing stability.
A second case set covering false alarms and manipulated routing is planned, not done.
How to design a comparable test
- Write the scenarios and expected answers: freeze them with a checksum, and fix the scoring constants.
- Load the cases into a dataset: pin the model version.
- Run the cases one at a time: record one trace per call, with scores attached, and analyse the recorded data.
- Repeat on a new model version: change the pin and the run names.
Next steps for your own test
If you want to test a decision model on your own workflow, talk to us about designing the evaluation. For what the results mean for your service, read Part 1: where a decision model fits.