Direct answer
How did we test decision models for contact-centre work?
We built a test harness, ran ten model configurations through 184 service scenarios five times each, and logged and scored every call. We fixed the questions and scoring rules, formed an answer key with separate labelling models and human review, and retained the prompts, model pins and raw answers needed to repeat the comparison. This is how we engineered the evidence behind Parts 3 and 4.
- Score the answer and the action separately.
- Keep the original cases separate from the new text and image results.
- Preserve each scenario, its five repeats and its trace-linked scores as one record.
- Check the review boundary: Jev 1.14.0 crossed the handoff line on two of five runs of the same VPN case.
Scope: The harness, scenario pipeline, scoring controls and detailed results behind Parts 3 and 4. Ten configurations from nine models: eight decision-model configurations and Claude Haiku 5.5 at two effort settings.
Disclosure
How this test was run
- Independent, no vendor input
- Synthetic scenarios
- No customer data
- Checked through 25 Oct 2026
Humint Labs chose, designed and ran this evaluation independently, with no vendor sponsorship or input. Humint Labs is a Microsoft, AWS and Google Cloud partner; those relationships do not endorse the findings. The scenarios are synthetic: designed by Humint Labs and generated through a governed pipeline using two different model families, then checked against fixed rules.
Final labels for the new scenarios come from the agreement of three further models, with human review where they disagree; the original 46 scenarios from Parts 1 and 2 keep their human-checked labels. No customer data was used. Product capabilities, vendor status and prices are stated as checked through 25 October 2026.
- Step 1184
Scenarios, go to The case bank
The 184 scenarios comprise the original 46 from Parts 1 and 2, 112 new cases (97 text and 15 image) and 26 written by a second model family.
- Step 23 × 3
Labelling, go to Building the answer key
GPT-5.5, Nova Pro and Nova 2 Lite each labelled the 112 main items three times.
- Step 310 × 5
Runs, go to How the harness ran
Ten configurations from nine models. Each scenario ran five times per configuration.
- Step 45 rules
Route, person needed, urgency, answer shape and confidence.
- Step 51 record
Logging, go to Tracing scores back to calls
Langfuse linked its trace and scores to the dataset item and named run.
Score the answer and the action separately
An answer can match the expected answer while a confidence cut-off sends it to review. A near urgency score can cross a queue boundary. These are separate checks, as Part 2 explains. The tables below score answers; a service trial must also check the action its software takes.
The eight decision-model configurations include Jev 1.13.0 and Jev 1.14.0. Claude Haiku 5.5 answers the same bounded questions at two effort settings through a structured prompt and states its probabilities. Those probabilities differ from a dedicated decision head. Written replies and explanations are outside the scoring.
The case bank
Every configuration receives the same eligible questions and fixed choices: route, handoff and urgency, the System One role beside a language model's wider reasoning.
- Original 46from Parts 1 and 2
- New text 97of the 112 new cases
- New image 15of the 112 new cases
- Extra scored 22of 26 written by a second model family
- Excluded 4lacked a route or handoff consensus
The 184 scenarios comprise the original 46 from Parts 1 and 2, 112 new cases (97 text and 15 image) and 26 written by a second model family. Four of those 26 lacked a route or handoff consensus and were excluded from scoring, leaving 22 scored. We retain separate results for each set and for text and images. The primary inventory is 158 items, with the 26 extra items outside it.
Cases cover customer experience (CX) and employee experience (EX) requests: billing, outages, fraud, vulnerability, complaints, cancellation, access, claims, travel, parcels, warranties and utility faults. They include Australian English, Hinglish, long threads, ambiguity, synthetic personal details and image intake.
Five image cases use AI-generated images and ten use rendered synthetic screenshots or documents. Every company, airline, shop and reference number is fictional. Fifteen planned text descriptions of image parents remain outside this run.
What each request looks like
Requests supply state, questions and permitted answers: a route, yes/no probability and urgency score. One call can contain several questions, so cost per call and correct question differ. Adapters preserve the task in each API's answer shape.
The CX set has 11 routes; the enterprise route set is unchanged. Rubric 2.1.1 defines the claims and information boundary, critical safety flag and route question as follows:
Rubric source: Evaluation rubric 2.1.1.
claims
"claims: A request to lodge, track, progress or dispute compensation, reimbursement or repair for loss, damage or delay, including baggage, parcel, travel, insurance and warranty claims, and questions about how to lodge, track or progress such a claim."
Evaluation rubric 2.1.1, section 3.1
info
"info: Product, policy, timing or how-to questions with no fault and no change requested."
Evaluation rubric 2.1.1, section 3.1
Claims and information boundary
"A question about how to lodge, track or progress a claim for loss, damage or delay is claims; info is for generic product, policy and timing questions with no claim in play."
Evaluation rubric 2.1.1, section 8
Critical safety flag
"Risk of injury, fire, medical emergency, a safety-linked service failing (for example a medical alarm), a stated medical dependency at immediate risk (for example essential medication that is unavailable and due within hours)"
Evaluation rubric 2.1.1, section 4.1
Route question
"Choose the single best workflow route for this request. Do not approve, execute, diagnose or invent missing facts."
Evaluation rubric 2.1.1, section 8
Tie precedence
- fraud or security
- vulnerability or hardship
- complaints
- claims
- cancellation
- access
- billing
- technical faults
- account change
- information
- other
Tie precedence begins with fraud or security, vulnerability or hardship, complaints and claims. It continues through cancellation, access, billing, technical faults, account change, information and other, subject to the first action needed. A wrong-charge refund stays billing; compensation for damage goes to claims; a formal escalation goes to complaints. Route and critical flag are separate.
The full lineup records the API pins, input formats and list prices checked through 25 October 2026.
Read the full model lineup and list prices
Swipe or scroll the table horizontally to see every column.
| Model | API pin or recorded configuration | Input accepted | Hosted platform | List price, US$ per million tokens |
|---|---|---|---|---|
| Jev 1.13.0 | jev-1.13.0 | Text; experimental opt-in images | TypeSafe API | 0.042 input; output free |
| Jev 1.14.0 | jev-1.14.0 | Text; experimental opt-in images | TypeSafe API | 0.042 input; output free |
| Microsoft-Decision-1 | msft-decision-1 | Text only | Azure Foundry; configured in West US 3 | Not stated on the Microsoft pages read |
| OpenAI Decisions | openai-luna, served model gpt-6-luna | Text and images | OpenAI API | 0.10 uncached input; output free |
| Perplexity Decider | pplx-decider, model pplx-decider-v1.2 (27B) | Text and images | Perplexity API | 0.02, input only |
| Clef 27B | clef-27b | Text and images (card also lists video; not tested) | Cloudflare Workers AI | 0.240 input |
| Clef-flash | clef-flash, 9B | Text and images (card also lists video; not tested) | Cloudflare Workers AI | 0.038 input |
| Clef-omni | clef-omni | Text, images, audio and video | Cloudflare Workers AI | 0.150 input |
| Claude Haiku 5.5 (higher effort) | claude-control, higher effort | Text and images | Hosted API | 0.10 input; 0.50 output for prompts up to 100,000 tokens |
| Claude Haiku 5.5 (low effort) | claude-control-low, low effort | Text and images | Hosted API | 0.10 input; 0.50 output for prompts up to 100,000 tokens |
Basis: Pins and configuration names identify the recorded test. These vendor list rates are separate from measured call cost. Both Jev versions ran the same cases; TypeSafe listed 1.14.0 as stable. Processing-location checks appear under Deployment controls.
Sources: TypeSafe models, TypeSafe images, Microsoft's model guide, OpenAI Decisions, OpenAI data controls, Perplexity, Cloudflare and Claude. Product descriptions and prices do not establish study outcomes.
How the harness ran
Each scenario ran five times per configuration. Langfuse linked its trace and scores to the dataset item and named run. The record preserves model pins, dates, prompts, rubric version and case hashes. We discarded the first full run after it lost about 3% of logging traces and analysed the complete rerun on the same case files.
The harness ties together three jobs: present the same service decision to each API, preserve the returned answer, then apply the fixed scoring rules. The adapter handles the provider's request and response format. It does not choose a different service rule for a model. That separation makes a wrong route, an invalid probability sum and an unsupported image visible as different events in the record.
Tracing scores back to calls
We linked the scores back to the call that produced them. A reviewer can move from a result to its named run, scenario, prompt, model configuration and answer. That is the working evidence behind a summary percentage. It also gives the delivery team a concrete failure to inspect: the message that reached the wrong queue, the probability that crossed the handoff line or the answer that failed the required shape.
Building the answer key
We designed and governed the scenario pipeline using Claude for the 112 main cases and GPT-5.5 for the 26 extra cases. Because the main set shares Claude Haiku 5.5's model family, we report the sets separately and build the answer key with other models.
GPT-5.5, Nova Pro and Nova 2 Lite each labelled the 112 main items three times. Each model's most frequent answer contributed to the majority route, handoff and critical flag. These three labelling models come from two vendors and are outside the tested lineup. GPT-5.5 shares OpenAI Decisions' vendor; the two Nova models share theirs.
The answer key is produced separately from the tested configurations. Label runs used an Azure OpenAI Global Standard deployment in Australia East, and Amazon Bedrock in Sydney with cross-Region inference (Asia Pacific for Nova Pro, global for Nova 2 Lite), so labelling could be processed outside Australia.
Only the two Nova models voted on the extra items because GPT-5.5 wrote them. Defined disagreements went to a Humint Labs team member for human review. The reviewer also considered main items with no route or handoff majority and three fixed borderline items, rather than labelling the whole set.
The review sheet lists 14 items, including four excluded extra items that received no human review; one reviewed item's proposed consensus value changed. There were four no-majority cases and seven extra Nova disagreements.
- Stage 1ScenariosHumint Labs designs synthetic scenarios using two model families and fixed rules. A person checked the answer key for the original 46 scenarios.
- Stage 2Model labelsGPT-5.5, Nova Pro and Nova 2 Lite each answer eligible scoring questions three times. Extra scenarios use Nova votes only.
- Stage 3Reviewer decisionA Humint Labs reviewer resolves disagreements using the 25 October rule. Four of 26 extra scenarios have no final label: they were run but excluded from scoring.
- Stage 4Repeated testsTen model configurations each run five times on 184 scenarios: 46 original, 112 new and 26 extra. Jev 1.13.0 and Jev 1.14.0 use the same cases.
- Stage 5Call recordsThe first full run was discarded after the logging service dropped about 3% of records. A clean rerun on the same cases supplies four separate results groups.
Read the diagram as text
| Stage | Action |
|---|---|
| Scenarios | Humint Labs designs synthetic scenarios using two model families and fixed rules. A person checked the answer key for the original 46 scenarios. |
| Model labels | GPT-5.5, Nova Pro and Nova 2 Lite each answer eligible scoring questions three times. Extra scenarios use Nova votes only. |
| Reviewer decision | A Humint Labs reviewer resolves disagreements using the 25 October rule. Four of 26 extra scenarios have no final label: they were run but excluded from scoring. |
| Repeated tests | Ten model configurations each run five times on 184 scenarios: 46 original, 112 new and 26 extra. Jev 1.13.0 and Jev 1.14.0 use the same cases. |
| Call records | The first full run was discarded after the logging service dropped about 3% of records. A clean rerun on the same cases supplies four separate results groups. |
Basis: Majority labels from repeated model votes, with limited human review. Urgency uses one final value; the stored acceptable range is not scored. Four extra items without route or handoff consensus were excluded.
- 83.3%
Route and handoff agreement, each 115 of 138
- 60.1%
Urgency agreement, 83 of 138
- 0.86 · 0.78 · 0.78
Kappa values for route, handoff and urgency
We checked label consistency before scoring. Route and handoff agreement were each 115 of 138, or 83.3%; urgency was 83 of 138, or 60.1%. Their kappa values were 0.86, 0.78 and 0.78. Final-versus-draft matches were 22 of 22, 22 of 22 and 18 of 22 for extra route, handoff and urgency; 12 of 15, 15 of 15 and 10 of 15 for images; 81 of 97, 94 of 97 and 74 of 97 for new text.
Scoring rules
Route
The main score accepts the final route and allowed alternatives chosen by another labeller. Strict top-choice accuracy uses the single preferred route. Calibration-only cases have no route accuracy result.
Person needed
The yes or no answer must match. Recall counts required handoffs caught; false-handoff rate counts needless handoffs among answers requiring none. Critical recall isolates the most serious labelled cases.
Urgency
Exact match compares the rounded level with the final expected value. Within half a level uses the continuous score and keeps continuity with Part 2. Within one level is looser. Mean absolute error measures average distance. The stored acceptable range is not scored. Where the Nova labellers split, either final alternative counts only for the within-one measure.
Answer shape
Failed, blocked and malformed scored answers count as wrong for every configuration. Unsupported input is separate and excluded from accuracy. Claude's higher and low effort settings produced 29 and 42 malformed probability-sum outputs across all attempts. A secondary check renormalises sums from 1.02 to 1.10; it does not replace the strict headline rule.
Confidence
Calibration compares stated probabilities with observed matches. It does not establish a safe live cut-off.
This strict treatment of scored failures differs from Part 2's earlier answer counts, which separated schema failure. Fix the rule before comparison rather than choosing it after seeing the scores.
Results in detail
Route and urgency each use 89 eligible text scenarios, five runs each, or 445 answers. Handoff uses all 97 text scenarios, or 485. Eight deliberately ambiguous items contribute calibration rather than route or urgency accuracy.
- 96.6% to 96.6%
Accepted routing, Jev 1.13.0 to 1.14.0, unchanged
- 95.9% to 93.2%
Handoff, Jev 1.13.0 to 1.14.0
- 58.4% to 64.5%
Exact urgency, Jev 1.13.0 to 1.14.0
Swipe or scroll the table horizontally to see every column.
| Model | Accepted route, % | Preferred route, % | Macro-F1 |
|---|---|---|---|
| Jev 1.13.0 | 96.6% | 89.2% | 0.913 |
| Jev 1.14.0 | 96.6% | 92.1% | 0.923 |
| Microsoft-Decision-1 | 93.9% | 81.8% | 0.853 |
| OpenAI Decisions | 94.4% | 83.1% | 0.873 |
| Perplexity Decider | 97.8% | 92.1% | 0.925 |
| Clef 27B | 94.4% | 85.4% | 0.812 |
| Clef-flash | 92.1% | 80.9% | 0.865 |
| Clef-omni | 91.0% | 84.3% | 0.830 |
| Claude Haiku 5.5 (higher effort) | 93.5% | 89.4% | 0.889 |
| Claude Haiku 5.5 (low effort) | 89.9% | 85.4% | 0.890 |
Basis: Accepted route is the main score; preferred route is the single answer. Macro-F1 gives each route equal weight.
Swipe or scroll the table horizontally to see every column.
| Model | Recall, % | False-positive rate, % | Critical recall, % |
|---|---|---|---|
| Jev 1.13.0 | 95.2% | 3.6% | 100.0% |
| Jev 1.14.0 | 95.7% | 8.7% | 100.0% |
| Microsoft-Decision-1 | 92.9% | 5.5% | 96.7% |
| OpenAI Decisions | 90.5% | 1.8% | 96.7% |
| Perplexity Decider | 95.2% | 0.0% | 100.0% |
| Clef 27B | 95.2% | 3.6% | 100.0% |
| Clef-flash | 95.2% | 9.1% | 100.0% |
| Clef-omni | 97.6% | 21.8% | 100.0% |
| Claude Haiku 5.5 (higher effort) | 88.1% | 0.0% | 94.0% |
| Claude Haiku 5.5 (low effort) | 82.4% | 0.0% | 87.3% |
Basis: Recall uses required handoffs; false-positive rate uses answers needing none. These differ from exact yes/no match in Part 3's chart. Jev 1.13.0 caught all 150 critical answers and raised 10 unneeded handoffs among 275 answers needing none.
Swipe or scroll the table horizontally to see every column.
| Model | Exact, % | Within one level, % | Within half a level, % | Mean absolute error |
|---|---|---|---|---|
| Jev 1.13.0 | 58.4% | 100.0% | 59.1% | 0.39 |
| Jev 1.14.0 | 64.5% | 100.0% | 65.4% | 0.39 |
| Microsoft-Decision-1 | 67.9% | 99.6% | 67.9% | 0.37 |
| OpenAI Decisions | 71.9% | 100.0% | 71.9% | 0.36 |
| Perplexity Decider | 71.9% | 100.0% | 71.9% | 0.30 |
| Clef 27B | 67.4% | 100.0% | 67.4% | 0.38 |
| Clef-flash | 55.1% | 98.9% | 55.1% | 0.47 |
| Clef-omni | 62.9% | 98.9% | 62.9% | 0.41 |
| Claude Haiku 5.5 (higher effort) | 73.9% | 96.2% | 74.4% | 0.37 |
| Claude Haiku 5.5 (low effort) | 67.2% | 94.8% | 67.9% | 0.41 |
Basis: Within half uses distance from the final value; exact uses the rounded level. Jev 1.13.0 had 263 of 445 within half, versus 260 of 445 exact and all 445 within one.
Moving from Jev 1.13.0 to 1.14.0 left accepted routing unchanged at 96.6%. Handoff moved from 95.9% to 93.2%, while exact urgency rose from 58.4% to 64.5%. Both versions belong on a text shortlist; test the version against the review and priority boundaries your service uses.
The original-case and extra-set results remain separate; neither enters the headline comparison.
Read results for the original 46 scenarios
Swipe or scroll the table horizontally to see every column.
| Model | Routing top choice, % | Handoff recall, % | Urgency within half, % |
|---|---|---|---|
| Jev 1.13.0 | 95.5% | 100.0% | 83.3% |
| Jev 1.14.0 | 95.5% | 100.0% | 78.3% |
| Microsoft-Decision-1 | 90.9% | 80.0% | 76.7% |
| OpenAI Decisions | 100.0% | 93.3% | 91.7% |
| Perplexity Decider | 95.5% | 86.7% | 66.7% |
| Clef 27B | 95.5% | 80.0% | 75.0% |
| Clef-flash | 90.9% | 80.0% | 41.7% |
| Clef-omni | 86.4% | 93.3% | 41.7% |
| Claude Haiku 5.5 (higher effort) | 100.0% | 77.3% | 55.0% |
| Claude Haiku 5.5 (low effort) | 100.0% | 73.3% | 43.3% |
Basis: Eligible original questions retain their earlier wording and human-checked labels. These descriptive rates are separate from new text and image results. Jev 1.13.0's 83.3% within half a level here differs from 59.1% on new text.
The extra-set table covers the 22 scored items, with the four exclusions outside its accuracy denominator.
Read results for the separately generated extra set
Swipe or scroll the table horizontally to see every column.
| Model | Acceptable route, % | Handoff recall, % | Exact urgency, % |
|---|---|---|---|
| Jev 1.13.0 | 100.0% | 100.0% | 77.3% |
| Jev 1.14.0 | 100.0% | 100.0% | 72.7% |
| Microsoft-Decision-1 | 100.0% | 100.0% | 87.3% |
| OpenAI Decisions | 100.0% | 100.0% | 68.2% |
| Perplexity Decider | 100.0% | 100.0% | 72.7% |
| Clef 27B | 100.0% | 100.0% | 77.3% |
| Clef-flash | 100.0% | 100.0% | 68.2% |
| Clef-omni | 95.5% | 100.0% | 72.7% |
| Claude Haiku 5.5 (higher effort) | 93.6% | 100.0% | 66.4% |
| Claude Haiku 5.5 (low effort) | 90.0% | 93.3% | 65.5% |
Basis: Acceptable route, handoff recall and exact urgency use final labels for 22 items. GPT-5.5 wrote the 26 scenarios but did not vote on them; the two Nova models supplied final labels. Four items without route or handoff consensus were excluded. Seven decision-model configurations reached 100.0% routing, and Clef-omni reached 95.5%.
Repeat behaviour and manipulation
- 6.1% vs 2.6%
Attack success, Jev 1.13.0 against Claude Haiku 5.5 (higher effort)
- 2.1% vs 17.9%
Text flip rate, Jev 1.13.0 against Claude Haiku 5.5 (higher effort)
Nine new text pairs contain a clean request and a manipulated version. Attack success counts a pair only when the clean answer is right and its manipulated twin is wrong. Jev 1.13.0's rate was 6.1%, against 2.6% for Claude Haiku 5.5 (higher effort). Use these paired cases to test each candidate against the manipulations your workflow must resist.
A flip is a changed answer across repeated runs; an invalid answer also counts. Recorded text flip rates were 2.1% for Jev 1.13.0 and 17.9% for Claude Haiku 5.5 (higher effort). Check whether a change alters the action, not only the number.
Expected calibration error compares confidence with strict top-choice correctness. Jev 1.13.0's route value was 0.048, against 0.158 for Claude Haiku 5.5 (higher effort). Enough local examples in each score range are needed before choosing a live cut-off. The full table below preserves each model's manipulation and confidence measures.
Read manipulation and confidence results
Swipe or scroll the table horizontally to see every column.
| Model | Attack success, % | Route ECE | Handoff ECE |
|---|---|---|---|
| Jev 1.13.0 | 6.1% | 0.048 | 0.096 |
| Jev 1.14.0 | 10.6% | 0.040 | 0.095 |
| Microsoft-Decision-1 | 2.6% | 0.133 | 0.027 |
| OpenAI Decisions | 0.0% | 0.040 | 0.047 |
| Perplexity Decider | 0.0% | 0.035 | 0.071 |
| Clef 27B | 0.0% | 0.047 | 0.109 |
| Clef-flash | 13.6% | 0.085 | 0.114 |
| Clef-omni | 13.0% | 0.077 | 0.090 |
| Claude Haiku 5.5 (higher effort) | 2.6% | 0.158 | 0.081 |
| Claude Haiku 5.5 (low effort) | 2.9% | 0.164 | 0.088 |
Basis: Attack success uses only pairs whose clean answer was correct. Expected calibration error (ECE) compares confidence with strict top-choice matches for routing and with final labels for handoff. Lower values indicate closer agreement between stated confidence and observed matches; set the operating cut-off using your service cases.
Response times and cost
Swipe or scroll the table horizontally to see every column.
| Model | Latency p50 / p95, ms | Cost per 1,000 calls | Cost per 1,000 correct questions | Flip rate |
|---|---|---|---|---|
| Jev 1.13.0 | 289 / 370 | US$0.047 | US$0.020 | 2.1% |
| Jev 1.14.0 | Not separately timed | US$0.047 | US$0.019 | 2.7% |
| Microsoft-Decision-1 | 947 / 1,084 | US$0.030 | US$0.012 | 2.1% |
| OpenAI Decisions | 216 / 322 | US$0.102 | US$0.041 | 0.0% |
| Perplexity Decider | 718 / 1,007 | US$0.016 | US$0.006 | 0.0% |
| Clef 27B | 561 / 1,243 | US$0.246 | US$0.101 | 0.0% |
| Clef-flash | 483 / 635 | US$0.039 | US$0.017 | 0.0% |
| Clef-omni | 375 / 677 | US$0.153 | US$0.067 | 0.0% |
| Claude Haiku 5.5 (higher effort) | 2,762 / 3,763 | US$0.441 | US$0.180 | 17.9% |
| Claude Haiku 5.5 (low effort) | 1,887 / 2,524 | US$0.340 | US$0.146 | 22.0% |
Basis: Text timings use one repeat of 68 scenarios on 15 October 2026, with successful calls only: 66 for Claude higher effort and 64 for low effort. Jev 1.14.0 was not separately timed. Accuracy, flip rates and Langfuse call costs use the five-repeat study on 25 October. One call can answer several questions; correct questions supply the second cost denominator. p95 means 95 in 100 calls were this fast or faster.
We timed every call as a full round trip from Australia, from sending the request to receiving the answer, and logged each one in Langfuse. The response times come from our 15 October 2026 run over the same scenarios and test harness, and the accuracy and cost figures from the full five-repeat study on 25 October.
The 25 October run had a network delay that added about 0.6 seconds per call, so we use the clean 15 October timings. That timing run also covered the original 46 scenarios and ten image scenarios, once each. Jev 1.13.0 took 280 ms at the median and 379 ms at p95 on the original 46, and 376 ms and 433 ms on images.
OpenAI Decisions recorded the fastest text p95 at 322 ms, followed by Jev 1.13.0 at 370 ms; both are strong choices.
The cost difference follows the request size. Part 1 averaged about 412 input tokens, against 1,107 for three-question new text calls, at the same US$0.042 per million input tokens. The later original-case run cost US$0.0167 per 1,000 calls, in line with Part 1's roughly US$0.017. Longer prompts explain the new US$0.047 rate. Output is free for Jev.
For the general-purpose comparison we used Claude Haiku 5.5, because the GPT and Nova models from Parts 1 and 2 were building our answer key this time, and a model shouldn't mark its own work.
Part 4 applies these rates to stated call mixes, drafting shares and a 5% retry allowance. It models costs for each design using those measured rates and declared assumptions.
Self-hosting is a separate calculation. The AWS Price List API listed Sydney SageMaker real-time ml.g6.xlarge at US$1.465 per hour on 25 October 2026, SKU B7T8N72WK4MJU7T5. That is US$35.16 per 24 hours while provisioned, including idle time, excluding storage and other charges.
Strands Decider's repository reports about 115 ms median and 299 ms p95 per JevBench question for an earlier v19 checkpoint on an RTX 3090 under WSL2. This vendor result is not a Humint Labs Sydney measurement or a benchmark of the current checkpoint.
Its model card lists Apache-2.0; base-model terms also need checking. No calls-per-dollar figure follows without throughput and utilisation.
Images, audio and video
Image results stand alone. Each Jev version supported 13 image items; two oversized inputs were unsupported. Microsoft-Decision-1 supported none of the 15. Unsupported inputs are excluded from accuracy, rather than counted wrong. TypeSafe says opt-in image input is not yet ready for production.
Swipe or scroll the table horizontally to see every column.
| Model | Image route top choice, % | Image handoff recall, % | Image urgency within half, % | Unsupported items |
|---|---|---|---|---|
| Jev 1.13.0 | 92.3% | 66.7% | 53.8% | 2 |
| Jev 1.14.0 | 92.3% | 73.3% | 53.8% | 2 |
| Microsoft-Decision-1 | Unsupported | Unsupported | Unsupported | 15 |
| OpenAI Decisions | 86.7% | 75.0% | 73.3% | 0 |
| Perplexity Decider | 93.3% | 100.0% | 60.0% | 0 |
| Clef 27B | 93.3% | 75.0% | 66.7% | 0 |
| Clef-flash | 86.7% | 25.0% | 53.3% | 0 |
| Clef-omni | 86.7% | 100.0% | 73.3% | 0 |
| Claude Haiku 5.5 (higher effort) | 92.0% | 50.0% | 68.0% | 0 |
| Claude Haiku 5.5 (low effort) | 93.3% | 50.0% | 66.7% | 0 |
Basis: Five runs per supported image case. Clef-omni caught every required handoff, but matched only 30 of 75 yes/no answers and raised 45 unneeded handoffs among 55 needing none. Recall alone hides that review burden.
Only Clef-omni was in scope for audio and video capability. Neither format has an independent benchmark here. A model accepting a format does not establish its decision quality on it.
Deployment controls
The benchmark establishes a routing and handoff shortlist. A delivery team then connects it to service rules, application permissions and review queues. Test rare and costly classes separately, keep a person responsible for consequential approvals, and log corrections alongside the model's original answer.
Measure speech recognition, transfer, resolution, review work and customer outcomes in the contact centre or digital service trial.
Confirm the deployment and contract before sending service data.
- Microsoft lists Australia East for Data Zone Standard deployments, which may process data anywhere in its Asia Pacific zone; this test used West US 3.
- OpenAI documents Decisions processing in the US and EEA/Switzerland.
- Cloudflare's Clef pages name no processing region.
- Perplexity's API FAQ says its compute is hosted in North America; its Decider pages name no region.
Check retention, access, availability guarantees and current prices as part of the implementation.
We scoped the benchmark to hosted APIs with the common task and access requirements. Strands Decider is a self-hosted option costed separately above. Solar Decide, Liquid d1, Together variants, Laya, GLiNER2.5-Decide, Span-01, Mercury Decide, GLiDE and Celeris sat outside that scope. Laya, Kev and Bespoke Nimble, named in Part 1, remain untested here.
Statistics note: We resample scenarios with their five repeats together, pair model versions on the same scenarios and adjust the text comparisons for multiple tests. Those checks do not establish a single accuracy winner; the full analysis retains the uncertainty ranges.
How to design a comparable test
Fix the inputs
Fix the cases, expected answers and scoring constants before running. Preserve the complete question text and all route definitions. Include close boundaries, rare classes, false alarms and clean/manipulated pairs. Use separate results for differing inputs and answer-key methods.
Build in the action
Build the expected action into the test design. For a route, name the queue and the fallback when no queue fits. For a handoff, name the person or team receiving it and what evidence they need. For urgency, identify the score boundary that changes priority. A model can return an acceptable answer while the application takes the wrong action; exercising that boundary is part of delivery.
Keep it as a change check
Use the retained cases as a change check. A model upgrade, revised route definition or new review rule should be tested against the same inputs before it enters the service. Inspect changed answers beside the original messages, then add the new service cases that motivated the change. This turns the evaluation into an operating asset that the service team can maintain, with an inspectable reason for each decision about rollout.
Pin, repeat and retain
Pin model versions and adapters, repeat each case five times, and attach every score to its trace, case and run. Retain failed and blocked calls. Check completeness before analysis so a clean-looking subset cannot replace the planned run. Measure the answer and the resulting action separately, then time the complete workflow.
Next steps for your own test
The canonical export SHA-256 is 9afa683c952153cbea30f072f33b11553b5c06b0100fe6ca543cca75cd4aadb7. The dataset item-hash SHA-256 is f45f74a9ed9511388f1c35865777030d7a1660a3b73c4f268393c0371f32d279. Preserve report, analysis, paired comparisons, manifest, prompts, rubric version, source dates and model pins together.
Choose one service decision and fix its cases, expected answers and action rules before the first call. Contact Humint Labs to build the harness and run a comparable test.