Direct answer
What evidence is needed before substituting an AI model in production?
Test the candidate against the workloads, prompts, tools and failure conditions that define the service, then compare it with the current model using declared acceptance thresholds. A general benchmark cannot establish that a substitution is safe for a particular enterprise workload.
Scope: This article covers substitution evidence and release decisions. It does not define a universal benchmark or acceptance threshold.
How do you know a model substitution is safe?
This is where portfolios fail most often, and they fail the same way each time. Somebody proposes a swap. The case for it is a benchmark table and a price per million tokens. The discussion becomes an argument about whether the new model is better in general, which is both unanswerable and irrelevant. The only question that matters is whether it is at least as good at your work, and you cannot answer that from anyone else's evaluation.
It is worth being precise about why the benchmark table is weak evidence for this decision, because "benchmarks are not everything" is a platitude and the actual reasons are specific and documented.
The first is saturation. A benchmark only discriminates while the candidates are spread across its range, and the headline reasoning and knowledge benchmarks have spent years being climbed. When the models you are choosing between all sit near the ceiling, a ranking that separates them by a point or two is not telling you they differ by a point or two. It is telling you the instrument has run out of resolution and you are reading noise as signal.
The second is contamination. Public benchmarks are public, which means their contents are reachable by exactly the crawls that assemble training corpora. This is not a suspicion, it is the stated motivation for a body of work building contamination-resistant alternatives: the MMLU-CF benchmark, published by Zhao and colleagues in December 2024, was constructed because, as its abstract puts it, "the open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to benchmark contamination, resulting in unreliable evaluation results". Their decontaminated construction reports materially lower scores than the original benchmark for the same models.
The third is construct validity, and it is the deepest of the three. Fodor's Line Goes Up?, published in February 2025, argues that inherent limitations of the benchmarking paradigm make benchmark performance unsuitable as a measure of generalisable competence across cognitive tasks. You do not have to accept the strongest form of that argument to act on the operational version of it, which is that a benchmark measures performance on the benchmark, and the distance between that and your work is the one quantity nobody outside your organisation can estimate for you.
None of this makes public benchmarks useless. They are a reasonable screen for building a shortlist, and a genuinely useful signal when a model is far off the pace. They are simply not a substitution decision, and treating a leaderboard position as one is how a swap gets approved on evidence that was never about your workload.
Answering it needs a harness, and the property the harness has to give you is comparability. Two runs are comparable when their manifests differ in exactly one field and in no other way. For a model swap, the one field that moved is the model identifier, and everything else has to be pinned: the corpus version, the prompt set, the retrieval index build, the tool schemas and the judge. If two things moved, the comparison cannot settle the argument, and it is better to know that before the meeting than during it.
There is a trap specific to model swaps that does not arise elsewhere, and it is about prompts. A production prompt is not a neutral instruction. It is accumulated compensation for one particular model's habits, built up over months of small corrections. Swap the model and keep the prompt, and what you have measured is how the old prompt performs on the new model. That is a genuinely useful measurement, because it tells you whether you could drop the new model in tomorrow. It is simply not the same measurement as whether the new model is better at the task once it has been tuned to. Run both. They answer different questions and teams routinely conflate them.
The gap between those two runs is usually larger than teams expect. On our own harness, a drop-in swap that keeps the incumbent prompt fails at least one per-class floor in a meaningful proportion of attempts, against a materially lower rate once the prompt has been revised for the new model. Budget for the revision rather than hoping for the drop-in.
Gate the result on per-class floors rather than on an aggregate score, for the same reason the routing metric has to be per route. An aggregate can be paid off: a model that became noticeably safer and slightly less helpful reads as a regression, and a model that became much more helpful and quietly less safe reads as an improvement. Decide in advance which classes of failure block a release outright and which report without blocking.
Then there is the set of second-order differences that no public benchmark will show you and that only surface in your own harness. Refusal behaviour, and where the new model draws its safety boundary, which can be tighter or looser than the one your acceptance tests were written against. Instruction-following on structured output, meaning how often you get schema-conformant JSON without a repair step. Tool-call formatting and its reliability under load. Tokenisation, which changes how many tokens your actual corpus costs and therefore moves both your bill and your context budget independently of the advertised rate. And the latency distribution rather than its mean, because a customer waiting on a turn experiences the tail.