Direct answer
How should an enterprise route work between AI models and handle fallback?
Route requests by the workload characteristics that matter, such as consequence, latency, cost and required capability. Treat fallback as a separate continuity control, and exercise it against the same acceptance criteria before a real incident forces the change.
Scope: This article covers routing and fallback design for a known workload portfolio. It does not replace service-specific capacity, security or recovery planning.
The serving path, the fallback path, and the checks that sit off both
The system splits cleanly into two paths that answer different questions, plus a third band of checks that keeps both honest.
The serving path starts with a single request, one unit of work, arriving at a router. The router decides in order: first by the call site, then by attributes already held about the request, and only as a last resort by a classifier. From there the request fans out to one of three tiers: a fast tier for classification, extraction and routing, where the output is checkable; a general tier carrying drafting, reasoning and most of the traffic; and a high-stakes tier reserved for work that is costly to get wrong or genuinely hard. All three tiers converge on one validator, which checks schema, totals and downstream state, before the reply goes out.
The fallback path is a different question entirely. A primary becomes unavailable through a timeout, throttling, a region failure or a quota limit. The first move is always the same tier somewhere else, another region or another provider, never a weaker model. Where that is genuinely not possible, the behaviour degrades instead: the task is shortened, the turn is handed to a person, or the system declines. Every one of those events raises an alert and a labelled record, never just a line in a log.
Sitting off both paths, inside a boundary of its own, is the set of checks that keeps either one safe to change: a substitution gate requiring a harness run before any model enters or leaves the estate; a comparability rule that pins the run manifest except for the model identifier; a release gate set on per-class failure floors rather than an aggregate score; and a lifecycle watch queried on a schedule rather than awaited by email.
What decides which model handles which work?
Not size, and not benchmark rank. The property that sorts work reliably is how much the cost of being wrong varies across it, and three questions get you there faster than any leaderboard.
What does a wrong answer cost here? An internal summary that a person reads and corrects before acting costs close to nothing when it is poor. A letter that goes to a customer about a decision on their money costs a great deal, and the cost is not evenly distributed either, because the expensive failures cluster in the awkward cases rather than the ordinary ones.
Is there a person on this turn? A great deal follows from this alone. Work with a reviewer in the loop tolerates a cheaper model and a higher error rate, because the error is caught. Work that reaches a customer unmediated does not, and no amount of aggregate accuracy changes that.
How long can it take? A turn inside a live conversation has a budget measured in low single-digit seconds and the customer experiences the tail, not the average. An overnight enrichment job has a budget measured in hours. Those are different constraints and they should not resolve to the same model by default.
Answer those three across your workloads and the tiers draw themselves. The shape most estates settle into is a fast tier for classification, extraction, routing and structured transformation, where the output is checkable by something other than a human reading it; a general tier carrying most of the drafting and reasoning traffic; and a high-stakes tier held back for work where being wrong is expensive or the task is genuinely hard. There is a fourth tier that is not on the serving path at all, which is the judge used for evaluation, and it belongs in the portfolio precisely because forgetting it is a common and painful mistake.
The size of the prize here is public, and it is wider than most teams assume. On the Amazon Bedrock on-demand pricing page on 28 July 2026, published on-demand rates for text models spanned more than two orders of magnitude, from around US$0.04 per million input tokens at the smallest models to several dollars per million at the frontier ones, with output tokens priced higher again and rates varying by region. Exact endpoints are deliberately not quoted. The page lists dozens of models in a mix of per-thousand and per-million units, changes often, and any pair of numbers pinned here would be wrong within weeks; read the page rather than this sentence if you need a rate. That spread sits across models behind a single API. The durable observation is the shape of the range: the gap between the bottom and the top is not a margin, it is a multiple, and a workload that routes everything to the top of it pays that multiple on every request that never needed it.
How much of a real workload can sit at the bottom of that range is a question a portfolio answers by measurement rather than assertion. In the systems we have built this way, the fast tier carries the largest share of production traffic, with the remainder split between the general and high-stakes tiers.
The cheapest work to move down a tier is the work whose output is verifiable. If a schema validates, a total reconciles, a date parses, or a classification can be checked against a downstream system that already knows the answer, then a small model plus a validator usually beats a large model plus trust, and it beats it on cost, latency and auditability at the same time. The tiering question and the verification question are the same question asked twice.
The escalation pattern that follows from this, running a cheap model first and promoting the request to a stronger one when a check fails, is not new and it is worth naming its provenance rather than presenting it as insight. It was set out as an LLM cascade in the FrugalGPT paper by Chen, Zaharia and Zou in May 2023. Two honest cautions come with it. The escalation signal is the hard part, and a model's self-reported confidence is a poor one, so cascades work best where the check is external to the model rather than asked of it. And a cascade adds latency to exactly the requests that were already the hardest, which is the wrong place to add it if there is a customer waiting.
How should the routing rule decide?
The router is a classifier, and it is the component teams reliably under-build. There are three designs, and the useful advice is mostly about not skipping to the third.
Static routing by call site is the first, it is boring, and it is correct far more often than it gets credit for. The code path already knows whether it is summarising an internal note or drafting an outbound letter. Route on that. No inference, no added latency, no new failure mode, and it is trivially explainable to anyone who asks why a given request went where it went.
Attribute routing is the second. Route on properties you already hold at the point of the request: the channel, the customer segment, whether a person is on the turn, the length of the input, whether the request touches a regulated topic. Still deterministic, still explainable, still testable without a model in the loop.
Model-based routing is the third, where a small model reads the request and picks the tier. It is the most expressive option and it is the only one of the three that introduces a genuinely new silent failure: the router itself can now be wrong in a way that produces a fluent, plausible answer from the wrong tier, with no error raised anywhere in the system. Build it when the first two have run out, not before, and never without the tests below.
Whichever design you use, the routing decision needs a test set and a record. The test is the same shape as a tool-selection test: a held-out set of real requests, each labelled with the tier it should have taken, replayed against the router, with the tier it actually took compared against the tier it owed. The record is per request, and it carries which model served it and which rule sent it there.
The failure this prevents is the silent downgrade, and it is worth being specific about why aggregate metrics cannot catch it. Suppose the router sends a small share of high-stakes work to the fast tier. Your overall quality number barely moves, because the volume is dominated by ordinary traffic that the fast tier handles perfectly well. Your cost number looks excellent. And the entire harm is concentrated in the requests you cared most about. An aggregate metric does not fail to detect that by accident; it is arithmetically incapable of detecting it. Measure quality per tier and per route, or do not claim to be measuring it.
Fallback is not the same decision as routing
Routing and fallback get written in the same file and they answer different questions. Routing asks which model should do this work. Fallback asks what happens when the model that should do this work is not available. Collapsing the two produces a specific and common failure.
The failure is the silent quality fallback. The primary tier times out or gets throttled, the system quietly serves the request from a cheaper tier instead, the request succeeds, nobody is paged, and for the rest of the afternoon a proportion of customer-facing output is produced by a model that was never approved for that work. Every dashboard reports a good day. The incident, if it is ever noticed, is noticed by a customer.
So the first rule is to fall back across availability rather than across quality. Fall back to the same tier in another region, or to an equivalent model on another provider, before you fall back to a weaker model. Where that genuinely is not possible, degrade the behaviour rather than the model: shorten the task to something the available tier can do safely, hand the turn to a person, or decline. Declining is a legitimate design output and it is badly under-used. The same argument applies here as applies to an empty retrieval result, where an answer confidently assembled from nothing is worse than no answer at all.
The second rule is that fallback has to be loud. Every fallback event is an alert and a labelled record, not a line in a log that nobody queries. If your fallback rate can climb from nothing to a fifth of traffic without anyone finding out, you do not have a fallback. You have an undeclared second production system with no approval attached to it.
The third rule is about capacity, and it is the one that gets skipped. Check what your fallback actually shares with your primary. A secondary model on the same account, in the same region, drawing on the same token throughput quota is not a fallback from a throttling event, because the throttling event is the shared quota. The binding constraint on inference is almost never compute you own; it is the endpoint and its quota.