LLM Routing Isn't One Thing. It's Three.

The first time an LLM feature goes properly into production, two things tend to happen within a month. The bill looks wrong, not just higher than expected but wrong: a share of requests that could have been cheap ends up hitting the most expensive model, because that was the model used in the prototype and nobody revisited it. And reliability becomes random: a provider has a bad hour, elevated errors or a regional issue, and the application inherits it directly, leaving on-call editing configuration under pressure or redeploying just to swap a model string.

Routing exists to prevent both problems, but it's often treated like a single toggle, and that's the trap. In practice there are three separate decisions happening at different times for different reasons: which model should answer a request, which provider should serve that model right now, and what happens when the first plan fails. A setup that collapses those three into one blob tends to look reasonable in configuration and still fail in exactly the ways routing was supposed to prevent.

TL;DR

• Treat routing as three knobs: model selection, provider selection, and fallbacks. Shipping only one or two is how pager fatigue happens.
• Use fastrouter/auto when a codebase has defaulted into sending everything to the most capable model because nobody wants to maintain an if/else model matrix.
• Use provider routing when the same model is available from multiple providers and the application cares about latency variance or rate limiting, not just average latency.
• Configure fallbacks per virtual key and test them. A fallback chain that has never been forced to activate is a diagram, not a safety net.

Why routing becomes mandatory once there's real traffic

Single-model, single-provider setups are fine until they aren't, and the breaking points are predictable.

No single model wins everywhere. Real workloads are rarely uniform: short classification, extraction, rewriting, harder multi-source synthesis, and tool-heavy flows all show up in the same backend. When everything routes to one frontier-tier model, the application pays frontier prices for boring work, which is what happens by default when the prototype's model choice becomes the production default. Automatic model selection is the structural fix: pushing the decision into routing configuration instead of relying on every engineer to remember to pick the cheaper model every time.

"Model" and "provider" have also stopped being the same thing. In open-weight ecosystems, the same weights are frequently hosted by more than one provider, and the model name staying constant doesn't mean the experience does: tail latency, rate-limit behavior, transient error frequency, and streaming stability can all differ by provider for an identical model. That's why provider routing exists as its own layer, separate from picking a model.

Providers also fail independently of model quality. An outage says nothing about whether a model is good, but an application that hardcodes a single provider goes down anyway when that provider does. That's what makes fallbacks necessary rather than a nice-to-have.

The mental model: three layers that compose

Routing works when these are treated as separate, composable layers: model selection decides what should answer, provider routing decides who runs it, and fallbacks decide what happens when reality disagrees with the plan. Each layer has its own failure modes and tradeoffs, and treating routing as one decision is how misconfiguration happens.

Automatic model selection: fastrouter/auto

Automatic model selection fits teams that have grown past one endpoint mapped to one model and now have mixed complexity flowing through the same service. The common trajectory: an endpoint starts as “summarize this text,” then adds extraction, then adds chunking for long inputs, then becomes a pseudo-agent that summarizes, extracts, classifies, and maybe calls tools. At that point the endpoint needs different models depending on the request, and most teams don’t want that branching logic hardcoded in application code or a redeploy required every time the routing policy changes. That’s what fastrouter/auto is for: a request sent to it is scored on complexity, domain, and cost, and handed to whichever underlying model fits that profile, rather than to a single model chosen ahead of time. 

1curl https://api.fastrouter.ai/api/v1/chat/completions \
2  -H "Authorization: Bearer $FASTROUTER_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "model": "fastrouter/auto",
6    "messages": [
7      { "role": "user", "content": "Summarize this contract clause." }
8    ]
9  }'
Copy

The tradeoff is less determinism. Auto-selection trades explicit control for a runtime policy decision, which is a good fit for broad, heterogeneous endpoints and a poor fit when prompts are tuned to a specific model's quirks, when downstream code expects a very stable output style, or when debugging needs to answer "which model actually answered this request" and logging isn't set up to say. Treating fastrouter/auto as a sensible default rather than a rule for every endpoint tends to work best in practice: use it broadly for general-purpose work, pin explicit models for the smaller set of flows where behavior needs to stay stable, such as compliance gates or brittle parsers, and make sure telemetry logs the resolved model and provider so debugging doesn't rely on guesswork.

Auto-selection rarely fails loudly. It fails by drifting: a request distribution shifts (longer documents, a new product area, different user language), auto-selection starts favoring a different underlying model, and output quality changes slightly without anyone noticing until a downstream metric moves. Reviewing routing policy periodically, the same way any other production dependency gets reviewed, is what catches this before it compounds.

Provider routing: choosing who serves the model

Once a model is chosen, explicitly or through fastrouter/auto, a separate question remains for any model backed by more than one provider: which provider actually handles this request? FastRouter offers seven strategies for this: lowest price, lowest latency, highest throughput, weighted shuffle, priority order, category routing, and its AI Auto Router.

Which strategy is worth using depends on the workload's actual pain point. Provider routing earns its keep when the application is user-facing and tail latency matters more than average latency, when batch jobs hit rate limits at predictable times, when one provider has periodic brownouts (slow responses that eventually time out rather than fail cleanly), or when multiple providers host the same model with no reason to favor one over another. For low-QPS internal tooling it's still useful but less urgent; fallbacks are usually the bigger first win there.

A strategy is set through the provider object on a request. Sorting by price:

1curl https://api.fastrouter.ai/api/v1/chat/completions \
2  -H "Authorization: Bearer $FASTROUTER_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "model": "deepseek/deepseek-v4.1-flash",
6    "messages": [{ "role": "user", "content": "Extract key dates and obligations from this text." }],
7    "provider": { "sort": "price" }
8  }'
Copy

The OpenAI Python and TypeScript SDKs don't have a typed provider field, so FastRouter-specific parameters pass through the SDK's own extension point (extra_body in Python) rather than as a first-class argument.

Sorting by lowest price is a strong default for background and offline workloads, but can hurt tail latency on interactive endpoints if the cheapest available provider happens to be cheapest because it's overloaded or aggressively rate-limited; starting interactive traffic on lowest latency and batch traffic on lowest price is a reasonable split. Weighted shuffle helps prevent a single provider from absorbing all traffic and triggering rate-limit storms, but it complicates debugging, since the same logical request can land on a different upstream across retries, which makes clear per-request provider logging necessary rather than optional. Priority order is the easiest to reason about and the least adaptive, a reasonable starting point for teams that want explicit control before introducing dynamic behavior.

Provider routing doesn't protect against brownouts on its own if requests are allowed to hang indefinitely. A provider that's technically up but slow can leave a request waiting until the client times out, and a router can't reroute a request that never actually fails. Sane client timeouts, a retry policy that doesn't amplify rate limits, and fallbacks that trigger on timeouts as well as hard errors are what make provider routing actually protective.

Fallbacks: the part that gets skipped until the first incident

Model selection and provider routing both optimize the happy path. Fallbacks exist for the unhappy path: rate limits, provider outages, timeouts, or a model becoming unavailable on a specific provider. FastRouter supports fallback chains configured per virtual key, and that scoping matters, since a development key and a production key shouldn't necessarily share a policy, and neither should an internal batch job and a customer-facing chat endpoint.

Fallbacks aren't free reliability, because they can change output behavior, not just availability. The more disruptive failure isn't a fallback returning an error, it's a fallback returning a successful response in a subtly different shape. A common example in structured-extraction pipelines: a primary model reliably returns bare JSON, while a fallback model occasionally wraps the same JSON in markdown fences or adds a short preamble sentence. The call itself succeeds, but downstream parsing fails outright, or worse, parses successfully with fields quietly missing. For any pipeline where output format isn't negotiable, fallbacks should be chosen for behavioral compatibility, not just availability, validated against the same golden test set used for the primary model, and in some cases a hard failure is preferable to a fallback that returns something that only looks like valid output.

A fallback chain that has never been forced to activate isn't a fallback chain, it's a diagram. A simple drill closes that gap: configure the chain, intentionally break the primary path in staging (disable the primary provider or force a failure), run a representative batch of requests through it, and confirm that the fallback actually activates, that outputs still meet format constraints, and that telemetry correctly shows the resolved model and provider. Running that drill once in staging catches the "looked configured but didn't work" class of mistake well before it happens during a real incident.

What breaks when one layer is missing

• Only model selection: spend improves, but a single provider outage still takes the application down.
• Only provider routing: provider performance improves, but trivial requests still hit the most expensive model by default.
• Only fallbacks: incidents improve, but the happy path can still be needlessly expensive and tied to a suboptimal provider.
• All three together: cost is controlled by default, provider variance is handled deliberately, and outages turn into degraded behavior instead of hard downtime.

Routing is operational policy, not a one-time setting

Routing needs revisiting because the inputs to it keep changing: request shapes shift as prompts get longer, providers change behavior on rate limits and latency, and new endpoints get added that inherit the wrong policy by default. For any system where LLM calls sit on the critical path, a quarterly review of routing policy is not overkill.

What to do this week

• Inventory LLM calls and tag them by risk: interactive versus batch, strict structured output versus freeform text, latency-sensitive versus cost-sensitive, and must-not-fail versus can-degrade.
• Move one mixed-complexity endpoint to fastrouter/auto. Pick something representative but non-critical, log the resolved model and provider per request, and keep an escape hatch to pin a model if behavior regresses.
• Add provider routing where it matches the workload: lowest price for batch jobs, lowest latency for interactive endpoints, and weighted distribution instead of a single cheapest winner if rate limits show up.
• Configure a minimal fallback chain on the most critical virtual key, starting with one fallback that's behaviorally compatible with the primary's output constraints, and set sane timeouts so brownouts turn into retries or fallbacks rather than hanging requests.
• Run a failure drill in staging: force the primary provider to fail, verify fallback activation and output correctness, and add alerting on fallback activation rate so a degraded system doesn't run silently.

Getting started

Full configuration details for automatic model selection, all seven provider routing strategies, and fallback chains are documented at docs.fastrouter.ai/explore-features.