Best LLM Gateways in 2026: 10 Platforms Compared (Without the Brochure)

A team ships an LLM feature with one provider. It works. Then reality shows up: a provider has a partial outage and streaming stalls mid-response, someone "fixes it" with retries and the service ends up hammering itself, token spend jumps and nobody can answer which endpoint caused it, and security asks for RBAC, audit logs, and key scoping while the actual setup is a shared API key sitting in an environment file.

That's the moment "LLM gateway" becomes a serious search term, and the label is overloaded enough to cause real buying mistakes. Some gateways route traffic. Some only observe it. Some are extensions of infrastructure a team already runs. Comparing them by feature checklist alone is how teams end up buying the wrong thing and rebuilding it under incident pressure a few months later.

Ten platforms below cover the range teams actually run into: FastRouter, OpenRouter, Portkey, LiteLLM, Helicone, Langfuse, Requesty, TrueFoundry, Cloudflare AI Gateway, and Kong AI Gateway.

TL;DR

• Pick the failure that's tolerable. Routing gateways help during a provider or model incident; observability tools help explain what happened afterward. Most production stacks eventually need both.
• Pricing shape matters more than the sticker price. Flat platform fee, percentage of spend, and request-based SaaS behave very differently. Run the math at ten times current volume, not today's volume.
• Self-hosting is a trade, not a discount. A self-hosted proxy can be the right answer, but it turns the gateway into a tier-zero service someone has to operate.
• A team already standardized on Cloudflare or Kong often gets the best outcome by extending that platform rather than adopting a new vendor.
• Model swaps aren't free. Fallbacks can break JSON contracts, tool calling, and latency assumptions. A gateway routes; it doesn't normalize behavior between models on its own.

"LLM gateway" is three different products

Routing gateways sit in the hot path: they accept a request, pick a provider or model, and forward or transform the response. This is what a team wants when the goal is "fail over automatically when a provider is having a bad day." FastRouter, OpenRouter, Requesty, TrueFoundry, and LiteLLM fall into this category.

Observability-first platforms watch traffic rather than drive it: logging prompts and outputs, tracing agent steps, tracking cost and latency, running evaluations. They don't save an application during a provider outage, because they aren't in a position to reroute anything. Helicone and Langfuse are the two here. FastRouter also ships this same depth of observability, tracing, alerts, dynamic tags, natively, so the distinction here is really about who else also routes traffic, not who else also watches it. 

Infrastructure-native gateways extend a platform a team is already running rather than introducing a new standalone vendor. Cloudflare AI Gateway and Kong AI Gateway fit this category, built for teams already on Cloudflare's edge platform or Kong's API management stack respectively.

Portkey sits in a middle category of its own: routing bundled with heavier enterprise governance and workflow features. That can be exactly right for a larger organization, or it can be more product than a team needed if routing was the only real requirement.

The ten platforms, at a glance

Routing gateways: the ones that take the pager

FastRouter

FastRouter is a managed routing gateway with a single OpenAI-compatible endpoint across 160+ models. The pricing stance is the differentiator: a flat platform fee by tier (Starter free, Pro at $199/month, Business at $799/month, Enterprise custom) with zero markup on token costs at every tier.

What matters more than the model count is what ships without an enterprise negotiation. RBAC, project and key limits, input/output guardrails, and weekly cost Insights are included on every tier, not reserved for the top plan, which stands out against platforms that treat governance as a paid add-on. The tradeoff to weigh regardless: a routing gateway is still a third party sitting in the hot path, so it's worth validating directly how routing changes, timeouts, and failover behave under real conditions rather than taking any vendor's description of it at face value.

1curl https://api.fastrouter.ai/api/v1/chat/completions \
2  -H "Content-Type: application/json" \
3  -H "Authorization: Bearer $FASTROUTER_API_KEY" \
4  -d '{
5    "model": "deepseek/deepseek-v4.1-flash",
6    "messages": [
7      { "role": "user", "content": "Reply with the single word: ok" }
8    ]
9  }'
10from openai import OpenAI
11import os
12
13client = OpenAI(
14    api_key=os.environ["FASTROUTER_API_KEY"],
15    base_url="https://api.fastrouter.ai/api/v1",
16)
17
18resp = client.chat.completions.create(
19    model="deepseek/deepseek-v4.1-flash",
20    messages=[{"role": "user", "content": "Reply with the single word: ok"}],
21)
22
23print(resp.choices[0].message.content)
24import OpenAI from "openai";
25
26const client = new OpenAI({
27  apiKey: process.env.FASTROUTER_API_KEY,
28  baseURL: "https://api.fastrouter.ai/api/v1",
29});
30
31const resp = await client.chat.completions.create({
32  model: "deepseek/deepseek-v4.1-flash",
33  messages: [{ role: "user", content: "Reply with the single word: ok" }],
34});
35
36console.log(resp.choices[0].message.content);
Copy

Swapping providers or models is a change to the model string; the surrounding application code stays the same.

Real failure mode worth planning for: a routing gateway can fail "successfully." The gateway reaches the provider and gets output back, but the client times out first on a mismatched timeout setting. The user sees an error while tokens still get billed. Aligning timeouts across client, gateway, and provider, and testing streaming timeouts specifically, catches this before production does.

OpenRouter

OpenRouter is the largest public model marketplace in this list, strong for breadth and speed: sign up, pick from a large menu, ship an experiment quickly. Pricing carries no markup on the inference rate itself; fees show up elsewhere, 5.5% on card-funded purchases, 5% on crypto, and 5% on bring-your-own-key usage past a free monthly allowance, all as published on OpenRouter's own site.

The marketplace model means quality and operational stability can vary by route, so treating each model route like its own dependency, tested and monitored rather than trusted by default, matters more here than on a platform with a smaller, more curated catalog. Governance features built for organizations, RBAC, audit trails, scoped keys, are generally a secondary focus compared to gateways built specifically for team operations. A common failure mode: a model that works in a demo turns out to be inconsistent about structured output in production. A gateway can swap the model; it can't make an unreliable model reliably emit valid JSON without app-side validation and retries.

Requesty

Requesty's pitch is simplicity: a flat 5% markup on model cost covers routing, caching, failover, guardrails, and analytics, dropping to 0% with bring-your-own-key. That simplicity is real, one number to reason about, but the downside is equally real: a percentage fee scales with spend by design. A team expecting significant usage growth should decide upfront whether a gateway cost curve that scales with tokens is acceptable, particularly for agentic workloads where one user action can fan out into many internal model calls, each carrying its own percentage fee.

TrueFoundry

TrueFoundry publishes both managed SaaS tiers (Pro at $499/month for 1M requests) and a self-hosted infrastructure cost range, roughly $600 to $1,000 a month on top of the platform fee, for teams needing VPC or on-prem deployment. That transparency is uncommon in this market; self-hosting is often described as "available on Enterprise" with no numbers attached until late in a sales process. Here, a team can build an early spreadsheet and know whether the self-hosted path is even in budget before a conversation starts. Self-hosting still implies ongoing operations, upgrades, scaling, incident response, so the published infrastructure number is a starting point, not the whole cost.

LiteLLM

LiteLLM is the self-hosted answer: an open-source proxy and SDK supporting a large number of providers, with virtual keys, budgets, and fallbacks included. It's a strong fit when self-hosting is a hard requirement, a data boundary, a network policy, procurement friction, or when full control over the data plane matters more than convenience.

The tradeoff that's easy to skip past: self-hosting means operating a gateway, which means monitoring, scaling, upgrades, and someone on call able to debug it when it starts returning errors. "Free" describes the license, not the operational cost. A concrete failure mode: a self-hosted gateway deployed in the wrong region, or without enough headroom, becomes a bottleneck. Latency stacks across application region, gateway region, and provider region, and placing the proxy far from application compute can add hundreds of milliseconds to time-to-first-token on every single request.

Observability platforms: the ones that explain the bill and the weird outputs

Helicone

Helicone is observability, not a routing layer: request logging, cost tracking, analytics. This distinction matters because teams sometimes shop for observability when what they actually needed was failover. Helicone won't switch providers during an outage, since it isn't in the request path to do so, but it will make it obvious which prompt version started generating five times the expected tokens. Using it as a proxy adds a hop and a system that needs its own security posture: data retention, PII handling, and access control all become considerations. For deep agent-step tracing specifically, Langfuse tends to be more purpose-built.

Langfuse

Langfuse focuses on tracing and evaluation, particularly useful for agentic applications with multi-step tool calls, loops, and retries. Like Helicone, it observes traffic rather than routing it. Retention and evaluation pricing mean a team should decide deliberately what to keep and for how long; keeping every trace indefinitely is a cost trap that creeps up slowly. It's a complement to a routing layer, not a substitute for one.

Infrastructure-native gateways: best when the platform is already there

Cloudflare AI Gateway

For a team already on Cloudflare Workers, Cloudflare AI Gateway is the low-friction choice. Core routing, caching, and analytics are free with any Cloudflare account; cost shows up in log volume beyond Workers plan limits and a 5% fee specifically on credit purchases made through its Unified Billing option, with underlying provider rates passed through unchanged. The fit is strong specifically because the platform is Cloudflare-shaped: excellent if the application already is, an added learning curve if it isn't, and worth reconsidering if multi-cloud neutrality becomes a priority later.

Kong AI Gateway

Kong AI Gateway extends Kong's established API management platform into LLM, MCP, and agent-to-agent traffic governance, for organizations that want AI traffic governed under the same platform as everything else rather than as a separate point solution. Pricing isn't published; it runs through Kong's existing enterprise sales motion. The fit is narrow and specific: an organization not already standardized on Kong is unlikely to adopt it purely for AI gateway functionality, since that's a much larger platform decision than a weekend integration.

How to actually choose

What pricing curve is acceptable at ten times current volume?

This question eliminates options faster than any feature comparison. A flat fee keeps platform cost predictable while token spend still scales on its own. A percentage of spend is easy to accept early and becomes a permanent tax on growth. Request-based SaaS pricing can work fine for simple chat traffic and turn surprising fast for agentic workloads that generate many internal calls per user action.

Does the team need routing, observability, or both?

A common production stack pairs a routing gateway with an observability layer, Helicone or Langfuse, for deep tracing and evaluation on top of it. Routing alone can be acceptable early, as long as spend is still attributable and incidents are still debuggable. Observability alone is fine when routing is already stable on a single provider, but it won't prevent an outage from reaching users.

Is self-hosting, VPC, or on-prem non-negotiable?

If so, the shortlist narrows quickly to LiteLLM's self-hosted path, TrueFoundry's published self-host option, or FastRouter's Enterprise tier, which offers a dedicated deployment path through the same kind of conversation. A percentage-markup SaaS gateway generally doesn't offer this route at all.

Is governance needed now or only "at scale"?

Governance requirements, per-environment keys, per-team budgets, RBAC, audit logs, tend to get pushed to "later" until a security review makes them urgent. Some platforms include these broadly from the start; others gate them behind an enterprise tier. If a team already knows it needs them, migrating onto a platform that gates them behind a higher tier later is more disruptive than choosing one that includes them from day one.

How does it actually fail?

This is diligence work, not something a pricing page answers. Worth testing directly: how provider rate-limit responses and their headers survive the proxy, how streaming stalls and timeouts are handled, whether a retry can double-charge tokens when a first attempt succeeded but the client had already disconnected, and how fallback behaves when the response has to be valid, schema-conformant JSON. A routing gateway can swap models on error; it can't guarantee output compatibility between the models it swapped between. That validation stays the application's responsibility either way.

Where FastRouter fits, and where it doesn't

FastRouter’s differentiation in this list is the combination of flat-fee pricing, zero token markup, and Routing Intelligence, FastRouter’s own layer of Evaluations, Insights, and Prompt Optimization, delivered alongside governance and cost visibility as baseline features rather than upsells. That combination fits a team that expects meaningful growth and wants the platform cost curve to stay flat while it happens, without negotiating separately for the routing intelligence, guardrails, and observability that already come standard on every tier. .

It isn't automatically the best fit for every case: a solo developer who mainly wants the largest possible public model marketplace may move faster on OpenRouter; a team under a strict no-SaaS-in-the-hot-path policy will end up evaluating LiteLLM or a self-hosted path regardless; and a team already deep in the Cloudflare or Kong ecosystem may find extending that platform cheaper in integration effort than adopting any new vendor, FastRouter included.

What to do this week

• Write a one-page requirements doc covering routing needs (failover, caching, per-model rules), governance needs (RBAC, per-environment keys, budgets, audit logs), and the hard deployment constraint (is SaaS in the hot path acceptable at all).
• Measure current traffic shape: requests per user action, especially for agentic flows, token distribution at p50 and p95 per endpoint, and today's actual error modes (429s, 5xxs, stream stalls).
• Shortlist one router and one observability tool rather than a long list of every platform in this piece. Run a real pilot instead of reading documentation indefinitely; include Langfuse if the application is agentic, include Helicone if the immediate pain is not knowing where spend is going.
• Run an incident simulation: force a 429 and confirm backoff still works, force a provider 5xx and confirm failover actually happens, force slow streaming and confirm timeouts don't produce the specific failure of a user-visible error alongside billed tokens.
• Decide based on operability: who gets paged when it breaks, how quickly routing can change without a client redeploy, and whether keys and budgets can be scoped without custom glue code.

A platform that can't make those specific, boring workflows easy isn't functioning as a gateway, whatever its marketing page calls it.

Getting started

FastRouter's full feature list and current pricing are documented at fastrouter.ai and docs.fastrouter.ai.

Pricing and feature details above reflect each vendor's own published pages as of October 2026. This is a fast-moving market; reverify every figure and tier detail against the live vendor page before publishing.