What Engineers Building With LLMs Are Actually Struggling With — And What Actually Fixes It

FastRouter reads the communities where developers, ML engineers, and engineering leaders talk about real production problems with AI systems. Not announcements, not benchmark releases — the conversations that happen when something breaks at 3am or when the invoice arrives and nobody can explain it.

We do this because the problems people describe publicly are the same ones that show up in our customer calls and audit reports. When the same question surfaces across r/FinOps, r/LLMDevs, r/mlops, r/AI_Agents, and a dozen X threads in the same period, it is not noise. It is signal.

This is our take on what is coming up right now, what the community is getting right about diagnosing the problem, and where we think the actual fix lies. All quotes are paraphrased and anonymised from public discussions.

Single-provider dependency stopped being a theoretical risk

For most of 2025, the risk of depending on a single LLM provider was treated as something to plan for eventually. That changed in the first half of 2026. Multiple major providers had visible outages, rate limit cuts without warning, and latency spikes that pushed into production. The conversations that followed were different from previous ones.

Overnight the provider cut limits significantly with no warning, no email, no announcement. Your production app that was working yesterday is now throwing errors every 30 seconds. This is why you never bet your entire stack on one API. — Builder on X

After 30 minutes fighting provider errors I gave up. The real problem is I have no fallback. When the API goes down, the whole workflow stops. — Developer in r/AI_Agents

What shifted was the framing. Outages used to generate frustration at the provider. They increasingly generate frustration at the architecture. Engineers who hit a single-provider outage are asking not when things will be fixed but why their entire application depended on one provider being up in the first place.

The latency variance problem is quieter but compounds. One engineer shared real-time latency data showing a specific provider becoming significantly more volatile following a model update while another's latency decreased. Managing this manually takes hours. By the time a human notices, decides to switch, updates configuration, and redeploys, the damage has already happened.

Our take: this is an architecture problem, not a vendor problem. Every provider will have outages. The question is whether the application is built to handle them automatically or whether it requires manual intervention every time. FastRouter's automatic failover routes around degraded providers in under 10ms. When a provider's error rate crosses a threshold or latency spikes, traffic shifts to the next available provider without any code change or manual rerouting. The application does not know anything changed. The team finds out from the dashboard rather than from user complaints.

Token leaderboards and the measurement trap

Amazon shut down an internal leaderboard called KiroRank after realising it was incentivising engineers to burn through AI tokens just to rank higher. Uber burned through its annual Claude Code budget in a quarter. Meta closed a similar leaderboard after employees competed for a title called Token Legend.

The community reaction to these stories was more sophisticated than the headlines suggested. Most engineers did not treat this as a story about AI being overhyped. They treated it as a story about what happens when organisations measure activity instead of outcome.

The real problem is not the leaderboard. It is that most teams still have no way to tell a useful token from a wasted one at the moment the request is made. The leaderboard just made the underlying gap visible. — Engineering leader in r/EngineeringManagers

This is Goodhart's Law running at scale. Token count is trivially easy to track, which made it an appealing proxy for AI adoption. It is also trivially easy to inflate. A prompt that dumps an entire source directory into the context window and runs a ten-minute reasoning loop burns far more tokens than a clean, tightly scoped request that solves the actual problem. The expensive request was the useless one.

Our take: token volume growing is not the problem. Token volume growing with no visibility, no attribution, and no limits is the problem. The fix is not a memo telling people to be more mindful. The fix is infrastructure that makes the right behaviour the default. FastRouter's per-project budget caps, hard stops at configurable limits, and real-time dashboard showing spend by team and model give teams the structural layer that stops runaway consumption before it shows up on the invoice.

The finance question nobody can answer — and why attribution has to happen before the call

The most consistent theme across FinOps communities has been a deceptively simple question: how do you break down AI spend in a way that finance actually trusts? The honest answer from most teams is that they cannot do it well, and the reason is structural.

Traditional cloud cost allocation works because infrastructure has tags. An EC2 instance has an owner, an environment, a project. The tag travels with the resource and the invoice maps cleanly to teams. LLM API spend has none of that. A single invoice arrives as one number. There is no resource-level metadata, no default tags, just a token count and a price.

Token level attribution for finance reporting is genuinely unsolved at most organisations. Getting to a number the CFO actually trusts requires building something custom on top of whatever the provider gives you. — FinOps practitioner in r/FinOps

One engineer described a framework built internally tracking what they called cost per thought — the full loaded cost of an LLM trace including retries, context overhead, and reasoning tokens, broken down by squad and budget owner. The sophistication of that setup was notable. What was more notable was how few people in the thread had anything comparable.

Our take: the fix is not a better reporting dashboard. It is moving the attribution point upstream. Every LLM call needs to be tagged with team, project, and feature before it reaches the provider — not reconstructed afterward from logs. FastRouter does this at the gateway level. Every request carries that metadata, and the dashboard breaks spend down by those dimensions in real time. The finance question becomes answerable because the data is captured at the only moment it can be captured cleanly: the point of the request. FastRouter's Projects and Virtual Keys structure makes this the default rather than something that has to be custom-built.

Prompt caching: the highest-leverage optimisation most teams have never configured

A thread in r/LLMDevs asked why it feels like providers are hiding prompt caching. The frustration is understandable. Prompt caching is one of the most impactful cost optimisations available at scale and the documentation for it is buried in ways that make it easy to miss entirely.

The economic case is straightforward: a cached prompt prefix costs a fraction of a full input token. For applications that send the same system prompt on every request — which is almost every production LLM application — the savings on a high-volume workload are significant. The catch is that the cache only works when stable content comes first. System instructions, fixed rules, examples, and formatting requirements need to be at the start. Dynamic user input goes at the end. Reverse that order and the cache never warms up.

The real issue is nobody puts it in the quickstart. Keep your static context up front and the cache hits actually show up. — Engineer in r/LLMDevs

The friction multiplies for teams using more than one provider. Anthropic requires manual cache_control breakpoints. OpenAI caches automatically for repeated prefixes past a token threshold. Gemini has its own implementation. A team routing across all three has to maintain three different mental models for something that should be a single configuration decision.

Our take: the part most teams miss is not just setting up caching — it is preserving it across subsequent requests. A cache hit on one request only saves money if the next request goes to the same provider where that cache is already warm. FastRouter's sticky routing ensures exactly that. Once a cache benefit is established with a provider, FastRouter routes subsequent requests in that session to the same provider automatically, so the warm cache is reused rather than abandoned. The Activity Log shows cache hit rates per provider so teams can see whether caching is actually working rather than inferring it from the bill. 

Hardcoded model names are locked-in policy decisions

This observation from X landed precisely because it names a failure mode that almost every team hits eventually:

Every time a model name is written into production code it locks in policy decisions about cost, quality, and availability that should be runtime decisions, not compile-time ones. — Developer on X

The way this plays out: a team picks a model during development, writes the model ID into the code, ships it, and moves on. Six months later the model has been updated, a cheaper alternative exists, and the provider has changed their rate limits — but the application is still calling the same string it was calling at launch. Changing it requires a code review, a pull request, a deployment. The operational cost of revisiting the decision is high enough that most teams never do it.

The second failure mode is more acute. When a provider cuts rate limits without warning, a hardcoded model ID means the application fails with no fallback. The team finds out from user complaints rather than from the infrastructure routing around the problem automatically.

Our take: Virtual Model Aliases in FastRouter let teams route to a logical name like production-default rather than a specific model ID. When the routing decision needs to change — because a better model launched, because a provider is degraded, because a cheaper model proved equivalent for this workload — it changes at the gateway level without touching application code. The Auto Router goes further: when fastrouter/auto is used as the model ID, FastRouter selects the most cost-efficient capable model per request based on configured routing rules.

Teams moving off LiteLLM are doing it for specific reasons

Conversations about moving off LiteLLM have been recurring for months. What became clearer recently is the specificity of the failure modes driving these decisions. Earlier conversations were general. More recent ones identify three concrete problems.

The first is config management at team scale. When multiple squads are modifying routing rules simultaneously, merge conflicts on LiteLLM config files become a real operational problem. The absence of any meaningful access model — team A cannot manage their routing config without being able to modify team B's — only becomes painful at a certain scale, and then it becomes acute quickly.

The second is the SSO and enterprise feature gate. LiteLLM puts Okta and SAML support behind an enterprise license. For teams that need SSO to satisfy security or compliance requirements, this creates an uncomfortable choice.

We moved off LiteLLM after eight months. The config scaling problem, the SSO gate, and a Redis incident during a load test all happened within a few months of each other. None of them were catastrophic individually but together they made us want something different. — Engineer in r/LLMDevs

The third is a Redis failure mode. LiteLLM uses Redis for distributed rate limiting. When Redis has an availability issue, the rate limits fail open — requests go through without enforcement. Several engineers described discovering this behaviour during load tests in staging and being uncomfortable with what it would mean in production during a cost spike.

Our take: FastRouter is fully managed SaaS. There is no Redis to operate, no config file to merge, and SSO is available without an enterprise gate. For teams that genuinely need self-hosted or on-prem, LiteLLM is still the right answer and we say so honestly. For teams that want the gateway capability without the operational overhead, FastRouter is the alternative that is worth evaluating.

MCP governance went from theoretical to operational

Discussions about MCP in production shifted noticeably from what is MCP to how do we govern this. The questions being asked moved from the protocol itself to the operational layer that needs to sit around it.

Most observability tools were built for single LLM calls and agent chains break that model. You end up with good debugging data but no real governance over what the agent was actually allowed to do or cost. — ML engineer in r/mlops

We need rate limiting per agent, an audit trail of which agent called which tool, cost attribution per agent, and failover if a provider degrades. Most gateways assume one client calling one model. That assumption breaks immediately in a multi-agent setup. — Engineer in r/kubernetes

The conversation distinguished clearly between observability and governance — a distinction that vendor marketing frequently blurs. Observability means seeing what happened: latency, token usage, error rates, traces. Governance means controlling what is allowed to happen: who can call which model, rate limits per team or agent, PII filtering, audit logs, cost allocation. Most gateways are strong on observability. The governance layer, particularly across multi-hop agent workflows, remains largely unsolved.

A specific concern that came up around MCP supply chain security: an MCP server returns tool schemas that extend what an agent believes it can do, which means a compromised or misconfigured server can hand an agent capabilities that were never in scope. This is not a problem that can be governed at runtime alone.

Our take: FastRouter's MCP Gateway handles credential vaulting so agents never see raw provider keys. Every tool call passes through the vault, and every interaction is in the audit log. For cost attribution in multi-agent chains, every request carries the project tag from the key used to make the call — even when agent A calls agent B which calls three different LLMs, the attribution flows through the project structure without requiring custom per-agent tagging logic.

Prompts are infrastructure and most teams are not treating them that way

A thread in r/opencodeCLI raised a specific version of a problem that appears in many forms. When routing requests to different models through a gateway, the prompt that works well for one model often does not work as well for another. The engineer wanted provider-specific prompts to activate automatically based on which model was selected, without maintaining separate agent configurations per model.

This is the narrower version of a broader problem. Prompts written for one model accumulate over time. When routing decisions change — because a cheaper model became available, because a provider went down, because an experiment was running — those prompts do not automatically adapt. Output quality degrades in ways that are hard to diagnose because the prompt looks fine. It is just on the wrong model.

A prompt is not a static string. It is configuration that controls the most expensive and least predictable component in the stack. Teams that treat it like a comment in the code discover that the hard way. — From community discussions

Our take: FastRouter's Prompt Hub stores prompts outside the codebase with full version history. The application references a prompt by ID rather than by pasting the string inline. When the routing layer selects a different model, the appropriate prompt variant can be applied. When a prompt needs to be updated, it is updated once in the Hub and the change propagates everywhere. Rollback is selecting the previous version — not a git revert and a deployment. The person best placed to improve a prompt — often a product manager or domain expert, not an engineer — can make that change directly without touching the codebase.

What the Cloudflare AI Gateway question is really asking

A thread in r/SelfHostedAI asked a question that came up in several forms across different communities: when does Cloudflare AI Gateway stop being sufficient and when does a more fully featured gateway become necessary? The framing that came back was useful.

Cloudflare AI Gateway is a traffic optimisation layer. For a solo developer building a side project who wants low-latency routing and basic logs without operating additional infrastructure, it is a reasonable starting point. The breaking point arrives when LLM traffic becomes something the platform or security team needs to govern rather than just something the application needs to call.

The breaking point is usually when someone asks for per-app or per-team budgets, BYOK per tenant, provider failover policies, audit logs tied back to specific features, or PII checks before content leaves your infrastructure. At that point Cloudflare is the wrong tool. — Engineer in r/SelfHostedAI

Our take: the practical advice that emerged from this thread is worth stating plainly. Design the abstraction layer now even if switching the implementation later. Keep provider calls behind a consistent internal interface, avoid Cloudflare-specific assumptions in application code, and define what logs, policies, and budgets will be needed before they become urgent. A team that does this can move to a more capable gateway when governance becomes important without touching application code. A team that does not creates technical debt that is painful to unwind.

The pattern across all of these problems is the same. AI usage scaled fast. The infrastructure layer underneath it did not scale at the same pace. Teams that moved quickly now have production systems with no attribution, no fallback, no governance, and prompts treated like comments in the code. The work for most engineering organisations right now is not building more features on top. It is building the layer underneath the features that makes everything manageable. That is the problem FastRouter is built to solve.

FastRouter is an LLM gateway giving engineering teams a single OpenAI-compatible endpoint across 160+ models. Per-team cost attribution, budget caps, automatic failover, Prompt Hub, MCP Gateway, and full observability — with zero markup on API calls. Start with a free 7-day audit at fastrouter.ai.