Back
How to Reduce LLM API Costs (Without Breaking Quality)

How to Reduce LLM API Costs (Without Breaking Quality)

Five pricing levers before switching models, plus Prompt Optimization to make a cheaper model match the quality bar you already clear.

author Andrej
Andrej Gamser
11 Min Read|Latest -

LLM costs rarely blow up because someone picked an expensive model once. They blow up because a system does the same expensive thing over and over: every request drags along a giant system prompt, tool schemas get resent verbatim thousands of times a day, background jobs run like interactive traffic at full price, a model available from multiple providers ends up pinned to the most expensive one by accident, and nobody looks at real traffic closely enough for the waste to become visible.

Switching to a cheaper model is a real lever, but it's also the bluntest one, and it often trades cost for reliability problems: more retries, more prompt scaffolding, more debugging of output format drift. The better order of operations is to stack the levers that don't touch quality first, then only switch models once the system is already efficient, and tune the prompt before accepting a quality hit at all. What follows is six levers, in the order they're worth doing. Most of them are boring. That's why they work.

TL;DR

  • Sort provider selection by price for models with multiple providers behind them. Same model, often cheaper, worth watching latency tails.
  • Make repeated prefixes actually repeat (stable system prompts, stable tool schemas) so prompt caching can apply.
  • Move non-urgent traffic to :flex pricing (OpenAI and Gemini models): same model, cheaper, slower and less predictable latency.
  • Batch offline volume. If nothing is waiting on the response, there's no reason to pay real-time prices for it.
  • Use Prompt Optimization before switching models. A better-tuned prompt can get a cheaper model to match the quality bar the current model clears.
  • Check Insights weekly. It surfaces waste specific to a system's own traffic, which a generic checklist can't find.

Step 1: Route to the cheapest provider, same model, less money

"The model" and "the provider serving the model" are two separate decisions, and it's easy to miss that they're separate. Open-weight models especially are commoditized: multiple providers often host the same weights at different prices and different latency characteristics. FastRouter can pick the cheapest qualifying provider at request time without changing which model answers.

1curl https://api.fastrouter.ai/api/v1/chat/completions \
2  -H "Authorization: Bearer $FASTROUTER_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "model": "deepseek/deepseek-v4.1-flash",
6    "messages": [{ "role": "user", "content": "Extract people, organizations, and products from the text and return JSON." }],
7    "provider": { "sort": "price" }
8  }'
9
10import os
11from openai import OpenAI
12
13client = OpenAI(
14    base_url="https://api.fastrouter.ai/api/v1",
15    api_key=os.environ["FASTROUTER_API_KEY"],
16)
17
18resp = client.chat.completions.create(
19    model="deepseek/deepseek-v4.1-flash",
20    messages=[{"role": "user", "content": "Extract people, organizations, and products from the text and return JSON."}],
21    extra_body={"provider": {"sort": "price"}},
22)
23
24print(resp.choices[0].message.content)

Latency and rate-limit behavior are the two things worth watching: the cheapest provider isn't always the fastest, so enabling price-sorting per workload, rather than globally, and watching P95/P99 after the change catches most of the downside. Keeping a way to pin a provider for the handful of critical request paths covers the rest. This is still the first lever worth pulling, since it doesn't touch prompts, temperature, tools, or output format. It only stops overpaying when the exact same model is available for less.

Step 2: Stop paying full price for repeated context

If an application sends a 1,000-to-10,000-token system prompt on every call, that's not prompt engineering, that's a subscription nobody meant to buy. Prompt caching helps whenever a request contains a prefix that repeats across calls: a system prompt that's stable for an app version, a tool-schema blob that never changes, shared policy or rubric text, or a chunk of RAG context reused across multiple steps.

On FastRouter, caching behavior depends on the upstream provider. For OpenAI, DeepSeek, and Google models, sending the same prefix repeatedly is commonly enough for the provider-side caching discount to apply with no extra configuration. For Anthropic models, caching is explicitly controlled, following Anthropic's own prompt-caching mechanics, since the caller has to mark cacheable content rather than have it inferred automatically. The exact request shape for enabling caching is worth confirming against the current provider documentation before shipping, since it's the kind of parameter that occasionally changes shape.

Caching is unforgiving. Prefix matching needs the prefix to actually match, and a small amount of dynamic content at the front of a prompt, a timestamp, a request ID, a reordered instruction, is enough to break it silently while the request still succeeds.

The most common ways caching quietly stops working: dynamic content placed inside the prefix instead of after it, user-specific data (a name, an account tier) embedded in the system message, a tool schema regenerated per request with different whitespace or key ordering even though it's logically identical, and RAG chunks returned in a different order across calls. The fix is boring but effective: treat the cacheable prefix as a versioned artifact. Move dynamic fields to the end of the message list, store the long system prompt as a literal constant rather than building it from fragments at runtime, version it deliberately, and normalize tool schemas (stable key order, stable whitespace) before sending them. Logging a hash of the stable prefix, its token count, and whether that request class is expected to be cacheable turns caching from a mystery into something that can be verified.

Step 3: Move non-urgent traffic to Flex pricing

Flex pricing offers roughly half-price inference on OpenAI and Google Gemini models in exchange for slower, near-real-time processing that may queue rather than return immediately. Appending :flex to a model slug is the entire integration:

1curl https://api.fastrouter.ai/api/v1/chat/completions \
2  -H "Authorization: Bearer $FASTROUTER_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "model": "openai/gpt-5.6-terra:flex",
6    "messages": [{ "role": "user", "content": "Classify this support ticket as billing, bug, outage, or account, and return JSON." }]
7  }'

The tradeoff is latency, not quality: the same model, the same output, at a slower and less predictable response time. That makes Flex a good fit for background enrichment, batch-adjacent workloads, evaluation runs, and anything else that isn't sitting in front of a waiting user, and a poor fit for chat UX, autocomplete, or synchronous endpoints with a user-visible latency budget. A common failure pattern is turning Flex on globally, getting user complaints, and then compensating with aggressive timeouts and retries, at which point the cheaper tier ends up costing more and being less reliable. Flex works best kept behind a clear split between asynchronous and interactive traffic, applied deliberately rather than as a blanket setting.

Step 4: Batch what doesn't need to run in real time

A job that runs from a nightly cron has no reason to pay real-time request overhead per item; that's an architecture mismatch, not a pricing problem. Batch Processing uploads a file of requests and returns results asynchronously at a reduced cost compared to running the same volume as individual real-time calls, which also smooths throughput and reduces exposure to rate-limit spikes.

Batching is worth it once volume is high (thousands or more requests per run), the work is genuinely offline, results can arrive minutes to hours later, and requests are homogeneous enough to write and read as a file. What teams tend to underestimate is the plumbing around it: partial failures need to be parsed out of a large batch and retried through a separate path, retries need to be idempotent so they don't duplicate downstream side effects, and monitoring has to shift from endpoint latency to job-completion-time distribution and per-batch error rate. Batching adds asynchronous plumbing; it's still the right move for offline volume.

Step 5: Use Prompt Optimization before accepting a quality hit

Every lever so far changes how a request is routed or billed without touching the model or the prompt. This one touches the prompt on purpose, because it's usually faster than hand-tuning and it's the step that makes switching to a cheaper model viable instead of just cheaper.

FastRouter's Prompt Optimization feature (GEPA, a reflection-driven optimization method) iterates on a prompt against real task examples, looking for a version that closes the gap between what a cheaper candidate model produces and what the current, more expensive model already produces. The output is a better prompt, not a different model and not a different request shape.

The useful order of operations: pick the cheaper candidate model first, run Prompt Optimization against that model using the task's actual inputs and acceptance criteria, and only then decide whether to switch. A cheaper model with an optimized prompt can land at the same output quality as a pricier model with a generic one; a cheaper model with an unoptimized prompt is usually where "switching models to save money" quietly turns into a quality complaint. Treat the optimized result the same way a Model Switch recommendation gets treated below: as a candidate to validate against the application's real constraints, not a result to ship on the strength of the optimization run alone.

Step 6: Use Insights to find waste specific to real traffic

The first five levers apply generically to any system. This one is different. FastRouter's Insights engine, found under Optimize → Insights in the dashboard, analyzes an organization's actual traffic on a rolling seven-day window and surfaces ranked, evidence-backed recommendations scoped to real projects, keys, and models. Nothing it recommends is applied automatically.

Caching flags a system-prompt prefix repeating across many requests on the same key where prompt caching isn't in use. Always on, free to generate. This is usually a prompt-stability issue rather than a platform one: fixing prefix stability is what surfaces the saving.

Flex Tier flags high-volume traffic sitting on a provider's standard tier where the same model is available on the cheaper Flex tier. Always on, free to generate. A confirmed example from FastRouter's own docs: a scope costing $3,236.62 a week on standard tier drops to $1,641.62 a week on Flex, a saving of $1,595.00 weekly.

Model Switch flags a use case where a different model would be both cheaper and, by an automated judge's scoring, comparable or better in quality. It's opt-in and billed, since it costs real inference to run the comparison, and must be enabled under Settings on the Insights page. A confirmed example from the docs: a switch cutting cost by $1.2074 per sampled request, a 79% reduction, which projects to $1,064.50 in weekly savings at that scope's volume.

An automated judge is useful, but it doesn't know a system's own constraints. It can score two outputs as equally good while missing that one of them breaks a downstream parser: a model that reliably returns bare JSON versus a model that needs explicit no-markdown instructions and still occasionally wraps its answer in code fences. The judge might call both responses fine; production only works with one of them. A positive Model Switch delta is a signal to run a real evaluation against the application's actual constraints (JSON validity rate, tool-call correctness, refusal behavior, retry rate under messy input), not a verdict to act on directly. It's also a natural trigger for Step 5: if Model Switch flags a cheaper model with a quality gap, Prompt Optimization against that candidate is the next thing to try before giving up on the switch.

A worked example: three levers on one workload

A background job that summarizes internal documents for search indexing doesn't need to run in real time, uses the same instructions every call, and runs at high volume. It can stack Flex pricing, provider price-sorting, and prompt caching in a single request:

1curl https://api.fastrouter.ai/api/v1/chat/completions \
2  -H "Authorization: Bearer $FASTROUTER_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "model": "openai/gpt-5.6-luna:flex",
6    "messages": [
7      { "role": "system", "content": "Summarize the document for engineers. Return JSON only, no markdown, no code fences." },
8      { "role": "user", "content": "Document: <document text>" }
9    ],
10    "provider": { "sort": "price" }
11  }'

None of those three settings change which model answers the request or the quality of its output. All three touch cost: one decides who serves the model, one decides whether the provider can discount a repeated prefix, and one decides whether the request is allowed to queue.

Common pitfalls

  • Applying Flex to user-facing, latency-sensitive traffic. The savings are real, but the queuing delay is felt immediately by users waiting on a response.
  • Assuming caching is enabled when the hit rate says otherwise. Almost always traceable to an unstable prefix: a timestamp, a request ID, or per-request schema formatting.
  • Treating Model Switch recommendations as final. The quality delta comes from an automated judge on a sample, not from a project's own evaluation criteria. Validate before rolling a switch out broadly.
  • Skipping Prompt Optimization and switching models anyway. A cheaper model evaluated with the old, unoptimized prompt looks worse than it actually is; the quality gap often closes with a better prompt, not a different model.
  • Optimizing for the cheapest model instead of the cheapest correct model. A cheaper model that needs more retries, longer prompts to behave, or generates more support tickets isn't actually cheaper once that cost is counted.

What to do this week

  • Turn on price-sorting behind a flag for one high-volume path. Watch latency P95/P99 and error rate for 48 hours before rolling it out further.
  • Stabilize one cacheable prefix on purpose. Pick the largest repeating system prompt or tool schema, turn it into a versioned constant, and remove dynamic fields from it.
  • Split traffic into interactive and asynchronous paths in code. Move exactly one asynchronous job to Flex and measure the completion-time distribution, not just average latency.
  • Move one offline volume job to batching. Start with the single largest nightly enrichment or classification run, plus minimal retry handling for failed items.
  • Run Prompt Optimization against one candidate cheaper model for one workload, before deciding whether to switch to it.
  • Put a weekly Insights review on the calendar. Check Optimize → Insights, prioritize Caching and Flex Tier recommendations first since they're free and always on, and only then consider Model Switch, backed by a real evaluation.

Getting started

Provider routing, prompt caching, Flex pricing, Batch Processing, Prompt Optimization, and Insights are all documented in full at docs.fastrouter.ai/explore-features.

Related Articles

How much does prompt caching save on LLM API costs?
How much does prompt caching save on LLM API costs?
Cost & Optimization

How much does prompt caching save on LLM API costs?

Prompt caching savings on LLM API costs depend on reuse and billing rules. Learn to calculate net savings, validate cache usage, and compare costs.

F
FastRouter Team
11 Min Read◆November, 6 2026
FastRouter vs TrueFoundry pricing
FastRouter vs TrueFoundry pricing
Cost & Optimization

FastRouter vs TrueFoundry: Pricing Compared

FastRouter vs TrueFoundry pricing, side by side. See what you pay for LLM gateway access and where costs can add up.

author Andrej
Andrej Gamser
12 Min Read◆October, 9 2026