FastRoutervs.Langfuse

A request from your app crosses three logical layers on its way to a response:

1. Request path. Your code talks to something that turns a payload into a completion. This is where a gateway lives. FastRouter handles this layer: provider selection, fallback, retries, key vaulting, budget enforcement, response back to caller.

2. Trace plane. After the call, the inputs, outputs, model used, cost, latency, and any tool calls get persisted somewhere queryable. This is Langfuse's home turf: a session view with spans, scores, prompt versions, and dataset workflows.

3. Eval loop. Periodically you score historical traces or run a curated dataset against candidate models to decide what to ship next. Both products have an opinion here, but they take it from different angles.

The two products do not compete in layer 1 (Langfuse does not serve completions) and do not compete in layer 2 (FastRouter logs but does not provide a full trace UI). They overlap in layer 3, with different shapes — FastRouter optimizes live routing using eval signal; Langfuse runs offline experiments against datasets.

What FastRouter records

Every request through the gateway produces a structured log entry: timestamp, model requested, model used, provider, input tokens, output tokens, cost, latency, status, retry count, and routing decision. You can search by API key, workspace, model, or status. Logs export over OpenTelemetry to whichever back-end you prefer. This is enough to answer "what did this user spend last week" and "which model handled this request."

What Langfuse records

A full trace. One trace can contain many spans — a retrieval call, a tool invocation, the LLM call itself, a post-processor — all linked under a session and a user. You can attach metadata, scores (LLM-as-judge, human, programmatic), and prompt versions to each span. The UI lets you reconstruct exactly what an agent did across multi-step interactions, replay it, and compare runs.

Where they overlap

Both can tell you what model was called, what it cost, and how long it took. FastRouter does this for traffic that went through FastRouter. Langfuse does this for any traffic instrumented to report to it — including FastRouter traffic, direct provider calls, framework calls (LangChain, LlamaIndex), or in-house agents.

Where they diverge

If your question is "the model returned the wrong answer on this user's session last Tuesday, walk me through every span," Langfuse is built for that and FastRouter is not. If your question is "should this request go to Haiku 4.5 or Sonnet 4.5 right now," Langfuse cannot answer it in time because it does not sit in the path.

FastRouter's three eval primitives

Smart Evaluations. LLM-as-judge scoring on live traffic. Scores update as requests flow, with no datasets to prepare.

Automatic Evaluations. Background sampling that benchmarks candidate models against each other on your real workload. Surfaces "Haiku 4.5 is now beating Sonnet 4.5 on your extraction prompts" without an A/B test you wrote by hand.

GEPA — Generative Evolutionary Prompt Architecture. Walks the joint space of prompt variants × model choices and returns Pareto-optimal pairs for a workload. Continuous, not one-shot.

All three feed the routing decision. If Smart Evals says Sonnet wins on this category, Auto Router favors it next request.

Langfuse's eval surface

Datasets & experiments. Curate a dataset of inputs (with expected outputs if available), run candidate prompts or models against it, and compare scores side by side. The bedrock of pre-release eval work.

LLM-as-judge on traces. Define an evaluator (prompt + model), point it at filtered historical traces, and write scores back. Schedule it to run on incoming traffic.

Human scores. Open a trace in the UI and tag it. Build a labeled set over time. Useful for training small classifiers or for ground-truth in dataset experiments.

The honest comparison

FastRouter optimizes the routing decision in production. Langfuse runs structured experiments against history. A team doing serious quality work usually wants both: Langfuse for "should we ship this new prompt next week," FastRouter for "which model handles this request right now."

For teams that run both, the wiring usually looks like this:

1. Application code calls FastRouter at /v1/chat/completions. FastRouter picks a provider, retries on failure, enforces the workspace budget, returns the completion.

2. FastRouter emits an OpenTelemetry span for each request: model used, provider, tokens, cost, latency, routing decision, status.

3. An OTel collector forwards those spans to Langfuse (Cloud or self-hosted). Langfuse stitches them into sessions when the application also reports a session ID.

4. For multi-step agents, the application reports its own spans to Langfuse directly so retrieval, tool calls, and the FastRouter LLM call all end up under one trace.

5. Evaluators in Langfuse score traces on a schedule. Smart Evals in FastRouter score live traffic. The two signals live in different planes but tell the same quality story.

What this stack gets you

One bill for inference (FastRouter), one trace store for product debugging (Langfuse), one place to enforce a budget (FastRouter), one place to run dataset experiments before a release (Langfuse). Nothing duplicated, nothing left out.

What it does not get you

If you wanted a single product that did all of the above, this is not it. You will run two systems, two dashboards, two SDKs in your code base, and two billing relationships if you choose Cloud for both. For most teams that is a fair trade. For some it is the wrong shape, and that is worth being honest about.

If you must pick one

Pick by your most painful failure mode. If outages and bad provider behavior are hurting users, start with FastRouter — Langfuse cannot fix a 5xx. If quality regressions are landing in production unnoticed, start with Langfuse — FastRouter cannot reconstruct what an agent did three days ago across five spans.

You cannot do an apples-to-apples cost comparison because the products charge for different things. A realistic monthly bill for a team running both:

• FastRouter — flat managed-service fee plus inference at pass-through (0% markup on BYOK).

• Langfuse Cloud — free tier covers a generous baseline; production volumes move you to a paid tier based on events ingested per month.

• Langfuse self-hosted — software is free, but you pay for Postgres, ClickHouse, Redis, blob storage, and the engineer time to keep them healthy. Most teams that pick this route do so because they need data sovereignty, not because the math is cheaper.

A worked example

A team spending $40K/month on inference with around 20M trace events:

• FastRouter: $40K pass-through inference + the platform fee. No percentage markup.

• Langfuse Cloud Pro: low four figures per month at this event volume.

• Langfuse self-hosted: software free; expect roughly the same all-in once you cost in infra and on-call.

The two line items live in different categories of spend (inference vs observability) so finance will not treat them as alternatives. They will treat them as complements, which is the right framing.