Prompt Compression: The Same Answer for a Fraction of the Tokens

A support agent reads a 4,000 line log file to find one error. A RAG pipeline stuffs eight retrieved chunks into context when the answer lives in two of them. A coding assistant re-sends the same 600 line system prompt on every single turn of a conversation. None of this is a bug. It is just what happens when context windows got big enough that nobody had to think about what they were putting in them.

FastRouter's new Prompt Compression feature is built for exactly that gap, the space between what a request actually needs and what it ends up sending. It is opt-in, it is one block added to an existing request, and it fails open, so turning it on cannot make a request behave worse than it already does.

Where this actually came from

Worth saying plainly: FastRouter did not invent these compression techniques. Headroom, Caveman, and LLMLingua all exist as open source approaches to the same problem, LLMLingua in particular comes out of Microsoft Research, where it was published as a peer reviewed method for token level prompt compression. What FastRouter built is the integration layer, one opt-in parameter that routes a request to whichever of these approaches fits the content, with consistent fail-open behavior and a way to measure the impact before committing to it. The value here is not a new algorithm. It is not having to evaluate, wire up, and maintain three separate libraries just to get the benefit of each one.

Three engines, three real situations

The tool output problem: headroom

An agent calls a tool, the tool returns a JSON array of 400 rows, and all 400 rows go straight into the next request whether the model needs three of them or all of them. Headroom is built for exactly this, structural, repetitive payloads where the shape matters more than any individual sentence.

A tool result like this:

1{"orders": [
2  {"id": 1001, "status": "shipped", "region": "us-east", "amount": 42.50},
3  {"id": 1002, "status": "shipped", "region": "us-east", "amount": 19.99},
4  {"id": 1003, "status": "pending", "region": "eu-west", "amount": 88.10},
5  ... 397 more rows ...
6]}
Copy

gets restructured rather than summarized, the repeated keys and structural noise get compressed out, while every row's actual data stays intact. Nothing the model might need to reference later is dropped, which matters when the very next question is 'which of those had status pending.'

The verbose system prompt problem: caveman

System prompts have a way of accumulating politeness and hedging that never mattered to the model in the first place. A prompt that reads:

"You are a helpful assistant that specializes in helping customers with their billing questions. Please make sure to always be polite and professional, and try your best to fully understand what the customer is asking before you respond to them."

carries the same instruction as something closer to: "Assistant, billing questions. Be polite, professional. Understand the question before responding." Caveman strips the filler, the hedging, and the redundant phrasing, while explicitly preserving negations, numbers, modals, and anything in quotes or code, the parts where dropping a word actually changes the instruction rather than just its politeness.

The RAG chunk problem: llmlingua

This is the case where the content is prose, but a lot of it is genuinely low information relative to the question being asked, retrieved documentation, meeting transcripts, long form articles pulled in for a narrow question. LLMLingua uses a small model to score each token's contribution to meaning and drops the ones that do not carry weight, rather than applying a blanket rule. It is the right tool when a five page document got retrieved for a two sentence answer.

Measuring it before trusting it

The honest way to introduce compression into a production path is not to turn it on and hope. Setting mode to audit runs the full pipeline and reports exactly what it would have saved, without changing a single byte of the actual outgoing request.

1{ "optimize": { "compress": { "engine": "all", "mode": "audit" } } }
Copy

That request comes back with the normal response plus the X-FastRouter-Tokens-Saved and X-FastRouter-Savings-Percent headers, so the first real decision, whether this is worth turning on for a given endpoint, can be made from actual numbers on actual traffic rather than a vendor's benchmark chart.

Stacking all three

For mixed content, a single request carrying a system prompt, a chat history, and a retrieved document, engine can be set to all, which runs headroom, then llmlingua, then caveman in sequence. Each engine gets the part of the payload it is actually good at, rather than one generic compressor doing a mediocre job on all three.

1{
2  "model": "openai/gpt-5.5",
3  "messages": [ ... ],
4  "optimize": {
5    "compress": {
6      "engine": "all",
7      "llmlingua_rate": 0.75
8    }
9  }
10}
Copy

The part that matters most: it cannot break anything

If the compression service is unreachable, times out, errors, or returns something that does not line up with the original message count, the original request goes out unchanged. No error, no failed call, just the request you would have sent anyway. Combined with audit mode for testing ahead of time, that means adopting this is close to zero risk to try, the worst case is that nothing happens, not that something breaks.

Prompt Compression is available now through FastRouter at fastrouter.ai. Start with mode: audit on a real endpoint, look at what it would have saved, and decide from there.