Fewer input tokens
Eligible messages are compressed at the gateway before they reach the provider, and every response reports exactly how many tokens you saved.
Prompt Compression
Send fewer tokens, keep every fact
Opt in per request and FastRouter compresses eligible messages before they reach the provider - cutting input tokens without affecting response quality, and failing open so a request is never broken.
No credit card required · Free to start
Compression engine
optimize.compress
engine: "all"
Prompt you sendOriginal
Prompt the provider getsCompressed
Eligible messages are compressed at the gateway - same request shape, same response quality.
X-FastRouter-Tokens-Saved: 32826.23% saved
Why prompt compression
Smaller prompts, same answers, zero risk
Shrink what you send to every provider through one gateway - with engines matched to your content and safety guarantees on every request.
An engine for every payload
Lossless structural compression for prompts and schemas, rule-based compaction for prose, ML token pruning for RAG - run one engine or stack all three.
Opt-in and fail-open
Compression is off by default and enabled per request. If the service errors or times out, your original messages are sent - a request can never be broken.
How it works
From your request to an optimized payload
Send a standard chat request with an optimize block. FastRouter compresses eligible messages, forwards the optimized payload to the provider, and returns the standard response plus compression metrics in the headers.
Request
Chat completion
optimize.compressmessages [ ]
- A standard OpenAI or Anthropic chat request, streaming or not.
- Off by default - the optimize block opts in per request.
FastRouter
Compression engine
headroomcavemanllmlingua
- Compresses eligible messages with the engines you pick.
- Fails open - on any error the original messages are sent.
To the provider
Optimized payload
OpenAI, Anthropic, and the rest receive fewer input tokens in the same request shape.
Back to you
Response + savings
The standard provider response, plus compression metrics in the response headers.
Compression settings
engine
allheadroom → llmlingua → caveman
mode
optimizeaudit
llmlingua_rate0.75
AggressiveDefaultGentler 1.0
cache_align
compress is an open container - any engine parameter passes through unchanged.
One compress object, full control
Everything lives in optimize.compress - an open key-value container where any engine parameter passes through unchanged. Start with the lossless default, then tune engines, rates, and scope as you learn what your traffic saves.
- mode: "audit" measures potential savings without changing a byte - the stats still land in your response headers.
- headroom and caveman run with no load cost; llmlingua loads its ML model once (~70s), then executes quickly.
Engines
Pick the right compressor for each kind of content
Different content compresses differently. Choose an engine per request with the engine parameter - a single value, a preset, or an explicit list - and the pipeline always runs in the same order.
headroom - lossless structure
The default. Restructures system prompts, rules, and tool schemas without losing information.
caveman - rule-based prose
Compacts chat prose and verbose instructions while never dropping negations, modals, quantifiers, code, URLs, numbers, or quoted strings.
llmlingua - ML token pruning
Scores and removes low-value tokens from RAG chunks, docs, and transcripts, with llmlingua_rate controlling how much is retained.
Pick an engine
engine- Lossless
headroomdefault
System prompts, rules, tool schemas
- Rule-based
caveman
Chat prose, verbose instructions
- ML pruning
llmlingua
RAG chunks, docs, transcripts
allheadroomllmlinguacaveman
Targeting & protection
Control what gets compressed - and what never does
Compression is precise, not indiscriminate. Scope engines to specific roles, shield the messages that matter most, and let non-text content pass through untouched.
Target by role
caveman_roles and llmlingua_roles restrict engines to specific roles, and headroom can toggle user and system messages independently.
Protect what matters
protect_recent leaves the N most recent turns untouched, protect_analysis_context shields reasoning blocks, and min_tokens_to_compress skips small blocks.
Tune the pressure
llmlingua_rate defaults to 0.75 - the fraction of tokens retained - or set llmlingua_target_token for an absolute token budget.
What gets compressed
- Ssystem messageCompressed
Long system prompt with rules and tool schemas
headroom
- Uuser messageCompressed
Retrieved RAG chunks and reference docs
llmlingua
- Uuser messageBypassed
Image and tool_result blocks
- Uuser messageProtected
The 2 most recent turns
Content marked with cache_control always skips compression.
Audit & observability
Measure savings before you change a single byte
Every compressed response tells you what happened, and audit mode lets you quantify the win on real traffic before turning compression on.
Audit mode
Set mode: "audit" to measure potential savings without modifying requests - statistics still appear in response headers.
Savings on every response
X-FastRouter-Compressed, X-FastRouter-Tokens-Saved, and X-FastRouter-Savings-Percent report exactly what compression did.
Fail-open, always
If the compression service is unreachable, times out, or errors, the original messages are sent and the headers are simply absent.
200 OK · response headers
CompressedX-FastRouter-Compressed: true
X-FastRouter-Tokens-Saved: 328
X-FastRouter-Savings-Percent: 26.23
mode: "audit"Same stats, zero bytes changed
Engine comparison
Three engines, one pipeline
Each engine trades differently between savings and fidelity. Run the one that fits your content - or stack them all and let each take its pass.
| Engine trait | headroomLossless structural | cavemanRule-based prose | llmlinguaML token pruning |
|---|---|---|---|
| Approach | |||
| How it compresses | Structural restructuring | Rule-based compaction | Scored token removal |
| Lossless | Included | Not includedFacts preserved | Not includedRate-controlled |
| Default engine | Included | Not included | Not included |
| Best for | |||
| Content type | System prompts, rules, tool schemas | Chat prose, verbose instructions | RAG chunks, docs, transcripts |
| Controls | |||
| Key parameters | target_ratio, protect_recent | caveman_level, caveman_roles | llmlingua_rate, llmlingua_roles |
| Runtime | |||
| First-request model load | None | None | ~70 seconds |
| Position in the "all" pipeline | 1st | 3rd | 2nd |
engine: "all" runs headroom → llmlingua → caveman in a fixed order; the "both" and "hybrid" presets run headroom + caveman.
Use cases
Built for prompts that carry extra weight
Wherever input tokens pile up - schemas, retrieved context, long histories - compression trims the payload before it's billed.
Agents with heavy tool schemas
Long system prompts, rules, and tool schemas compress losslessly with headroom, so agents carry the same instructions in fewer tokens.
RAG and document pipelines
Prune retrieved chunks, docs, and transcripts with llmlingua at a tunable rate, keeping the context that matters within budget.
Long-running conversations
Compact verbose chat history with caveman while protecting the most recent turns, so ongoing threads stay cheap without losing facts.
Savings audits before rollout
Run mode: "audit" on production traffic to measure potential savings in response headers before compression changes a single request.
FAQ
Prompt compression questions, answered
Compression is off by default. Add an optimize block with a compress object to a standard chat request - for example {"optimize": {"compress": {"engine": "headroom"}}} - and FastRouter compresses eligible messages before forwarding them. It works on OpenAI chat completions and Anthropic Messages routes, streaming and non-streaming; native Responses and Gemini handlers are not covered.
headroom is the lossless default, best for system prompts, rules, and tool schemas. caveman is rule-based compaction for chat prose and verbose instructions. llmlingua is ML token pruning for RAG chunks, docs, and transcripts. Set engine to a single value, a preset ("both" or "hybrid" runs headroom + caveman; "all" runs the full pipeline), or an explicit list - the processing order stays constant either way.
No. Compression reduces input tokens without affecting response quality, and it fails open: if the compression service is unreachable, times out, errors, or the result fails validation, your original messages are sent unchanged. A request can never be broken by compression. caveman also never drops negations, modals, quantifiers, code, URLs, paths, numbers, quoted strings, or ALL-CAPS codes.
They cut cost at different points. Prompt compression shrinks the input tokens you send on each request. Prompt caching keeps the tokens but bills repeated context at a discounted rate. Response caching skips the model call entirely for repeated requests. They compose, too: the cache_align option improves provider prompt-cache hits without reducing tokens, and content marked with cache_control is always left untouched.
Set mode: "audit" in the compress object. Audit mode measures potential savings without modifying the request - not a byte changes - while the statistics still appear in the X-FastRouter-Tokens-Saved and X-FastRouter-Savings-Percent response headers.
headroom and caveman carry no load cost. The first llmlingua request loads the ML model, which takes around 70 seconds; subsequent requests execute quickly.
The top-level system prompt is compressed like other eligible content, while non-text blocks - images, tool_use, and tool_result - pass through unchanged. Content with cache_control markers always skips compression, and caveman and llmlingua are recommended for structure preservation.
Start sending fewer tokens today
Add an optimize block, audit your savings on real traffic, and cut input tokens across every provider - with fail-open safety on every request.