Asynchronous by design
Send high-volume work as one file instead of thousands of live calls. Batches run in the background, so bulk jobs avoid rate limits.
Batch Processing
Thousands of requests, one file.
Upload a JSONL file of up to 50,000 chat completion or embedding requests and let FastRouter process them asynchronously across OpenAI, Anthropic, and Gemini-results land as a downloadable file within 24 hours.
No credit card required · Free to start
Batch processing
Input fileJSONL
embeddings-backfill.jsonl12,480 rows
endpoint: /v1/embeddings
Processing asynchronously
8,320 / 12,480 completedIn Progress
output file
results.jsonl
mixed models in one file
openaianthropicgemini
Why batch processing
Built for the work that doesn't need an instant answer
When volume matters more than latency, batching turns thousands of individual calls into a single file you submit once and collect later.
Mix models in one file
Combine models from OpenAI, Anthropic, and Gemini inside a single batch, as long as every line targets the same endpoint.
Results as a file
Requests complete within 24 hours-often much quicker-and come back as a downloadable JSONL keyed by your own custom_id.
How it works
From a JSONL file to a results file
Prepare your requests, run the batch against one endpoint, follow its progress, and download the output-through the dashboard or the API.
Step 1
Upload your file
JSONLcustom_id
- One JSON object per line, each with a unique custom_id.
- Upload in the dashboard or POST it to /v1/files.
Step 2
Run the batch
endpointAPI key
- Pick the endpoint and the API key used to bill every request.
- Requests are queued and processed asynchronously.
Step 3
Monitor progress
In ProgressCompleted
- Track completion percentage from the Batch Jobs view.
- Failed requests are listed with their error details.
Step 4
Download results
results.jsonl
- Results arrive as JSONL, one line per custom_id.
- Typically ready within hours, up to 24 hours.
File format
One JSONL file, one request per line
A batch is just a JSONL file: each line is a complete request with your own identifier. That makes batches easy to generate from a script, a database export, or an existing dataset.
Unique custom_id per line
Give every request an identifier such as request-1. It must be unique across the file and is how results are matched back.
provider, method, url, body
Each line names the provider slug, the POST method, the endpoint, and the request payload with its model and parameters.
Chat completions or embeddings
Point every line at either /v1/chat/completions or /v1/embeddings-one endpoint per file, though models can differ.
requests.jsonl
JSONL{"custom_id": "request-1",
"provider": "anthropic",
"method": "POST",
"url": "/v1/chat/completions",
"body": { "model": "…" }}
{"custom_id": "request-2", "provider": "openai", …}
One endpoint per fileModels can differ
Dashboard
Create a batch and watch it finish
Upload the file in the FastRouter dashboard, choose the API key that should be billed, and submit. Progress is tracked for you-there is nothing to poll by hand.
Upload and configure
Open Batch Processing, click Create Batch, and upload your JSONL file for the endpoint type you are running.
Pick the billing key
Select an API key from your account; that key is used for every request in the batch.
In Progress, Completed, or Failed
Batch Jobs shows each batch with a progress bar, and failed requests come with error details you can inspect.
Batch Jobs
Dashboardsupport-replies.jsonlCompleted
4,200 requests100%
embeddings-backfill.jsonlIn Progress
12,480 requests67%
eval-run-mar.jsonlFailed
860 requests42%
Batch API
Automate the whole run in four calls
Everything the dashboard does is available over REST, so scheduled jobs and data pipelines can submit batches without a human in the loop.
Upload, then trigger
POST your file to /v1/files for a file_id, then POST to /v1/batches with the input_file_id, endpoint, and a 24h completion window.
Poll the batch status
GET /v1/batches/{batch_id} returns the current state so your pipeline knows when the work is done.
Fetch the output file
GET /v1/files/{file_id}/content downloads the results JSONL, with one line per custom_id including any errors.
Batch API
REST1POST/v1/files
Upload the JSONL file→ file_id
2POST/v1/batches
Trigger with endpoint + 24h window→ batch_id
3GET/v1/batches/{batch_id}
Check status and progress→ status
4GET/v1/files/{file_id}/content
Download the results file→ results.jsonl
Same flow as the dashboardFully automatable
Batch vs real-time
When to batch and when to call live
Real-time requests answer immediately, one call at a time. Batch processing trades that immediacy for volume: one file, many requests, collected when the run finishes.
| Behavior | Batch processingOne JSONL file | Real-time requestsLive API calls |
|---|---|---|
| How the work runs | ||
| Execution | Asynchronous | Synchronous |
| Requests per submission | Up to 50,000 | One |
| Completion window | Within 24 hours | Immediate |
| Rate limits on bulk jobs | Avoided | Apply per call |
| Inputs and outputs | ||
| Input | JSONL file | Request body |
| Models per submission | Mixed providers | One |
| Endpoint | One per file: chat or embeddings | Any, per call |
| Results | JSONL file by custom_id | Inline response |
Batches are processed within 24 hours, often much quicker, and are billed at the batch pricing of the underlying model.
Use cases
Where batches earn their place
Any workload that is large, repetitive, and patient is a good batch-nothing is waiting on the response.
Embed a whole dataset
Generate embeddings for large corpora in one submission instead of trickling requests through a live endpoint.
Run bulk evaluations
Send a fixed set of prompts across several models in one file and compare the responses once the batch lands.
Large-scale content generation
Produce descriptions, summaries, or translations for thousands of records where nothing depends on an instant reply.
Overnight data processing
Queue classification or enrichment work that can finish while nobody is waiting, then collect the output file.
FAQ
Batch processing, answered
Batch processing lets you upload a JSONL file containing many chat completion or embedding requests, which FastRouter processes asynchronously across supported providers. It is built for high-volume work such as generating embeddings for large datasets or running many completions at once, and the results come back as a downloadable file.
Use JSONL-one JSON object per line. Each line needs a custom_id that is unique across the file, a provider slug such as openai or anthropic, a method of POST, a url of either /v1/chat/completions or /v1/embeddings, and a body with the model and the parameters for that endpoint. Every line in a file must use the same url.
Batches currently run on OpenAI, Anthropic, and Gemini, with more providers coming. Supported endpoints are /v1/chat/completions for chat requests and /v1/embeddings for vector representations. You can mix models from the supported providers within a single file as long as they all target the same endpoint.
Batches are processed within 24 hours and are often much quicker. A single batch can contain up to 50,000 requests, and processing time scales with batch size. It is worth starting with a small batch to confirm your file format before submitting a large one.
Each batch uses the FastRouter API key you select when creating it, and requests are billed at the batch pricing for the underlying model-tokens for chat completions and inputs for embeddings. Check the individual model pages for the batch pricing of a specific model, and make sure the key has sufficient credits before submitting.
The results file is JSONL with one line per request, matched by custom_id, containing either the response data or the error for that request. The dashboard also shows failed requests with more detail, so you can correct just those lines and resubmit them in a new batch.
Run your first batch today
Upload a JSONL file, pick an endpoint and an API key, and let FastRouter work through it while you get on with something else.