100k requests per job
Batch API (beta)
Evals, backfills, and bulk scoring — async, without holding a connection.
Batch runs through the same gateway as live traffic: submit up to 100,000 requests per job, let it run asynchronously, and download results when the job completes. Same models, same metering, same ledger — only the response path changes. Each request bills at the same model rates the model page publishes.
DeepSeek V4.1 Flash (`deepseek-v4-flash`) is the batch workhorse for bulk scoring: 1.05M context, $0.22/1M input, $0.022/1M cached input, $0.66/1M output, and a measured 218 tok/s. Repeated prompts across a backfill hit the cached rate, and long documents fit because the context window is the largest in the catalog.
Batch workloads trade latency for throughput: no SSE connection to hold open, no per-request streaming overhead, and a 24-hour turnaround. Evals, embeddings, and rerank requests share the same job envelope and the same model ids as live calls, so a backfill does not need a second integration.
There is no batch discount while the API is in beta: spend lands in the same ledger and analytics as live traffic, so compare the model's token rates before you size the job. Submit a batch from the console, or read the batch evals tutorial first if you are still shaping the request set.
- ✓Async jobs, 24h turnaround
- ✓100k requests per job
- ✓Evals + embeddings + rerank
- ✓Unified ledger + analytics
Keep exploring
Production inference
Serverless tokens or warm dedicated GPUs — one gateway, OpenAI-compatible.
Fine-tuning that ships
SFT → DPO → RL, then deploy the checkpoint in seconds.
Agents that stay up
Low-latency tool loops, streaming, and spend guardrails for autonomous workloads.
Enterprise RAG that cites
Embed, retrieve, rerank, answer — with cached prefixes and audit trails.
Long-context production
Text models behind one key — vision is not served yet.