Beginner8 min
Run nightly evals with the Batch API
Nightly SWE-bench-style evals without holding a connection open. Submit async, download results.
1. Package your prompts
JSONL: one object per case — input, expected, rubric. Up to 100k requests per job.
{"input": "…", "expected": "…", "id": "case-001"}2. Submit from console
Console → Batch API → pick model + count. Same gateway, same per-token rate while in beta.
# billed at the standard serverless rate — shows in the same ledger
3. Score on download
Join results with expected, track pass-rate per model version in /console/usage. Promote winners to serverless.
# usage API groups cost by model — compare candidates