Beginner8 min
Run nightly evals with the Batch API
Nightly SWE-bench-style evals without holding a connection open. Beta demo today — the pipeline shape is final, execution lands with GA.
1. Package your prompts
JSONL: one object per case — input, expected, rubric. Up to 100k requests per job.
{"input": "…", "expected": "…", "id": "case-001"}2. Submit from console
Console → Batch API → pick model + count. DEMO: nothing is processed or billed yet — jobs simulate progress so you can design the pipeline.
# beta demo — no ledger entries until GA
3. Score on download
Design your join now (results × expected → pass-rate per model version); scoring against real results activates at GA.
# plan the join: results ⋈ expected → pass-rate per model