← All tutorials
Beginner8 min

Run nightly evals with the Batch API

Nightly SWE-bench-style evals without holding a connection open. Beta demo today — the pipeline shape is final, execution lands with GA.

1. Package your prompts

JSONL: one object per case — input, expected, rubric. Up to 100k requests per job.

{"input": "…", "expected": "…", "id": "case-001"}

2. Submit from console

Console → Batch API → pick model + count. DEMO: nothing is processed or billed yet — jobs simulate progress so you can design the pipeline.

# beta demo — no ledger entries until GA

3. Score on download

Design your join now (results × expected → pass-rate per model version); scoring against real results activates at GA.

# plan the join: results ⋈ expected → pass-rate per model

Next tutorials