412 tok/s peak, fastest model
Serverless tokens or warm dedicated GPUs — one gateway, OpenAI-compatible.
Serverless inference serves the catalog's open models through one OpenAI-compatible endpoint: send a chat completion, stream tokens over SSE, and pay per token. Cached prompt prefixes bill at the cached rate — on DeepSeek V4.1 Flash (`deepseek-v4-flash`) $0.022/1M cached input against $0.22/1M uncached input and $0.66/1M output.
When traffic steadies, move the same model id to a dedicated deployment. The `b200-4x-fp4` preset is 4× B200 180GB on NVFP4 at $40/hr, which the catalog sizes for ≤70B dense models; `b300-8x-fp8` is 8× B300 288GB on FP8 at $96/hr for maximum quality on frontier dense models. Billing follows the preset rate with scale-to-zero, and the endpoint URL does not change.
Context is the first sizing constraint. DeepSeek V4.1 Flash accepts 1.05M tokens, GLM-5.3 accepts 203K, and Qwen3.8-27B Flash accepts 33K. DeepSeek V4.1 Flash carries the catalog's measured output speed at 218 tok/s; GLM-5.3 is listed at 118 tok/s estimated. Choose the preset from the model footprint and precision, not from the request rate: NVFP4 on four B200s for ≤70B dense models, FP8 on eight B300s when frontier dense weights need more memory and quality.
Start with the model page for the context window and token rates, then compare the two GPU presets before committing to dedicated capacity. Go serverless first; move to dedicated when the bill is predictable.
Fine-tuning that ships
SFT → DPO → RL, then deploy the checkpoint in seconds.
Agents that stay up
Low-latency tool loops, streaming, and spend guardrails for autonomous workloads.
Batch API (beta)
Evals, backfills, and bulk scoring — async, without holding a connection.
Enterprise RAG that cites
Embed, retrieve, rerank, answer — with cached prefixes and audit trails.
Long-context production
Text models behind one key — vision is not served yet.