Serving GLM-5.3-flash at 412 tok/s on RTX PRO 6000
Inside the lazarus fleet — SGLang tuning, spot VMs, checkpoint-wake, and why scale-to-zero makes cheap inference possible.
runaii.cloud runs its own fleet. No upstream vendor, no resold capacity — the models in the catalog run on RTX PRO 6000 Blackwell cells we operate with our lazarus serving platform. That is the whole reason the prices can be what they are.
The cell
Each production cell is a spot VM with two Blackwell GPUs running SGLang with a tuned configuration per model (tensor parallelism, chunked prefill, scheduler politics). The GLM-5.3-flash cell sustains ~412 tokens/sec on standard-tier traffic with 1M-token context enabled.
Scale from zero in seconds
Idle cells suspend to a CRIU checkpoint (GPU state snapshotted, VM released). The
first request after an idle period wakes the cell from checkpoint — minutes of GPU
billing avoided per idle hour, and the caller sees a clean 529 upstream_warming
with a Retry-After instead of a hang. The autoscaler keeps one cell warm per
published model during business-hours traffic patterns and lets everything else go
to zero.
Failure is normal
Spot VMs get preempted — we plan for it. Nodes register with the fleet host and heartbeat; a dead node's traffic re-routes to warm replicas while the autoscaler replaces capacity. Requests in flight on a dying cell are reversed (not charged) and the client gets a retryable error. The ledger invariant suite includes a kill-the-database-mid-settle test for exactly this class of failure.
What this means for your bill
- No vendor margin stacked on top — the fleet is the cost floor
- Cache-friendly routing: repeat prefixes tend to land on warm caches
- Honest 529s instead of silent queueing — you pay for delivered tokens only