Streaming

Streaming requests

Set "stream": true and consume server-sent events exactly like the OpenAI API:

from openai import OpenAI

client = OpenAI(
    api_key="sk-runaii-...",                    # your runaii key
    base_url="https://api.runaii.cloud/v1",     # ← the only change
)

stream = client.chat.completions.create(
    model="glm-5.3",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="")

The terminal usage chunk

When the client sets stream_options: {"include_usage": true}, runaii emits one extra chunk after the final content chunk — and before [DONE] — carrying the full usage and cost for the request (OpenAI-compatible: usage is only sent when requested):

{
  "id": "rq-...",
  "object": "chat.completion.chunk",
  "choices": [],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 32,
    "total_tokens": 46,
    "prompt_tokens_details": { "cached_tokens": 0 },
    "cost": 0.000017
  }
}

This means streaming clients can display spend in real time without a second API call — the same usage fields as non-streaming responses.

Billing on streams

Streaming requests are billed identically to non-streaming: estimate reserved at start, settled from the provider-reported token counts at stream end, refund of any unused reservation — all in one transaction. If your client disconnects mid-stream, the reservation is reversed; partial delivery is never charged.

Errors mid-stream

If the upstream cell dies mid-stream, the stream ends with a retryable error chunk and the request is fully refunded (usage.cost will not appear — nothing was charged). Safe pattern: catch the error and re-issue; the ledger guarantees you were not double-charged.

Anthropic-dialect streaming

stream: true on POST /api/messages returns native Anthropic SSE events — the Anthropic SDK consumes it with zero configuration:

event: message_start        → the message envelope (usage fills in later)
event: content_block_start  → a text / thinking / tool_use block opens
event: content_block_delta  → text_delta | thinking_delta | input_json_delta
event: content_block_stop   → block closes
event: message_delta        → stop_reason + final output_tokens
event: message_stop         → done

Details worth knowing:

• Tool use streams as tool_use blocks: the model's tool call arrives as a content_block_start ({type:"tool_use", id, name, input:{}}) followed by input_json_delta frames carrying the argument JSON incrementally.
• Truncated output is never a clean finish: if the upstream breaks mid-stream, stop_reason arrives as max_tokens (the Anthropic incomplete-output signal) rather than a misleading end_turn.
• Keep-alive comments (: runaii keep-alive) may appear during long prefills — every spec-compliant SSE parser ignores them.

Responses-dialect streaming

stream: true on POST /v1/responses returns the native Responses event sequence — the OpenAI SDK / Agents SDK consume it with zero configuration:

event: response.created            → the response envelope (status: in_progress)
event: response.in_progress
event: response.output_item.added  → a message / function_call item opens
event: response.content_part.added → (messages) the output_text part opens
event: response.output_text.delta  → text fragments
event: response.output_text.done   → full text for the part
event: response.content_part.done
event: response.output_item.done   → item closes (status: completed)
event: response.function_call_arguments.delta / .done  → (tool calls)
event: response.completed          → final status + usage (input/output/total)

Notes:

• Usage rides response.completed — input_tokens, output_tokens, total_tokens — the same numbers the other dialects report, from the same metered settle.
• Truncated output is signaled, not hidden: if the upstream breaks or the budget caps the answer, the final envelope carries status: "incomplete" with incomplete_details: { reason: "max_output_tokens" }.
• Tool calls stream as function_call output items with incremental function_call_arguments.delta frames — the round-trip (function_call / function_call_output in the next request's input) mirrors the chat-completions tool_calls / role:"tool" flow.