Intermediate12 min
Ship RAG — retrieval on you, answers on us
Embed + rerank with your own retrieval stack, then synthesize answers on a Flash model. (runaii's /v1/embeddings + /v1/rerank are not live yet — check GET /api/v1/models for what's callable today.)
1. Embed your docs (your stack)
Chunk to ~512 tokens and embed with any embedding model you host or subscribe to — qwen3-embedding-8b runs well on a single GPU VM. Store vectors in Postgres + pgvector.
# your embedding step — e.g. self-hosted # qwen3-embedding-8b on a runaii GPU VM emb = embed(chunks) # -> pgvector
2. Retrieve, then rerank
Pull top-20 by cosine, rerank to top-5 with a cross-encoder (same box works). Rerank quality is where most RAG wins hide.
docs = vec_top20(q) top5 = reranker(q, docs) # cross-encoder, same VM
3. Answer cheap — on runaii
Feed the 5 passages to qwen3.8-flash. Cache the system prompt + corpus prefix — repeat questions bill at cached rates.
resp = client.chat.completions.create(
model="runaii/qwen3.8-flash",
messages=[{"role": "system", "content": SYS},
{"role": "user", "content": q + ctx}])