Optimize retrieval, prompt size, model choice, caching, and streaming — and lean on the inference-level wins too, like KV cache and batching.

Latency comes from network calls, retrieval, reranking, model generation, and tool use. Concrete levers: use smaller models for simple classification, extraction, routing, or formatting rather than the largest model for every task; shrink prompts by removing unnecessary instructions, summarizing history, and retrieving only relevant chunks; limit output length with a max-token cap; stream the response so the user sees output earlier, which improves perceived latency even when total generation time is unchanged; rely on KV cache so autoregressive generation reuses previously computed Key/Value vectors instead of recomputing the whole context on every token; batch requests where possible for throughput; and route simple tasks to a small model while reserving a larger model for complex or high-risk cases. Agent-heavy systems should also cap the number of tool-call loops, since repeated tool calls silently add latency on top of generation time.