What actually drives RAG latency.
If your retrieval layer is slow, the LLM call on top makes the whole experience feel broken. Latency varies a lot by corpus, query mix, and the LLM you choose, so this page focuses on the stages that add up to a response time—and how CogniWeave is architected to keep cited retrieval fast—rather than quoting benchmark numbers that wouldn't generalize to your workload.
The stages that add up to RAG latency
- • Query embedding: turning the question into a vector.
- • Retrieval: approximate nearest-neighbor search over the index.
- • Reranking (optional): scoring the shortlist with a stronger model.
- • Prompt assembly: stitching cited passages into the LLM context.
- • Answer generation: the LLM call—usually the largest single component.
- • Network + serialization: client to API and back.
What drives the numbers (qualitatively)
Retrieval latency grows sublinearly with corpus size on a well-built ANN index, so doubling your corpus does not double query time. Embedding latency is roughly constant per query. Reranking trades a small latency cost for higher precision. The LLM call dominates total wall-clock time for most assistants—pick the fastest model that meets your quality bar.
How CogniWeave is architected for fast cited retrieval
- • Ingest-time work (parsing, chunking, embedding, consolidation) is done once, so query-time work stays small.
- • Compute, index, and embedding live in the same region to remove cross-region hops.
- • Active workspaces are kept warm so production queries hit the hot path.
- • Citations are produced during retrieval—no extra round-trip to resolve sources.
Illustrative ranges (industry, not CogniWeave benchmarks)
- • Query embedding for short questions: typically single-digit to low double-digit milliseconds on managed APIs.
- • ANN search on a well-tuned in-memory index: typically tens of milliseconds at moderate corpus sizes.
- • Cross-encoder reranking of a shortlist: typically tens of milliseconds.
- • LLM answer generation: hundreds of milliseconds to several seconds, depending on the model and tokens generated.
- • These are general industry ranges to set expectations—not measured CogniWeave results.
How to measure your own latency
- • Time each stage independently on a representative sample of real queries.
- • Track p50, p95, and p99—averages hide tail behavior that users feel.
- • Separate warm and cold queries; report them distinctly.
- • Measure end-to-end from the client, not just server-side, to include network.
Common questions
Does CogniWeave publish measured benchmark numbers?
Not on this page. End-to-end latency depends on your corpus size, query distribution, network path, and the LLM you call on top of retrieval. We describe what drives latency and how CogniWeave is architected for fast cited retrieval, rather than quoting numbers that wouldn't generalize to your workload.
What dominates RAG latency in practice?
For most workloads the largest single component is the LLM call that composes the final answer. Inside the retrieval layer, approximate nearest-neighbor search and any reranking step are the next largest contributors; query embedding and serialization are typically minor.
How does cold start affect a first query?
A first query after an idle period is generally slower than warm queries because the index has to page into memory. CogniWeave keeps active workspaces warm so production traffic typically sees warm-path latency.
How should I measure latency for my own workload?
Time each stage independently—query embedding, retrieval, reranking, prompt assembly, LLM generation, network—on a representative sample of real queries. Whole-pipeline averages hide which stage to optimize.