Prompt patterns that make RAG actually grounded.
The retrieval layer pulls the right passages. The prompt decides whether the model uses them. Here are the patterns that hold up across Claude, GPT, and Llama in production.
Basic pattern
System prompt defines the role and grounding rule. User prompt injects retrieved context followed by the question. Output instructions are tight: cite, decline when unsupported, no synthesis from prior knowledge.
Advanced patterns
- • Explicit citation: ask for `[source:document_name:page]` markers.
- • Confidence scoring: ask for a 0–100 confidence based on context coverage.
- • Multi-turn memory: retrieve fresh context every turn; never reuse stale passages.
- • Fallback: instruct the model to say 'Not found in documents' instead of hallucinating.
- • Reranking signal: ask the model to rate passage relevance before using it.
Example prompts
- • Finance assistant: 'Answer questions about company policies using only provided documents. Cite each fact.'
- • Legal assistant: 'Extract key terms from the contract. For each term, cite the relevant clause.'
- • Code assistant: 'Explain this codebase feature using only the provided source files. Cite line numbers.'
Chunking for prompts
- • Small chunks (~128 tokens): higher precision, more context switches.
- • Large chunks (~512 tokens): fewer fragments, more noise per fragment.
- • Hybrid: retrieve small, expand to parent chunks before injection.
Long-context handling
Compress earlier turns into a running summary once they exceed ~30% of the window. Re-retrieve fresh context each turn instead of carrying it forward; passages become stale quickly in conversational flow.
Anti-patterns
- • Don't ask the model to synthesize facts not present in the context.
- • Don't silently drop low-relevance passages; either keep with a low-confidence flag or remove with logging.
- • Don't reuse a single mega-prompt for every task—task-specific system prompts measurably outperform.
Common questions
How do I get structured JSON out of a RAG response?
Ask for JSON explicitly in the system prompt, provide a schema, and validate with Zod or Pydantic on the way out. Retry once with the validation error appended if parsing fails.
How do I prevent the model from citing irrelevant passages?
Filter retrieval results by final_score before injection, and instruct the model to omit citations when no passage supports a claim.
How do I count tokens for a multi-passage prompt?
Sum tokens for system, each passage, the question, and the formatting wrappers. Use the provider's tokenizer—token counts vary across Claude, GPT, and Llama.
Should the LLM rerank passages itself?
Only as a fallback. A dedicated cross-encoder is faster and cheaper; reserve LLM reranking for high-stakes queries where the extra latency is acceptable.