ModelRefs / How to choose an AI model for RAG
How to choose an AI model for RAG
A practical framework for selecting a generation model for retrieval-augmented generation based on task fit, grounding, evaluation, latency, cost, and deployment constraints.
Overview
The generation model is one component in a RAG pipeline. A stronger general-purpose model cannot compensate for irrelevant retrieval, missing permissions, poor chunking, or an evaluation set that does not represent production questions.
Who this guide is for
Teams selecting a language model for a new or existing RAG system, especially when retrieval quality, grounded answers, operating cost, and deployment constraints matter more than headline model capability.
The generation model is one component in a RAG pipeline. A stronger general-purpose model cannot compensate for irrelevant retrieval, missing permissions, poor chunking, or an evaluation set that does not represent production questions.
Decision framework
Define the answer task
Separate extraction, synthesis, comparison, classification, and multi-step reasoning. Record required output structure and acceptable abstention behavior.
Measure retrieval independently
Check whether relevant passages reach the prompt before judging generation. Track retrieval recall, ranking quality, and permission filtering.
Set context requirements
Estimate retrieved passage volume, instructions, conversation history, and output length. Treat context-window size as a capacity limit, not proof of effective long-context use.
Test grounding and citations
Evaluate whether answers stay within supplied evidence, cite the correct passages, and decline when evidence is insufficient.
Evaluate reasoning and tool support
Use representative multi-document questions and structured-output tests. Include function or tool calls only when the workflow actually depends on them.
Measure latency and cost
Capture time to first token, total response time, input/output volume, retries, and retrieval overhead under realistic concurrency.
Check deployment constraints
Review data handling, regional availability, hosting model, observability, quotas, fallback options, and operational ownership.
Run a controlled comparison
Use the same retrieval results, prompts, scoring rubric, and failure review across candidates before choosing a rollout path.
Trade-offs to weigh
Capability versus operating cost
More capable models may improve difficult synthesis while increasing latency and cost. Route simple and complex questions differently when evidence supports it.
Large context versus retrieval discipline
Sending more context can reduce omission risk but may add noise, cost, and slower responses. Better ranking can matter more than a larger window.
Managed API versus controlled deployment
Managed services reduce infrastructure work; controlled hosting can offer different privacy and customization options but adds operational responsibility.
Fluent answers versus calibrated abstention
A useful RAG system must identify missing evidence. Evaluate refusal and uncertainty behavior alongside answer quality.
Requirements
Document the target questions, data sensitivity, answer format, grounding expectations, latency budget, cost envelope, tool requirements, concurrency, and deployment constraints before comparing candidates.
Candidate model categories
Start with language models that support the required context, output controls, deployment path, and language coverage. Add reasoning-oriented or smaller models only when task tests justify the additional branch.
Evaluation criteria
Score grounded correctness, citation accuracy, completeness, abstention, structured-output adherence, latency, cost, and operational failures on a representative question set.
Selection process
Shortlist by hard constraints, compare on fixed retrieval results, inspect failures, test end-to-end behavior, then stage a monitored rollout with a documented fallback.
Limitations and coverage notes
The framework is model-neutral and provisional. It does not provide current scores, prices, or a universal winner.
Sources and methodology
Source coverage is expanding. This guide remains provisional while evidence and editorial review mature.
Limitations and method
This guide triangulates the foundational RAG paper, an official engineering treatment of advanced retrieval pipelines, and official evaluation guidance. It converts those sources into a provider-neutral decision checklist; it does not compare model performance or reproduce mutable limits and prices.
- This guide does not rank individual models or reflect current provider pricing and service limits.
- RAG quality depends on retrieval, data preparation, prompting, and evaluation—not only the generation model.
- Model behavior should be tested on representative private data before implementation decisions are finalized.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks arXiv · accessed 2026-06-22
- Build Advanced Retrieval-Augmented Generation Systems Microsoft Learn · accessed 2026-06-22
- Evaluation best practices OpenAI · accessed 2026-06-22
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to How to choose an AI model for RAG.