GyaanStack. Blog

Articles / raggenai-system-design

RAG architectures compared: naive, hybrid, agentic, GraphRAG and multimodal

Five RAG architectures side by side, with diagrams, latency and cost estimates, failure modes, how to evaluate each, and when to upgrade.

By Raghunath · · 13 min read

Most "types of RAG" lists name the patterns and stop there. That leaves the questions you face when building one: what each architecture costs per query, how it fails, how you would know it is working, and which metric should push you to the next one. This article answers those for five architectures, plus two research variants, so you can pick one on purpose.

The patterns are not levels you graduate through. They solve different problems, and in production they get combined: a hybrid retriever inside an agentic loop is common. I'll treat each one as a set of components you can add when a measured problem calls for it.

All latency and cost figures below are estimates for a typical setup (a hosted LLM, a managed vector index, a few million chunks). I show the reasoning so you can redo the math for your own stack.

The baseline numbers I use#

To compare architectures I need one unit. Mine is a single question answered with about 4,000 input tokens of retrieved context and 300 output tokens.

Step Typical latency (estimate) Why
Embed the query 10 to 50 ms One short input to an embedding model
Vector search (ANN) 5 to 50 ms HNSW-style index lookup, top 20 to 50
BM25 keyword search 5 to 30 ms Inverted index, no model call
Cross-encoder rerank of 50 passages 100 to 400 ms 50 short model passes, batched on a GPU
Small LLM call (rewrite, route, grade) 300 to 1,000 ms Short prompt, short output
Answer generation 1 to 5 s Dominated by output tokens and time to first token

Generation dominates. Retrieval tricks cost tens to hundreds of milliseconds; every extra LLM call costs most of a second. Keep that in mind when you read the agentic section.

1. Naive RAG#

question ──► embed ──► vector index ──► top-k chunks ──► prompt + LLM ──► answer

This is the setup from the original RAG paper (Lewis et al., 2020) reduced to its practical form: chunk documents, embed them, retrieve the top k by similarity, and paste them into the prompt.

What it solves. The model can answer from your documents instead of its training data, and you can cite the chunks.

When not to use it. When questions use exact identifiers (error codes, SKUs, legal clause numbers), when users phrase things very differently from the documents, or when answers need facts spread across many documents.

Latency and cost. One embedding call, one search, one LLM call. Roughly 1 to 5 seconds end to end, almost all of it generation. Cost is one LLM call with about 4,000 input tokens.

How it fails.

  • Exact-match misses. Dense embeddings blur "ERR-4012" and "ERR-4021" together.
  • Wrong chunk boundaries. The answer is split across two chunks and only one is retrieved.
  • Confident answers from weak context. Nothing checks that the retrieved text supports the answer.
  • Position effects. Models use information at the start and end of a long context better than the middle (Liu et al., 2023), so the order you paste chunks in changes answers.

How to evaluate it. Build 30 to 100 real questions, each labelled with the chunk or document that answers it. Measure recall@k and MRR for retrieval, then faithfulness (is every claim supported by the retrieved text?) and answer relevance for generation. RAGAS defines reference-free versions of the generation metrics if you have no gold answers yet.

Start here. Naive RAG with good chunking and a real eval set beats a fancy architecture with no measurement. The RAG cheat sheet covers the chunking and indexing defaults I use.

2. Hybrid RAG: keyword plus dense, with reranking and query rewriting#

                ┌──► BM25 top 50 ─────┐
question ──► (rewrite?) ──►           ├──► fuse (RRF) ──► rerank top 50 ──► top 5-8 ──► LLM ──► answer
                └──► vector top 50 ───┘

"Advanced RAG" in most lists means some mix of these three upgrades. They are independent; add the one your metrics point to.

Hybrid search. Run BM25 and vector search in parallel and merge the two ranked lists. Reciprocal rank fusion (Cormack et al., 2009) scores each document as the sum of 1/(k + rank) over the lists, with k = 60 in the paper. It needs no score normalization between the two lists, which makes it a simple default.

Reranking. Retrieve wide (top 50), then let a cross-encoder read each question-passage pair and reorder them. Keep the top 5 to 8. Cross-encoders see the question and passage together, so they catch relevance that a single embedding vector misses.

Query rewriting. Use a small LLM call to turn a chatty follow-up ("what about for contractors?") into a standalone query, or to split a compound question into parts. HyDE (Gao et al., 2022) is a variant: generate a hypothetical answer and embed that instead of the question.

What it solves. Exact-match misses (BM25), noisy top-k (reranker), and conversational or vague queries (rewriting).

When not to use it. Reranking a corpus of a few hundred short FAQ entries is wasted latency. Rewriting hurts when users already type precise queries; the rewrite can drop the one keyword that mattered.

Latency and cost. BM25 runs in parallel with vector search, so it adds almost nothing. The reranker adds 100 to 400 ms. A rewrite adds one small LLM call, 300 to 1,000 ms. Answer generation cost stays the same, and it may drop, because a reranked top 5 is often better than a raw top 10, so you can send fewer tokens.

How it fails.

  • The fusion buries a perfect BM25 hit under several mediocre vector hits. Check fused rankings on your exact-ID questions.
  • The reranker was trained on web text and misjudges your domain jargon.
  • Rewrites drift from the user's intent. Log the rewritten query next to the original so you can see it.

How to evaluate it. Same eval set, but compare variants: dense only, BM25 only, hybrid, hybrid plus rerank. Report recall@k and MRR per variant, and split the results by question type (exact identifier vs natural language). If hybrid does not beat dense on your identifier questions, something is wrong with tokenization or the fusion.

This is where I would put most production systems. If answers are wrong, check retrieval first: when the right passage never reaches the prompt, no prompt or model change can fix the answer.

3. Agentic RAG#

question ──► agent (LLM) ──► decide: answer / search / which source / ask user
                 ▲                │
                 │                ▼
                 └── observe ◄── tools: vector search, SQL, web, APIs
                                  │
                       (stop when enough evidence or budget hit) ──► answer with citations

An LLM decides at query time what to retrieve, from which source, and whether it has enough. The simplest form is a router that picks one index per question. The full form is a loop: search, read, decide to search again with a better query, then answer.

What it solves. Questions that need several lookups ("compare the refund policy for plan A and plan B"), questions that span structured and unstructured data (a SQL table plus policy documents), and corpora split across systems with different access rules.

When not to use it. When 80 percent of questions are single-hop lookups. A fixed pipeline answers those faster and more predictably. Also when you cannot afford variable latency: an agent's response time depends on how many steps it decides to take.

Latency and cost. Each loop iteration is at least one LLM call plus a retrieval. A typical answer takes 2 to 5 steps, so expect roughly 3 to 15 seconds and 2 to 5 times the tokens of a single-pass answer. That multiplier is the main reason to cap steps and spend.

How it fails.

  • Loops. The agent keeps searching with near-identical queries. Cap the number of steps and detect repeated queries.
  • Early stopping. It answers after one weak result. Ask it to state what evidence it has before answering, and grade that.
  • Wrong tool. It queries the vector store for something that lives in the SQL table. Tool descriptions decide this more than the model does.
  • Unbounded cost on adversarial or vague questions.

How to evaluate it. Final-answer metrics are not enough, because a right answer can come from a wasteful or lucky path. Also measure the trajectory: was the right tool called, how many steps, total tokens and cost per answer, and the rate of hitting the step cap. I wrote up a small harness for this in evaluating an AI agent in Python, and the support-agent and text-to-SQL designs in agentic system design show where agentic retrieval fits a larger system.

4. GraphRAG#

INDEXING (offline, expensive)
documents ──► chunks ──► LLM extracts entities + relationships ──► knowledge graph
                                                                        │
                                          community detection (Leiden) ◄┘
                                                    │
                                     LLM writes a summary per community

QUERY
local question  ──► find entities ──► neighbours + their source chunks ──► LLM ──► answer
global question ──► map over community summaries ──► reduce into one answer

Microsoft's GraphRAG (Edge et al., 2024; code) uses an LLM to extract an entity-relationship graph from the corpus, groups the graph into communities with the Leiden algorithm, and writes a summary for each community at several levels. Global questions are answered map-reduce style over those summaries.

What it solves. Two things plain chunk retrieval cannot do well. The first is global questions about the whole corpus ("what are the main themes in these 2,000 customer interviews?"), where no single chunk holds the answer. The second is multi-hop questions that follow relationships ("which suppliers of our top product had quality incidents last year?").

When not to use it. For lookup questions whose answer sits in one passage. Hybrid RAG answers those faster and cheaper. Also avoid it on corpora that change every hour, because the graph and summaries have to be rebuilt or patched.

Latency and cost. Indexing is where the cost goes. Entity extraction is at least one LLM call per chunk, often more if you run extra extraction passes, and then one call per community summary. For 10,000 chunks, that is on the order of 10,000 to 30,000 LLM calls before you answer a single question. Global queries are also heavy: the map step calls the LLM once per batch of community summaries. Local queries are closer to hybrid RAG in cost.

How it fails.

  • Extraction errors become graph errors. If the LLM merges two different people named "A. Kumar" into one node, every downstream answer inherits it.
  • Stale graphs. New documents are invisible until re-indexed.
  • Over-general answers. Community summaries can smooth away the specific detail the user wanted.

How to evaluate it. Keep two question sets. For global questions, compare answers against a hybrid RAG baseline with an LLM judge on comprehensiveness and diversity, which is the approach the GraphRAG paper uses. For multi-hop questions, write gold answers by hand and check exact facts. Also spot-check the graph itself: sample 50 extracted relationships and count how many are correct.

5. Multimodal RAG#

PDF / slides / scans
   │
   ├──► layout parser ──► text blocks ─────────► text embeddings ──┐
   ├──► tables ──► kept as structured rows/markdown ──► embeddings ─┼──► retrieve ──► vision-capable LLM ──► answer
   └──► page images / figures ──► image embeddings (or captions) ───┘

Real documents carry answers in tables, charts, diagrams and scanned pages. Text-only RAG loses most of that when the PDF is flattened to plain text.

There are two main approaches.

  • Parse and caption. A layout parser splits text, tables and figures. Tables are kept as structured rows; figures get captions from a vision model. Everything becomes text and goes through your normal pipeline.
  • Embed the page image. Models like ColPali (Faysse et al., 2024) embed page screenshots directly with a vision-language model, so retrieval works on what the page looks like, skipping OCR. A vision-capable LLM then reads the retrieved pages.

What it solves. Questions answered by a table cell, a chart value, or a diagram label. It also helps with scanned documents where OCR quality is poor.

When not to use it. When your documents are mostly clean prose. A layout-aware text parser gets most of the benefit at a fraction of the cost.

Latency and cost. Indexing costs a vision-model call per figure (captioning) or per page (image embedding). At query time, sending page images to a vision LLM uses far more input tokens than text chunks; image token counts depend on the provider and resolution, so measure on your own pages before committing.

How it fails.

  • Tables lose their header row when chunked, so "Q3 revenue" ends up next to the wrong number. Keep tables whole or repeat headers in every chunk.
  • Captions describe a chart in general terms and miss the one value the user asked about.
  • Citations point to a page but not to the region, so users cannot verify quickly.

How to evaluate it. Label questions by where the answer lives: prose, table, chart, image. Report accuracy per type. A single blended score hides the fact that table questions are failing. The PDF question-answering bot walk-through shows page-level citations, which this architecture needs from day one.

Two research variants worth knowing: Self-RAG and Corrective RAG#

Self-RAG (Asai et al., 2023) trains a model to emit special reflection tokens that decide whether to retrieve, judge whether retrieved passages are relevant, and critique whether its own output is supported. It needs a fine-tuned model, so most teams borrow the idea rather than the method: add an explicit "is this supported?" check after generation.

Corrective RAG (CRAG) (Yan et al., 2024) adds a lightweight evaluator that grades retrieved documents. If retrieval looks wrong, it falls back to web search; if ambiguous, it combines both. The practical version is a grading step between retrieval and generation that can trigger a retry or a refusal.

Both add one or more LLM or classifier calls per question, so they sit between hybrid and agentic RAG in cost.

Which architecture for which situation#

Situation Start with Add when
FAQ or docs site, questions answered by one passage Naive RAG Recall@5 below your target, or users search exact terms: add hybrid
Product codes, error IDs, legal clauses Hybrid (BM25 + dense) Right document retrieved but ranked low: add a reranker
Chat interface with follow-up questions Hybrid + query rewriting Rewrites drift: log and evaluate them separately
Questions that need several lookups or several systems Agentic RAG with a step cap Cost per answer too high: route single-hop questions to a fixed pipeline
"What are the themes across all of this?" GraphRAG (global search) Corpus changes often: schedule incremental re-indexing
Answers live in tables, charts, scans Multimodal (parse + caption first) Captions miss values: try page-image retrieval
High cost of a wrong answer (finance, health, legal) Hybrid + a grading step (CRAG-style) Grader blocks too much: tune its threshold on labelled cases

The upgrade path I follow#

Each step is triggered by a metric, never by a new framework release.

  1. Naive RAG plus an eval set. Without the eval set, none of the later steps can be judged.
  2. Retrieval recall is low on exact-term questions → add BM25 and fuse with RRF.
  3. Recall@50 is fine but recall@5 is low → the right chunk is retrieved but ranked badly; add a reranker.
  4. Follow-up questions fail → add query rewriting, and evaluate the rewrites.
  5. Faithfulness is low even when retrieval is good → fix the prompt and context order first; then add a support check after generation.
  6. Single-pass retrieval cannot answer multi-part questions → add an agentic loop for those questions only, routed by a classifier, with a step and cost cap.
  7. Users ask corpus-wide questions → add GraphRAG global search next to the existing pipeline, not instead of it.
  8. Answers live in tables and figures → add layout parsing, then page-image retrieval if needed.

For many products, steps 1 to 4 are enough. If you are preparing for interviews, being able to explain why you stopped there, with numbers, is worth more than listing all seven types. For sources to go deeper, I keep a reading order in free resources to learn RAG.

Sources#