Advanced RAG

The intuition

Classic RAG is a vending machine. You put in a query, it drops one set of chunks, and the generator has to make do — even when what came out is wrong. It never looks at what it dispensed and never tries again.

The two advanced families each replace a different part of that machine:

• Agentic RAG replaces the fixed dispensing with a researcher. A researcher searches, reads what came back, notices it's not the right thing, searches differently, follows a lead from the first result into a second search, and stops when satisfied. Same index — completely different control flow.
• Graph RAG replaces the shelf of loose pages with a map of how things relate. Some questions aren't about finding a passage at all: "which of our vendors' suppliers are affected?" has an answer that exists only in the connections between documents, and "what are the main themes this quarter?" has an answer that is a property of the whole corpus. No top-k of chunks contains either.

Why it exists

Plain RAG makes two assumptions that are invisible until they break.

Assumption 1: one retrieval is enough. It embeds the query, gets chunks, and generates — regardless of whether those chunks are any good, and with no way to use what it learned from them. That's fatal for multi-hop questions, where the second query is unknowable until you've read the first result. "Which team owns the service that caused last week's outage?" requires finding the service, then looking up its owner. No single embedding of the original question retrieves both.

Assumption 2: the corpus is a bag of independent chunks. Similarity search answers "what text resembles this text?" — nothing more. Two question shapes fall straight through:

• Relational / multi-hop: the answer path crosses documents linked only by shared entities.
• Global / aggregative: "what are the main themes across all support tickets this quarter?" The answer is a property of 40 000 documents. Retrieving the 10 most similar ones doesn't approximate it — it isn't even the right kind of operation.

Agentic RAG breaks assumption 1. Graph RAG breaks assumption 2.

The core idea

Classic RAG is a fixed one-shot pipeline: retrieve once, generate once, done — even if retrieval returned garbage. Two families of advanced RAG relax different assumptions:

Agentic RAG relaxes the control flow. Retrieval becomes a tool the LLM can call zero, one, or many times inside an agent loop. The model decides whether to retrieve (skip it for "summarize what I just pasted"), what to query (it writes and rewrites the search query itself), and whether the results are good enough — if not, it reformulates and retrieves again, or escalates to another source (different index, SQL, web search). The pipeline becomes a loop with self-correction: retrieve → grade → re-query or answer. Cost: multiple LLM calls per question, so latency and spend multiply; it pays for multi-hop and ambiguous questions where one-shot retrieval demonstrably fails.

Graph RAG relaxes the data model. Vector RAG treats the corpus as independent chunks and answers "find me text similar to this query." It structurally fails at two question types: multi-hop ("which suppliers of our vendors are affected by the sanction?" — the answer spans linked documents no single chunk covers) and aggregation/global ("what are the main themes across all support tickets this quarter?" — the answer is a property of the whole corpus, not of any top-k chunks). Graph RAG builds a knowledge graph at ingest — LLM extracts entities and relationships from chunks — and retrieval becomes graph traversal and/or querying pre-computed community summaries (Microsoft GraphRAG's key move: cluster the graph, summarize each cluster with an LLM at index time, answer global questions map-reduce over summaries).

Positioning line: these aren't upgrades you apply by default — they're responses to specific failure classes. Vector RAG for "find the passage"; agentic RAG when questions need iteration and judgment; graph RAG when questions need relationships or corpus-wide synthesis. Cost and complexity go up an order of magnitude with each step.

How it actually works

Agentic RAG mechanics. The retriever is exposed via tool/function calling: search_kb(query, filters) -> chunks. The agent loop: model emits a tool call → runtime executes retrieval → results go back as a tool message → model either answers or calls again with a refined query. The patterns worth naming:

• Routing — the model (or a cheap classifier) picks among sources: vector index A vs B, SQL, web search, or no retrieval at all.
• Iterative / multi-hop retrieval — answer to hop 1 ("who acquired X?") parameterizes the query for hop 2 ("that acquirer's CEO"). Fixed pipelines cannot do this; the loop makes retrieval conditional on intermediate reasoning.
• Self-correction (CRAG-style) — a grader (small LLM or classifier) scores retrieved docs for relevance; on low confidence: re-write the query, retry, fall back to web search, or decline to answer. Corrective RAG formalized this: lightweight retrieval evaluator → {correct, incorrect, ambiguous} → different action per verdict.
• Self-RAG (research, know the name) — a model trained to emit reflection tokens deciding when to retrieve and critiquing whether each generated segment is supported by the evidence.

Failure modes: loops that never terminate (cap iterations, budget tokens), the agent rephrasing the same failing query five times, and compounding latency — every hop is a full LLM round-trip.

Microsoft GraphRAG pipeline, concretely. Index time: (1) chunk the corpus; (2) LLM extracts entities and typed relationships per chunk (no schema required — prompted extraction), plus claims; (3) build the entity-relationship graph and merge duplicate entities; (4) run Leiden community detection to cluster the graph hierarchically (communities of communities); (5) LLM writes a summary for every community at every level. Query time, two modes: local search — locate entities matching the query, traverse their neighborhood (relations, claims, source chunks) and answer from that subgraph; global search — map-reduce: generate partial answers from each relevant community summary, then reduce them into one final answer. The paper (Edge et al., 2024) frames this as query-focused summarization over the whole corpus, which top-k chunk retrieval fundamentally can't do.

Costs to quote. GraphRAG indexing makes LLM calls proportional to corpus size (extraction per chunk + summarization per community) — an order of magnitude above embedding-only ingest — and the graph must be maintained as documents change. Global search touches many summaries per query. This is why the decision rule is question-shape-driven, not "graphs are better."

The flows

Three control flows, selected by question shape — not by which technique sounds more advanced.

The operational model: vector RAG is the workhorse; a router sends the small fraction of iterative or relational questions elsewhere. These are failure-class responses, not default upgrades.

A worked example

One corpus — the governed enterprise platform's internal knowledge base of runbooks, incident reports, and service ownership pages. Three questions of increasing shape difficulty, and what each architecture does with them.

Question A — "What's the rollback procedure for the payments service?"

Vector RAG. One embedding, top-k, generate. The answer sits in one runbook section. Total: ~2 s, one LLM call. An agent here would burn three extra calls to arrive at the same place.

Question B — "Which team owns the service that caused last Tuesday's outage?"

Vector RAG fails, and instructively. The query embeds near outage, ownership, and Tuesday — so it retrieves the incident report or an ownership table, rarely both, and never the right ownership row, because the query doesn't contain the service name yet. The service name is the missing link and it's only discoverable from hop 1.

Agentic RAG:

Two retrievals, three LLM calls, ~6 s. Hop 2's query literally could not have been written before hop 1 returned — that's the capability, stated in one sentence.

Question C — "What are the recurring causes of Sev-1 incidents this quarter?"

Both of the above fail, and for the same reason: there is no top-k that contains this answer. 340 incident reports exist; the answer is a property of all of them. Retrieve the 10 most similar to the query and you get 10 arbitrary incidents plus a model confidently generalizing from a 3% sample.

Graph RAG global search: at ingest, entities (services, causes, teams) and relations were extracted from all 340 reports, Leiden clustered the graph, and an LLM pre-wrote a summary per community — one of which is effectively "deploy-pipeline-related failures in payments services." At query time, map over the relevant community summaries, reduce into one answer. The expensive aggregation happened once at ingest, not per query.

Production concerns

• Latency/cost ceilings on agents: hard-cap loop iterations (2–3 hops covers most value), set token budgets, and stream a partial answer if the budget is hit. Log every hop — an agent trace you can't replay is undebuggable. (See agent-reliability.md.)
• Grader quality is the hinge of self-correction: a lenient grader passes garbage (no better than naive RAG); a harsh one triggers endless retries (cost blowup). Tune the relevance grader on labeled examples like any classifier, and monitor its trigger rate as a production metric.
• Graph freshness: document updates require re-extracting entities, patching the graph, and re-summarizing affected communities — incremental updates are much harder than upserting vectors. Common compromise: nightly/weekly graph rebuilds over a slow-moving corpus, vector index for the fast-moving remainder.
• Extraction quality drift: entity dedup errors ("IBM" vs "International Business Machines" as two nodes) silently fragment the graph and break multi-hop answers; evaluate extraction (entity/relation precision-recall on a sample) separately from end-to-end answers.
• Hybrid architectures are the realistic answer: most production systems keep vector search as the workhorse and add (a) an agentic router in front and/or (b) a graph over the entity-rich slice of the corpus. Nobody should replace a working vector pipeline with a graph wholesale — that judgment is the distinction that matters in practice.
• When NOT to use these: if eval says one-shot retrieval already answers 95% of real queries, agentic/graph machinery adds cost, latency, and failure surface for the remaining 5% — sometimes rightly, but by measurement, not fashion.

Common drill-downs

What makes RAG "agentic," concretely? Retrieval moves from a fixed pipeline stage to a tool inside an LLM loop. The model decides whether to retrieve, writes its own queries, judges result quality, and iterates — retrieve → grade → refine → retrieve again → answer. The capability unlocked is conditional retrieval: hop 2's query depends on hop 1's answer.

When does one-shot RAG fail where an agent succeeds? Multi-hop questions ("Which team owns the service that caused last week's outage?" — first find the service, then its owner), ambiguous queries needing reformulation after seeing bad first results, and questions mixing sources (docs + SQL + web). Anything where the right second query is unknowable until you've seen the first results.

Walk me through what Microsoft GraphRAG actually builds. Ingest: LLM extracts entities + relationships per chunk → knowledge graph → Leiden algorithm clusters it hierarchically → LLM pre-writes a summary per community. Query: local search traverses a matched entity's neighborhood; global search map-reduces over community summaries for corpus-wide questions. The pre-computed summaries are the trick — "what are the main themes?" becomes reading summaries, not scanning millions of chunks.

When does graph beat vector retrieval? Two shapes: multi-hop relational questions, where the answer path crosses documents connected only through shared entities; and aggregation/sensemaking questions, where the answer is a global property no top-k chunks contain. For "find the relevant passage," vector search wins on cost, simplicity, and freshness — the honest answer includes that half.

Costs and risks of GraphRAG? Indexing: LLM calls scale with corpus size (extraction + community summaries) — 10x+ embedding-only ingest. Maintenance: updates need graph patching and re-summarization; most teams rebuild periodically. Quality risk: entity extraction/dedup errors fragment the graph and silently break traversals. Only justified when eval shows relational/global questions failing on vector RAG.

What is corrective RAG (CRAG) / self-correction in one minute? A lightweight evaluator grades retrieved docs before generation. Confident-correct → generate; incorrect → discard, fall back (query rewrite, web search); ambiguous → blend both, filtering irrelevant spans. It converts "retrieval silently failed" into an explicit branch. Failure knobs: grader calibration and retry caps.

How do you evaluate an agentic RAG system beyond final-answer quality? Trace-level metrics: retrieval-tool precision/recall per hop, grader accuracy vs labels, loop count and token spend distributions, and unnecessary-retrieval rate (retrieved when parametric answer sufficed). Final-answer evals miss an agent that gets right answers at 4x the needed cost.

You have 10M docs and users ask both "what does contract #123 say about termination?" and "what termination-clause patterns do we have across all contracts?" Split by question shape: vector (+ metadata filter on contract ID) for the first; the second is aggregation — GraphRAG-style global summaries over clause entities, or a periodic offline map-reduce summarization job if a full graph isn't justified. A router classifies the query shape up front. One retrieval strategy cannot serve both.

Test yourself

Go deeper

• From Local to Global: A Graph RAG Approach to Query-Focused Summarization — Edge et al., 2024 — the Microsoft GraphRAG paper: extraction → Leiden communities → summaries → map-reduce.
• GraphRAG documentation — Microsoft — the open-source implementation: indexing pipeline, local/global/DRIFT search modes.
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Asai et al., 2023 — retrieval-on-demand plus self-critique via reflection tokens.
• Corrective Retrieval Augmented Generation — Yan et al., 2024 — the retrieval-evaluator → {correct/incorrect/ambiguous} branching pattern.
• What is Agentic RAG? — IBM Technology — concise popular explainer of the agent-loop framing.
• GraphRAG: The Marriage of Knowledge Graphs and RAG — Emil Eifrem (AI Engineer) — 19-minute conference talk on when graphs add accuracy and explainability.

Where this connects

• Agent Foundations — the loop, terminating conditions, and state management that agentic RAG inherits wholesale.
• Agent Reliability — iteration caps, no-progress detection, and the compounding-error maths behind "2–3 hops covers most of the value".
• RAG Evaluation — the measurement that tells you whether you have a failure class worth this much machinery.
• Design: Document Q&A at Scale — routing between question shapes, in a full system design walkthrough.