Advanced RAG

The intuition

Classic RAG is a vending machine. You put in a query, it drops one set of chunks, and the generator has to make do — even when what came out is wrong. It never looks at what it dispensed and never tries again.

Two advanced families each replace a different part of that machine. Agentic RAG replaces fixed dispensing with a researcher: search, read, notice the miss, try a different query or source, stop when satisfied. Same index — different control flow. Graph RAG replaces the shelf of loose pages with a map of how things relate. Questions like "which of our vendors' suppliers are affected?" live only in connections between documents; "what are the main themes this quarter?" is a property of the whole corpus. No top-k of chunks contains either.

Why it exists

Plain RAG makes two assumptions that stay invisible until they break.

One retrieval is enough. The pipeline embeds, retrieves, and generates with no way to use what it learned from the chunks. That is fatal for multi-hop questions, where the second query is unknowable until you have read the first result. "Which team owns the service that caused last week's outage?" needs the service name first, then its owner. No single embedding of the original question retrieves both.

The corpus is a bag of independent chunks. Similarity search answers "what text resembles this text?" — nothing more. Two shapes fall through: relational multi-hop (the answer path crosses documents linked only by shared entities) and global / aggregative questions ("main themes across all support tickets this quarter?"), where the answer is a property of tens of thousands of documents and the ten most similar ones are not even the right kind of operation.

Agentic RAG breaks the first assumption. Graph RAG breaks the second. Query rewriting, multi-query, and HyDE still sit inside one-shot retrieval — they improve the vending machine's aim; they do not add a second attempt or a graph.

The core idea

Agentic RAG turns retrieval into a tool inside an agent loop. The model decides whether to retrieve, what to query (it writes and rewrites the search string), and whether the results are good enough. If not, it reformulates, hops, or escalates to another source. The pipeline becomes retrieve → grade → re-query or answer. Multiple LLM round-trips per question — justified when one-shot retrieval demonstrably fails on multi-hop or ambiguous traffic.

Graph RAG changes the data model. At ingest, an LLM extracts entities and relationships into a knowledge graph. Retrieval becomes graph traversal and/or queries over pre-computed community summaries — Microsoft GraphRAG's key move: cluster the graph, summarize each cluster at index time, answer global questions by map-reduce over summaries rather than raw chunks.

Vector RAG for "find the passage"; agentic when questions need iteration; graph when they need relationships or corpus-wide synthesis. Cost steps up roughly an order of magnitude with each move. Failure-class responses, not default upgrades.

How it actually works

Agentic mechanics

Expose the retriever as a tool — search_kb(query, filters) -> chunks — via function calling. The model emits a tool call; the runtime retrieves; results return as a tool message; the model answers or calls again. Four patterns you will meet in codebases and papers:

Routing picks among sources (index A vs B, SQL, web, or no retrieval). Most production value is here: only the hard fraction enters the loop. Iterative / multi-hop retrieval uses hop 1's answer to write hop 2's query — fixed pipelines cannot do this. Corrective RAG (CRAG) adds a lightweight retrieval evaluator that branches: correct → generate; incorrect → discard and fall back (rewrite, web search, refuse); ambiguous → blend and filter spans. Self-RAG (know the name) trains a model to emit reflection tokens that decide when to retrieve and whether each segment is evidence-supported. Production systems more often approximate that with a separate grader and harness caps, without fine-tuning those tokens.

Failure modes: unbounded loops (cap iterations and tokens), the same failing rephrase five times (detect no-progress), and latency that compounds because every hop is a full LLM round-trip.

Graph RAG mechanics

Microsoft's open-source GraphRAG is the reference most teams mean by the name. Index time: chunk → LLM-extract entities, typed relationships, and claims (schema optional) → merge duplicates into a graph → Leiden community detection (hierarchical clusters) → LLM summary per community. Query time, two primary modes:

• Local search — match entities in the query, traverse their neighbourhood (relations, claims, source chunks), answer from that subgraph. Entity-centric multi-hop.
• Global search — map-reduce over community summaries into one final answer. Edge et al. (2024) frame this as query-focused summarization over the whole corpus — the operation top-k chunk retrieval structurally cannot do.

DRIFT search mixes both: community context first, then local follow-ups. A quality/cost middle ground, not a third failure class.

Graph indexing makes LLM calls proportional to corpus size — typically an order of magnitude above embedding-only ingest — and the graph must stay fresh as documents change. Selection is by question shape, not "graphs are better."

The flows

Selected by question shape, not by which technique sounds advanced.

Vector RAG is the workhorse; a router sends the iterative or relational fraction elsewhere. Illustrative order-of-magnitude: vector is one retrieval + one generation; agentic multi-hop is N LLM calls and often several seconds; graph local is near-vector at query time but paid 10x+ at ingest; global map-reduce is heavier per query.

A worked example

One corpus — the governed enterprise platform's knowledge base of runbooks, incident reports, and ownership pages. Three questions, three shapes.

A — "What's the rollback procedure for the payments service?" Vector RAG. One embedding, top-k, generate. Answer sits in one runbook section. Illustrative: ~2 s, one LLM call. An agent here burns extra calls for the same result.

B — "Which team owns the service that caused last Tuesday's outage?" Vector RAG fails instructively. The query embeds near outage and ownership, so it retrieves the incident report or an ownership table — rarely both, and almost never the right ownership row, because the query does not contain the service name yet.

Hop 2's query could not have been written before hop 1 returned — that is the capability.

C — "What are the recurring causes of Sev-1 incidents this quarter?" Both above fail: there is no top-k that contains this answer. Hundreds of reports exist; the answer is a property of all of them. Graph RAG global search: at ingest, entities and relations were extracted, Leiden clustered the graph, and an LLM pre-wrote community summaries — one effectively "deploy-pipeline failures in payments." Query time maps and reduces those summaries. Aggregation happened once at ingest, not per query.

Production concerns

Cap agents hard. Two to three hops cover most multi-hop value; beyond that returns fall while latency and cost climb. Set token budgets and wall-clock timeouts. Log every hop with queries and chunk IDs — an unreplayable agent trace is undebuggable. See agent reliability.

The grader is a classifier. A lenient CRAG-style grader passes garbage (plain RAG plus an extra call); a harsh one triggers endless retries. Label (query, chunks) pairs, measure precision/recall, and monitor trigger rate against known retrieval failure rate — not 2% or 98%.

Graph freshness is harder than vector upserts. Updates need re-extraction, entity merge, and re-summarization of affected communities. Incremental patching is still immature in many GraphRAG stacks; common compromise is periodic rebuild over a slow-moving corpus, vector index for the rest. Evaluate extraction separately (entity/relation precision-recall) or you never notice silent fragmentation.

Hybrid is the realistic default. Vector for most traffic; agentic router for iterative questions; graph over the entity-rich slice. If eval shows one-shot retrieval already handles most real queries, this machinery is cost and surface for a thin tail — sometimes right, but by measurement, not fashion.

Common drill-downs

Self-RAG vs CRAG in production — which do people ship? CRAG-style: separate evaluator plus harness logic for rewrite, fallback, and caps. Self-RAG needs training reflection tokens into the generator; few teams do that when tool-calling loops already approximate retrieve-on-demand plus critique. Know Self-RAG for papers; ship the harness pattern.

How do you evaluate agentic RAG beyond final-answer quality? Trace metrics: retrieval precision/recall per hop, grader accuracy vs labels, loop-count and token-spend distributions, unnecessary-retrieval rate. Final-answer evals miss an agent that is right at 4× the needed cost.

10M docs, both "what does contract #123 say about termination?" and "what termination-clause patterns exist across all contracts?" Split by shape. Vector + metadata filter on contract ID for the first. The second is aggregation — GraphRAG-style global summaries, or a periodic offline map-reduce if a full graph is not justified. One retrieval strategy cannot serve both.

When does graph beat an agent for multi-hop? When hops ride stable typed relations extraction can capture ("supplier-of", "owned-by") and entity linking is reliable. When hops are ad-hoc English bridges that only appear after reading prose ("the service that caused the outage"), an agent that rewrites queries often wins without graph ingest. Measure both before committing.

Test yourself

Go deeper

• From Local to Global: A Graph RAG Approach to Query-Focused Summarization — Edge et al., 2024 — the Microsoft GraphRAG paper: extraction → Leiden communities → summaries → map-reduce.
• GraphRAG documentation — Microsoft — open-source implementation: indexing pipeline, local / global / DRIFT search modes.
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Asai et al., 2023 — retrieval-on-demand plus self-critique via reflection tokens.
• Corrective Retrieval Augmented Generation — Yan et al., 2024 — retrieval evaluator → {correct / incorrect / ambiguous} branching.
• What is Agentic RAG? — IBM Technology — concise popular explainer of the agent-loop framing.
• GraphRAG: The Marriage of Knowledge Graphs and RAG — Emil Eifrem (AI Engineer) — conference talk on when graphs add accuracy and explainability.

Where this connects

• Agent Foundations — the loop, terminating conditions, and state management that agentic RAG inherits wholesale.
• Agent Reliability — iteration caps, no-progress detection, and why 2–3 hops cover most of the value.
• RAG Evaluation — the measurement that tells you whether you have a failure class worth this much machinery.
• Design: Document Q&A at Scale — routing between question shapes, in a full system design walkthrough.