Ingestion & Chunking

The intuition

Think of chunking as cutting a book into index cards, where each card must satisfy two people who want opposite things:

• The librarian (retrieval) wants small cards. One idea per card means the card is easy to file in exactly the right place and easy to find again. A card covering nine topics gets filed somewhere vague and found by nobody.
• The reader (the LLM) wants big cards. A card reading "Revenue grew 3%" is useless — whose revenue, which quarter, compared to what?

Every chunking strategy is a negotiated settlement between those two. And where you cut matters as much as how big the pieces are: cutting a sentence in half serves neither party.

Why it exists

Two hard constraints force chunking on you whether you want it or not:

1. An embedding is a fixed-size vector. Feed it a sentence, get 1024 floats. Feed it fifty pages, get 1024 floats. The second vector is an average of everything on those pages, so a specific question ("what's the notice period for contractors?") lands nowhere near it. Meaning gets averaged away.
2. The context window is a budget, and it is also your bill. Even if you could embed a whole document usefully, you can't afford to paste fifty pages into every prompt.

So documents must become retrieval-sized units. And the moment you cut, you create a new problem — each fragment loses the context it was sitting in — which is why half of this page is about putting that context back.

There's also a brutal practical reason this section matters more than it looks: retrieval can only surface what ingestion preserved. A shredded table is unanswerable no matter how good your reranker is, and it fails silently — your recall metrics just look mysteriously mediocre.

The core idea

Ingestion is the offline half of RAG: parse documents to clean text, split them into chunks, attach metadata, embed, and index. It's unglamorous and it's where most real-world RAG quality is won or lost — retrieval can only surface what ingestion preserved.

Parsing is the first trap. PDFs are a layout format, not a text format: text arrives in draw order, not reading order; two-column layouts interleave; tables flatten into word soup; headers/footers repeat into every chunk; scanned pages need OCR. Good pipelines use layout-aware parsers (or a vision model for hard documents) and preserve structure — headings, table boundaries, lists — because chunking wants those boundaries.

Chunking is a precision/context tradeoff. A chunk must be small enough that its embedding is specific (one topic → one clean vector, and retrieval returns only relevant text) but large enough that the LLM gets usable context (a lone sentence out of context is unanswerable). The strategies, cheapest first:

1. Fixed-size — every N tokens, blind to content. Baseline only; splits mid-sentence and mid-thought.
2. Recursive — try splitting on paragraphs, then sentences, then words until pieces fit the size budget. Respects natural boundaries at near-zero cost; the default that's genuinely hard to beat in benchmarks.
3. Structural / layout-aware — split on document structure: headings, sections, table boundaries, code blocks (Markdown/HTML/code-aware splitters). Wins whenever documents have structure — which is most enterprise content.
4. Semantic — embed sentences, cut where consecutive-sentence similarity drops. Content-adaptive but costs embedding calls at ingest and benchmarks inconsistently — it can fragment text into tiny chunks that retrieve poorly.

Typical numbers: 256–512 tokens per chunk as a starting point, 10–20% overlap, tuned by measuring retrieval recall on an eval set — never by intuition. And every chunk carries metadata (source, section, timestamp, ACL tags) because production retrieval is always filtered retrieval.

How it actually works

Why chunk size moves retrieval quality. An embedding is a fixed-size vector regardless of input length. Small chunks → each vector represents one idea → high retrieval precision, but the generator may lack surrounding context. Large chunks → each vector is an average of several topics → specific queries match diluted vectors → recall drops; and each retrieved chunk drags more irrelevant tokens into the prompt. This tension is fundamental, which is why "decouple what you match from what you send" patterns exist (small-to-big, parent-child — see retrieval-patterns.md).

Overlap repeats the tail of chunk n at the head of chunk n+1 so facts straddling a boundary survive in at least one chunk intact. It costs storage and can return near-duplicate chunks (dedupe at retrieval); past ~20% it's waste. Structural chunking needs less overlap because its boundaries are real topic boundaries.

The context-loss problem and its fixes. A chunk saying "Revenue grew 3%" is ambiguous — which company, which quarter? Chunking amputates the document context the embedding needed. Three fixes:

Tables and figures need special handling: never split a table mid-row; either keep it whole (Markdown-serialized), or generate an LLM text summary of the table and embed that with a pointer to the raw table. Same summarize-and-point trick for images/charts in multimodal corpora (see multimodal.md).

Metadata design at ingest — decide before indexing, because backfilling means re-processing the corpus:

• Identity: doc_id, chunk_id, source_uri, content hash (drives incremental re-indexing).
• Filterable: department, product, doc type, language, updated_at — everything you'll ever want in a WHERE clause.
• Security: ACL/tenant tags — permissions must be enforceable as retrieval filters (see production-rag.md and filtering-and-metadata.md).
• Provenance: page/section for citations and for fetching parent context.

The flows

Three operational paths through ingestion — the steady-state path, the change path, and the expensive full rebuild.

The operational model: parsing and metadata decisions are almost free to get right once and very expensive to reverse; chunk-boundary experiments always mean re-embedding.

A worked example

One page from an employee handbook, ingested three ways. The source looks like this:

## Leave Policy
### Annual Leave
Employees accrue 1.75 days per completed month of service. Unused
days carry forward to a maximum of 30.
### Parental Leave
Primary caregivers are entitled to 26 weeks at full pay. ...

Now the query: "how many leave days carry over?"

chunk 7:  "...Employees accrue 1.75 days per completed month of service. Unused"
chunk 8:  "days carry forward to a maximum of 30. ### Parental Leave Primary..."

The lesson generalizes: most of the win comes from not destroying structure, not from clever algorithms. Semantic chunking and contextual retrieval are real improvements on top of this — but they're improvements on top of this, not substitutes for it.

Production concerns

• Parsing is the top silent failure. Retrieval metrics look "mysteriously bad" because chunks contain interleaved columns or shredded tables. Eyeball random chunks per document type before ever tuning embeddings; add per-source parse-quality checks (extracted-text length vs page count, table count).
• Ingest cost scales with cleverness: recursive splitting is CPU-only; semantic chunking adds an embedding pass; contextual retrieval adds an LLM pass over the whole corpus. At 10M documents that ordering is the difference between an afternoon and a real budget line — reserve LLM-per-chunk work for high-value corpora.
• Idempotency and incremental updates: hash each doc; re-parse → re-chunk → re-embed only changed docs, and delete the old chunks (orphaned stale chunks are a classic wrong-answer source). Chunk IDs deterministic from (doc_id, position) keep updates clean.
• One corpus, many formats: route by type — structural splitter for Markdown/HTML, code-aware for source files, layout parser for PDF. A single one-size splitter across mixed content is a common root cause of bimodal retrieval quality.
• Changing chunking = full re-index. Chunk experiments require re-embedding the corpus, so build the offline eval harness (retrieval recall on a golden set) first, and version your index so you can A/B old vs new chunking safely. (See rag-evaluation.md.)

Common drill-downs

How do you choose chunk size? Start at 256–512 tokens with 10–15% overlap, then treat it as a measured hyperparameter: build a query→relevant-chunk eval set and sweep size against recall@k and answer quality. Push smaller when queries are factoid and precision matters; larger when answers need synthesis. State the tradeoff: small = precise vectors but starved context; large = diluted vectors and bloated prompts.

Fixed vs recursive vs semantic chunking — when does each win? Fixed: never, except as a baseline. Recursive: the default — respects sentence/paragraph boundaries, free, benchmarks near the top. Structural: wins when documents have exploitable structure (headings, HTML, code) — usually the best real-world choice. Semantic: only when content has no structure and topics shift unpredictably (transcripts, chat logs); costs embeddings at ingest and can over-fragment.

Why is PDF parsing hard? PDF encodes glyph positions, not semantics. Reading order must be inferred (multi-column breaks naive extractors), tables are just positioned text with no row/column markup, headers/footers repeat into content, and scanned PDFs are images requiring OCR. That's why layout-aware parsers and increasingly vision-LLM parsing exist as a distinct pipeline stage.

What do you do with tables in RAG? Never let the splitter cut one. Keep the table atomic (Markdown-serialized so the LLM reads it), and if it's large, embed an LLM-generated summary of the table while storing the full table as the payload to send at generation time. Row-level questions over big tables are often better served by text-to-SQL than by embedding rows.

What is overlap for, and what's the downside? It preserves facts that straddle chunk boundaries so at least one chunk holds them whole. Downsides: storage/embedding cost and near-duplicate retrieval results crowding the top-k (dedupe before prompting). 10–20% suffices; structure-aware boundaries need less.

A chunk in isolation loses document context — how do you fix that? Cheapest: prepend the heading path. Better: contextual retrieval — LLM-generate a short situating blurb per chunk, prepend before embedding and BM25 indexing (Anthropic: ~49% retrieval-failure reduction combined with hybrid search, ~$1 per million doc tokens with caching). Alternative: late chunking with a long-context embedding model.

What metadata do you attach at ingest and why then? Source/IDs for citations and re-indexing, filterable fields for scoping queries, ACL tags for security-by-filter, timestamps for freshness/recency ranking. At ingest because filters only work if every chunk carries the fields — backfilling means reprocessing the corpus.

How do you evaluate a chunking change? Offline: golden set of (query, relevant-doc/passage) pairs; measure recall@k, MRR, and token-level overlap of retrieved vs relevant text before/after (document-level hit rate hides partial-chunk misses). Then end-to-end answer quality on the same queries, since chunking changes what the generator sees. Ship behind an index version so rollback is instant.

Test yourself

Go deeper

• Chunking Strategies for LLM Applications — Pinecone — the standard survey of methods and selection criteria.
• Evaluating Chunking Strategies for Retrieval — Chroma Research — token-level eval methodology; shows recall swings up to ~9% from chunking alone and that popular defaults are suboptimal.
• Introducing Contextual Retrieval — Anthropic — the contextual-chunk technique, failure-rate numbers, and cost math.
• Building Performant RAG Applications for Production — LlamaIndex — decoupling retrieval chunks from synthesis chunks, structured retrieval.
• The 5 Levels Of Text Splitting For Retrieval — Greg Kamradt — the widely-cited walkthrough from character splits to semantic and agentic chunking, with code.

Where this connects

• Retrieval Patterns — parent-child and contextual retrieval are the direct answers to the context-loss problem introduced here.
• Filtering & Metadata — the metadata you design here is what makes filtered retrieval possible at query time.
• RAG Evaluation — the harness that has to exist before any chunking experiment means anything.
• Multimodal — when parsing hits scanned pages, charts, and diagrams, ingestion becomes a vision problem.
• Production RAG — incremental re-indexing, deletes, and the document lifecycle around this pipeline.