Ingestion & Chunking

The intuition

Chunking is cutting a book into index cards for two people who want opposite things.

The librarian (retrieval) wants small cards: one idea each, easy to file and find. A card that covers nine topics gets filed somewhere vague and found by nobody. The reader (the LLM) wants big cards: "Revenue grew 3%" alone is useless — whose revenue, which quarter, compared to what?

Every strategy is a settlement between those two. Where you cut matters as much as size: a mid-sentence break serves neither party.

Why it exists

Two constraints force chunking whether you want it or not.

An embedding is a fixed-size vector. Feed a sentence or fifty pages and you still get the same length vector (say 1024 floats). The long input is averaged into one point, so a specific question ("what's the notice period for contractors?") rarely lands near it. Meaning gets diluted.

The context window is a budget and a bill. Even if whole-document embeddings worked, you cannot afford to paste fifty pages into every prompt.

So documents become retrieval-sized units — and the moment you cut, each fragment loses document context. Half of good ingestion is putting that context back. Stakes: retrieval can only surface what ingestion preserved. A shredded table is unanswerable no matter how good your reranker is, and it fails silently.

The core idea

Ingestion is the offline half of RAG: parse to clean text, split into chunks, attach metadata, embed, and index. It is unglamorous, and it is where most real-world RAG quality is won or lost.

Parsing is the first trap. PDFs are a layout format, not a text format. Glyphs arrive in draw order, not reading order; two-column pages interleave; tables flatten into word soup; headers and footers repeat into every chunk; scans need OCR. Good pipelines use layout-aware parsers (or a vision model on hard pages) and keep structure — headings, table boundaries, lists — because chunking wants those boundaries.

Chunking balances precision against context. Small enough that the embedding is specific; large enough that the LLM can answer. Four strategies, cheapest first:

1. Fixed-size — every N tokens, blind to content. Baseline only; splits mid-sentence.
2. Recursive — paragraphs, then sentences, then words until pieces fit the budget. Near-zero cost; the default that is hard to beat in benchmarks.
3. Structural / layout-aware — split on headings, sections, tables, code blocks. Wins when documents have structure — most enterprise content.
4. Semantic — embed consecutive sentences, cut where similarity drops. Content-adaptive, but costs an embedding pass and can over-fragment into tiny chunks that retrieve poorly.

Start around 256–512 tokens with 10–20% overlap, then tune on an eval set — never by feel. Every chunk carries metadata (source, section, timestamp, ACL tags) because production retrieval is almost always filtered retrieval.

How it actually works

Size moves quality because compression is lossy. Small chunks → precise vectors, but the generator may starve for context. Large chunks → several topics averaged into one vector, so specific queries miss and each hit drags extra tokens into the prompt. That is why production systems often decouple match unit from generation unit (parent-child / small-to-big; see retrieval patterns).

Overlap copies the tail of chunk n onto the head of chunk n+1 so a boundary-straddling fact still appears whole somewhere. Cost: storage and near-duplicate hits (dedupe at retrieval). Past ~20% is mostly waste; structural boundaries need less. Token-level evals show heavy overlap can hurt efficiency without always buying recall — measure on your corpus.

Context loss and three fixes. After the cut, "Revenue grew 3%" no longer says which company or quarter:

Heading paths first. Escalate to contextual retrieval when eval still shows referent failures on a high-value corpus. Late chunking is the embedding-side alternative when you control the model.

Tables and figures. Never split a table mid-row. Keep it atomic (Markdown-serialized), or embed an LLM summary and store the full table as the generation payload. Same pattern for charts (see multimodal). Row-level questions over large tables often belong in text-to-SQL, not embedded rows.

Metadata at ingest — decide before indexing; backfill means reprocessing:

• Identity: doc_id, chunk_id, source_uri, content hash (drives incremental re-index).
• Filterable: department, product, doc type, language, updated_at.
• Security: ACL/tenant tags as retrieval filters (see production RAG and filtering & metadata).
• Provenance: page/section for citations and parent-context fetch.

The flows

Parsing and metadata are cheap to get right once and expensive to reverse. Chunk-boundary experiments always mean re-embedding.

A worked example

One page from an employee handbook on the governed enterprise platform, ingested three ways:

## Leave Policy
### Annual Leave
Employees accrue 1.75 days per completed month of service. Unused
days carry forward to a maximum of 30.
### Parental Leave
Primary caregivers are entitled to 26 weeks at full pay. ...

Query: "how many leave days carry over?"

chunk 7:  "...Employees accrue 1.75 days per completed month of service. Unused"
chunk 8:  "days carry forward to a maximum of 30. ### Parental Leave Primary..."

Most of the win is not destroying structure. Semantic and contextual methods improve on top of this base; they do not replace it.

Production concerns

Parsing is the top silent failure. Metrics look mysteriously bad because chunks contain interleaved columns or shredded tables. Sample random chunks per document type before you touch embedding models. Add per-source sanity checks: text length vs page count, table count, header/footer repetition.

Ingest cost scales with cleverness. Recursive is CPU-only; semantic adds an embedding pass; contextual retrieval adds an LLM pass. Reserve per-chunk LLM work for high-value corpora where eval proves the gain.

Idempotency. Hash each doc; re-process only what changed; delete old chunks by doc_id before writing the new set. Orphaned stale chunks are a classic wrong-answer source. Make chunk IDs deterministic from (doc_id, position).

Route by format. Structural for Markdown/HTML, code-aware for source, layout parser for PDF. A single splitter across mixed content often yields bimodal quality.

Changing chunking means a full re-index. Build the offline eval harness first; version indexes for A/B and rollback. Popular vendor defaults (large chunks, heavy overlap) often score poorly on token-level evals — treat defaults as hypotheses. See RAG evaluation.

Common drill-downs

How do you choose chunk size? Start around 256–512 tokens with modest overlap (or none if boundaries are structural). Sweep against recall@k and answer quality on a golden set. Smaller for factoid precision; larger when answers need synthesis.

Fixed vs recursive vs structural vs semantic? Fixed: baseline only. Recursive: free default near the top of independent evals. Structural: usual enterprise choice when headings/HTML/code exist. Semantic: unstructured streams with topic shifts — pay the cost and watch over-fragmentation.

How do you restore document context after the cut? Cheapest: heading path. Stronger: contextual retrieval (LLM blurb; numbers above). Alternative: late chunking. Parent-child (match small, return large) is the online cousin — see retrieval patterns.

How do you evaluate a chunking change? Golden set of (query, relevant passage) pairs; recall@k, MRR, and token-level overlap of retrieved vs relevant text (document-level hits hide partial misses). Then end-to-end answer quality. Ship behind an index version.

Test yourself

Go deeper

• Chunking Strategies for LLM Applications — Pinecone — survey of methods and selection criteria.
• Evaluating Chunking Strategies for Retrieval — Chroma — token-level eval; chunking can swing recall by ~9%; large vendor defaults often underperform.
• Introducing Contextual Retrieval — Anthropic — mechanism, failure-rate numbers, cost math with prompt caching.
• Late Chunking — Jina — embed first, pool per chunk; primary source for late chunking.
• Production RAG tips — LlamaIndex — decoupling retrieval chunks from synthesis chunks.
• 5 Levels Of Text Splitting — Greg Kamradt — character splits through semantic and agentic chunking.

Where this connects

• Retrieval Patterns — parent-child and contextual retrieval answer the context-loss problem introduced here.
• Filtering & Metadata — the metadata you design here is what makes filtered retrieval possible at query time.
• RAG Evaluation — the harness that must exist before any chunking experiment means anything.
• Multimodal — when parsing hits scans, charts, and diagrams, ingestion becomes a vision problem.
• Production RAG — incremental re-indexing, deletes, and the document lifecycle around this pipeline.