Multimodal
When the input isn't text — and the moment your RAG pipeline meets a PDF full of charts.
Prerequisites
- Embeddings — multimodal search is the same geometry trick with a shared image/text space.
- Ingestion & Chunking — this is where most real-world multimodal work actually happens.
- Tokens & Context Windows — images bill as tokens, and the arithmetic surprises people.
Why this section exists
Most engineers meet multimodality not because someone asked for an image feature, but because the corpus fought back. Enterprise documents are scanned PDFs, slide decks, tables that a text extractor turns into word soup, and diagrams carrying the information the surrounding paragraph only alludes to. Ignore all of that and your retrieval quality has a ceiling no reranker can lift.
So the question that actually matters is rarely "how does a vision transformer work." It is:
- How do you get searchable text out of a page whose meaning is partly in its layout?
- When do you describe an image with a model and index the description, versus index the image itself in a shared embedding space?
- What does a picture cost, in tokens and in latency?
- Where does audio fit — transcription in, speech out, and what breaks in between?
Key insight
The highest-leverage multimodal technique in most RAG systems isn't multimodal retrieval at all. It's using a vision model at ingestion time to convert charts, tables, and scanned pages into rich text descriptions, then indexing that text with everything else. You keep one retrieval path and one evaluation story, and you stop losing half your corpus.
What's in here
| Page | The question it answers |
|---|---|
| Multimodal AI | Vision in pipelines, ingesting PDFs with charts and tables, speech to and from text — at survival depth |
Where this connects
- RAG — parsing and chunking are where multimodal ingestion pays off.
- Cost — image tokens are the line item people forget to model.
- Design: Document Q&A at Scale — real document corpora are never plain text.