Multimodal AI

The intuition

A text-only pipeline is a librarian who can only read the words on a page. Hand them a financial report and they index every paragraph carefully — and silently drop the bar chart that holds the only number that matters.

Multimodal work is teaching that librarian to look. Once a vision encoder turns an image into a block of visual tokens, the model treats it like any other stretch of context: billed, attended, and limited by the same window. Speech runs the idea in reverse: sound becomes text, text becomes a reply, the reply becomes sound — and the whole chain must finish before the other person feels the lag.

The engineering question is almost never "can the model see?" Modern vision-language models (VLMs) can. It is where you spend the vision pass (once at ingestion vs every query), how many visual tokens you buy with resolution, and which failure mode you accept when the chart is hard to read.

Why it exists

Text-only pipelines lose on corpora that real organizations own:

1. Meaning lives outside the text layer. Charts, architecture diagrams, forms where position encodes fields, tables with merged headers — a PDF parser returns empty or mangled strings and the index never sees the answer.
2. OCR is not a free lunch. Scans get some text, but errors embed cleanly, retrieve confidently, and poison answers without looking broken. Complex tables still collapse.
3. Cross-modal search is a product need. "Find slides that look like this mockup" cannot be answered by a text embedder alone.
4. Voice has a hard latency budget. Users judge a voice agent as natural under roughly 800 ms voice-to-voice. Cascaded STT → LLM → TTS and native speech-to-speech answer that budget differently.

The alternatives lose in practice. Ignoring non-text caps recall no reranker can lift. OCR-everything is cheap until charts and layout-dependent meaning appear. Sending every page to a VLM at query time works for demos and dies on cost and latency at corpus scale. Native speech-to-speech for every channel sounds ideal until telephony (8 kHz PSTN) erases much of the audio-quality advantage while the audio-token premium remains.

The core idea

For an applied AI engineer, multimodal means four practical capabilities: send images to an LLM, ingest documents whose meaning lives in charts and tables, retrieve across image and text with shared embeddings, and move between speech and text for voice.

Images are tokens. A VLM runs the image through a vision encoder — usually a Vision Transformer (ViT) that cuts it into fixed-size pixel patches, embeds each patch, and projects those into the same space as text tokens. From the LLM's perspective the image is a block of hundreds to thousands of visual tokens in the context window: billed like input, consuming budget, attended by the same transformer. Token count scales with resolution (roughly quadratically), so resolution is the cost dial.

The RAG-relevant use case is PDF ingestion. Text extraction is cheap and exact for body text, but returns nothing for charts and diagrams and mangles complex tables. The fix is either a vision pass at ingestion (render the page, VLM emits Markdown tables and figure captions, index that text) or vision-native retrieval (ColPali-style: embed the page image and skip parsing). Most production systems hybridize: text for prose, vision for figure and table regions.

Multimodal embeddings (CLIP and successors) map images and text into one vector space via contrastive training, so "text query, image results" is nearest-neighbor search — the same vector-DB machinery as text RAG, with a different encoder.

Voice is speech-to-text (STT) plus text-to-speech (TTS). A voice agent is either a cascade (STT → LLM → TTS, each streaming) or a native speech-to-speech model (audio in, audio out). Cascade gives control and observability; native S2S gives lower floor latency and preserved prosody. The whole game is the latency budget.

How it actually works

All four capabilities share one pattern: convert a non-text modality into tokens, embeddings, or both, then apply ordinary retrieval, generation, and cost controls.

Image → tokens (as of mid-2026; verify before shipping). Same idea, different arithmetic:

Cost for the same JPEG varies several-fold across providers. Providers auto-downscale oversized images, so dense screenshots may silently lose legibility — pre-resize so you control what the model sees. In multi-turn loops, images re-bill every turn until you replace them with extracted text. See tokens and context and cost.

PDF ingestion — three architectures:

1. Text-first. Parser for digital text, OCR for scans, layout-aware tools (Docling, Unstructured, Azure Document Intelligence) for reading order and tables. Cheap and near-deterministic. Fails on charts, merged-header tables, and low-quality scans. See ingestion and chunking.
2. Vision-at-ingestion. Render pages at roughly 150–200 DPI. Prompt a VLM for structured output: tables as Markdown, figures as dense captions with numbers. Index as ordinary text; keep a page-image pointer for re-inspection. On the order of 1.5k–6k input tokens per page — offline batch. Pairs with structured output.
3. Vision-native retrieval (ColPali). Embed each page image as a grid of patch embeddings (multi-vector). Embed the query as token embeddings. Score with late interaction (ColBERT-style MaxSim): for each query token, max similarity over page patches, then sum. No OCR, no chunking. On visually rich corpora it often wins — illustrative financial-PDF comparison: ~62% recall dense text vs ~84% ColPali-family. On heavy scans it beats hybrid because OCR errors never enter. Tradeoffs: multi-vector storage ~100× a single vector per page; generation needs a VLM too.

Route by page content. Vision earns its cost only where meaning is visual. Born-digital prose stays on the cheap, exact text path.

CLIP (survival depth). Two encoders — image ViT, text transformer — trained contrastively on hundreds of millions of image-caption pairs: pull matches together, push non-matches apart. One shared space for zero-shot classification and cross-modal retrieval. Classic CLIP limits: 77-token text context, weak text-in-image reading, bag-of-words-ish composition. Successors (SigLIP, provider multimodal embedding APIs) improve the loss and text handling; ColPali is the multi-vector evolution for documents. Similarity math: similarity metrics. Storage and ANN: vector databases.

Speech. Whisper is the open-source STT reference — encoder-decoder over 30-second log-mel windows, tiny (~39M) to large (~1.55B), trained on ~680k hours. Metric: WER (word error rate). Frontier APIs sit around single-digit WER on clean audio and degrade on jargon. Whisper is batch — no native streaming. Live products use streaming STT with partials in ~150–300 ms. TTS streams first audio in ~100–300 ms; cascade it sentence-by-sentence from the LLM output.

Voice agent shapes. A cascade streams every stage concurrently. You own turn detection (VAD + semantic endpointing), barge-in (cancel gen, flush TTS), and free transcripts for logging, evals, and guardrails. Native speech-to-speech (OpenAI Realtime over WebRTC/WebSocket/SIP, Gemini Live) is one model audio-in/audio-out: lower floor latency, hears tone, natural prosody — at the cost of less mid-stream control, harder evals, and premium audio tokens. Over 8 kHz PSTN much of the audio edge evaporates while the price premium remains; telephony often stays cascaded.

Target ≤800 ms voice-to-voice (human gaps ~200–300 ms; beyond ~1 s feels broken). Illustrative failing budget: 280 ms STT + 600 ms LLM TTFT + 320 ms TTS + 200 ms transport ≈ 1.4 s. Lever order: LLM TTFT first, then endpointing, then TTS TTFB. Full tactics: latency.

The flows

A worked example

Trace hybrid PDF ingestion through the governed enterprise platform. A regulated firm indexes ~10k internal policy and financial PDFs for ~10–15k daily users. An employee asks:

"What was Q3 operating margin on the regional performance pack?"

The answer lives in a bar chart on page 7 of a slide deck — no data labels in the text layer.

Token arithmetic if the page had been sent raw at query time (Claude-style 28-px patches, order of magnitude): ~1000×1000 ≈ 1,296 visual tokens; denser full page toward ~5k. Batch ingest pays once per page version; query time pays on every ask and still risks provider downscaling.

Production concerns

Cost. Pre-resize to the smallest resolution where the task still works (legible axis labels, not 4K). Crop to the region of interest. Use low-detail modes for classification. Batch ingest through batch APIs (often roughly half-price). Images in multi-turn history re-bill every turn — replace with extracted text once extraction succeeds. Full treatment: cost.

Latency. Vision adds encoder time and payload upload; TTFT on image-heavy prompts is worse than text-only. For voice, stream every stage and measure p95 voice-to-voice, not component averages. Full treatment: latency.

Ingestion failures. OCR noise poisons the index silently. Vision transcription hallucinates plausible table values — worse than missing because nothing looks broken. Sample-audit extractions; track parse-confidence; route low-confidence pages to vision or humans. See production RAG and reliability.

Chart and table misreads. Models interpolate off axes, confuse legend colors, misalign merged headers, and count poorly. Force structured output with a "not legible" hatch. Require quoting the cell or label. Sanity-check numbers. For high stakes, two passes and flag disagreement. Always keep the source page image.

Resolution truncation is the silent killer: a 3,000-px scan shrinks below legibility and the model confabulates. Test densest real documents.

Voice. Turn detection is the top complaint — interrupt too early or wait too long. Tune endpointing per use case. Confirm names and IDs. On barge-in, cancel generation, flush audio, keep the transcript consistent with what the user heard. Wire into observability and evals and testing. Native S2S audio tokens often run ~10× text; cascaded pipelines price STT/TTS per minute plus LLM tokens — model both against average call length.

Common drill-downs

Financial reports full of charts — text extraction misses them. What do you do? Hybrid ingestion: text for prose; detect figure/table regions; VLM to Markdown and number-bearing captions; index with a page-image pointer. Heavily scanned or figure-dense corpora may prefer ColPali instead.

ColPali vs hybrid text + vision-at-ingest? ColPali embeds page images as patch multi-vectors and retrieves with MaxSim — no OCR, no chunking. Choose it when parsing is the bottleneck or OCR poisons the index. Accept multi-vector storage and a VLM at generation. Keep hybrid text for born-digital prose.

Cascade vs speech-to-speech? Cascade for transcripts, guardrails, component swaps, telephony. Native S2S when latency and naturalness dominate — less mid-stream control, costlier audio tokens.

Why not vanilla Whisper for a live voice agent? Batch model, no native streaming — seconds of turn latency. Live agents need streaming STT with partials in ~150–300 ms. Whisper fits offline transcription.

Precise number from a bar chart with no data label — trust it? No. Models interpolate off axes. Require label-vs-estimate, allow "not legible", sanity-check totals, and for high stakes run a second pass and show the source image.

Test yourself

Go deeper

• Anthropic — Vision docs — patch-based token accounting, resolution tiers, limits, and image-quality guidance.
• OpenAI — Images and vision guide — detail levels (low / high / original) and patch/tile token calculation across model families.
• OpenAI — Realtime API guide — native speech-to-speech voice agents over WebRTC/WebSocket/SIP.
• CLIP paper — Learning Transferable Visual Models From Natural Language Supervision — the contrastive shared-space idea behind multimodal embeddings.
• ColPali paper — Efficient Document Retrieval with Vision Language Models — vision-native page retrieval with late interaction (ICLR 2025).
• openai/whisper (GitHub) — the open-source STT reference: model sizes, architecture, usage.
• How AI 'Understands' Images (CLIP) — Computerphile — visual explainer of CLIP's shared embedding space.

Where this connects

• Ingestion & Chunking — where hybrid text/vision PDF pipelines actually live in the offline path.
• Production RAG — freshness, ACLs, and degradation when parse-confidence and vision routes fail in production.
• Cost — image tokens, batch discounts, and the multi-turn re-billing trap for images left in history.
• Design: Document Q&A at Scale — end-to-end design when real corpora are never plain text.