Multimodal AI

The intuition

A text-only pipeline is like a librarian who can only read the words on a page and must pretend the diagrams, photos, and margin notes do not exist. Hand them a financial report and they will index every paragraph carefully — and silently drop the bar chart that holds the only number that matters.

Multimodal work is teaching that librarian to look. An image is not a special side channel; once a vision encoder has turned it into patches and projected those patches into the model's embedding space, the image is just another stretch of tokens in the context window — billed like text, attended like text, subject to the same context budget. Speech is the same idea in reverse: sound becomes text, text becomes a reply, the reply becomes sound again, and the whole chain has to finish before the other person feels the conversation lag.

The engineering question is almost never "can the model see?" — modern VLMs can. It is where you spend the vision pass (once at ingestion vs every query), how many visual tokens you buy with resolution, and which failure mode you prefer when the chart is hard to read.

Why it exists

Text-only pipelines lose on corpora that real organizations actually own. The pain is concrete:

1. Meaning lives outside the text layer. Charts, architecture diagrams, forms where position encodes fields, tables with merged headers — a PDF parser returns empty or mangled strings and the index never sees the answer.
2. OCR is not a free lunch. Scans get some text, but errors embed cleanly, retrieve confidently, and poison answers without looking broken. Complex tables still collapse.
3. Cross-modal search is a product need. "Find slides that look like this mockup" or "images of red safety equipment" cannot be answered by a text embedder alone.
4. Voice interfaces have a hard latency budget. Users judge a voice agent as natural under roughly 800 ms voice-to-voice. Cascaded STT → LLM → TTS and native speech-to-speech are different answers to the same budget, with different control and cost tradeoffs.

The alternatives lose in practice. Ignoring non-text caps recall no reranker can lift. OCR-everything is cheap until charts and layout-dependent meaning appear. Sending every page to a VLM at query time works for demos and dies on cost and latency at corpus scale. Native speech-to-speech for every channel sounds ideal until telephony (8 kHz PSTN) erases the audio-quality advantage while the audio-token premium remains.

Multimodal techniques exist so each of those constraints gets a deliberate path — not a hope that the text layer was enough.

The core idea

Multimodal, for an applied AI engineer, means four practical capabilities: sending images to an LLM, ingesting documents whose meaning lives in charts and tables, retrieving across image and text with shared embeddings, and moving between speech and text for voice interfaces.

Images are just tokens. A vision-language model (VLM) runs the image through a vision encoder (a Vision Transformer) that cuts it into fixed-size pixel patches, turns each patch into an embedding, and projects those embeddings into the same space as text token embeddings. From the LLM's perspective the image is a block of a few hundred to a few thousand "visual tokens" sitting in the context window like any other tokens — billed like input tokens, consuming context budget, and attended to by the same transformer. Token count scales with resolution (roughly quadratically), so resolution is the cost dial: downscale and you save tokens but small text becomes illegible; send full resolution and a single 4K page can cost ~5,000 tokens.

The RAG-relevant use case is PDF ingestion. Text extraction (parsers plus OCR for scans) is cheap and exact for body text, but it returns nothing for charts and diagrams and mangles complex tables — a large fraction of the signal in enterprise documents silently never reaches the index. The fix is either a vision pass at ingestion (render each page to an image, have a VLM transcribe tables to Markdown and describe figures, index that text) or vision-native retrieval (ColPali-style: embed the page image directly and skip parsing entirely). Most production systems run a hybrid: text pipeline for prose, vision pipeline for figure/table regions.

Multimodal embeddings (CLIP and successors) map images and text into one vector space via contrastive training, so "text query, image results" is just nearest-neighbor search — the same vector-DB machinery as text RAG, with a different encoder.

Voice is two conversions: speech-to-text (Whisper-class models, or streaming APIs when latency matters) and text-to-speech. A voice agent is either a cascaded pipeline (STT → LLM → TTS, each streaming into the next) or a native speech-to-speech model that takes audio in and emits audio out. The cascade gives control and observability; native speech-to-speech gives lower latency and preserved prosody. The whole game is the latency budget: users perceive a voice agent as natural under roughly 800 ms voice-to-voice, and the LLM's time-to-first-token is usually the largest line item.

How it actually works

The four capabilities share one pattern: convert a non-text modality into something the rest of the stack already knows (tokens, embeddings, or both), then apply ordinary retrieval, generation, and cost controls.

Image → tokens, per provider (mid-2026). Same idea, different arithmetic — know one cold and name the others:

Consequences: (1) cost varies several-fold across providers for the same JPEG; (2) providers auto-downscale oversized images, so a dense spreadsheet screenshot may silently lose legibility — pre-resize yourself so you control what the model actually sees; (3) in multi-turn agent loops, every prior image is resent with the conversation each turn, so image-heavy histories inflate per-turn cost until you evict or replace images with their extracted text. Token accounting connects directly to tokens and context windows and cost.

PDF ingestion — three architectures:

1. Text-first: PDF parser for digital text, OCR for scans, layout-aware tools (Docling, Unstructured, Azure Document Intelligence) to recover reading order and table structure. Cheap, fast, near-deterministic. Fails on: charts/diagrams (no text to extract), tables with merged cells or multi-row headers, layout-dependent meaning, low-quality scans where OCR errors propagate into the index. See ingestion and chunking.
2. Vision-at-ingestion: render each page at ~150–200 DPI, prompt a VLM to emit structured output — tables as Markdown/HTML, figures as dense captions with the numbers read off, prose transcribed. Index the result as ordinary text chunks; keep a pointer to the page image so the generator (or a human) can re-inspect the original. Costs 1.5k–6k input tokens per page and seconds of latency, so it's an offline batch job. Structured emission pairs well with structured output.
3. Vision-native retrieval (ColPali): embed each page image as a grid of patch embeddings (multi-vector, one per patch) using a VLM, embed the query as token embeddings, score with late interaction (ColBERT-style MaxSim). No OCR, no chunking, no parsing pipeline to maintain. On visually rich corpora it wins outright — one benchmark on financial PDFs: 62% recall for dense text embeddings vs 84% for a ColPali-family model — and on heavily scanned corpora it beats even hybrid pipelines because OCR errors never enter the system. Costs: multi-vector storage is ~100× a single vector per page, and retrieved "chunks" are page images, so the generator must also be a VLM and generation input costs rise.

When does a vision model reading the page beat text extraction? When the answer is in the visual: a trend in a line chart, a value in a bar chart with no data labels, a table whose header spans rows, an architecture diagram, a form where position encodes meaning. When the document is born-digital prose, text extraction is strictly better: cheaper, exact, no hallucination risk.

CLIP mechanics (survival depth): two encoders — image ViT, text transformer — trained contrastively on ~400M web image-caption pairs: maximize cosine similarity of matched pairs, minimize it for the other pairs in the batch. Result: one shared space where text and image vectors are directly comparable, enabling zero-shot classification and cross-modal retrieval. Known weaknesses to name: the text encoder is short-context (77 tokens), it reads text inside images poorly, and it behaves bag-of-words-ish on composition ("a red cube on a blue cube" vs the reverse). Successors (SigLIP, provider multimodal-embedding APIs) improve training loss and text handling; ColPali is the multi-vector evolution for documents. Similarity arithmetic is the same as in similarity metrics; storage and ANN concerns sit in vector databases.

Speech-to-text: Whisper is the open-source reference — an encoder-decoder transformer over 30-second log-mel spectrogram windows, sizes from tiny (39M) to large (1.55B) plus turbo (809M), robust across accents/noise because it trained on 680k hours of weakly supervised audio. Metric is WER (word error rate); frontier APIs sit around 6–9% on clean benchmarks but degrade sharply on domain jargon and long-form audio. Whisper is batch — it has no native streaming; real-time products use streaming APIs (Deepgram, AssemblyAI, ElevenLabs Scribe) that emit partial transcripts with 150–300 ms first-partial latency, then finalize. Accuracy-vs-latency is the axis: batch models re-attend over the whole clip; streaming models commit early and correct.

Text-to-speech: streaming TTS engines return first audio in ~100–300 ms and synthesize ahead of playback; quality knobs are prosody/naturalness vs speed vs price. TTS consumes the LLM's output stream sentence-by-sentence — you don't wait for the full completion.

Voice agent shapes:

• Cascade (STT → LLM → TTS): every stage streams concurrently. You own turn detection (VAD + semantic endpointing: has the user finished the thought?), interruption handling (user barge-in must cancel LLM generation and flush the TTS buffer), and transcripts fall out for free — which means logging, evals, guardrails, and swapping any component independently.
• Native speech-to-speech (OpenAI gpt-realtime over WebRTC/WebSocket/SIP, Gemini Live): one model, audio-in/audio-out. Lower floor latency, hears tone and hesitation, natural prosody out. Costs you: less mid-stream control (no text to inspect before it's spoken), harder evals, premium audio-token pricing — and over ordinary phone lines (8 kHz PSTN) the audio-quality advantage largely evaporates while the price premium remains, which is why telephony deployments often stay cascaded.
• Latency budget: target ≤800 ms voice-to-voice (human turn-taking gaps are ~200–300 ms; beyond ~1 s feels broken). A realistic failing breakdown: 280 ms STT finalization + 600 ms LLM time-to-first-token + 320 ms TTS first audio + 200 ms transport ≈ 1.4 s. The lever ordering: LLM TTFT dominates (use a small/fast model, trim the prompt, start TTS on the first sentence), then endpointing delay, then TTS TTFB. Full latency tactics: latency.

The flows

A worked example

Trace hybrid PDF ingestion and a chart question through the governed enterprise platform. A regulated firm indexes ~10k internal policy and financial PDFs for ~10–15k daily users. An employee asks:

"What was Q3 operating margin on the regional performance pack?"

The answer lives in a bar chart on page 7 of a slide deck — no data labels in the text layer.

Token arithmetic if the page had been sent raw at query time instead (Claude-style 28-px patches, order of magnitude): a ~1000×1000 render ≈ 1,296 visual tokens; a denser full page toward ~5k. At batch ingest you pay once per page version; at query time you pay on every ask and still risk provider downscaling.

Common drill-downs

These are the questions worth asking one level down once the core idea is solid.

How does an LLM "see" an image? A vision encoder (ViT) splits it into fixed pixel patches, embeds each patch, and a projection layer maps those into the LLM's token-embedding space. The image becomes a block of visual tokens in the context — billed and attended like text. Token count scales with resolution, which is why providers cap or downscale large images.

You're building RAG over financial reports full of charts and tables. Text extraction misses them — what do you do? Hybrid ingestion: keep text extraction for prose; detect figure/table regions and run a VLM pass that transcribes tables to Markdown and captions charts with the actual numbers, then index that text with a pointer back to the page image. If the corpus is heavily scanned or figure-dense, consider ColPali-style visual retrieval instead — embed page images directly and skip parsing.

What is ColPali and when would you choose it? A VLM that embeds each document-page image as a grid of patch embeddings and retrieves with late-interaction (MaxSim) scoring — no OCR, no chunking. Choose it for figure-heavy or scanned corpora where parsing is the bottleneck or OCR errors poison the index; accept multi-vector storage costs and a VLM requirement at generation time.

How does CLIP enable text-to-image search? Contrastive training on hundreds of millions of image-caption pairs pushes matched image and text vectors together in one shared space. Embed the query text, nearest-neighbor over image vectors — standard vector search, cross-modal. Caveats: weak on text inside images and on compositional queries.

Cascaded voice pipeline vs a speech-to-speech model — how do you choose? Cascade (STT → LLM → TTS) when you need control: transcripts for logging/evals/guardrails, component swaps, telephony (where S2S's audio advantage dies over PSTN). Native S2S when latency and naturalness dominate — one model, hears prosody, speaks with it — at the price of less mid-stream control and costlier audio tokens.

What's your latency budget for a voice agent and where does it go? Target ≤800 ms voice-to-voice; ~1 s+ feels broken. Typical spend: STT finalization ~200–300 ms, LLM time-to-first-token (the biggest item, often 400–800 ms), TTS first audio ~100–300 ms, transport ~100–200 ms. Optimize the LLM first: smaller model, shorter prompt, stream TTS from the first sentence.

Why doesn't vanilla Whisper work for a live voice agent? It's a batch model over 30-second windows with no native streaming — you'd add seconds of turn latency. Live agents use streaming STT that emits partials in ~150–300 ms and finalizes on endpoint detection; Whisper-class batch models fit transcription jobs, not conversations.

The model read a bar chart and gave you a precise number. Do you trust it? No — models interpolate off axes and report false precision. Require it to state whether the value was a printed data label or an estimate, give it a "not legible" out, sanity-check against totals or adjacent text, and for high-stakes figures cross-check with a second pass or second model and surface the source image.

How do image inputs change your token costs, concretely? A ~1-megapixel image is ~1,300 tokens on Claude (28-px patches) and similar order on OpenAI (32-px patches with a per-model budget and multiplier); a full-resolution page can hit ~5k. The same image can cost several times more on one provider than another, and in agent loops each image is re-billed every turn it stays in history — so resize, crop, choose detail level, and evict.

When is text extraction strictly better than a vision model for PDFs? Born-digital prose documents: extraction is near-free, exact, and cannot hallucinate. Vision earns its cost only where meaning is visual — charts, diagrams, complex tables, scans — so route by page content, don't send every page to a VLM.

Production concerns

• Cost control for images: pre-resize to the smallest resolution where the task still works (chart reading needs legible axis labels, not 4K); crop to the region of interest instead of sending the full page; use the provider's low-detail mode for classification-grade tasks. Batch ingestion jobs through batch APIs (~50% discount). Watch multi-turn loops: images in history are re-billed every turn — replace an image with its extracted text once extraction succeeds. Full treatment: cost.
• Latency: vision requests add encoder time plus payload upload (a base64 page image is hundreds of KB); time-to-first-token on image-heavy prompts is noticeably worse than text-only. For voice, every serial stage adds latency — colocate services, stream everything, and measure p95 voice-to-voice, not component averages. Full treatment: latency.
• Ingestion failure modes: OCR noise poisons the index silently (garbage text embeds fine and retrieves wrong); vision transcription hallucinates plausible table values, which is worse than missing values because nothing looks broken. Sample-audit extracted tables against source pages; track per-document parse-confidence and route low-confidence pages to the vision (or human) path. See production RAG and reliability.
• Chart/table misreads at query time: models interpolate values off axes ("about 42" becomes "42.3"), confuse legend colors, misalign table cells across merged headers, and are unreliable at counting objects. Mitigations: force structured output with an explicit "not legible" escape hatch; require the model to quote the cell/label it read; run numeric sanity checks (row sums, totals, units); for high-stakes numbers, run two models or two passes and flag disagreement; always retain the source page image for verification.
• Resolution truncation: provider auto-downscaling is the silent killer — a 3,000-px-wide scanned page gets shrunk below legibility and the model confabulates instead of failing. Detect by testing your densest real documents, not clean samples.
• Voice reliability: turn detection is the top complaint source (agent interrupts the user, or waits awkwardly); tune endpointing per use case. Handle STT errors on names/IDs with explicit confirmation ("Did you say A-C-M-E?"). Plan interruption semantics: on barge-in, cancel generation, flush audio buffers, and keep the transcript consistent with what was actually heard by the user, not what was generated. Observability and evals: observability, evals and testing.
• Speech pricing reality: audio tokens for native speech-to-speech run roughly an order of magnitude above text tokens; cascaded pipelines price per-minute for STT/TTS plus normal LLM tokens — model both against your average call length before choosing an architecture.

Test yourself

Go deeper

• Anthropic — Vision docs — patch-based token accounting, resolution tiers, limits, and image-quality guidance.
• OpenAI — Images and vision guide — detail levels and patch/tile token calculation across model families.
• OpenAI — Realtime API guide — native speech-to-speech voice agents over WebRTC/WebSocket/SIP.
• CLIP paper — Learning Transferable Visual Models From Natural Language Supervision — the contrastive shared-space idea behind multimodal embeddings.
• ColPali paper — Efficient Document Retrieval with Vision Language Models — vision-native page retrieval with late interaction.
• openai/whisper (GitHub) — the open-source STT reference: model sizes, architecture, usage.
• How AI 'Understands' Images (CLIP) — Computerphile — the canonical visual explainer of CLIP's shared embedding space.
• What is Multimodal RAG? Unlocking LLMs with Vector Databases — IBM Technology — concise overview of multimodal retrieval architecture.

Where this connects

• Ingestion & Chunking — where hybrid text/vision PDF pipelines actually live in the offline path.
• Production RAG — freshness, ACLs, and degradation when parse-confidence and vision routes fail in production.
• Cost — image tokens, batch discounts, and the multi-turn re-billing trap for images left in history.
• Design: Document Q&A at Scale — end-to-end design when real corpora are never plain text.