ModelRefs / Multimodal Architecture — AI Glossary
Multimodal Architecture — AI Glossary
A model architecture that processes multiple data types (text, images, audio, video) by projecting each modality into a shared embedding space.
Overview
Multimodal LLMs add modality-specific encoders (e.g., ViT for images, Whisper encoder for audio) connected via a projection layer to the text decoder. GPT-4o, Gemini 1.5, Claude 3 Opus, and Llama 3.2 Vision follow this pattern. The shared transformer backbone enables cross-modal reasoning without separate model stacks.
Reference details
| Topic | architecture |
|---|---|
| Also known as | vision-language model, VLM, multimodal LLM, MLLM |
| Last reviewed | 2026-06-24 |
Related terms
Example: An image is context, and it is not cheap
A vision encoder splits an image into a grid of patches — say 16×16 = 256 patches — and emits one vector per patch. A projection layer maps each vector into the language model's hidden size so the decoder can attend to them as if they were tokens. That one image therefore occupies roughly 256 positions in the context window before a single word of your prompt. Two images and a system message can consume more context than the question you are actually asking, which is why high-resolution modes, which raise the patch count, cost noticeably more per call.
Commonly confused with
A multimodal architecture is not a multimodal pipeline. Running OCR or a captioning model first and pasting text into a prompt is a pipeline: the LLM never sees the image, so layout, colour and spatial relationships are already lost. A true multimodal model attends to the image representation directly, which is why it can answer “what is in the top-right cell of this table”.
When to use it
Reach for it when:
- The answer depends on layout, spatial position or visual detail a caption would discard
- Documents, screenshots and charts, where structure carries meaning
- You want one model and one prompt path rather than a brittle OCR-then-LLM chain
Reach for something else when:
- The images are always the same known format — a dedicated extractor is cheaper and more accurate
- Context budget is tight: images crowd out retrieved text
- You need character-perfect transcription of dense text; a specialist OCR engine still wins
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Multimodal Architecture — AI Glossary.
Frequently asked questions
What is Multimodal Architecture?
A model architecture that processes multiple data types (text, images, audio, video) by projecting each modality into a shared embedding space.
Is Multimodal Architecture the same as vision-language model?
Yes — vision-language model, VLM, multimodal LLM, MLLM are common aliases for Multimodal Architecture.
What concepts are related to Multimodal Architecture?
Closely related concepts include visual encoder, cross attention, encoder decoder.