ModelRefs / Vision-Language Model (VLM) — AI Glossary
Vision-Language Model (VLM) — AI Glossary
A model that accepts both image (and optionally video) and text inputs within a single unified context. 2 Vision are VLMs. Also called VLM or vision model.
Overview
GPT-4o, Claude 3.5 Sonnet, Gemini 2.5 Pro, and Llama 3.2 Vision are VLMs. The dominant architecture: a vision encoder (ViT) projecting image patches into the LLM's token embedding space. Enables OCR, chart understanding, image Q&A, and visual agent tasks.
Reference details
| Topic | multimodal |
|---|---|
| Also known as | VLM, vision model, multimodal LLM |
| Last reviewed | 2026-06-24 |
Related terms
Example: It infers the number, it does not read it
Asked for a value from a table screenshot, a VLM produces the most probable token given the image representation — which is inference, not extraction. On crisp printed text that is usually right. On a low-resolution scan, a handwritten digit, or a bar without a printed label, it will still return a confident number, because the model has no mechanism for saying “those pixels are ambiguous”. A dedicated OCR engine returns a per-character confidence you can threshold. The practical pattern for documents is to use both: OCR for exact strings, the VLM for layout, structure and meaning.
Commonly confused with
A VLM takes images and text in; it is not a diffusion model, which generates images out. It is also narrower than multimodal architecture generally, which spans audio and video — and its scope is what makes the distinction worth keeping: an image-and-text model is not automatically an audio model.
When to use it
Reach for it when:
- Screenshots, charts, diagrams and documents where layout carries meaning
- Visual agent work, where the screen is the interface
- Open-ended questions about an image that a fixed extractor cannot anticipate
Reach for something else when:
- Character-exact transcription of dense or degraded text — use OCR and check confidence
- Where extracted numbers feed calculations without a verification step
- High-volume identical documents, where a trained extractor is cheaper and more reliable
Referenced by
This term is used by the following ModelRefs references:
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Vision-Language Model (VLM) — AI Glossary.
Frequently asked questions
What is Vision-Language Model (VLM)?
A model that accepts both image (and optionally video) and text inputs within a single unified context.
Is Vision-Language Model (VLM) the same as VLM?
Yes — VLM, vision model, multimodal LLM are common aliases for Vision-Language Model (VLM).
What concepts are related to Vision-Language Model (VLM)?
Closely related concepts include llm, multimodal, computer use.