Multimodal AI — AI Glossary

Reference details

Related terms

• Vision-Language Model (VLM)
• ASR (Automatic Speech Recognition)
• TTS (Text-to-Speech)
• Large Language Model

Example: Why native beats a pipeline

A bolted-together pipeline runs OCR over a chart, extracts “Q3: 42”, and hands text to the model — the axis labels, the trend line and the legend are gone before the model sees anything. A natively multimodal model tokenises the image into the same embedding space as text, so “is the Q3 dip seasonal?” can be answered from the shape of the line, not just the extracted number.

Commonly confused with

Accepting an image is not the same as reasoning over it. Many systems described as multimodal are a text model behind an OCR or captioning step, which discards layout, colour and spatial relationships. Native multimodality means the modality reaches the model as tokens, not as someone else's transcription.

When to use it

Reach for it when:

• Layout, position or visual structure carries meaning — charts, forms, screenshots
• OCR would destroy the information you actually need
• The task spans modalities: describe this diagram, then write the code

Reach for something else when:

• The document is clean text — OCR plus a text model is cheaper and often more accurate
• Images arrive at volume; vision tokens are expensive and inflate context fast
• You need deterministic extraction from a fixed form, where a parser beats a model

Referenced by

This term is used by the following ModelRefs references:

• vision language stack
• speech pipeline stack
• voice realtime stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Multimodal AI — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path