MMMU-Pro Leaderboard

What this benchmark measures

• multimodal reasoning
• genuine image dependence
• vision-only comprehension

Relevant to:

• vision-language model screening
• multimodal shortcut detection

Failure modes it exercises:

• reliance on text shortcuts instead of reading the image

Method and its limits

A robust version of MMMU built in three steps: filtering out questions text-only models can answer, expanding the candidate answer set, and adding a vision-only setting with the question embedded in the image.

• Still multiple choice, so it does not test open-ended multimodal generation.
• Expanding the option set lowers scores for reasons partly unrelated to understanding.

Dataset

Dataset

MMMU-Pro

Type

filtered multi-discipline multimodal multiple-choice questions

Freshness

aging

Benchmark release

2024-09-04

Derived from MMMU via the three-step process registered in the primary paper. The date records paper submission.

How to read this score

The paper reports that model performance falls by 16.8% to 26.9% relative to MMMU, so MMMU-Pro scores must never be compared against MMMU scores.

Similarity to real tasks: Medium — Established from the primary methodology: MMMU-Pro removes questions answerable by text-only models, increases the number of candidate answers, and adds a vision-only setting in which the question is embedded inside the image. Requiring the model to read a question rendered in an image resembles document and screenshot workloads more closely than standard multimodal multiple choice. It remains academic multiple choice rather than an applied task.

Data contamination risk: Medium — MMMU-Pro was constructed specifically to defeat shortcut answering, which mitigates some leakage, but the source MMMU items are public. No run-specific audit is registered.

Benchmark gaming risk: Medium — Prompt format and whether the vision-only or standard setting is used change results substantially, and ModelRefs does not register which setting produced the score shown.

What you still need to test yourself

• Test on your own image types and resolutions.
• Evaluate open-ended multimodal output, which this benchmark does not cover.

This benchmark supports decisions about:

• Check whether a multimodal model genuinely uses the image rather than text shortcuts.
• Screen candidates for document and diagram comprehension.

Limitations

• Scores are not comparable with MMMU.
• Not evidence of open-ended multimodal generation quality.

Primary source reviewed 2026-10-11. MMMU-Pro is treated as an aging 2024 benchmark.

What this evidence does not cover

• Only one model is currently scored on this page, and the evaluation setting behind that score is not registered.

Sources

• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark Yue, Zheng, Ni, Wang, Zhang, Tong, Sun, Yu, Zhang, Sun, Su, Chen, Neubig · accessed 2026-10-11Supports the three-step construction, the vision-only setting, and the stated 16.8-26.9% performance drop relative to MMMU; it does not support any model score shown here.
• MMMU-Pro evaluation code MMMU-Benchmark (MMMU-Pro authors) · accessed 2026-10-25The authors' evaluation repository, which states it contains the evaluation code for the MMMU-Pro paper cited beside it and carries a dedicated mmmu-pro directory covering the standard and vision-only input settings. It establishes the official evaluation settings; it does not establish which setting produced the score shown here.

How this benchmark is scored

Primary source: https://mmmu-benchmark.github.io/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MMMU-Pro Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path