ModelRefs / VQAv2 Leaderboard
VQAv2 Leaderboard
Open-ended visual question answering over COCO images. Current leaders, methodology, and citation sources for VQAv2.
Overview
Open-ended visual question answering over COCO images.
How it is measured: VQA accuracy with 10-annotator agreement.
What this benchmark measures
- open-ended visual question answering
- image grounding
Relevant to:
- vision-language model screening
- image question answering
Failure modes it exercises:
- answering image questions from language priors rather than the image
Method and its limits
Open-ended questions over COCO photographs, each with ten independently collected ground-truth answers, scored by an automatic accuracy metric over that answer set.
- Answers are short and open-ended, so the metric rewards matching common phrasing rather than understanding.
- The official source documents the dataset release but does not publish the test-dev split definition or the accuracy formula, so the exact protocol behind any reported figure is not established from it.
Dataset
- Dataset
- VQA v2.0
- Type
- balanced open-ended visual question answering over COCO images
- Freshness
- aging
- Benchmark release
- 2017-04-01
204,721 COCO images, 1,105,904 questions and 11,059,040 ground-truth answers in the April 2017 v2.0 full release, as registered on the official site. Earlier v1.0 and v1.9 releases are separate and not interchangeable.
How to read this score
Accuracy is interpretable only against a stated split and answer-normalisation rule. Because frontier multimodal models now score far above the 2017 baselines, the benchmark functions as a floor check.
Similarity to real tasks: Medium — Established from the official dataset source: VQA v2.0 pairs 204,721 COCO images with 1,105,904 open-ended questions, each carrying ten ground-truth answers, and the v2 release is explicitly balanced so that language priors alone are insufficient. Open-ended questions about photographs resemble real image-understanding requests. Answers are short, so it does not test description, reasoning over documents, or multi-image comparison.
Data contamination risk: High — VQAv2 has been public since 2017 and is standard pretraining and evaluation data for vision-language models. No run-specific overlap audit is registered.
Benchmark gaming risk: Medium — The v2 balancing was introduced specifically to defeat language-prior shortcuts, which mitigates the original failure mode, but answer-normalisation rules still affect reported accuracy.
What you still need to test yourself
- Test on your own imagery and question types; COCO photographs are not documents or screenshots.
- Establish the split and answer-normalisation rule before comparing any published figures.
This benchmark supports decisions about:
- Use as a floor check that a multimodal model grounds answers in the image.
Limitations
- Short-answer format; not evidence of description or multimodal reasoning quality.
- Effectively a floor check for current models.
Official dataset source reviewed 2026-10-12. VQAv2 is treated as an aging 2017 benchmark.
What this evidence does not cover
- Only one model is currently scored on this page, so no comparison is supported.
- The split behind the registered score is not stated by its source, so it cannot be placed against a VQAv2 figure whose split is stated.
Sources
- Visual Question Answering — official dataset site VQA team · accessed 2026-10-12Supports the v2.0 dataset release of 204,721 images, 1,105,904 questions and ten ground-truth answers per question, and the release history; it does not document any individual split or the accuracy formula, and it does not support any model score.
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering Goyal et al. · accessed 2026-10-25The primary methodology paper for VQA v2, establishing the balanced question-image construction that distinguishes it from v1. It does not establish the definition of any individual evaluation split.
How this benchmark is scored
| Category | vision |
|---|---|
| Maximum score | 100 % accuracy |
| Direction | Higher is better |
Primary source: https://visualqa.org/
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to VQAv2 Leaderboard.
Frequently asked questions
What does the VQAv2 benchmark measure?
Open-ended visual question answering over COCO images.
What is the methodology?
VQA accuracy with 10-annotator agreement.
Which AI model is the current VQAv2 leader?
Pixtral 12B (Mistral) currently leads with 78.6 % accuracy.