ModelRefs / ChartQA Leaderboard
ChartQA Leaderboard
Question answering over real-world charts. Current leaders, methodology, and citation sources for ChartQA.
Overview
Question answering over real-world charts.
How it is measured: Relaxed accuracy across human + augmented splits.
What this benchmark measures
- chart visual understanding
- chart question answering
- logical and arithmetic reasoning over chart data
Relevant to:
- chart question-answering screening
- document and dashboard analysis evaluation
- visual data extraction portfolios
- financial reporting and variance-chart failure-case screening
Failure modes it exercises:
- misreading chart labels or values
- incorrect arithmetic or logical operations over chart data
- failure across human-authored and generated questions
Method and its limits
Evaluate question answering over chart images using human-authored and machine-generated question sets, with visual, tabular, logical, and arithmetic reasoning under the pinned ChartQA split and metric.
- Human-authored and generated question subsets have different construction and difficulty characteristics and should be reported separately when available.
- Relaxed accuracy, answer normalization, OCR, chart rendering, and access to underlying tables can materially affect results.
- Repository annotations include acknowledged noise, and the chart-type distribution does not cover every real dashboard or visualization.
Dataset
- Dataset
- ChartQA
- Type
- chart images with human-authored and generated visual question-answer pairs
- Freshness
- aging
- Benchmark release
- 2023-11-27
The paper reports 9.6K human-written and 23.1K generated questions. The official repository separates human and augmented splits and documents chart annotations and known noise.
How to read this score
Inspect human and augmented splits, chart types, answer categories, OCR conditions, and error classes. Aggregate relaxed accuracy is not proof of reliable chart interpretation.
Similarity to real tasks: Medium — The benchmark uses chart images and questions relevant to visual analytics, but a controlled single-chart answer does not reproduce messy dashboards, provenance checks, multi-chart synthesis, or consequential business decisions.
Data contamination risk: High — The dataset and charts are public; no model-specific training-overlap audit is registered. ModelRefs cannot rule out training overlap for any specific model run, so contamination exposure should be treated as high unless a score source documents decontamination.
Benchmark gaming risk: High — Benchmark-specific OCR, table extraction, answer normalization, and tuning can affect performance; no run-level gaming assessment is registered. Those are run conditions rather than properties of the model, so cross-run comparison requires matched conditions.
What you still need to test yourself
- Test the target dashboard and report formats, OCR quality, accessibility variants, chart types, units, annotations, and source provenance.
- Measure extraction accuracy, calculation accuracy, abstention, citation to chart elements, latency, and human-review outcomes separately.
- Use reconciled internal reports and charts to test materiality, period/entity selection, source-of-truth links, and reviewer correction.
This benchmark supports decisions about:
- Screen multimodal candidates for chart reading and bounded chart reasoning under a matched ChartQA protocol.
- Identify chart types, visual features, and arithmetic operations that require deeper workload testing.
- Expose chart-label, unit, legend, OCR, and arithmetic risks that finance-report and variance-analysis test sets should cover.
Limitations
- ChartQA does not establish reliable decision-making from enterprise dashboards or high-stakes charts.
- A high aggregate score can hide failures on uncommon chart types, OCR, units, legends, visual clutter, or multi-step calculations.
- It does not establish financial-statement accuracy, variance causality, materiality, control effectiveness, or audit evidence.
Sources reviewed 2026-07-02. Dataset freshness remains unknown; repository update dates are not treated as chart-content freshness dates.
What this evidence does not cover
- No authoritative current-world or dataset-freshness cutoff is established for the public chart corpus.
- Every score still needs exact split, metric, OCR or table-access condition, prompt, model version, and source-scoped run evidence.
Sources
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning ChartQA authors / ACL Findings 2022 · accessed 2026-07-02Supports the human and generated question sets, chart reasoning scope, and benchmark design; it does not establish production dashboard reliability.
- ChartQA official repository vis-nlp / ChartQA authors · accessed 2026-07-02Supports the dataset splits, chart images, underlying tables, annotations, and documented annotation noise; exact repository revision and evaluation conditions remain run-level requirements.
How this benchmark is scored
| Category | vision |
|---|---|
| Maximum score | 100 % accuracy |
| Direction | Higher is better |
Primary source: https://github.com/vis-nlp/ChartQA
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ChartQA Leaderboard.
Frequently asked questions
What does the ChartQA benchmark measure?
Question answering over real-world charts.
What is the methodology?
Relaxed accuracy across human + augmented splits.
Which AI model is the current ChartQA leader?
Llama 4 Scout (Meta) currently leads with 90 % accuracy.