DROP Leaderboard

What this benchmark measures

• reading comprehension
• discrete numerical reasoning over text

Relevant to:

• document question answering screening
• numerical extraction from prose

Failure modes it exercises:

• arithmetic and counting errors over retrieved passages

Method and its limits

Crowdsourced, adversarially created reading comprehension over paragraphs, scored on a generalized F1 accuracy metric.

• Passages are single paragraphs, so multi-document and long-context reasoning are not tested.
• The benchmark is from 2019 and frontier models score far above the original baselines, compressing differences near the top.

Dataset

Dataset

DROP

Type

adversarially crowdsourced reading comprehension questions

Freshness

aging

Benchmark release

2019-03-01

96k questions as registered in the primary paper. The date records paper submission, not a knowledge cutoff.

How to read this score

F1 is interpretable against the original protocol. The paper reports expert human performance at 96.0% and the best systems of the time at 32.7% F1, so the headroom described in the paper no longer reflects current models.

Similarity to real tasks: Medium — Established from the primary methodology: DROP requires resolving references in a question and performing discrete operations such as addition, counting, or sorting over a paragraph. That resembles extracting figures from a document more closely than multiple-choice recall does. It does not reproduce a production document pipeline, because retrieval, multi-document reconciliation, table parsing, and citation are not represented.

Data contamination risk: High — DROP has been public since 2019 and is widely included in instruction-tuning mixtures. No run-specific overlap audit is registered for the score shown here.

Benchmark gaming risk: Medium — Prompt format and answer normalisation materially affect the F1 metric, and ModelRefs registers no per-score protocol detail.

What you still need to test yourself

• Test on your own document types, including tables and multi-page sources.
• Measure citation and abstention behaviour, which DROP does not score.

This benchmark supports decisions about:

• Screen basic numeric reasoning over short passages.
• Use as a floor check rather than a differentiator between frontier models.

Limitations

• Not evidence of long-document or multi-source reasoning.
• Saturation near the top makes small differences uninformative.

Primary source reviewed 2026-10-11. DROP is treated as an aging 2019 benchmark.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.

Sources

• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs Dua, Wang, Dasigi, Stanovsky, Singh, Gardner · accessed 2026-10-11Supports the 96k question count, the discrete-reasoning design, and the 32.7% baseline and 96.0% expert human figures; it does not support any model score shown here.
• Official DROP evaluation script Allen Institute for AI · accessed 2026-10-25AI2's own DROP scoring implementation, which the institute's DROP dataset page resolves to. It establishes the official scoring procedure; the repository is archived, so it records that method rather than a maintained harness.

How this benchmark is scored

Primary source: https://arxiv.org/abs/1903.00161

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to DROP Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path