LongBench v2 Leaderboard

What this benchmark measures

• long-context understanding
• reasoning over very long inputs

Relevant to:

• long-context model selection
• document and repository comprehension

Failure modes it exercises:

• comprehension collapse as context length grows

Method and its limits

503 multiple-choice questions across six task categories, with contexts from 8k to 2M words, scored by accuracy and reported both with and without chain-of-thought and broken down by length band.

• Multiple choice allows a correct answer by elimination without genuine comprehension of the context.
• 503 questions spread across six categories and three length bands leaves small per-cell samples.

Dataset

Dataset

LongBench v2

Type

long-context multiple-choice comprehension tasks

Freshness

aging

Benchmark release

2024-12-19

503 questions, six task categories, contexts from 8k to 2M words with most under 128k, and length bands of Short 0-32k, Medium 32k-128k and Long 128k-2M, as registered on the official site. The site states no release date; the freshness date recorded here is the primary paper's publication date.

How to read this score

The official source reports human experts at 53.7% under a 15-minute limit, the best directly-answering model at 50.1%, and o1-preview with longer reasoning at 57.7%. A near-human score therefore reflects a time-limited human baseline, not a solved benchmark.

Similarity to real tasks: High — Established from the official source: LongBench v2 covers six task categories including single- and multi-document question answering, long in-context learning, long dialogue history, code repository comprehension and structured data analysis, over contexts from 8k to 2M words. Those are the real shapes of long-context work. The multiple-choice answer format is narrower than the open-ended output such work usually requires.

Data contamination risk: Medium — The question set is public. Very long contexts are harder to memorise wholesale than short items, but no run-specific overlap audit is registered.

Benchmark gaming risk: Medium — Reported figures differ substantially with and without chain-of-thought, which the official leaderboard separates, so a single quoted number is ambiguous without that setting.

What you still need to test yourself

• Test open-ended output over your own long documents; multiple choice flatters comprehension.
• Measure cost and latency at your actual context length.

This benchmark supports decisions about:

• Select models for genuinely long inputs rather than nominal context windows.
• Compare degradation across length bands rather than a single figure.

Limitations

• Multiple choice understates the difficulty of open-ended long-context work.
• The human baseline is time-limited, so exceeding it does not mean exceeding human comprehension.

Official benchmark source reviewed 2026-10-12. The source states no release date; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.
• The registered score is a single aggregate across contexts from 8k to 2M words, so it does not establish behaviour at any specific context length.

Sources

• LongBench v2 — official benchmark site LongBench authors · accessed 2026-10-12Supports the 503-question composition, the six task categories, the 8k to 2M word range and length bands, the with and without chain-of-thought reporting, and the 53.7% human expert, 50.1% best direct model and 57.7% o1-preview figures; it does not support any model score recorded on this page.
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks Bai et al. · accessed 2026-10-25The primary methodology paper, establishing the benchmark's task construction and release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://longbench2.github.io/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LongBench v2 Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path