∞Bench Methodology and Evidence

What this benchmark measures

• very long context processing
• reasoning over 100k-plus token inputs

Relevant to:

• long-document model selection
• book and codebase scale analysis

Failure modes it exercises:

• failure on inputs an order of magnitude longer than typical benchmarks

Method and its limits

Twelve tasks over 100k-plus token contexts spanning summarisation, question answering, multiple choice, dialogue speaker identification, code execution simulation, maths, and key or key-value retrieval, scored by accuracy with ROUGE variants for the free-text tasks.

• The suite mixes real-world and synthetic tasks with very different difficulty profiles, so an aggregate is not meaningful.
• The official page's per-task average input lengths range widely and several fall below the headline 100k framing, so the stated scale is not uniform across tasks.

Dataset

Dataset

InfiniteBench

Type

mixed real and synthetic very-long-context tasks

Freshness

aging

Benchmark release

2024-02-21

Twelve tasks with per-task average input lengths reported on the official page. The page states no release date; the freshness date recorded here is the primary paper's publication date.

How to read this score

Per-task results are essential: the official page records GPT-4 at 100% on passkey retrieval but 23.25% on code execution simulation, and several models below 5% on the hardest tasks. A single aggregate would hide a 20-fold spread.

Similarity to real tasks: Medium — Established from the official source: InfiniteBench covers twelve tasks over contexts of 100k-plus tokens, mixing real-world material such as books, scripts and code documents with synthetic constructs such as key retrieval and code execution simulation. The real-world half resembles genuine long-document work; the synthetic half is diagnostic.

Data contamination risk: Medium — Real-world source material such as books and scripts is likely represented in training corpora, while the synthetic retrieval tasks are not memorisable. No run-specific audit is registered.

Benchmark gaming risk: Low — Synthetic retrieval and execution tasks require processing the supplied context rather than recalling an answer, which limits shortcut strategies.

What you still need to test yourself

• Measure cost and latency at your real context length, which the benchmark does not report.

This benchmark supports decisions about:

• Compare long-context models on tasks beyond simple retrieval.
• Identify which long-context capability a candidate actually lacks.

Limitations

• An aggregate across twelve heterogeneous tasks is not meaningful.
• Not evidence of cost or latency at long context.

Official repository reviewed 2026-10-12. The source states no release date; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.
• The release date is not stated, and per-task input lengths are not uniform with the 100k-plus framing.

Sources

• InfiniteBench — official repository OpenBMB · accessed 2026-10-12Supports the twelve-task composition, the 100k-plus token framing, the mix of real-world and synthetic tasks, the accuracy and ROUGE metrics, and the reported GPT-4 figures of 100% on passkey retrieval and 23.25% on code execution; it does not support any model score, and no score is registered on this page.
• $\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens Zhang et al. · accessed 2026-10-25The primary methodology paper, establishing the benchmark's task construction and release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://github.com/OpenBMB/InfiniteBench

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ∞Bench Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path