BIG-Bench Hard Leaderboard

What this benchmark measures

• multi-step symbolic and logical reasoning
• chain-of-thought benefit

Relevant to:

• reasoning evaluation portfolios
• chain-of-thought regression checks

Failure modes it exercises:

• reasoning failures on tasks that defeat direct answering

Method and its limits

23 BIG-Bench tasks selected because prior model evaluations did not outperform the average human rater, evaluated with and without chain-of-thought prompting.

• The suite is selected for difficulty, so aggregate performance is not representative of BIG-Bench or of general capability.
• The paper's own point is that few-shot prompting without chain-of-thought substantially underestimates capability, so a BBH figure is meaningless without the prompting protocol.

Dataset

Dataset

BIG-Bench Hard

Type

selected hard subset of BIG-Bench tasks

Freshness

aging

Benchmark release

2022-10-17

23 tasks as registered in the primary paper. The date records paper submission.

How to read this score

The paper reports that with chain-of-thought, PaLM surpassed average human-rater performance on 10 of 23 tasks and Codex on 17 of 23. Per-task results matter more than the aggregate, because the 23 tasks differ widely in kind.

Similarity to real tasks: Low — Established from the primary methodology: BBH is the suite of 23 BIG-Bench tasks on which prior language-model evaluations did not beat the average human rater. The tasks are deliberately artificial probes of symbolic and logical reasoning, chosen for difficulty rather than resemblance to work, so transfer to production tasks is not implied.

Data contamination risk: High — BIG-Bench and BBH are public and widely used in evaluation suites. The BIG-Bench project distributes a canary string as a leakage mitigation, but no run-specific overlap audit is registered here.

Benchmark gaming risk: High — Results depend heavily on whether chain-of-thought is used, which the paper demonstrates directly, and BBH is a common leaderboard component that is optimised against.

What you still need to test yourself

• Evaluate on your own reasoning tasks; BBH tasks are artificial by design.
• Fix a prompting protocol before comparing models, since the protocol dominates the result.

This benchmark supports decisions about:

• Check whether chain-of-thought prompting still improves a candidate model.
• Compare reasoning on tasks selected to defeat direct answering.

Limitations

• Selected for difficulty, so not representative of general capability.
• Figures are not comparable across different prompting protocols.

Primary source reviewed 2026-10-12. BBH is treated as an aging 2022 benchmark.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.
• The registered score is the base pretrained model, because its source reports BBH only for that variant and not for the instruction-tuned one.

Sources

• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them Suzgun, Scales, Schärli, Gehrmann, Tay, Chung, Chowdhery, Le, Chi, Zhou, Wei · accessed 2026-10-12Supports the 23-task selection criterion and the reported chain-of-thought results of 10 of 23 for PaLM and 17 of 23 for Codex against average human raters; it does not support any model score.
• BIG-Bench Hard task release Suzgun et al. (BBH authors) · accessed 2026-10-25The authors' release of the BBH task subset and its chain-of-thought prompts, matching the paper's title. It establishes the task selection and prompting procedure; it does not establish per-model results.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2210.09261

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BIG-Bench Hard Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path