MuSR Leaderboard

What this benchmark measures

• multistep soft reasoning over narrative
• long-form commonsense inference

Relevant to:

• narrative reasoning screening
• chain-of-thought stress testing

Failure modes it exercises:

• inference failures across a long natural-language narrative

Method and its limits

Free-text narratives of roughly 1000 words produced by a neurosymbolic synthetic-to-natural generation algorithm, requiring multistep soft reasoning to answer.

• Narratives are generated rather than naturally occurring, so linguistic variety and messiness are bounded by the generator.
• The abstract reviewed reports no numeric model or human scores, so the difficulty gap is described qualitatively rather than measured here.

Dataset

Dataset

MuSR

Type

neurosymbolically generated reasoning narratives

Freshness

aging

Benchmark release

2023-10-24

Murder-mystery narratives of roughly 1000 words are the example registered in the primary paper. The date records first paper submission; the paper was revised in March 2024.

How to read this score

The paper's stated design point is that instances challenge GPT-4 while remaining tractable for human annotators at high accuracy, so a low model score indicates reasoning difficulty rather than item ambiguity.

Similarity to real tasks: Medium — Established from the primary methodology: MuSR instances are free-text narratives of roughly 1000 words, generated by a neurosymbolic synthetic-to-natural algorithm, requiring multistep reasoning over the story. Reasoning over a long piece of prose resembles case-file and report analysis more closely than short multiple-choice items do. The narratives are synthetic constructions such as murder mysteries rather than real documents.

Data contamination risk: Low — The generation algorithm is designed to scale as models improve, so fresh instances can be produced, which limits leakage compared with a fixed public set. No run-specific audit is registered.

Benchmark gaming risk: Low — Because instances are generated rather than fixed, optimising against a specific test set is harder than for a static benchmark.

What you still need to test yourself

• Test on your own documents; generated narratives are cleaner than real case files.

This benchmark supports decisions about:

• Stress-test reasoning over long prose rather than short items.
• Compare models on a benchmark designed to resist saturation.

Limitations

• Synthetic narratives, not real-world documents.
• Not evidence of reasoning over messy or contradictory real sources.

Primary source reviewed 2026-10-12. MuSR was first published in 2023 and revised in 2024; its generator is designed to produce fresh instances.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.
• No human accuracy baseline is established; the primary abstract says only that the tasks are tractable for humans with high accuracy, and gives no figure.

Sources

• MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning Sprague, Ye, Bostrom, Chaudhuri, Durrett · accessed 2026-10-12Supports the neurosymbolic synthetic-to-natural generation, the roughly 1000-word narrative length, and the design aim of challenging GPT-4 while remaining tractable for humans; it reports no numeric scores here and does not support any model score.
• MuSR dataset and evaluation release Sprague et al. (MuSR authors) · accessed 2026-10-25The authors' MuSR release. It establishes the generated reasoning domains and the evaluation assets; it does not establish the model and human accuracies reported in the paper.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2310.16049

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MuSR Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path