SimpleQA Leaderboard

What this benchmark measures

• short-form factual accuracy
• abstention behaviour

Relevant to:

• factual assistant screening
• hallucination risk assessment

Failure modes it exercises:

• confident incorrect answers to short factual questions

Method and its limits

Short fact-seeking questions adversarially collected against GPT-4 responses, each with a single indisputable answer, graded as correct, incorrect, or not attempted.

• Questions were adversarially collected against GPT-4, so the difficulty profile is tuned against one model family.
• A single indisputable answer excludes the ambiguous or contested questions that cause real-world factuality failures.

Dataset

Dataset

SimpleQA

Type

adversarially collected short fact-seeking questions

Freshness

aging

Benchmark release

2024-11-07

The primary paper registers the adversarial collection method and the three-way grading scheme. The date records paper submission; answers encode world knowledge as of that period. The published test set contains 4,326 questions, counted from the official CSV that OpenAI's reference implementation loads, parsing quoted fields rather than counting lines.

How to read this score

The benchmark is designed to measure whether models know what they know, so the correct, incorrect, and not-attempted rates must be read together. A single accuracy number omits the abstention behaviour that is the point of the benchmark.

Similarity to real tasks: Medium — Established from the primary methodology: SimpleQA contains short fact-seeking questions each with a single indisputable answer, graded as correct, incorrect, or not attempted. Short factual lookup resembles a common assistant request, and the three-way grading captures whether a model declines when unsure. It does not test retrieval, citation, or multi-step research.

Data contamination risk: Medium — The dataset is published by the benchmark authors and has been public since 2024. No run-specific overlap audit is registered.

Benchmark gaming risk: Medium — Because abstention is graded separately, a model can raise its correct-when-attempted rate by declining more often; the headline figure chosen therefore changes the ranking.

What you still need to test yourself

• Add retrieval and grounding, which the benchmark deliberately excludes.
• Define your own tolerance for a wrong answer versus a refusal.

This benchmark supports decisions about:

• Compare how often models answer incorrectly rather than decline.
• Estimate hallucination exposure for ungrounded factual questions.

Limitations

• Not evidence of grounded or retrieval-augmented factual accuracy.
• Single-answer design excludes the ambiguous questions where factuality failures are most costly.

Primary source reviewed 2026-10-11. SimpleQA is treated as an aging 2024 benchmark whose answers encode knowledge from that period.

What this evidence does not cover

• Two models are currently scored on this page, and which of the three grades each score represents is not registered.

Sources

• Measuring short-form factuality in large language models Wei, Karina, Chung, Jiao, Papay, Glaese, Schulman, Fedus · accessed 2026-10-11Supports the adversarial collection against GPT-4, the single-indisputable-answer design, and the correct/incorrect/not-attempted grading; it does not establish the question count and does not support any model score shown here.
• SimpleQA reference implementation and published test set OpenAI · accessed 2026-10-25OpenAI's reference implementation, which loads the published SimpleQA test set and defines the CORRECT / INCORRECT / NOT_ATTEMPTED grading. It establishes the grading scheme and the dataset's question count: the published test set contains 4,326 questions. It does not establish which grade any score shown here represents.

How this benchmark is scored

Primary source: https://openai.com/index/introducing-simpleqa/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to SimpleQA Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path