GAIA Methodology and Evidence

What this benchmark measures

• tool use
• web browsing
• multi-modality handling
• multi-step task completion

Relevant to:

• general assistant evaluation
• agent capability screening

Failure modes it exercises:

• failure to complete multi-step real-world assistant tasks

Method and its limits

466 curated questions requiring reasoning, multi-modality handling, web browsing and tool use, of which the answers to 300 are withheld to power a leaderboard.

• Each question resolves to one checkable answer, so open-ended assistant work is not represented.
• Scores depend heavily on the agent scaffolding and tool access supplied, which the benchmark does not fix.

Dataset

Dataset

GAIA

Type

curated general-assistant questions with withheld answers

Freshness

aging

Benchmark release

2023-11-21

466 questions with 300 answers withheld for the leaderboard, as registered in the primary paper. The date records paper submission.

How to read this score

The paper reports human respondents at 92% against 15% for GPT-4 with plugins, a gap the authors frame as the point of the benchmark. Any quoted figure must name the scaffolding that produced it.

Similarity to real tasks: High — Established from the primary methodology: GAIA questions require reasoning, multi-modality handling, web browsing and general tool-use proficiency, and are described by the authors as conceptually simple for humans yet challenging for assistants. Needing to browse, use tools and combine sources is what a real assistant request looks like. Questions are curated with a single checkable answer, which real requests rarely have.

Data contamination risk: Low — Withholding the answers to 300 questions is a deliberate leakage mitigation, though the questions themselves are released. No run-specific audit is registered.

Benchmark gaming risk: High — Results are dominated by agent scaffolding, tool access and browsing capability rather than the base model, so figures are not comparable across differing harnesses.

What you still need to test yourself

• Rebuild with your own tools and browsing permissions, which dominate the result.
• Evaluate open-ended requests, which single-answer grading excludes.

This benchmark supports decisions about:

• Compare agent systems, not base models, on realistic assistant tasks.
• Estimate the gap between assistant systems and human performance.

Limitations

• Measures an agent system including its scaffolding, not a model.
• Single-answer grading excludes open-ended assistant work.

Primary source reviewed 2026-10-12. GAIA is treated as an aging 2023 benchmark.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.
• The difficulty level structure is not established from the abstract reviewed.

Sources

• GAIA: a benchmark for General AI Assistants Mialon, Fourrier, Swift, Wolf, LeCun, Scialom · accessed 2026-10-12Supports the 466-question set with 300 withheld answers, the required capabilities, and the stated 92% human versus 15% GPT-4-with-plugins figures; it does not support any model score, and no score is registered on this page.
• GAIA dataset distribution gaia-benchmark (GAIA authors) · accessed 2026-10-25The dataset distribution published under the gaia-benchmark organization and citing the GAIA paper. It states that each of the three levels is divided into a fully public dev set and a test set with private answers, and totals more than 450 questions. Access is gated behind accepting the dataset's conditions, so the files themselves are not anonymously retrievable.

How this benchmark is scored

Primary source: https://huggingface.co/gaia-benchmark

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GAIA Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path