Spider 2.0 Methodology and Evidence

What this benchmark measures

• enterprise text-to-SQL
• long-context schema reasoning
• agentic SQL workflows

Relevant to:

• analytics assistant selection
• database agent screening

Failure modes it exercises:

• incorrect SQL against large real schemas

Method and its limits

632 real-world text-to-SQL workflow problems over enterprise databases across several systems and dialects, requiring long-context schema handling and, in some settings, agentic interaction rather than one-shot query generation.

• Settings differ in dialect and task shape — Snowflake, DuckDB with DBT, BigQuery and SQLite — so results across settings are not interchangeable.
• The official page's per-setting example counts do not reconcile with its stated total, so the exact composition is not established from it.

Dataset

Dataset

Spider 2.0

Type

enterprise text-to-SQL workflow problems

Freshness

aging

Benchmark release

2024-11-12

632 problems in total as registered on the official page, across Spider 2.0-Snow (Snowflake), Spider 2.0-DBT (DuckDB with DBT) and Spider 2.0-Lite (BigQuery, Snowflake, SQLite). The page is listed as ICLR 2025 but states no dataset release date; the freshness date recorded here is the primary paper's publication date.

How to read this score

The official page reports o1-preview solving only 17.1% and GPT-4o only 10.1%, against 86.6% for GPT-4o on Spider 1.0. The contrast is the finding: performance on earlier text-to-SQL benchmarks does not transfer to enterprise schemas.

Similarity to real tasks: High — Established from the official source: Spider 2.0 is 632 text-to-SQL workflow problems derived from enterprise database use cases, over databases that often carry more than a thousand columns, requiring SQL that frequently exceeds 100 lines and agentic interaction with the environment rather than a single query. That is markedly closer to real analytics engineering than single-table benchmarks.

Data contamination risk: Low — Tasks require generating long SQL against large specific schemas and, in some settings, interacting with a live environment, which memorised text cannot satisfy. No run-specific audit is registered.

Benchmark gaming risk: Medium — Agentic settings depend on scaffolding and retry behaviour, which the benchmark does not fix and which are not registered per score here.

What you still need to test yourself

• Test against your own schema, dialect and permissions.
• Measure what a wrong query costs in your environment, which no benchmark captures.

This benchmark supports decisions about:

• Screen models for SQL against realistic enterprise schemas rather than toy databases.
• Set expectations: the published baselines are far below Spider 1.0 figures.

Limitations

• Results are not interchangeable across dialect settings.
• Agentic settings measure a system including scaffolding, not a model.

Official project source reviewed 2026-10-12. The source states no dataset release date; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.
• The per-setting example counts on the official page do not reconcile with the stated total, and no release date is given.

Sources

• Spider 2.0 — official project page Spider 2.0 authors · accessed 2026-10-12Supports the 632-problem total, the Snowflake, DuckDB/DBT, BigQuery and SQLite settings, the large-schema and long-SQL characterisation, and the reported 17.1% o1-preview and 10.1% GPT-4o figures against 86.6% on Spider 1.0; its per-setting counts do not reconcile with the total, and it does not support any model score, and no score is registered on this page.
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows Lei et al. · accessed 2026-10-25The primary methodology paper, establishing the benchmark's enterprise task construction and release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://spider2-sql.github.io/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Spider 2.0 Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path