DS-1000 Methodology and Evidence

What this benchmark measures

• data science code generation
• library API usage

Relevant to:

• analyst assistant screening
• data pipeline code generation

Failure modes it exercises:

• incorrect or non-idiomatic use of data science libraries

Method and its limits

1,000 data science problems derived from 451 unique StackOverflow questions across seven libraries, scored by multi-criteria automatic evaluation combining test-case execution with surface-form constraints on API and keyword usage.

• Problems reflect what StackOverflow users asked, which skews toward difficulties people got stuck on rather than routine work.
• Surface-form constraints make the metric stricter than functional correctness alone, so scores are not comparable with execution-only benchmarks.

Dataset

Dataset

DS-1000

Type

StackOverflow-derived data science coding problems

Freshness

aging

Benchmark release

2022-11-18

1,000 problems from 451 unique StackOverflow problems across seven libraries, as registered on the official page. The page states no release date; the freshness date recorded here is the primary paper's publication date.

How to read this score

The official page reports that only 1.8% of accepted Codex-002 solutions were in fact incorrect, which is an unusually strong statement of metric reliability, and that Codex-002 reached 43.3%.

Similarity to real tasks: High — Established from the official source: DS-1000 draws 1,000 problems from 451 unique StackOverflow questions across NumPy, SciPy, Pandas, TensorFlow, PyTorch, Scikit-learn and Matplotlib, and evaluates both functional correctness and surface-form constraints such as restricted API usage. Problems that practitioners actually asked, checked for both correctness and acceptable form, is close to real analyst work.

Data contamination risk: Medium — The authors proactively perturb problems against memorisation — the official page registers 152 surface perturbations, 235 semantic perturbations and 162 difficult rewrites, with more than half of all problems modified. The page does not quantify how effective those defences are, and the StackOverflow originals remain public.

Benchmark gaming risk: Low — Surface-form constraints plus perturbed problems make it harder to satisfy by recall or by a single memorised idiom than an execution-only benchmark.

What you still need to test yourself

• Test against your own library versions; API changes break idioms these problems assume.

This benchmark supports decisions about:

• Screen assistants for practical data science code rather than puzzle-style tasks.
• Prefer a benchmark with explicit anti-memorisation design when contamination is a concern.

Limitations

• StackOverflow-derived problems skew toward where people got stuck.
• Scores are not comparable with execution-only code benchmarks.

Official project source reviewed 2026-10-12. The source states no release date; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.
• The effectiveness of the perturbation defences is asserted by design rather than measured, and the release date is not stated.

Sources

• DS-1000 — official project page DS-1000 authors · accessed 2026-10-12Supports the 1,000 problems from 451 unique StackOverflow questions, the seven libraries, the multi-criteria evaluation, the perturbation counts, and the 1.8% false-acceptance and 43.3% Codex-002 figures; it does not support any model score, and no score is registered on this page.
• DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation Lai et al. · accessed 2026-10-25The primary methodology paper, establishing the benchmark's problem sourcing and release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://ds1000-code-gen.github.io/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to DS-1000 Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path