ModelRefs / SWE-Bench Methodology — Methodology

SWE-Bench Methodology — Methodology

SWE-Bench measures whether a model can resolve real GitHub issues end-to-end in popular Python repositories.

Overview

What it measures: Real-world software-engineering capability: navigating a repo, editing files, and passing the project's own tests.

How it works

  • 2,294 issue-pull-request pairs scraped from 12 high-quality Python repos.
  • Model receives the issue text plus repo state.
  • Model must produce a patch that passes the hidden tests attached to the merged PR.
  • Variants: SWE-Bench Lite (300), SWE-Bench Verified (500, human-validated).

Strengths

  • End-to-end realism
  • Test-grounded scoring
  • Harder than code-snippet benchmarks

Limitations

  • Python only
  • Repo-shape bias toward Django/sklearn-style projects
  • Sandbox infrastructure non-trivial

Best use cases

  • Coding-agent evaluation
  • Tool-use + long-context assessment

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to SWE-Bench Methodology — Methodology.

Frequently asked questions

What does SWE-Bench measure?

Real-world software-engineering capability: navigating a repo, editing files, and passing the project's own tests.

What are its main limitations?

Python only Repo-shape bias toward Django/sklearn-style projects Sandbox infrastructure non-trivial

When should I use this benchmark?

Coding-agent evaluation Tool-use + long-context assessment