ModelRefs / SWE-Bench — AI Glossary
SWE-Bench — AI Glossary
A benchmark that tests AI systems' ability to resolve real GitHub issues in Python open-source repositories. Also called SWE-Bench Verified.
Overview
SWE-Bench tasks an agent with understanding a bug report, navigating a codebase, and writing a code patch verified by existing tests. SWE-Bench Verified (human-validated) is the authoritative version. Scores reflect end-to-end software engineering ability.
Reference details
| Topic | evaluation |
|---|---|
| Also known as | SWE-Bench Verified |
| Last reviewed | 2026-06-24 |
Related terms
Example: The number describes a system, not a model
The task is: read an issue, find the relevant files in a real repository, write a patch, and have the maintainers' existing tests pass. Almost none of that is generation. Retrieving the right files, deciding how many attempts to make, whether to run tests between edits, how much of the repository to put in context — all of it lives in the scaffold around the model, and all of it moves the result. Two reported figures for the same model can differ substantially because the harnesses differ. Read any result as a claim about model *and* scaffold, and treat a comparison that does not describe the harness as uninterpretable.
Commonly confused with
The Verified subset is human-validated for solvable, well-specified issues; the full set contains tasks with underspecified reports and tests that cannot be passed from the issue alone. They are different exams and figures are not interchangeable. This is also not a code-generation benchmark in the usual sense — writing a correct function from a clear spec is a different skill from locating what to change in an unfamiliar codebase.
When to use it
Reach for it when:
- Assessing agentic engineering ability where navigation and iteration are the point
- Comparing scaffolds against a fixed model, which is what the harness sensitivity makes it good for
- Alongside a self-contained coding benchmark, which isolates generation from navigation
Reach for something else when:
- Comparing models across papers without identical harnesses and the same subset
- Generalising beyond Python and this repository set
- As a hiring-style proxy for engineering judgement — passing tests is not the same as a good patch
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to SWE-Bench — AI Glossary.
Frequently asked questions
What is SWE-Bench?
A benchmark that tests AI systems' ability to resolve real GitHub issues in Python open-source repositories.
Is SWE-Bench the same as SWE-Bench Verified?
Yes — SWE-Bench Verified are common aliases for SWE-Bench.
What concepts are related to SWE-Bench?
Closely related concepts include evaluation benchmark, humaneval.