RepoBench Methodology and Evidence

What this benchmark measures

• repository-level code completion
• cross-file context retrieval

Relevant to:

• coding assistant selection
• repository context handling

Failure modes it exercises:

• completion failures when the needed context is in another file

Method and its limits

Three sub-tasks over real repositories: RepoBench-R for retrieving cross-file context, RepoBench-C for next-line prediction from cross-file and in-file context, and RepoBench-P for the combined pipeline.

• Coverage is Python and Java only, so other ecosystems are not represented.
• Next-line prediction is narrower than the multi-line, multi-file edits an assistant actually performs.

Dataset

Dataset

RepoBench

Type

repository-level completion and retrieval tasks

Freshness

aging

Benchmark release

2023-06-05

Python and Java, across the three registered sub-tasks. The date records first submission; the paper was revised in October 2023.

How to read this score

The three sub-tasks should be read separately: a strong pipeline score can come from retrieval quality rather than completion quality, and the benchmark is designed to let those be distinguished.

Similarity to real tasks: High — Established from the primary methodology: RepoBench evaluates repository-level auto-completion through three sub-tasks — retrieving relevant cross-file snippets, next-line prediction using cross-file and in-file context, and a combined pipeline. The paper's stated motivation is that existing benchmarks cover single-file tasks and miss multi-file programming, so this is markedly closer to assistant work than function-level synthesis.

Data contamination risk: High — The tasks are drawn from public repositories, which are core training data for code models. No run-specific overlap audit is registered.

Benchmark gaming risk: Medium — Results depend on the retrieval component and context window supplied, so scaffolding rather than the model can dominate the pipeline result.

What you still need to test yourself

• Test on your own repositories and languages; coverage here is Python and Java.
• Evaluate multi-line edits, which next-line prediction does not represent.

This benchmark supports decisions about:

• Compare assistants on multi-file context rather than isolated functions.
• Separate retrieval quality from completion quality.

Limitations

• Python and Java only.
• Next-line prediction is narrower than real assistant edits.

Primary source reviewed 2026-10-12. RepoBench is treated as an aging 2023 benchmark.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.

Sources

• RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems Liu, Xu, McAuley · accessed 2026-10-12Supports the three sub-task structure, the repository-level multi-file motivation, and the Python and Java coverage; it does not support any model score, and no score is registered on this page.
• RepoBench dataset and evaluation release Liu et al. (RepoBench authors) · accessed 2026-10-25The authors' release, matching the paper's title. It establishes the retrieval and completion task construction; it does not establish per-model results.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2306.03091

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to RepoBench Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path