ModelRefs / LiveCodeBench — AI Glossary
LiveCodeBench — AI Glossary
A contamination-free code generation benchmark drawing problems from timed competitive programming contests after a model's training cutoff.
Overview
LiveCodeBench (Jain et al. 2024) continuously scrapes new problems from LeetCode, AtCoder, and Codeforces published after each model's training cutoff, preventing memorization. Evaluates code generation, execution, and self-repair. More realistic than HumanEval; frontier models score 55–75% Pass@1.
Reference details
| Topic | evaluation |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: The same benchmark, a different exam per model
Problems are timestamped from when the contest published them, and a model is evaluated only on problems dated after its own training cutoff. That is the point — it cannot have memorised them. It is also the caveat: a model released in March and one released in September are scored on different problem sets, of different difficulty, drawn from different contests. Two papers can both report “LiveCodeBench” and not be comparing the same exam. Any comparison you make must state the evaluation window, or it is not a comparison.
Commonly confused with
It is not a replacement for SWE-bench. LiveCodeBench asks for a self-contained solution to a stated competitive-programming problem; SWE-bench asks for a patch to a real repository that makes a failing test pass. Competitive-programming skill and codebase navigation are different capabilities, and models rank differently on them.
When to use it
Reach for it when:
- Checking algorithmic coding ability without contamination from memorised training data
- Tracking one model family over time, where the window can be held fixed
- Alongside a repository-level benchmark, not instead of one
Reach for something else when:
- Cross-model league tables that do not state the problem window
- Predicting performance on production engineering work — the task shape is unrelated
- Evaluating a model on problems predating its cutoff, which silently reintroduces contamination
Primary source
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LiveCodeBench — AI Glossary.
Frequently asked questions
What is LiveCodeBench?
A contamination-free code generation benchmark drawing problems from timed competitive programming contests after a model's training cutoff.
What concepts are related to LiveCodeBench?
Closely related concepts include swe bench, passatk, bigcodebench.