CRUXEval Leaderboard

What this benchmark measures

• code execution reasoning
• input and output prediction

Relevant to:

• code comprehension screening
• debugging assistant evaluation

Failure modes it exercises:

• mispredicting what a function actually does

Method and its limits

800 Python functions with input-output pairs, split into CRUXEval-I for input prediction and CRUXEval-O for output prediction, reported as pass@1 and pass@5.

• Functions were generated by Code Llama 34B and filtered to what a good human programmer could handle without extra memory in about a minute, so difficulty is bounded by that construction.
• The official page reports pass@1 and pass@5 without describing the scoring procedure, so protocol detail is not established from it.

Dataset

Dataset

CRUXEval

Type

generated Python functions with input-output pairs

Freshness

aging

Benchmark release

2024-01-05

800 samples randomly selected after filtering, as registered on the official page. The page states no dataset release date; the freshness date recorded here is the primary paper's publication date.

How to read this score

The official page reports GPT-4 at 67% on input prediction and 63% on output prediction, rising to 74.8% and 81.9% with chain-of-thought, and notes that fine-tuned performance plateaus under 70%. A figure is therefore meaningless without its prompting mode.

Similarity to real tasks: Medium — Established from the official source: CRUXEval asks a model to predict the input that produces a given output, or the output of a given input, over 800 Python functions. Reasoning about what code does without running it is what a developer does when reviewing or debugging. The functions are short and self-contained, so no repository context, state or side effects are involved.

Data contamination risk: Medium — The function set is published. Execution prediction is harder to satisfy by recall than answer lookup, but no run-specific overlap audit is registered.

Benchmark gaming risk: Medium — The official page reports that fine-tuning Code Llama 34B on similar assertions matches GPT-4 on both tasks, so the benchmark can be trained for directly.

What you still need to test yourself

• Test on your own code, which carries state, dependencies and side effects these functions do not.

This benchmark supports decisions about:

• Screen whether a model reasons about execution rather than pattern-matching syntax.
• Compare chain-of-thought benefit on code understanding.

Limitations

• Short self-contained functions, not real codebases.
• Figures are not comparable across prompting modes.

Official benchmark source reviewed 2026-10-12. The source states no release date; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.
• The benchmark's own pass@k scoring procedure is not established from the official source reviewed, and the registered score is reported as accuracy rather than pass@k.
• The registered score covers CRUXEval's input-prediction task only; output prediction is scored separately and is not registered here.

Sources

• CRUXEval — official benchmark page CRUXEval authors · accessed 2026-10-12Supports the 800-function composition, the input and output prediction tasks, the Code Llama 34B generation and human-solvability filtering, and the reported GPT-4 and chain-of-thought figures; it does not support any model score.
• CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution Gu et al. · accessed 2026-10-25The primary methodology paper, establishing the benchmark's task construction and release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://crux-eval.github.io/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to CRUXEval Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path