MBPP+ Leaderboard

What this benchmark measures

• code correctness under rigorous testing
• robustness of generated solutions

Relevant to:

• code generation screening
• detecting fragile solutions

Failure modes it exercises:

• solutions that pass weak tests but fail edge cases

Method and its limits

The MBPP problem set re-tested under EvalPlus, which the project states carries 35x more tests than the original, so solutions that pass only weak tests are exposed.

• The benchmark's value is the test rigour, not the tasks, which remain short synthetic problems.
• The task set has changed across versions: the project records MBPP+ moving from 399 to 378 tasks when broken tasks were removed, with an expected pass@1 rise of about 4 points from that removal alone, so figures across versions are not comparable.

Dataset

Dataset

MBPP+

Type

rigorously re-tested short programming problems

Freshness

aging

Benchmark release

2023-05-02

The project records MBPP+ v0.2.0 reducing the task set from 399 to 378 by removing broken tasks. No current total or release date is stated on the page reviewed; the freshness date recorded here is the primary paper's publication date. The official repository establishes the task total: v0.2.0 reduced MBPP+ from 399 to 378 tasks, and applies a 35x test multiplier to MBPP against the 80x used for HumanEval+.

How to read this score

The meaningful quantity is the gap between MBPP and MBPP+ for the same model: the project frames a small drop as rigour in code generation and a large drop as fragile code. A plus score quoted alone discards that.

Similarity to real tasks: Low — Established from the official source: MBPP+ is MBPP with substantially more tests — the EvalPlus project states 35x more tests than the original. Harder testing better reflects whether code actually works, but the underlying tasks remain short self-contained programming problems with no repository, dependency or review context.

Data contamination risk: High — MBPP has been public for years and is standard training and evaluation data. The added tests are also published. No run-specific overlap audit is registered.

Benchmark gaming risk: Medium — The headline value is the drop between base and plus scores, which is harder to game than a single number, but the underlying problems are heavily exposed.

What you still need to test yourself

• Write tests for your own edge cases; a benchmark's test suite is not yours.

This benchmark supports decisions about:

• Detect models whose code passes weak tests but fails under rigorous ones.
• Compare base-to-plus drop rather than the plus score alone.

Limitations

• Short synthetic problems; not evidence of engineering ability.
• Figures are not comparable across benchmark versions.

Official repository reviewed 2026-10-12. The source states no release date for the current version; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.
• Version changes make cross-version figures incomparable: v0.2.0 reduced MBPP+ from 399 to 378 tasks, so a figure from an earlier version is not comparable with one measured after it.
• The source table for the registered score states no metric, so it is recorded under the benchmark's own pass@1 unit rather than a metric its source confirms.

Sources

• EvalPlus — official repository EvalPlus · accessed 2026-10-12Supports the 35x test multiplier for MBPP+, the 80x multiplier for HumanEval+, the v0.2.0 reduction from 399 to 378 tasks and its expected pass@1 effect, and the base-to-plus drop as the intended reading; it does not support any model score.
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation Liu et al. (EvalPlus authors) · accessed 2026-10-25The primary methodology paper for EvalPlus, which introduced the augmented test suites behind MBPP+, and establishes its release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://github.com/evalplus/evalplus

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MBPP+ Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path