MultiPL-E Leaderboard

What this benchmark measures

• multilingual code generation
• cross-language transfer

Relevant to:

• polyglot assistant screening
• language coverage planning

Failure modes it exercises:

• code quality collapse outside Python

Method and its limits

Python benchmark problems translated into 18 programming languages by per-language compilers that convert unit tests, doctests and function signatures, so the same problems can be compared across languages.

• The problems are translations of a Python set, so they test what Python exercises happen to cover rather than what is idiomatic in each target language.
• The official page does not state the scoring metric, reporting only a success rate, so pass@k protocol is not established from it.

Dataset

Dataset

MultiPL-E

Type

translated multi-language code generation benchmark

Freshness

aging

Benchmark release

2022-08-17

18 programming languages as registered on the official page, translated from the Python benchmarks of Chen et al. 2021. The page states no release date; the freshness date recorded here is the primary paper's publication date.

How to read this score

Per-language results are the point; an aggregate across 18 languages mixes ecosystems with very different training-data volumes and hides exactly the gap the benchmark exists to show.

Similarity to real tasks: Low — Established from the official source: MultiPL-E translates Python benchmark problems into 18 programming languages by compiling their unit tests, doctests and function signatures into each target language. Comparing the same problem across languages is a controlled experiment rather than a real task: the problems remain short, self-contained function exercises.

Data contamination risk: High — The source Python problems are among the most widely reproduced code benchmarks, and the translations are published. No run-specific overlap audit is registered.

Benchmark gaming risk: Medium — Because the problems derive from a saturated Python set, gains can reflect exposure to the originals rather than multilingual competence.

What you still need to test yourself

• Evaluate idiomatic code in your target language, which translated Python exercises do not represent.

This benchmark supports decisions about:

• Compare a model's code quality across languages on matched problems.
• Identify languages where a candidate degrades sharply.

Limitations

• Translated Python exercises, not idiomatic per-language tasks.
• Aggregate scores hide the per-language differences the benchmark exists to expose.

Official project source reviewed 2026-10-12. The source states no release date; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.
• The registered score covers MultiPL-E's HumanEval translations only; the MBPP translations are scored separately and are not registered here.

Sources

• MultiPL-E — official project page Northeastern University Programming Research Laboratory · accessed 2026-10-12Supports the 18-language coverage and the compiler-based translation of unit tests, doctests and function signatures from the Python benchmarks of Chen et al. 2021; it does not state the scoring metric and does not support any model score.
• MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation Cassano et al. · accessed 2026-10-25The primary methodology paper, establishing the translation approach behind the benchmark's language coverage and its release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://nuprl.github.io/MultiPL-E/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MultiPL-E Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path