HumanEval-X Methodology and Evidence

What this benchmark measures

• multilingual code synthesis
• cross-language code translation

Relevant to:

• polyglot code-generation screening
• non-Python code quality checks

Failure modes it exercises:

• functional errors in generated code outside Python

Method and its limits

A multilingual extension of HumanEval with hand-written solutions in C++, Java, JavaScript and Go alongside the original Python, used to evaluate code generation and translation.

• Function-level tasks do not represent repository-scale engineering.
• The abstract reviewed does not state how many problems HumanEval-X contains, so per-language sample size is not established from this source.

Dataset

Dataset

HumanEval-X

Type

multilingual function-level synthesis tasks with reference solutions

Freshness

aging

Benchmark release

2023-03-30

C++, Java, JavaScript and Go solutions hand-written to extend the Python-only HumanEval, as registered in the primary paper. The date records first submission; the paper was revised in July 2024.

How to read this score

Per-language results matter more than any aggregate, since the benchmark exists specifically to expose differences between Python and other languages.

Similarity to real tasks: Low — Established from the primary methodology: HumanEval-X extends HumanEval beyond Python by hand-writing solutions in C++, Java, JavaScript and Go, and is used for code generation and translation. Function-level synthesis against hidden tests is a narrow slice of engineering: no repository context, build system, dependency, or review step is represented.

Data contamination risk: High — HumanEval and its derivatives are among the most widely reproduced code datasets, and solutions are published. No run-specific overlap audit is registered.

Benchmark gaming risk: High — Function-level synthesis benchmarks are directly optimised against, and pass@k depends on sampling temperature and attempt count, which are not registered per score here.

What you still need to test yourself

• Evaluate on your own repository and build system, which function-level tasks do not represent.
• Fix sampling temperature and attempt count before comparing pass rates.

This benchmark supports decisions about:

• Check whether code quality holds outside Python.
• Screen candidates for code translation tasks.

Limitations

• Not evidence of repository-scale engineering ability.
• Scores are not comparable across different sampling protocols.

Primary source reviewed 2026-10-12. HumanEval-X is treated as an aging 2023 benchmark.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.
• The problem count is not established from the abstract reviewed.

Sources

• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X Zheng, Xia, Zou, Dong, Wang, Xue, Wang, Shen, Wang, Li, Su, Yang, Tang · accessed 2026-10-12Supports the extension of HumanEval into C++, Java, JavaScript and Go with hand-written solutions and its use for generation and translation; it does not state the problem count and does not support any model score, and no score is registered on this page.
• HumanEval-X release in the CodeGeeX repository CodeGeeX authors · accessed 2026-10-25The authors' repository for the CodeGeeX paper that introduced HumanEval-X, carrying the multilingual problem set and the execution-based evaluation. It establishes the benchmark's languages and harness; the problem count stated on this page is not established from it.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2303.17568

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to HumanEval-X Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path