Pass@k — AI Glossary

Reference details

Related terms

• LiveCodeBench
• BigCodeBench
• Code Generation

Example: Why pass@10 flatters, in one line of arithmetic

If a model solves a problem with probability 0.3 on any single attempt, and attempts are independent, then at least one of ten succeeds with probability 1 − (1 − 0.3)^10 = 1 − 0.7^10 ≈ 0.97. A model that is right under a third of the time reports 97%. Nothing is wrong with the metric — it is answering “can it ever get there”, which is the right question when a verifier exists to pick the passing sample. It is the wrong question when the user sees one answer, and pass@1 is the number that describes that.

Commonly confused with

Pass@k is not accuracy, and the two diverge fast as k grows. It also depends on sampling temperature: higher temperature raises diversity and therefore pass@k while often lowering pass@1. Reporting pass@k without the temperature, the sample count and the value of k describes almost nothing.

When to use it

Reach for it when:

• Settings with a cheap verifier — unit tests, a type checker, a proof assistant — that can pick the winner
• Measuring headroom: how often the ability is present but not reliably surfaced
• Comparing sampling and search strategies against a fixed model

Reach for something else when:

• Predicting single-shot user experience, which pass@1 describes
• Where no verifier exists, since nobody can tell which of the k samples was the passing one
• Comparing across papers without k, n and temperature stated

Primary source

• Chen et al., Evaluating Large Language Models Trained on Code (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pass@k — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path