IFEval Leaderboard

What this benchmark measures

• instruction following
• constraint compliance in generation

Relevant to:

• format-constrained output screening
• assistant reliability checks

Failure modes it exercises:

• ignoring explicit formatting or length constraints

Method and its limits

Around 500 prompts containing one or more of 25 verifiable instruction types, scored by objective programmatic checks rather than human or model judgement. The official implementation defines strict scoring on the response as given, and loose scoring across eight variants of it, with asterisks removed and with the first and/or last line removed, counting an instruction as followed if any variant satisfies it; the code documents loose scoring as an upper bound.

• Only mechanically checkable constraints are covered, so tone, relevance and correctness are out of scope entirely.
• The abstract reviewed does not describe the scoring procedure, so strict versus loose compliance variants are not established from this source.

Dataset

Dataset

IFEval

Type

prompts with programmatically verifiable instructions

Freshness

aging

Benchmark release

2023-11-14

25 verifiable instruction types across roughly 500 prompts as registered in the primary paper. The date records paper submission.

How to read this score

A compliance rate is directly interpretable and reproducible, which is the benchmark's stated purpose. It says nothing about whether the compliant output is useful.

Similarity to real tasks: High — Established from the primary methodology: IFEval uses around 500 prompts built from 25 types of verifiable instruction, such as writing in more than 400 words or mentioning a keyword at least three times, whose compliance can be checked objectively against the output. Honouring explicit constraints is exactly what production prompts demand, so similarity is high. It checks compliance rather than whether the content is any good.

Data contamination risk: Medium — The prompt set is public, but because scoring is a programmatic check on the output rather than a match against a reference answer, memorisation is a weaker concern than for fixed-answer benchmarks. No run-specific audit is registered.

Benchmark gaming risk: Medium — Compliance with a fixed set of 25 instruction types can be trained for directly without improving general instruction following.

What you still need to test yourself

• Test your own constraint types, which may not be among the 25 covered.
• Judge output quality separately; this benchmark only checks compliance.

This benchmark supports decisions about:

• Screen whether a model reliably honours explicit output constraints.
• Compare instruction compliance without human or model judges.

Limitations

• Covers only mechanically checkable constraints.
• Compliance is not evidence of output quality.

Primary source reviewed 2026-10-12. IFEval is treated as an aging 2023 benchmark.

What this evidence does not cover

• Only one model is currently scored on this page, so no comparison is supported.
• The scoring variant behind the registered score is not stated by its source, so it cannot be placed against another IFEval figure.

Sources

• Instruction-Following Evaluation for Large Language Models Zhou, Lu, Mishra, Brahma, Basu, Luan, Zhou, Hou · accessed 2026-10-12Supports the 25 verifiable instruction types, the roughly 500 prompts, and the stated aim of objective reproducible evaluation; it does not establish the scoring procedure and does not support any model score.
• IFEval official evaluation implementation Google Research · accessed 2026-10-25Google Research's IFEval implementation, which defines both scoring variants in code. It establishes the strict and loose scoring procedures: strict scores the response as given, while loose scores eight variants of it, with asterisks removed and with the first and/or last line removed, counting an instruction as followed if any variant satisfies it. It does not establish any model's score.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2311.07911

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to IFEval Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path