BLEU — AI Glossary

Reference details

Related terms

• ROUGE
• BERTScore
• Exact Match (EM)

Example: Score a correct paraphrase and watch it lose

Candidate “the cat sat on the mat”, reference “the cat is on the mat”. Unigrams: the (clipped to 2), cat, on and mat match; sat does not — 5 of 6, or 0.83. Bigrams: “the cat”, “on the” and “the mat” match; “cat sat” and “sat on” do not — 3 of 5, or 0.60. Both are six tokens, so the brevity penalty is 1. Now try a fluent human paraphrase — “the mat had a cat on it” — and bigram precision collapses to zero despite the meaning being intact. BLEU measures surface overlap with the references you supplied, and nothing else.

Commonly confused with

BLEU is precision-oriented; ROUGE is recall-oriented and is the summarisation convention. Neither measures meaning. BERTScore compares embeddings instead, and an LLM-as-judge rubric compares against criteria — both disagree with BLEU on paraphrase, which is usually a point in their favour.

When to use it

Reach for it when:

• Machine translation, where decades of published results make it the comparable metric
• Regression testing a system against its own previous output, where the reference is stable
• Cheap, deterministic scoring in CI, where you need a number that never drifts

Reach for something else when:

• Open-ended generation, chat or summarisation — many correct answers, few references
• Comparing systems that differ in style or verbosity rather than in accuracy
• Any claim about quality made from BLEU alone without reporting the tokenization used

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BLEU — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path