MTEB Leaderboard

What this benchmark measures

• text embedding quality
• retrieval ranking
• multilingual semantic similarity

Relevant to:

• embedding model selection for RAG
• semantic search shortlisting

Failure modes it exercises:

• poor retrieval ranking on out-of-domain corpora

Method and its limits

Aggregate score across 8 embedding task types and 58 datasets in 112 languages; the headline number is a mean over heterogeneous tasks.

• The aggregate mean hides large per-task and per-language variance.
• The paper reports that no particular text embedding method dominates across all tasks, so a leading average does not identify the best retriever for a specific corpus.

Dataset

Dataset

Massive Text Embedding Benchmark

Type

aggregated embedding task suite

Freshness

aging

Benchmark release

2022-10-13

58 datasets, 8 task types, and 112 languages as registered in the primary paper. The date records paper submission, not a corpus cutoff.

How to read this score

The average is interpretable only against the same task mix; per-task scores matter more than the headline for any specific retrieval workload.

Similarity to real tasks: Medium — Established from the primary methodology: MTEB aggregates 58 datasets across 8 embedding task types and 112 languages. Retrieval and reranking tasks resemble production retrieval more closely than multiple-choice formats do. It still does not reproduce a deployed pipeline, because chunking strategy, domain corpus, hybrid lexical scoring, and reranker choice are not represented.

Data contamination risk: Medium — Constituent datasets are public and widely used in embedding-model development. No run-specific overlap audit is registered for any score shown here.

Benchmark gaming risk: Medium — A public leaderboard creates direct selection pressure on the aggregate score. ModelRefs registers no per-score protocol detail, so comparability across rows is not established.

What you still need to test yourself

• Re-rank candidates on your own corpus with your own chunking; MTEB ordering frequently does not hold.
• Measure embedding cost, latency, and dimensionality, which MTEB does not cover.

This benchmark supports decisions about:

• Shortlist embedding models before testing them on your own corpus.
• Identify which task types a candidate model is weakest on.

Limitations

• Not evidence of retrieval quality on your corpus.
• Does not measure embedding cost, latency, or dimensionality trade-offs.

Primary source reviewed 2026-10-11. MTEB is treated as an aging 2022 benchmark.

What this evidence does not cover

• The models currently scored on this page are all OpenAI embedding models, so this is not a provider-neutral comparison.

Sources

• MTEB: Massive Text Embedding Benchmark Muennighoff, Tazi, Magne, Reimers · accessed 2026-10-11Supports the 58-dataset, 8-task, 112-language structure and the finding that no method dominates across all tasks; it does not support any model score shown here.
• MTEB evaluation implementation and task definitions embeddings-benchmark (MTEB maintainers) · accessed 2026-10-25The maintained evaluation implementation and task/dataset definitions, published under the embeddings-benchmark organization and citing the MTEB paper's authors. It establishes the benchmark's task inventory and harness; it does not establish the provenance of any individual model score.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2210.07316

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MTEB Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path