MMTEB: Massive Multilingual Text Embedding Benchmark Article Swipe

MAKE IMPACT

Kenneth Enevoldsen , Isaac Chung , Imene Kerboua , Márton Kardos , Ashwin Mathur , David Stap , Jay Gala , Wissam Siblini , Dominik Krzemiński , Genta Indra Winata , Saba Sturua , Saiteja Utpala , Mathieu Ciancone , Marion Schaeffer , Graça Sequeira , D. Misra , Shreeya Singh Dhakal , Jonathan Hvithamar Rystrøm , Roman Solomatin , Ömer Veysel Çağatan , Akash Kundu , Martin Bernstorff , Shitao Xiao , Akshita Sukhlecha , Bhavish Pahwa , Rafał Poświata , Kranthi Kiran GV , Shawon Ashraf , Daniel Auras , Björn Plüster , Jan Philipp Harries , Loïc Magne , Isabelle Mohr , Mariya Hendriksen , Dawei Zhu , Hippolyte Gisserot-Boukhlef , Tom Aarsen , Jan Kostkan , Konrad Wojtasik , Taemin Lee , Marek Šuppa , Crystina Zhang , Roberta Rocca , Mohammed Hamdy , Andrianos Michail , John Yang , Manuel Faysse , Aleksei Vatolin , Nandan Thakur , Manan Dey , Dipam Vasani , Pranjal A. Chitale , Simone Tedeschi , Nguyen Dinh Tai , Artem Snegirev , Michael Günther , Mengzhou Xia , Weijia Shi , Xing Han Lù , Jordan Clive , G. Krishnakumar , А. А. Maksimova , Silvan Wehrli , Maria Tikhonova , Henil Shalin Panchal , Aleksandr Abramov , Malte Ostendorff , Zheng Liu , Simon Clematide , Lester James V. Miranda , Alena Fenogenova , Guangyu Song , Ruqiya Bin Safi , Wen-Ding Li , Andrea Borghini , Federico Cassano , Hongjin Su , Jimmy Lin , H. W. Yen , Lasse Hansen , Sara Hooker , Chenghao Xiao , Vaibhav Adlakha , Orion Weller , Siva Reddy , Niklas Muennighoff · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2502.13595

Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2502.13595

PDF

https://arxiv.org/pdf/2502.13595

OA Status

green

Cited By

6

Related Works

10

OpenAlex ID

https://openalex.org/W4407764402

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4407764402

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2502.13595

Digital Object Identifier

Title

MMTEB: Massive Multilingual Text Embedding Benchmark

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-02-19

Full publication date if available

Authors

Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Graça Sequeira, D. Misra, Shreeya Singh Dhakal, Jonathan Hvithamar Rystrøm, Roman Solomatin, Ömer Veysel Çağatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafał Poświata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Björn Plüster, Jan Philipp Harries, Loïc Magne, Isabelle Mohr, Mariya Hendriksen, Dawei Zhu, Hippolyte Gisserot-Boukhlef, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Šuppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang, Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal A. Chitale, Simone Tedeschi, Nguyen Dinh Tai, Artem Snegirev, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lù, Jordan Clive, G. Krishnakumar, А. А. Maksimova, Silvan Wehrli, Maria Tikhonova, Henil Shalin Panchal, Aleksandr Abramov, Malte Ostendorff, Zheng Liu, Simon Clematide, Lester James V. Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Andrea Borghini, Federico Cassano, Hongjin Su, Jimmy Lin, H. W. Yen, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, Niklas Muennighoff

List of authors in order

Landing page

https://arxiv.org/abs/2502.13595

Publisher landing page

PDF URL

https://arxiv.org/pdf/2502.13595

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2502.13595

Direct OA link when available

Concepts

Benchmark (surveying), Computer science, Embedding, Natural language processing, Artificial intelligence, Geography, Cartography

Top concepts (fields/topics) attached by OpenAlex

Cited by

6

Total citation count in OpenAlex

Citations by year (recent)

2025: 6

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload