MTEB

MTEB is an evaluation package that benchmarks embedding quality across diverse tasks, languages, and modalities.

MTEB is a package for benchmarking and evaluating the quality of embedding models across a variety of tasks, modalities, and domains. It covers more than 1000 different tasks, ranging from historic Swedish patent classification to documentation retrieval for Python. These tasks spread across more than 1000 languages and cover image, text, and audio modalities. Users can run evaluations by selecting a model and tasks through Python or the command line interface. If a model is not natively implemented in MTEB, it loads using SentenceTransformer or CrossEncoder.

MTEB at a glance

Best for: Software engineer, Researcher
Runs on: CLI, open source
Pricing: Free
Released: April 2022
Category: AI research tool
Website: docs.mteb.org

Key features

Evaluates embedding models across diverse tasks
Supports text, image, and audio modalities
Covers over 1000 languages
Provides a command line interface for running evaluations
Integrates with SentenceTransformer and CrossEncoder
Includes over 1000 different benchmark tasks

Use it for

Evaluate the quality of text embeddings
Benchmark multilingual embedding models
Test image and audio embedding tasks
Run evaluations using the command line interface

Alternatives to MTEB

VLMEvalKit — Evaluate large vision-language models across numerous benchmarks with a single command.
FastEmbed — It uses ONNX Runtime for high speed and does not require a GPU or PyTorch.
WeMM Embedding — Provides state-of-the-art performance across multiple multimodal benchmarks.
LMMs-Eval — Provides open research and tooling for multimodal model evaluation and benchmarks.
Lighteval — Evaluates LLMs across multiple backends with thousands of built-in tasks.
LiveBench — Compare model performance and operational costs without contamination.
Overview — Learn language models by running hands-on experiments without writing scripts
BGE Embeddings — Provides specialized models for high-performance information retrieval.
Text Embeddings Inference — High-performance serving for open-source embedding models.
OpenVLHarness — Lifts six multimodal model backbones by 9.7 to 12.6 points across 22 benchmarks.
Benchpress — Evaluates document artificial intelligence models against specific user pages for accuracy, cost, and speed.
Finetuning With Sampling — Official implementation of the Finetuning with Sampling research paper from Harvard.

All MTEB alternatives