VLMEvalKit

VLMEvalKit is an open-source evaluation toolkit for large vision-language models that enables one-command evaluation across various benchmarks.

VLMEvalKit is an open-source evaluation toolkit designed for large vision-language models (LVLMs). It enables one-command evaluation on various benchmarks without requiring the heavy workload of data preparation under multiple repositories. The toolkit adopts generation-based evaluation for all LVLMs and provides results obtained with both exact matching and LLM-based answer extraction. It supports multiple commercial APIs, open-source models, and numerous image and video benchmarks. Users can develop custom benchmarks or models by implementing specific functions while the codebase handles data downloading, preprocessing, prediction inference, and metric calculation.

VLMEvalKit at a glance

Best for: Researcher, Software engineer
Runs on: Linux, open source, CLI
Pricing: Free
Released: December 2023
Category: AI research tool
Website: github.com/open-compass/VLMEvalKit

Key features

One-command evaluation of large vision-language models
Generation-based evaluation for all models
Exact matching and LLM-based answer extraction
Support for custom split-thinking functions
Prediction file saving in TSV format
Multi-node distributed inference using LMDeploy or VLLM
Support for commercial APIs and open-source models

Use it for

Evaluate large vision-language models on multiple benchmarks
Benchmark new vision-language models with minimal setup
Run distributed inference for large-scale models
Perform LLM-based answer extraction for evaluation

Alternatives to VLMEvalKit

Vlmhub — Run multimodal prompts across multiple local and cloud backends without rewriting inference code.
Lighteval — Evaluates LLMs across multiple backends with thousands of built-in tasks.
LMMs-Eval — Provides open research and tooling for multimodal model evaluation and benchmarks.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
EvalScope — It combines model capability benchmarks with inference performance testing in one tool.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
OpenVLHarness — Lifts six multimodal model backbones by 9.7 to 12.6 points across 22 benchmarks.
vLLM — Maximizes throughput with PagedAttention and continuous batching.
DeepEval — Provides over fifty research-backed metrics and pytest-native execution.
MTEB — Evaluates embedding models across over 1000 tasks and 1000 languages.
LMFlow — It offers memory-efficient training and full Accelerate support for large language models.
LightLLM — It delivers high-speed inference and serving performance for large language models.

All VLMEvalKit alternatives