OSWorld

OSWorld is a benchmark for evaluating multimodal agents on open-ended tasks within real computer environments.

OSWorld is a benchmarking platform designed to test multimodal agents on open-ended tasks in real computer environments. It provides virtual machine environments running on platforms like VMware, VirtualBox, Docker, Modal, Daytona, and AWS. Researchers use it to run agent baselines, evaluate success rates across categories like Office, Daily, and Professional, and conduct manual task examinations using screenshots and video recordings. The platform supports execution with custom API endpoints and handles complex tasks requiring configurations such as Google account access and proxy settings. It is built for researchers and developers working on multimodal AI agents.

OSWorld at a glance

Best for: Researcher, Software engineer
Runs on: Web, macOS, Windows, Linux, Docker, AWS, self-hosted, open source
Pricing: Free
Released: October 2023
Category: AI research tool
Website: github.com/xlang-ai/OSWorld

Key features

Benchmark multimodal agents on open-ended tasks
Support parallel evaluation on AWS
Provide virtual machine support via Docker, VMware, and VirtualBox
Record execution with screenshots and videos
Generate per-domain success rates and statistics
Include a manual task examination tool

Use it for

Evaluating multimodal agent performance on real computer tasks
Running parallel evaluations using cloud infrastructure
Analyzing agent success rates across Office, Daily, and Professional categories
Manually examining and verifying specific benchmark tasks

Alternatives to OSWorld

Video2World — It evaluates coding agents on geometry, dynamics, and functionality across 222 tasks.
OpenResearch — Run parallel research workflows locally or on remote compute.
Osaurus — Runs entirely on your Mac with local models, persistent memory, and complete data privacy.
OpenVLHarness — Lifts six multimodal model backbones by 9.7 to 12.6 points across 22 benchmarks.
OpenCompass — It offers an objective, comprehensive, and credible open-source AI evaluation system.
Navital Gym — Measure AI agent performance on realistic tasks using deterministic rules without LLM judges.
FAROS — Connects literature evidence, executable plans, and human review into an auditable research loop.
Senro — Tests WebMCP tools and monitors AI agent behavior without building infrastructure.

All OSWorld alternatives