whereismyai
Bugbout
Bugbout is a benchmarking tool that replays real project bugs to test and score coding agent performance.
Bugbout is a benchmarking tool designed to evaluate coding agents on a specific codebase. It operates by mining git history for past bug fixes and tests, then allowing different coding agents to race to solve the same historical issues. Each agent operates inside an isolated sandbox environment with a code checkout that excludes the original fix and tests. The system scores every generated patch using the hidden tests originally written by project maintainers. It supports various agents out of the box, including Codex, Claude Code, Gemini CLI, and mini-swe-agent, while allowing custom command-line agents. Additionally, bugbout cross-examines agents by running their generated tests against other patches to evaluate jury-based patch gating.
Bugbout at a glance
Key features
- Replays real bugs from project git history
- Scores patches with hidden maintainer tests
- Supports Codex, Claude Code, and Gemini CLI
- Runs agents inside an OS sandbox
- Cross-examines agent-generated tests
- Resumable execution flow
- Configurable via TOML
Use it for
- Compare different coding agents on custom codebases
- Benchmark local models against historical project bugs
- Evaluate agent patch reliability using maintainer tests
- Test agent compatibility with proprietary frameworks
Companions
Tools that articles about Bugbout recommend using with it, each with its job.
- Claude Code — Terminal-first agent that handles multi-hour refactors and migrations
- Cursor — Editor extension reviews changes inside Cursor.
- Linear — Reviews check the change against the issue it came from.
- Jira — Plans and reviews start from the Jira ticket.
- Claude — Strong writing and coding help, with projects, memory and connectors
Alternatives to Bugbout
- BugTape — Replays browser errors and failed requests for coding agents to fix automatically.
- Agent Trials — Test if an agent fixes your bugs in your stack at a sane price before deployment.
- bb — Runs locally with support for multiple AI models and full customization.
- pstack — Improves AI coding agent outcomes with rigorous workflows.
- CodeRabbit — Most-installed AI PR reviewer, built to check code from coding agents
- Freebuff — Access multiple coding AI models for free without subscriptions.
- Claude BugHunter — It loads 83 skills and 15 slash commands automatically by topic.
- OpenCode — Works with 75+ model providers, local models, and existing subscriptions
- Agentic Bug Hunter — It drives the reconnaissance, triage and reporting loop an agent would run.
- Agent Beacon — Retains coding agent knowledge across tools so errors do not repeat.
- Metabob — Provides proactive debugging and enforces safe implementation patterns for AI tools.
- jevgrep — Reduces coding agent costs by providing targeted, relevant repository context.
All Bugbout alternatives