whereismyai
Skill Placebo
Skill Placebo is a placebo-controlled benchmark test evaluates the most-starred skills for coding agents.
Skill Placebo tests the most-starred skills for coding agents to see if they outperform neutral text of the same length. It runs every skill in three arms on the same tasks: no skill, placebo, and skill. The benchmark measures cost, token usage, and pass rates across public tasks using specific models like claude-opus-5-5 in Claude Code and gpt-6-sol in Codex. Results show that most tested skills perform no better than a neutral placebo. Every trial is published with complete logs to ensure full reproducibility.
Skill Placebo at a glance
Key features
- Placebo-controlled testing of coding agent skills
- Comparison against same-length neutral text
- Cost and token tracking across trials
- Public trial records and agent logs
- Command-line reproduction via uvx
Use it for
- Benchmarking coding agent skills
- Evaluating cost impacts of agent instructions
- Comparing pass rates against neutral placebos
Alternatives to Skill Placebo
- Agent Skills — Encodes senior-engineer workflows and quality gates into 24 skills.
- Agent Trials — Test if an agent fixes your bugs in your stack at a sane price before deployment.
- Self Bench — Evaluates coding agents on internal repository tasks instead of generic public datasets.
- pstack — Improves AI coding agent outcomes with rigorous workflows.
- PUA Skill — It increases successful bug resolution by preventing premature surrender during debugging.
- AI Deploy Bench — Measures AI agent success, time, tokens, cost, and human steps for cloud deployments.
- jevgrep — Reduces coding agent costs by providing targeted, relevant repository context.
- QualityMax — Verifies AI-generated code with independent quality gates and auto-healing tests.
- Eval Skills — Guides coding agents from first execution to validated graders and experiments.
- Claude Skills Pack — 388 production-ready skills across 13 tools with zero-dependency Python tools.
- Playwright Skill — Generates and runs real, persistent Playwright code instead of simple scripts.
- Skill-Up — Measures behavior across cases and compares runs with and without skills.
All Skill Placebo alternatives