whereismyai
Evals Skills
Evals Skills is a collection of CLI-based AI coding agents that guide the development of product-specific evaluation pipelines for AI models.
Evals Skills provides a suite of tools that function as AI coding agents to assist in building project-specific evaluations. It includes specialized skills to audit existing pipelines, discover errors in traces, generate synthetic test data, and write code-based checks. The system automates tasks like building review interfaces, clustering data for analysis, and calibrating LLM-as-judge evaluators. It is designed for developers who need to evaluate RAG pipelines or subjective quality criteria. Users install skills via the command line, which then guides them through building, reviewing, and organizing failure modes.
Evals Skills at a glance
Key features
- Audit eval pipelines for potential problems
- Build custom human annotation interfaces for trace review
- Generate diverse synthetic test inputs
- Analyze dataset content and design visual review apps
- Cluster trace data to identify failure modes
- Write code checks for objective failure criteria
- Calibrate LLM judges against human labels
Use it for
- Auditing existing AI evaluation pipelines
- Analyzing model traces to discover failure modes
- Building annotation interfaces for human review
- Creating synthetic datasets for testing
- Validating LLM judge performance
Alternatives to Evals Skills
- Eval Skills — Guides coding agents from first execution to validated graders and experiments.
- Agent Skills — Encodes senior-engineer workflows and quality gates into 24 skills.
- Markdown Viewer Skills — Extends AI coding agents with structured visualization capabilities.
- Draw.io Skill — Synchronizes diagrams with code changes without discarding manual layout adjustments.
- Tech Leads Agent Skills — Extends AI coding agents with security-scanned, specialized workflows and tools.
- .NET Skills — Curated .NET skills and custom agents for AI coding assistants.
- Skill-Up — Measures behavior across cases and compares runs with and without skills.
- Skill Seekers — Converts 18 data sources into structured AI skills for 22 platforms.
- jevgrep — Reduces coding agent costs by providing targeted, relevant repository context.
- Waza — Embeds structured engineering habits into AI agents to prevent generic, imprecise output.
- Crit — It keeps human feedback localized and precise for AI agent code iterations.
- Refact.ai — Open-source, self-hosted, and integrates directly into your IDE.
All Evals Skills alternatives