RAG Evaluation Starter

RAG Evaluation Starter is a minimal Python script to evaluate retrieval-augmented generation pipelines against a golden set.

This product is a minimal Python script designed to evaluate retrieval-augmented generation pipelines against a golden set of questions and expected answers. It runs five metrics in a single execution, including Retrieval Recall at K, Answer Faithfulness, Answer Relevance, Exact Match or Token F1, and Latency plus Cost. Users supply two functions to plug in their own system, with an optional embedder function defaulting to sentence-transformers/all-MiniLM-L6-v2. It targets development teams looking to catch performance regressions and find system failure points in their pipelines locally. The software is released under the MIT license as an open source project.

RAG Evaluation Starter at a glance

Best for: Software engineer
Runs on: CLI, self-hosted, open source
Pricing: MIT
Released: October 2026
Category: AI coding tool
Website: github.com/infrixo-systems/rag-evaluation-starter

Key features

Measures Retrieval Recall at K
Calculates Answer Faithfulness via token overlap
Computes Answer Relevance via cosine similarity
Evaluates Exact Match and Token F1
Tracks query latency and cost
Outputs a color-coded terminal table

Use it for

Test retrieval-augmented generation pipelines against a golden set
Identify where a retrieval system fails before deployment
Measure answer faithfulness without an LLM judge
Check answer relevance using embedding cosine similarity

Alternatives to RAG Evaluation Starter

My RAG — Compare Python and Java implementations sharing identical data.
Sgrep — Runs locally without accounts or hosted indexes using small CPU models.
Quivr — It provides a pre-built RAG infrastructure for rapid application integration.
Code Graph RAG — It builds knowledge graphs from codebases using Tree-sitter parsing.
UltraRAG — Build complex RAG workflows using modular MCP components and YAML configuration.
RAGFlow — Provides an integrated platform combining data processing with agent workflows.
BGE Embeddings — Provides specialized models for high-performance information retrieval.
LightRAG — It uses a dual-layer architecture to reduce LLM calls and retrieval latency.
jevgrep — Reduces coding agent costs by providing targeted, relevant repository context.
llmware — Build private, local RAG pipelines with 300+ prepackaged models and integrated document parsing.
Evidently — Provides 100+ built-in metrics and test suites for machine learning and LLM systems.
RAG-Anything — It unifies text and multimodal document retrieval within a single framework.

All RAG Evaluation Starter alternatives