Laboratory / GenAI / [008] RAG Evaluation Template
[EVAL-001] · Template · Live

RAG Evaluation Template

Area · GenAI Stack · Ragas · LangChain · pandas · LLM judge Dataset · Synthetic, 25 credit card call-center questions Live
Overview
Architecture
GitHub

The Problem

Evaluating a RAG usually means trying a few questions by hand. That shows some answers are bad, but not how bad, how often, or why. A wrong answer can come from retrieval (the wrong chunks were fetched) or from generation (the model said something the chunks do not support), and the fixes are different: chunking and top-k in one case, prompt and model in the other.

This project is a template for evaluating any RAG system with Ragas, starting from a CSV. You bring the questions, the correct answers and what your RAG replied. The template validates the file, grades every answer with an LLM judge on the same three metrics, and labels each failure as a retrieval problem, a generation problem or an incomplete answer.

It never calls your RAG. The CSV is the contract, so it works the same for a LangChain pipeline, a vector database with a prompt, or a vendor product. The example data is a fictional bank's credit card call-center assistant, with seven failures planted on purpose.

What It Delivers

Three scores per answer
Context recall, faithfulness and factual correctness, each from 0 to 1, saved to data/eval_scores.csv with the mean, median and spread per metric.
A diagnosis, not just a score
Low context recall points to retrieval. Low faithfulness points to generation. Only low correctness means the answer is incomplete or wrong. The threshold defaults to 0.7 and lives in eval_config.py.
Charts
A boxplot per metric, a heatmap per question, and a recall × faithfulness scatter that separates retrieval failures from generation failures at a glance.
Worst cases
The lowest-scoring rows for each failure type, so the review starts with the answers that matter.

Example Run

First 10 rows of the synthetic CSV, judged by gpt-4o-mini: 30 judge calls, about one minute.

Context recall
Mean 0.90 · median 1.00 · 1 of 10 below 0.7
Faithfulness
Mean 0.68 · median 0.88 · 4 of 10 below 0.7
Factual correctness
Mean 0.75 · median 1.00 · 3 of 10 below 0.7

Retrieval is healthy, and the weak spot is generation. Row 6, where the answer invents "guaranteed approval", keeps recall at 1.00 but drops faithfulness to 0.33. Row 9, with irrelevant chunks and an invented answer, scores 0.00 on all three. The judge also scored two answers low that a human would probably accept, which is why the diagnosis is a rule, not a verdict.

Key Terms

Context recall
Did the retrieved chunks contain what the reference answer needs? Low means retrieval missed information.
Faithfulness
Is every claim in the answer supported by the retrieved chunks? Low means the model added or changed something.
Factual correctness
Does the answer say the same as the reference? Low means it is wrong or incomplete.
LLM judge
A model that grades the answers. It should not be the same model your RAG uses, and it has its own variance and bias.

Pipeline

RAG Evaluation Template Architecture
Fig. 1 · CSV → validate → Ragas judge → scores → charts → diagnosis. Only the grading step spends API calls.
1 · Validate
Checks the four columns (user_input, reference, response, retrieved_contexts), empty cells and the chunk list format before any API call.
2 · Grade
The LLM judge scores each row on the three metrics. This is the only step that costs money, capped by the LIMIT variable in the notebook.
3 · Summarize
Mean, median and spread per metric.
4 · Plot
Boxplot, heatmap per question, recall × faithfulness scatter.
5 · Diagnose
Labels each failure as retrieval, generation or incomplete answer, and lists the worst cases.

Key Decisions

CSV as the contract
The template does not call your RAG. Any system that can export questions, answers and retrieved chunks can be evaluated without integration code.
Validate before spending
A malformed file fails fast with a row-level message, before a single judge call is made.
Judge is not the RAG's model
Grading with the same model that produced the answer inflates scores. OpenAI (gpt-4o-mini) or Google (gemini-2.5-flash) are configurable as judge.
Failures don't crash the run
A failed judge call becomes an empty score and is counted in the summary, so one bad request never loses the whole evaluation.
Three metrics only
Enough to separate retrieval from generation, few enough to read. The diagnosis is a rule, not a verdict.
Logic in src/, not the notebook
Validation, evaluation and charts live in modules, so the same evaluation runs from a script or a scheduled job.

Limitations and Next Steps

The judge is an LLM with its own variance and bias, and it has not yet been validated against human labels. The example CSV is synthetic, references can themselves be wrong, and small samples are fragile. The v2 roadmap: label 100 to 200 answers by hand and compare the judge against them (confusion matrix, precision, recall, Cohen's kappa), add non-LLM retrieval metrics (Recall@k, MRR), and keep dated run folders to track drift over time.

Repository

eval-001-rag-evaluation-template github.com →
eval-001-rag-evaluation-template/
├── README.md
├── requirements.txt
├── .env.example
├── .gitignore
├── eval_config.py
├── src/
│ ├── __init__.py
│ ├── validation.py
│ ├── evaluation.py
│ └── visualization.py
├── notebooks/
│ └── quickstart.ipynb
├── data/
│ ├── eval_input.csv
│ └── eval_scores.csv
└── figures/

Stack: Python 3.12, Ragas 0.3.9, LangChain 0.3.x, pandas, Jupyter. Judge: OpenAI gpt-4o-mini or Google gemini-2.5-flash. Replace data/eval_input.csv with your own questions, references, answers and retrieved chunks, run python -m src.validation, then open notebooks/quickstart.ipynb with LIMIT = 10 for a first run.