GyaanStack. Blog

Articles / ragevalspython

Evaluate a RAG app in Python: retrieval and generation, scored separately

Score a RAG app in two halves (recall@k, MRR, hit rate for retrieval; faithfulness, correctness, citations for answers) and find which half broke.

By Raghunath · · 21 min read

A RAG answer can be wrong for two different reasons. The retriever never found the passage that holds the answer, or it found the passage and the model wrote something else. One average score over the final answers mixes those two cases, so when it drops you know something broke but not where to look. The fix for a retrieval miss (chunking, tokenization, hybrid search, a reranker) has nothing in common with the fix for a generation miss (the prompt, context order, a stricter model).

This post builds a small evaluation pipeline that scores the two halves separately. Retrieval gets hit rate, recall@k and MRR, computed from chunk ids. Generation gets faithfulness, answer correctness and a citation check, computed from text. Then a short rule labels each failing question as a retrieval or a generation failure. Everything runs on the Python standard library with no API key. The model is a stub behind one method, and the comments show where a real LLM call goes.

The RAG cheat sheet lists these metrics in one table. This post computes them and shows how to read them when a number moves.

Two halves, two kinds of label#

                 labelled question
                        |
                    retriever
                        |
  top-k chunk ids  ---- compare with gold chunk ids ---->  hit rate, recall@k, MRR
                        |
                 generator (LLM)
                        |
  answer + citations -- compare with retrieved text ---->  faithfulness, citation precision
                     -- compare with required facts ---->  correctness
                        |
  per-question diagnosis: retrieval miss / generation miss / ok

Each half needs its own label. For retrieval you label which chunks answer the question. For generation you label which facts a correct answer must contain. Faithfulness needs no gold label at all, because it compares the answer with whatever was retrieved, so it catches invented claims even on questions that have no reference answer yet.

The labelled set#

Ten questions over a ten-chunk staff handbook. Each row has the chunk ids that hold the answer and the short strings a correct answer must contain. The last row has no relevant chunk. The handbook says nothing about gym memberships, and the only correct output is a refusal. Keep a few rows like this in any real set; they are the only rows that test whether the model invents answers when retrieval comes back empty.

# data.py: a tiny handbook split into chunks, plus a labelled question set
CHUNKS = {
    "leave-1": "Employees get 24 days of paid leave per calendar year. Up to 10 unused days carry over to the next year.",
    "leave-2": "Leave requests go through the HR portal at least 5 working days in advance. Your manager approves them.",
    "sick-1": "Sick leave is separate from paid leave. You get 12 sick days a year. A doctor's note is needed for more than 3 consecutive sick days.",
    "it-1": "New joiners receive a laptop on day one. Laptops are replaced every 3 years.",
    "it-2": "The VPN is required for all access to internal tools from outside the office. Install the client from the IT portal.",
    "it-3": "Passwords must be at least 14 characters long and are rotated every 90 days.",
    "exp-1": "Travel expenses must be filed within 30 days of the trip. Receipts are required for any item above 500 rupees.",
    "exp-2": "Expense claims above 20,000 rupees need approval from a department head.",
    "wfh-1": "Employees may work from home up to 2 days a week with manager approval.",
    "wfh-2": "The remote work allowance is 1,500 rupees a month for home internet costs.",
}

# relevant: chunk ids that hold the answer. facts: strings a correct answer must contain.
GOLDEN = [
    {"id": "q01", "question": "How many days of paid leave do employees get per year?", "relevant": ["leave-1"], "facts": ["24"]},
    {"id": "q02", "question": "How many unused leave days carry over to next year?", "relevant": ["leave-1"], "facts": ["10"]},
    {"id": "q03", "question": "How far in advance must I submit leave requests?", "relevant": ["leave-2"], "facts": ["5 working days"]},
    {"id": "q04", "question": "When do I need a doctor's note for sick days?", "relevant": ["sick-1"], "facts": ["3 consecutive"]},
    {"id": "q05", "question": "How often are laptops replaced?", "relevant": ["it-1"], "facts": ["3 years"]},
    {"id": "q06", "question": "What is the minimum password length?", "relevant": ["it-3"], "facts": ["14 characters"]},
    {"id": "q07", "question": "What is the deadline to file travel expenses, and when are receipts required?", "relevant": ["exp-1"], "facts": ["30 days", "500"]},
    {"id": "q08", "question": "Who must approve an expense claim of 25,000 rupees?", "relevant": ["exp-2"], "facts": ["department head"]},
    {"id": "q09", "question": "How many days a week can I work from home, and is there an allowance for home internet?", "relevant": ["wfh-1", "wfh-2"], "facts": ["2 days", "1,500"]},
    {"id": "q10", "question": "Does the company pay for a gym membership?", "relevant": [], "facts": []},
]

Label chunk ids, not chunk text. Ids let you compute retrieval metrics without any model, and they survive prompt changes. They break when you re-chunk, so store the source document and character span next to each id if you expect to change chunk sizes. The ingestion pipeline post covers versioning chunks.

Ten rows is a demo. For a real app, 50 to 100 questions drawn from real user queries gives you numbers that move by more than noise. With 10 questions, one question is worth 10 points of accuracy; with 100, it is worth 1.

Retrieval metrics, worked by hand#

All three metrics read the ranked list of chunk ids the retriever returned and the set of relevant ids.

  • Hit rate@k: 1 if any relevant chunk is in the top k, else 0, averaged over questions.
  • Recall@k: the share of the relevant chunks that made the top k. It equals hit rate when every question has exactly one relevant chunk, and it falls below hit rate when a question needs two chunks and gets one.
  • MRR (mean reciprocal rank): 1 divided by the rank of the first relevant chunk, 0 if none came back, averaged. It was the main measure in the TREC-8 question answering track, and it rewards putting the right chunk first.

Three questions, k = 3. Question A finds its chunk at rank 1, B at rank 3, C not at all.

            rank of first relevant   hit@3   reciprocal rank
question A          1                  1        1/1 = 1.00
question B          3                  1        1/3 = 0.33
question C          -                  0              0.00
mean                                 0.67             0.44

Hit rate says two of three questions got their evidence. MRR says the evidence was often buried. Models use the start and end of a long context better than the middle (Liu et al. measured this), so a fine hit rate with a low MRR is a reason to add a reranker.

Now a two-chunk question: "How many days a week can I work from home, and is there an allowance for home internet?" needs both wfh-1 and wfh-2. If the retriever returns only wfh-2, hit rate is 1 and recall is 0.5. Hit rate alone would call that a success, and the answer will be half right.

Generation metrics#

Faithfulness is the share of the answer's claims that the retrieved text supports. The Ragas definition is the same: split the response into claims, check each one against the retrieved context, divide supported by total. Here each sentence is a claim and a cheap rule judges support. Every number in the claim must appear in the chunk, and at least 60% of its content words must too. In production you replace supports() with an LLM judge that has the same signature, (chunk, claim) -> bool.

Correctness checks the answer against the labelled facts. A string match on "24" or "department head" is crude, but it never drifts, and for short factual answers it catches most errors. For unanswerable questions, correctness means the model refused.

Citation precision is the share of cited chunk ids that were retrieved and back at least one sentence of the answer. A model that cites hr-9 when hr-9 was never in its context made the citation up, and a user who clicks it lands on an unrelated page.

Faithfulness and correctness measure different things. An answer can be faithful and wrong: it accurately repeats a retrieved chunk that does not answer the question. It can also be correct and unfaithful: the right fact, taken from the model's training data, which you cannot audit and which goes stale.

The pipeline#

The retriever is BM25 written out in full, so the tokenizer's behaviour is visible; that matters below. The generator is a stub. It picks the retrieved sentences that share at least two content words with the question, cites their chunks, and refuses when nothing matches. An invent switch simulates a prompt change that lets the model answer from outside knowledge.

# rag.py: a BM25 retriever and a stub generator behind one interface
import math, re
from collections import Counter

STOP = set("a an the of to for and or is are do does i my me can be per in on at with from any how what when who must there this that".split())

STEM = False  # crude plural stripping, switched on for the fix in run_eval.py

def tokens(text):
    out = [t for t in re.findall(r"[a-z0-9]+", text.lower()) if t not in STOP]
    if STEM:
        out = [t[:-1] if t.endswith("s") and len(t) > 3 else t for t in out]
    return out

class BM25:
    def __init__(self, chunks, k1=1.5, b=0.75):
        self.ids = list(chunks)
        self.docs = [tokens(chunks[i]) for i in self.ids]
        self.avg = sum(map(len, self.docs)) / len(self.docs)
        df = Counter(t for d in self.docs for t in set(d))
        n = len(self.docs)
        self.idf = {t: math.log(1 + (n - f + 0.5) / (f + 0.5)) for t, f in df.items()}
        self.k1, self.b = k1, b

    def search(self, query, k):
        q = tokens(query)
        scores = []
        for cid, d in zip(self.ids, self.docs):
            tf = Counter(d)
            s = sum(self.idf.get(t, 0) * tf[t] * (self.k1 + 1)
                    / (tf[t] + self.k1 * (1 - self.b + self.b * len(d) / self.avg)) for t in q)
            scores.append((s, cid))
        scores.sort(reverse=True)
        return [cid for s, cid in scores[:k] if s > 0]

def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.?!])\s+", text) if s.strip()]

REFUSAL = "I can't find that in the handbook."

class StubGenerator:
    """Stands in for an LLM. A real one would get the prompt below and return JSON.

    prompt = "Answer only from the context. Cite chunk ids. If the context does not
              contain the answer, say you can't find it.\n\n" + context + question
    """
    def __init__(self, invent=False):
        self.invent = invent  # simulates a prompt change that lets the model use outside knowledge

    def answer(self, question, chunks):
        q = set(tokens(question))
        picked = []
        for cid, text in chunks.items():
            for s in sentences(text):
                overlap = len(q & set(tokens(s)))
                if overlap >= 2:
                    picked.append((overlap, cid, s))
        picked.sort(key=lambda x: -x[0])
        picked = picked[:2]
        if not picked:
            if self.invent:
                return {"answer": "Yes, the company reimburses up to 2,000 rupees a month for a gym membership.", "citations": []}
            return {"answer": REFUSAL, "citations": []}
        text = " ".join(s for _, _, s in picked)
        if self.invent and "leave" in q:
            text += " Leave days are credited monthly."
        return {"answer": text, "citations": sorted({cid for _, cid, _ in picked})}

The metrics, retrieval on ids and generation on text, plus the diagnosis rule:

# metrics.py: retrieval metrics on ids, generation metrics on text
import re
from rag import tokens, sentences, REFUSAL

def hit(retrieved, relevant):
    return float(any(r in relevant for r in retrieved))

def recall(retrieved, relevant):
    return len(set(retrieved) & set(relevant)) / len(relevant)

def reciprocal_rank(retrieved, relevant):
    for rank, cid in enumerate(retrieved, start=1):
        if cid in relevant:
            return 1 / rank
    return 0.0

def numbers(text):
    return set(re.findall(r"\d[\d,]*", text))

def supports(chunk, claim):
    """Cheap judge: every number in the claim appears in the chunk, and most of its words do.
    Swap this for an LLM judge call in production (same signature: chunk, claim -> bool)."""
    words = set(tokens(claim))
    if not words or not numbers(claim) <= numbers(chunk):
        return False
    return len(words & set(tokens(chunk))) / len(words) >= 0.6

def faithfulness(answer, context):
    if answer == REFUSAL:
        return 1.0
    claims = sentences(answer)
    ok = sum(any(supports(c, claim) for c in context.values()) for claim in claims)
    return ok / len(claims)

def correct(answer, case):
    if not case["relevant"]:                  # unanswerable: the only right answer is a refusal
        return answer == REFUSAL
    return all(f.lower() in answer.lower() for f in case["facts"])

def citation_precision(citations, context, answer):
    """Share of cited ids that were retrieved and back at least one sentence of the answer."""
    if not citations:
        return 1.0
    good = sum(c in context and any(supports(context[c], s) for s in sentences(answer)) for c in citations)
    return good / len(citations)

def diagnose(row):
    if not row["answerable"]:
        return "ok" if row["correct"] else "generation: answered an unanswerable question"
    if not row["correct"]:
        return "retrieval: evidence missing" if row["recall"] < 1 else "generation: evidence present, answer wrong"
    if row["faithfulness"] < 1:
        return "generation: unsupported claim"
    return "ok"

diagnose() is the part that answers which half broke. If the answer is wrong and some relevant chunk was not retrieved, blame retrieval. If the answer is wrong while every relevant chunk was in the context, blame generation. If the answer is right but contains an unsupported sentence, that is still a generation problem, because the extra sentence is the hallucination a user will eventually trip on.

The runner evaluates four configurations. One is the baseline, two are plausible changes a team might ship, and the last is a fix.

# run_eval.py: run the golden set through one configuration and report both halves
import sys
from data import CHUNKS, GOLDEN
from rag import BM25, StubGenerator
import metrics as m
import rag

def evaluate(top_k=3, invent=False):
    retriever, gen = BM25(CHUNKS), StubGenerator(invent=invent)
    rows = []
    for case in GOLDEN:
        ids = retriever.search(case["question"], k=top_k)
        context = {cid: CHUNKS[cid] for cid in ids}
        out = gen.answer(case["question"], context)
        row = {"id": case["id"], "answerable": bool(case["relevant"]), "retrieved": ids,
               "answer": out["answer"],
               "faithfulness": m.faithfulness(out["answer"], context),
               "correct": m.correct(out["answer"], case),
               "citation_precision": m.citation_precision(out["citations"], context, out["answer"])}
        if case["relevant"]:
            row |= {"hit": m.hit(ids, case["relevant"]), "recall": m.recall(ids, case["relevant"]),
                    "rr": m.reciprocal_rank(ids, case["relevant"])}
        row["diagnosis"] = m.diagnose(row)
        rows.append(row)
    return rows

def report(name, rows):
    ans = [r for r in rows if r["answerable"]]
    mean = lambda key, rs: sum(r[key] for r in rs) / len(rs)
    print(f"== {name}")
    print(f"retrieval   hit_rate={mean('hit', ans):.2f}  recall={mean('recall', ans):.2f}  MRR={mean('rr', ans):.2f}")
    print(f"generation  faithfulness={mean('faithfulness', rows):.2f}  correct={mean('correct', rows):.2f}"
          f"  citation_precision={mean('citation_precision', rows):.2f}")
    for r in rows:
        if r["diagnosis"] != "ok":
            print(f"  {r['id']} {r['diagnosis']:45s} retrieved={r['retrieved']}")
    print()

if __name__ == "__main__":
    report("baseline: top_k=3", evaluate(top_k=3))
    report("change A: top_k=1 to save tokens", evaluate(top_k=1))
    report("change B: prompt allows outside knowledge", evaluate(top_k=3, invent=True))
    rag.STEM = True
    report("fix: strip plurals in the tokenizer", evaluate(top_k=3))

Reading the four runs#

Output of python3 run_eval.py:

== baseline: top_k=3
retrieval   hit_rate=0.89  recall=0.89  MRR=0.89
generation  faithfulness=1.00  correct=0.90  citation_precision=1.00
  q06 retrieval: evidence missing                   retrieved=[]

== change A: top_k=1 to save tokens
retrieval   hit_rate=0.89  recall=0.83  MRR=0.89
generation  faithfulness=1.00  correct=0.80  citation_precision=1.00
  q06 retrieval: evidence missing                   retrieved=[]
  q09 retrieval: evidence missing                   retrieved=['wfh-2']

== change B: prompt allows outside knowledge
retrieval   hit_rate=0.89  recall=0.89  MRR=0.89
generation  faithfulness=0.68  correct=0.80  citation_precision=1.00
  q01 generation: unsupported claim                 retrieved=['leave-1', 'sick-1', 'wfh-1']
  q02 generation: unsupported claim                 retrieved=['leave-1', 'sick-1', 'leave-2']
  q03 generation: unsupported claim                 retrieved=['leave-2', 'sick-1', 'leave-1']
  q06 retrieval: evidence missing                   retrieved=[]
  q10 generation: answered an unanswerable question retrieved=[]

== fix: strip plurals in the tokenizer
retrieval   hit_rate=1.00  recall=1.00  MRR=1.00
generation  faithfulness=1.00  correct=0.90  citation_precision=1.00
  q06 generation: evidence present, answer wrong    retrieved=['it-3']

Baseline. Correctness is 0.90 and the one failure is q06, "What is the minimum password length?". Nothing was retrieved. The chunk says "Passwords must be at least 14 characters long", and after stop words the query keeps minimum, password and length. None of those match passwords, characters or long exactly, so BM25 scores every chunk 0. Hit rate is 8 of 9 answerable questions, 0.89. MRR is also 0.89, because every chunk that was found came back at rank 1. The diagnosis says retrieval, and no prompt change would have fixed it.

Change A, top_k cut from 3 to 1. Hit rate stays at 0.89, but recall drops to 0.83. q09 needs two chunks and now gets one, so its recall is 0.5, and (7 + 0.5) / 9 = 0.83. Correctness falls to 0.80. Hit rate hides this regression and recall shows it.

Change B, the prompt allows outside knowledge. Retrieval numbers are identical to the baseline, which rules out retrieval at a glance. Faithfulness drops from 1.00 to 0.68 because three leave answers gained a sentence no chunk supports, and q10 now claims a 2,000-rupee gym benefit that exists nowhere in the handbook. Correctness only drops from 0.90 to 0.80, since q01 to q03 still contain their required facts. If you tracked correctness alone, you would ship a prompt that adds an invented sentence to a third of the answers.

The fix, plural stripping in the tokenizer. passwords becomes password, BM25 finds it-3, and every retrieval metric reaches 1.00. Correctness stays at 0.90, though, because q06 is still wrong. Its diagnosis moved from "retrieval: evidence missing" to "generation: evidence present, answer wrong". The stub needs two shared words and only password overlaps. A real LLM would usually get this one, since it can connect "minimum length" with "at least 14 characters". Fixing retrieval did not fix the question; it moved the failure to the other half, and the per-question label says the next fix belongs in generation. A single total score shows no change here, and you would conclude the tokenizer fix did nothing.

Which half broke: a lookup table#

What moved What it means Where to look
Hit rate and recall both down Evidence is not being found Chunking, tokenizer, embedding model, filters
Recall down, hit rate flat Multi-part questions lose a piece top_k, chunk size, query decomposition
Hit rate fine, MRR low Evidence is found but ranked low Reranker, hybrid fusion weights
Retrieval flat, faithfulness down Model adds claims not in context Prompt, temperature, model choice
Retrieval flat, correctness down, faithfulness flat Model has the evidence and misreads or omits it Prompt, context order, answer format
Refusal row now answered Model fills gaps with invented facts Refusal instruction in the prompt, faithfulness gate
Citation precision down Citations point at chunks that do not hold the claim Citation format in the prompt, a post-check

For the retrieval rows, RAG architectures compared has the upgrade path from naive to hybrid to reranked retrieval, and the cheat sheet has a runnable hybrid search example scored with recall@k and MRR.

Test the metrics before you trust them#

A metric function with a bug produces confident wrong numbers. Pin each one to a case you worked out by hand:

# test_metrics.py: pin the metric functions to numbers worked out by hand
import metrics as m

assert m.reciprocal_rank(["a", "b", "c"], ["b"]) == 0.5        # first relevant at rank 2
assert m.reciprocal_rank(["a", "b", "c"], ["z"]) == 0.0        # never found
assert m.hit(["a", "b"], ["b", "c"]) == 1.0                    # one of two is enough for a hit
assert m.recall(["a", "b"], ["b", "c"]) == 0.5                 # but recall counts both

ctx = {"exp-2": "Expense claims above 20,000 rupees need approval from a department head.",
       "it-1": "New joiners receive a laptop on day one. Laptops are replaced every 3 years."}
answer = "Claims above 20,000 rupees need approval from a department head."
assert m.faithfulness(answer, ctx) == 1.0
assert m.faithfulness(answer + " Approval takes 2 days.", ctx) == 0.5   # invented number
assert m.citation_precision(["exp-2"], ctx, answer) == 1.0
assert m.citation_precision(["exp-2", "it-1"], ctx, answer) == 0.5     # it-1 backs nothing
assert m.citation_precision(["hr-9"], ctx, answer) == 0.0              # cited a chunk never retrieved
print("all metric tests pass")
$ python3 test_metrics.py
all metric tests pass

These tests are the only place in this post where citation precision drops below 1.0. The stub cites only chunks it quoted from, so it never miscites. Real models do, which is why the check belongs in the pipeline even when the demo shows 1.00.

Replacing the stubs#

The generator. Replace StubGenerator.answer with a call to your model that sends the prompt in the docstring and asks for JSON with answer and citations. Keep the return shape and nothing else in the pipeline changes. Run each question a few times at your production temperature and keep the worst score per question, because a single run can hide a failure that shows up one time in three. The agent evaluation post shows that repeat-run pattern.

The judge. Replace supports() with an LLM call that receives one claim and one chunk and returns yes or no. Then hand-label 30 to 50 claim and chunk pairs and measure how often the judge agrees with you before trusting its averages. Zheng et al. found that strong judges agree with human raters about as often as humans agree with each other, and they also documented position and verbosity bias, so check on your own data.

Off-the-shelf metrics. Ragas (version 0.4.4 on PyPI when this was written) ships Faithfulness, ContextRecall and ContextPrecision in ragas.metrics.collections, each scored by an LLM you configure. Its context recall is not the recall above. By default it splits your reference answer into claims and checks how many the retrieved context supports, so it needs a reference answer instead of chunk ids; the same page documents an id-based variant too. Its context precision is a rank-weighted precision, closer to MRR in what it rewards. DeepEval has a faithfulness metric that takes input, actual_output and retrieval_context, and a contextual recall metric. Both libraries change their APIs between minor versions, so pin the version, and keep your own id-based retrieval metrics alongside them. Those cost nothing to run and never change meaning.

Tracking runs. Write each run's per-question rows to a file or a SQLite table, keyed by config and dataset version, and compare runs question by question rather than by average. The LLM evaluation dashboard post builds that storage and a regression view on top.

Ways the evaluation itself goes wrong#

  • Labels that go stale. Re-chunking changes ids, and old labels then point at chunks that no longer exist. Every retrieval metric collapses to zero overnight. Version the chunk set with the dataset.
  • Only answerable questions. A set with no unanswerable rows cannot catch the q10 failure from change B. Keep at least 10% unanswerable.
  • Questions copied from the chunks. Questions written while looking at a chunk reuse its exact words, so lexical retrieval looks better than it is. The q06 miss appeared only because the question was phrased the way a user would phrase it. Draw questions from real query logs where you can.
  • Faithfulness scored against the gold chunk. Score it against what was retrieved. Scoring against the gold chunk hides the cases where the model answered from a wrong chunk.
  • One number on the dashboard. Report the two halves separately and keep the per-question diagnosis. The tokenizer fix above took retrieval from 0.89 to 1.00 and left overall correctness flat; only the per-question view showed the progress.

Sources#