Articles / evalsllmopspythonrag
Detect and measure hallucinations in an LLM app, in Python
Types of hallucination, claim-level support checks, NLI, self-consistency, request-path guards, and a hallucination rate with honest confidence intervals.
"Our bot hallucinates sometimes" is not a measurement. It doesn't tell you which answers are wrong, in what way, how often, or whether last week's prompt change made it better. To act on hallucinations you need three things: a definition precise enough to label against, detectors that point at the wrong sentence, and a rate with an error bar so you know whether a change is real.
Most posts on this topic do one of those. They list a library with a hallucination_score() call, or they explain self-consistency in a paragraph. Few show what the detector gets wrong, and almost none put a confidence interval on the final number. This post covers all three, with small Python files that run offline. The model and the entailment check are stubs, and I say where to swap in real ones.
Some neighbouring topics have their own posts, so I link to them instead of repeating them. Guardrail layering, latency and online sampling math are in guardrails and observability. Building a golden set and a harness is in evaluating an AI agent. Storing runs and comparing prompt versions is in the evaluation dashboard post. Scoring retrieval quality is in the RAG evaluation pipeline, and building and calibrating a judge prompt is in LLM-as-a-judge in Python.
What counts as a hallucination#
The survey by Ji et al. splits hallucinations by their relation to a source. An intrinsic hallucination contradicts the source. An extrinsic one adds something the source neither supports nor contradicts. That split maps well onto RAG apps, where you have a source for every answer, and it tells you what each kind of check can see.
In application terms, four failures cover most of what users report:
| Type | Example (source says "5 to 7 business days") | Intrinsic or extrinsic | What catches it |
|---|---|---|---|
| Wrong number or entity | "Shipping takes 2 to 3 business days" | Intrinsic | Number and entity comparison, NLI contradiction |
| Unsupported addition | "Express shipping is free for premium members" | Extrinsic | Claim-level support check |
| Fabricated citation | "...[doc4]" when only doc1 and doc2 were retrieved | Either | ID check against the retrieved set, free and exact |
| Closed-book fabrication | A plausible founding year when no source was given | Extrinsic, no source at all | Self-consistency sampling, retrieval, or refusing |
The last row is the hard one. With no source to compare against, you can only look at the model's own behaviour, which is what sampling-based methods do.
An extrinsic claim isn't always false. "Express shipping is free for premium members" might be true on the website. In a RAG app it still counts as a failure, because the app promised answers grounded in its documents and this one isn't. Decide this up front and write it in your labelling guide. Otherwise two labellers will disagree on every extrinsic case and your rate will be noise.
The pipeline#
Every detector below follows the same steps. It splits the answer into claims, checks each claim, and then combines the per-claim labels.
answer ──► split into claims ──► for each claim:
├─ cited IDs exist in the retrieved set? (exact, ~0 ms)
├─ numbers / entities match the cited text? (exact, ~0 ms)
├─ NLI: does the cited text entail it? (small model, tens of ms)
└─ no source? sample k more answers, agree? (k model calls)
│
▼
label per claim: supported / contradicted / unsupported / fabricated_citation
│
┌────────────────────────────┴───────────────────────────┐
▼ ▼
request path: block, retry, or flag offline: answer-level label,
(cheap checks only) rate + confidence interval
Claim-level labels matter because a long answer can be 90% right. If you label whole answers, one wrong number in a five-sentence reply looks the same as a reply that's entirely made up, and a detector that points at the wrong sentence is far more useful to the person fixing the prompt. FActScore takes this to its end: it breaks a generation into atomic facts and reports the share supported by a knowledge source. In its human evaluation of biographies, ChatGPT scored 58%.
Offline detection: claims checked against sources#
Here is the core check. It splits an answer into sentences (a real system would ask a model for atomic claims), checks cited IDs, then runs an entailment check against each cited source. The nli_stub is a rule-based stand-in for a natural language inference model. It flags a number in the claim that the source doesn't contain, and treats high word overlap as entailment.
# checks.py
import re
NUM = re.compile(r"\d+(?:[.,]\d+)*%?")
CITE = re.compile(r"\[(doc\d+)\]")
STOP = {"the", "a", "an", "of", "to", "in", "is", "are", "and", "or", "for",
"on", "with", "within", "can", "be", "by", "it", "you", "your", "at"}
def split_claims(answer):
"""One claim per sentence. A real system asks a model for atomic claims."""
parts = re.split(r"(?<=[.!?])\s+", answer.strip())
return [p for p in parts if p]
def words(text):
return {w for w in re.findall(r"[a-z]+", text.lower()) if w not in STOP}
def numbers(text):
return {n.replace(",", "") for n in NUM.findall(CITE.sub("", text))}
def nli_stub(premise, hypothesis):
"""Stand-in for an NLI model. Returns entailment, contradiction or neutral.
Rules: a number in the claim that the premise does not contain is a
contradiction if the premise has some other number, else neutral.
Otherwise, high word overlap counts as entailment.
"""
h_nums, p_nums = numbers(hypothesis), numbers(premise)
if h_nums - p_nums:
return "contradiction" if p_nums else "neutral"
h = words(CITE.sub("", hypothesis))
overlap = len(h & words(premise)) / max(len(h), 1)
return "entailment" if overlap >= 0.6 else "neutral"
def check_answer(answer, sources):
"""Label every claim against the sources it cites (or all sources)."""
results = []
for claim in split_claims(answer):
cited = CITE.findall(claim)
missing = [c for c in cited if c not in sources]
if missing:
results.append((claim, "fabricated_citation", missing))
continue
pool = [sources[c] for c in cited] or list(sources.values())
labels = [nli_stub(p, claim) for p in pool]
if "entailment" in labels:
label = "supported"
elif "contradiction" in labels:
label = "contradicted" # intrinsic: conflicts with the source
else:
label = "unsupported" # extrinsic: source says nothing
results.append((claim, label, cited))
return results
Run it on an answer with one of each failure:
# demo_checks.py
from checks import check_answer
SOURCES = {
"doc1": "Refunds are available within 30 days of purchase for unused items.",
"doc2": "Standard shipping takes 5 to 7 business days within India.",
}
ANSWER = (
"You can get a refund within 30 days of purchase for unused items [doc1]. "
"Shipping takes 2 to 3 business days [doc2]. "
"Express shipping is free for premium members [doc2]. "
"Gift cards never expire [doc4]."
)
for claim, label, cited in check_answer(ANSWER, SOURCES):
print(f"{label:<20} {claim}")
supported You can get a refund within 30 days of purchase for unused items [doc1].
contradicted Shipping takes 2 to 3 business days [doc2].
unsupported Express shipping is free for premium members [doc2].
fabricated_citation Gift cards never expire [doc4].
Two of these checks are exact and you should run them everywhere. A citation ID that wasn't in the retrieved set is always a bug, and a string lookup finds it. A number in the answer that appears in none of the cited text is not always wrong ("a week" versus "7 days"), but it's wrong often enough that it's worth flagging every time. Wrong numbers also do the most damage, because a refund window or a price is something a user acts on.
Where the stub breaks, and why you need a real NLI model#
Word overlap is a weak stand-in for entailment. Here it is on three claims a reader would judge easily:
# stub_limits.py
from checks import nli_stub
premise = "Refunds are available within 30 days of purchase for unused items."
for claim in [
"Refunds are not available within 30 days of purchase for unused items.",
"You have thirty days after purchase to return unused items for a refund.",
"Refunds are available within 30 days of purchase for used items.",
]:
print(f"{nli_stub(premise, claim):<14} {claim}")
entailment Refunds are not available within 30 days of purchase for unused items.
neutral You have thirty days after purchase to return unused items for a refund.
entailment Refunds are available within 30 days of purchase for used items.
All three are wrong. The negation passes because "not" is one extra word and overlap stays high. The correct paraphrase fails because "thirty" is not "30" and the wording differs. "Used" for "unused" passes because five of six content words still match, which clears the 0.6 threshold. These are the cases an NLI model is trained on. A cross-encoder such as cross-encoder/nli-deberta-v3-base takes a (premise, hypothesis) pair and returns scores for contradiction, entailment and neutral, which is the same interface as nli_stub. Swapping it in is a one-function change:
# not run here: needs the sentence-transformers package and a model download
from sentence_transformers import CrossEncoder
_model = CrossEncoder("cross-encoder/nli-deberta-v3-base")
_LABELS = ["contradiction", "entailment", "neutral"] # order from the model card
def nli(premise, hypothesis):
scores = _model.predict([(premise, hypothesis)])[0]
return _LABELS[scores.argmax()]
Keep the exact number check in front of it anyway. NLI models trained on general text can miss a changed digit in a long premise, and the regex never does.
Three practical limits of NLI on RAG output:
- Long premises. NLI models are trained on sentence pairs. Feeding a 2,000-token chunk as the premise degrades them. Split the source into sentences and take the maximum entailment score across them.
- Claims that need two sources. "Refunds take 30 days and shipping takes a week, so a return cycle is about five weeks" needs doc1 and doc2 together, plus arithmetic. Per-source NLI will call it unsupported. That's arguably correct for a strict RAG policy, but your labelling guide must say so.
- Claim splitting is itself a model call. If you use an LLM to extract atomic claims, it can drop or merge claims. Check a sample of its splits by hand before trusting the claim counts.
Self-consistency, when there is no source#
For closed-book questions there is nothing to compare against. The idea of sampling several answers and trusting the one they agree on comes from Wang et al., who used a majority vote over reasoning paths to improve accuracy. SelfCheckGPT uses the model's own behaviour instead: if a model knows a fact, answers sampled at non-zero temperature tend to agree; if it is guessing, they diverge. The paper compares several ways to score agreement, including n-gram overlap, question answering and NLI between the main answer and the samples.
The stub model below always gives the same founding year and guesses the head count:
# consistency.py
import random
from collections import Counter
from checks import numbers
def fake_model(question, seed):
"""Stub for a model sampled at temperature > 0.
It 'knows' the founding year (same answer every time) but guesses
the head count, so that answer drifts between samples.
"""
rng = random.Random(seed)
if "founded" in question:
return "The company was founded in 2011."
return f"The company has about {rng.choice([120, 150, 150, 400, 90])} employees."
def consistency(question, k=5):
"""Share of k samples that agree with the most common set of numbers."""
keys = [frozenset(numbers(fake_model(question, s))) for s in range(k)]
top, count = Counter(keys).most_common(1)[0]
return count / k, sorted(top)
for q in ["When was the company founded?", "How many employees does it have?"]:
score, answer = consistency(q)
flag = "ok" if score >= 0.8 else "LOW AGREEMENT"
print(f"{score:.1f} {flag:<14} {answer} <- {q}")
1.0 ok ['2011'] <- When was the company founded?
0.6 LOW AGREEMENT ['150'] <- How many employees does it have?
Comparing extracted numbers works for factoid answers. For free text you need to decide when two answers mean the same thing. Farquhar et al. (Nature, 2024) cluster sampled answers by bidirectional entailment and compute entropy over the clusters ("semantic entropy"), which targets confabulations: answers that are both wrong and arbitrary.
Know what this method can't see:
- Consistent errors pass. If the model learned a wrong fact firmly, all five samples repeat it and agreement is 1.0. Self-consistency detects guessing, not being wrong.
- Cost scales with k. Five samples is five generations. For a 300-token answer that's about 1,500 extra output tokens per checked request, an estimate that follows directly from k times the answer length. That's fine offline and too slow for most request paths.
- Temperature matters. At temperature 0 the samples are near-identical whether the model knows the fact or not, so agreement is meaningless. Sample at the temperature the method was tested with, or at least well above zero, and keep it fixed across runs.
A cheaper relative is Chain-of-Verification: the model drafts an answer, writes verification questions about its own claims, answers them independently, and revises. It costs a few extra calls instead of k full samples, and it gives you a revised answer rather than only a score.
Online checks on the request path#
On the request path you can afford only the cheap checks, and you need to decide what each one does when it fires. A reasonable default:
| Label | Action | Why |
|---|---|---|
fabricated_citation |
Block, retry once with feedback | Always a bug, exact to detect |
contradicted (number mismatch) |
Block, retry once with feedback | High precision when the source has a number and the claim has a different one |
unsupported |
Ship, flag for review | Lower precision; blocking these causes many false refusals |
| Retry fails again | Fallback answer with source links | Saying "I couldn't confirm that" beats shipping a known contradiction |
# guard.py
from checks import check_answer
SOURCES = {
"doc1": "Refunds are available within 30 days of purchase for unused items.",
"doc2": "Standard shipping takes 5 to 7 business days within India.",
}
BLOCKING = {"fabricated_citation", "contradicted"}
def stub_generate(question, attempt, feedback=None):
"""Stub model: the first draft has a wrong number, the retry fixes it."""
if attempt == 0:
return "Shipping takes 2 to 3 business days [doc2]."
return "Standard shipping takes 5 to 7 business days within India [doc2]."
def answer_with_guard(question, max_retries=1):
feedback = None
for attempt in range(max_retries + 1):
draft = stub_generate(question, attempt, feedback)
results = check_answer(draft, SOURCES)
bad = [(c, label) for c, label, _ in results if label in BLOCKING]
soft = [c for c, label, _ in results if label == "unsupported"]
if not bad:
return {"answer": draft, "attempts": attempt + 1, "flag_for_review": bool(soft)}
feedback = "; ".join(f"{label}: {c}" for c, label in bad)
print(f"attempt {attempt + 1} blocked -> {feedback}")
return {"answer": "I couldn't confirm that from our documents. Here are the sources: doc1, doc2.",
"attempts": max_retries + 1, "flag_for_review": True}
print(answer_with_guard("How long does shipping take?"))
attempt 1 blocked -> contradicted: Shipping takes 2 to 3 business days [doc2].
{'answer': 'Standard shipping takes 5 to 7 business days within India [doc2].', 'attempts': 2, 'flag_for_review': False}
The retry doubles latency for the requests that hit it. If 5% of requests are blocked once, mean latency rises by about 5% and the p95 sits right at the edge of the slow group, so it moves a little. If 20% are blocked, the p95 request is one of the retried ones and p95 roughly doubles. These are estimates from treating the retry as one more full generation of the same length. Track the block rate as a metric, because a jump in it usually means retrieval broke, not that the model got worse. Where this guard sits among the other checks, and whether it should fail open or closed, is covered in the guardrails post.
Streaming makes this harder. If tokens go to the user as they are generated, a post-generation check can only append a correction. For answers where a wrong number causes harm (prices, dosages, deadlines), buffer the answer and check it before sending.
Measuring a hallucination rate#
Detectors produce flags. A rate needs labels. The minimum honest setup is a small set of real answers that a person labels using the written guide from earlier, and a detector whose agreement with that person you've measured.
Sixty labelled answers is a realistic first set. Here is what you can say from it:
# rate.py
import math
Z = 1.96 # 95% two-sided
def wilson(k, n, z=Z):
"""Wilson score interval for k successes out of n."""
p = k / n
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return centre - half, centre + half
def wald(k, n, z=Z):
p = k / n
half = z * math.sqrt(p * (1 - p) / n)
return p - half, p + half
def pct(lo_hi):
return f"[{lo_hi[0]:6.1%}, {lo_hi[1]:6.1%}]"
# 60 answers a person labelled, each paired with what the detector said.
# (human_says_hallucinated, detector_flagged)
LABELLED = ([(True, True)] * 5 + [(True, False)] * 2 +
[(False, True)] * 4 + [(False, False)] * 49)
n = len(LABELLED)
k = sum(h for h, _ in LABELLED)
print(f"human-labelled rate {k}/{n} = {k / n:.1%}")
print(f" Wilson 95% CI {pct(wilson(k, n))}")
print(f" Wald 95% CI {pct(wald(k, n))}")
tp = sum(h and d for h, d in LABELLED)
fn = sum(h and not d for h, d in LABELLED)
fp = sum(d and not h for h, d in LABELLED)
tn = n - tp - fn - fp
sens, spec = tp / (tp + fn), tn / (tn + fp)
print(f"detector recall {tp}/{tp + fn} = {sens:.0%} CI {pct(wilson(tp, tp + fn))}")
print(f"detector precision {tp}/{tp + fp} = {tp / (tp + fp):.0%} CI {pct(wilson(tp, tp + fp))}")
print(f"detector specificity {tn}/{tn + fp} = {spec:.0%}")
# The detector runs on 5,000 unlabelled production answers and flags 14%.
flagged = 0.14
corrected = (flagged + spec - 1) / (sens + spec - 1) # Rogan-Gladen
print(f"raw flag rate {flagged:.0%} -> corrected estimate {corrected:.1%}")
for margin in (0.05, 0.03, 0.02):
need = math.ceil(Z * Z * 0.1 * 0.9 / margin ** 2)
print(f"labels needed for +/-{margin:.0%} at a 10% rate: {need}")
human-labelled rate 7/60 = 11.7%
Wilson 95% CI [ 5.8%, 22.2%]
Wald 95% CI [ 3.5%, 19.8%]
detector recall 5/7 = 71% CI [ 35.9%, 91.8%]
detector precision 5/9 = 56% CI [ 26.7%, 81.1%]
detector specificity 49/53 = 92%
raw flag rate 14% -> corrected estimate 10.1%
labels needed for +/-5% at a 10% rate: 139
labels needed for +/-3% at a 10% rate: 385
labels needed for +/-2% at a 10% rate: 865
The rate is "somewhere between 6% and 22%". Seven failures out of sixty gives a point estimate of 11.7%, but the interval is wide. If a prompt change takes you to 5/60, the intervals overlap almost entirely and you can't claim an improvement. Report the interval next to the rate, every time.
Use Wilson, not the textbook interval. The Wald interval (p ± 1.96·√(p(1−p)/n)) is the one most people remember. At small n and rates near 0 it is too narrow and can go below zero. Brown, Cai and DasGupta (2001) show its coverage is poor even at moderate sample sizes and recommend Wilson for small n. With 0 failures out of 30, Wald gives [0%, 0%], which claims certainty you don't have. Wilson gives an upper bound near 11%.
The detector's own numbers are uncertain too. Recall of 71% rests on seven positive cases, so its interval runs from 36% to 92%. You can't tune a detector threshold on seven positives. To estimate recall to within about ±10 points you need on the order of 60 to 90 labelled hallucinations, which at a 10% base rate means 600 to 900 labelled answers. That's an estimate from the same sample-size formula applied to recall. If that's too many to label, oversample answers the detector flagged or users disliked, and report the uniform sample separately so the headline rate stays unbiased.
A raw flag rate is not a hallucination rate. The detector flagged 14% of production answers, but it has false positives and misses. If you know its sensitivity and specificity, the Rogan-Gladen estimator from epidemiology corrects for both: (flag rate + specificity − 1) / (sensitivity + specificity − 1). Here that gives 10.1%. With sensitivity and specificity measured on sixty labels, this correction inherits their uncertainty, so treat it as a rough check that the production rate is in the same range as your labelled set, not a precise figure.
Sample size grows with the square of the precision you want. ±5 points at a 10% rate needs about 139 labels, ±3 needs 385, ±2 needs 865. Pick the smallest change you would act on before you start labelling. Comparing two versions needs more than this, because both rates carry error; the two-proportion calculation is in the guardrails post.
One more rule saves a lot of confusion: decide whether you report a claim-level rate (share of claims unsupported) or an answer-level rate (share of answers with at least one bad claim), and don't switch. An answer with five claims and a 3% claim-level error rate fails at the answer level about 14% of the time (1 − 0.97⁵), assuming independent claims. Both numbers are honest. Mixing them across reports isn't.
Failure modes of the detection setup itself#
| Failure | Symptom | Fix |
|---|---|---|
| Labelling guide is vague on extrinsic claims | Two labellers agree on 70% of cases, the rate jumps between batches | Write the policy ("anything not in the cited text is a failure") and label 20 cases together first |
| Detector tuned on its own flags | Precision looks great, recall unknown | Label a uniform random sample, not only flagged answers |
| Retrieval failure scored as hallucination | Rate rises after an index rebuild with no model change | Log retrieved IDs with every answer, and separate "source missing" from "model made it up" (see the RAG evaluation pipeline) |
| Self-consistency at temperature 0 | Agreement is 1.0 on everything | Sample at a fixed non-zero temperature |
| Judge used as the ground truth | Rate moves when the judge prompt changes | Calibrate the judge against human labels first (LLM-as-a-judge) and version it |
| Rate reported without an interval | A 2-point "improvement" on 60 cases gets celebrated | Print Wilson bounds next to every rate |
| Claim splitter drops claims | Fewer claims per answer after a splitter change, rate "improves" | Track claims per answer as its own metric |
Where to start#
If you have nothing today, do these in order. Each step is useful on its own.
- Add the citation-ID check and the number check to the request path. They cost almost nothing and are exact.
- Write a one-page labelling guide with the four types from the first table, then label 60 real answers. Report the rate with its Wilson interval.
- Run the claim checker offline on those 60 and measure its precision and recall against your labels. Replace
nli_stubwith a real NLI model and measure again. - Grow the labelled set toward 300 to 400 answers, which gets you to about ±3 points at a 10% rate. Store each run so you can compare versions, as in the dashboard post.
- Add self-consistency only for the closed-book routes, where there is no source to check against.
Sources#
- Ji et al., Survey of Hallucination in Natural Language Generation, 2022 (intrinsic and extrinsic hallucination)
- Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, 2023
- Manakul, Liusie and Gales, SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models, 2023
- Farquhar et al., Detecting hallucinations in large language models using semantic entropy, Nature, 2024
- Dhuliawala et al., Chain-of-Verification Reduces Hallucination in Large Language Models, 2023
- Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022
- Model card, cross-encoder/nli-deberta-v3-base (Hugging Face)
- Brown, Cai and DasGupta, Interval Estimation for a Binomial Proportion, Statistical Science, 2001
- Rogan and Gladen, Estimating prevalence from the results of a screening test, American Journal of Epidemiology, 1978
- Binomial proportion confidence interval (Wikipedia, for the Wilson formula)