LLM-as-a-judge in Python: rubrics, biases and calibration against human labels
Build an LLM judge you can trust. Pass/fail rubrics, pairwise vs pointwise, position and length bias, and Cohen's kappa against 30 human labels.
An LLM judge is a second model call that grades the first one. It is easy to write: a prompt, a "rate this 1 to 5", a regex to pull out the number. It is much harder to know whether its grades mean anything. A judge that passes bad answers is worse than no judge, because now the dashboard is green and nobody looks.
This post builds a small judge for a customer support bot and then does the part most tutorials skip: it checks the judge against 30 answers a person has labelled, computes agreement and Cohen's kappa, and finds where the judge is wrong. Along the way it covers rubric design, why pass/fail beats a 1 to 5 scale for most app evals, pairwise versus pointwise grading, the biases that published papers have measured in LLM judges, and a case where embedding similarity gives the wrong answer with high confidence.
Two earlier posts cover the surrounding pieces, so this one does not repeat them. Evaluating an AI agent in Python builds a golden set and checks tool calls and arguments. The LLM evaluation dashboard stores runs and compares prompt versions. A judge is one scorer you plug into either.
The judge model here is a rule-based stub behind a one-function interface, so everything runs offline with no API key. The stub has a known flaw built in, and the point of the post is to show the calibration step catching it. Tested with Python 3.15.2, scikit-learn 1.9.1 and sentence-transformers 6.3.0 in a venv.
Where the judge sits#
eval case judge calibration (once per judge change)
--------- ----- -----------------------------------
question ─┐
context ─┼─> rubric prompt ─> model ─> JSON ─> parse ─> pass / fail per criterion
answer ─┘ │ │
└─> error (retry, │
never "0") v
30 human labels ──> agreement, kappa,
confusion matrix,
list of disagreements
The judge runs on every eval case. Calibration runs when you write the judge and again whenever you change its prompt or its model. Until calibration says the judge agrees with people, its scores are opinions.
Pass/fail per criterion, not a 1 to 5 score#
The usual first judge asks for a score from 1 to 5 and defines only the ends: 5 is perfect, 1 is a hallucination. Nobody defines 3. Two runs of the same model will disagree on whether an answer with one small error is a 3 or a 4, and so will two people. Then the team sets a gate like "average judge score must stay above 4.5" and spends weeks arguing about drops from 4.6 to 4.4 that are mostly noise.
Binary verdicts avoid most of that. Each criterion is one yes/no question that a person could answer in a few seconds, written down precisely enough that two people would agree. A case passes when every criterion passes. You lose some resolution, and in exchange each verdict is checkable: when the judge says "fail", you can read the answer and decide whether it was right. That property is what makes calibration possible.
Keep a scale only when the quality really is graded and you have written an anchor for every point. G-Eval is an example of that approach, with chain-of-thought steps and a form the model fills in. For a support bot, "is every claim in the context?" is a yes/no question, so treat it as one.
Three rules for the rubric text:
- One property per criterion. "Grounded" and "answers the question" are separate. An answer can be true and off-topic, or on-topic and invented.
- Say what fails. "Accurate" is vague. "A number, limit or condition that differs from the context is a fail" is something a model and a person can both apply.
- Reason before verdict. Ask for a short reason first and the verdict after it, in JSON. The reason is also what you read when the judge and a person disagree.
The judge#
# judge.py
import json
import re
RUBRIC = {
"grounded": (
"Every factual claim in the ANSWER is stated in the CONTEXT. "
"A number, limit or condition that differs from the CONTEXT is a fail. "
"Words like 'always' or 'never' are a fail unless the CONTEXT says them."
),
"answers_question": (
"The ANSWER responds to what the QUESTION asked. "
"A correct fact about a different topic is a fail. "
"Saying the information is not available passes only if the CONTEXT lacks it."
),
}
PROMPT = """You grade one answer from a customer support bot.
QUESTION: {question}
CONTEXT: {context}
ANSWER: {answer}
For each criterion below, write one sentence of reasoning, then a verdict.
Length and confident tone are not evidence. Judge only what the ANSWER claims.
{criteria}
Reply with JSON only, in this shape:
{{"grounded": {{"reason": "...", "verdict": "pass" or "fail"}},
"answers_question": {{"reason": "...", "verdict": "pass" or "fail"}}}}"""
def build_prompt(case):
criteria = "\n".join(f"- {name}: {text}" for name, text in RUBRIC.items())
return PROMPT.format(criteria=criteria, **case)
class JudgeParseError(ValueError):
pass
def parse_verdict(raw):
"""Turn the judge's text into {criterion: 'pass'|'fail'} or raise."""
match = re.search(r"\{.*\}", raw, re.DOTALL)
if not match:
raise JudgeParseError("no JSON object in judge output")
data = json.loads(match.group(0))
verdicts = {}
for name in RUBRIC:
verdict = data.get(name, {}).get("verdict")
if verdict not in ("pass", "fail"):
raise JudgeParseError(f"bad verdict for {name}: {verdict!r}")
verdicts[name] = verdict
return verdicts
def judge(case, call_model, retries=1):
"""Grade one case. Returns verdicts, or None if the judge never gave valid output."""
prompt = build_prompt(case)
for _ in range(retries + 1):
try:
verdicts = parse_verdict(call_model(prompt))
except (JudgeParseError, json.JSONDecodeError):
continue
overall = "pass" if all(v == "pass" for v in verdicts.values()) else "fail"
return {"criteria": verdicts, "overall": overall}
return None # count it as an error, never as a score of 0
Two details differ from the regex-and-a-number pattern. The output is JSON with a fixed shape, and anything else raises an error. And when the judge never produces valid output, judge() returns None. The common version sets the score to 0 when the regex misses. That quietly turns a formatting failure into a "terrible answer" grade and drags the average down for reasons that have nothing to do with the app. Count parse failures separately and alert on them.
With a real model, pass the JSON schema to the provider's structured-output option so the reply is valid JSON in the first place. Ollama supports this through the format field, and the hosted APIs have their own versions. If you use Ollama through LangChain, the current docs use the langchain-ollama package (OllamaLLM, ChatOllama) rather than the older langchain_community.llms.Ollama class. Keep parse_verdict anyway; schemas reduce bad output but do not check that the verdict values are the ones you expect.
The stub model#
judge() takes any function from prompt string to reply string. Here that function is a stub that reads the prompt and applies simple rules. Replace stub_judge_model with a function that sends the prompt to your judge model and returns its text; nothing else changes.
# stub_model.py
# A rule-based stand-in for the judge model, so the post runs with no API key.
# It reads the prompt like a model would and replies with JSON text.
# It has a built-in flaw on purpose: long answers get the benefit of the doubt.
import json
import re
ABSOLUTES = ("always", "never", "every", "guaranteed", "no receipt")
def _field(prompt, name):
return re.search(rf"^{name}: (.*)$", prompt, re.MULTILINE).group(1)
def stub_judge_model(prompt):
context = _field(prompt, "CONTEXT").lower()
answer = _field(prompt, "ANSWER").lower()
numbers_ok = set(re.findall(r"\d+", answer)) <= set(re.findall(r"\d+", context))
absolutes_ok = all(w not in answer or w in context for w in ABSOLUTES)
grounded = numbers_ok and absolutes_ok
if len(answer.split()) > 35: # the flaw: verbosity reads as thoroughness
grounded = True
on_topic = "sorry" not in answer
def item(ok, why):
return {"reason": why, "verdict": "pass" if ok else "fail"}
return json.dumps({
"grounded": item(grounded, "numbers and limits checked against context"),
"answers_question": item(on_topic, "checked the answer addresses the question"),
})
The rules are crude on purpose, and the length rule is a planted bug. It imitates a bias real judges have (more on that below). The question is whether calibration finds it without being told.
Calibrating against 30 human labels#
Calibration means grading a set of answers twice, once by a person and once by the judge, and measuring how often they agree. The labelled set should contain your app's real answers, including bad ones; a set of only good answers cannot tell you whether the judge catches failures. Thirty is a workable start: one person can label them in under an hour. The 30 cases here cover five short policy texts (returns, shipping, warranty, payment, cancellation), with 16 human passes and 14 human fails. Each line of labels.jsonl looks like this:
{"id": 1, "question": "What is the return window?", "context": "Items can be returned within 30 days of purchase with a receipt. Sale items cannot be returned.", "answer": "You can return items within 30 days if you have the receipt.", "human": "pass"}
Raw agreement is the share of cases where the two match. It is misleading when one label is much more common than the other, so the script also computes Cohen's kappa, which subtracts the agreement you would get by chance from each side's label rates:
kappa = (p_observed - p_chance) / (1 - p_chance)
# metrics.py
import random
def cohen_kappa(a, b):
"""Cohen's kappa for two lists of labels."""
n = len(a)
labels = sorted(set(a) | set(b))
p_observed = sum(x == y for x, y in zip(a, b)) / n
p_chance = sum((a.count(k) / n) * (b.count(k) / n) for k in labels)
if p_chance == 1:
return 1.0
return (p_observed - p_chance) / (1 - p_chance)
def bootstrap_kappa(a, b, rounds=2000, seed=7):
rng = random.Random(seed)
idx = range(len(a))
stats = []
for _ in range(rounds):
pick = [rng.choice(idx) for _ in idx]
stats.append(cohen_kappa([a[i] for i in pick], [b[i] for i in pick]))
stats.sort()
return stats[int(0.025 * rounds)], stats[int(0.975 * rounds)]
The bootstrap resamples the 30 cases with replacement 2,000 times and reports the middle 95% of kappa values. With so few labels the interval will be wide, and you should see how wide.
# calibrate.py
import json
from judge import judge
from metrics import bootstrap_kappa, cohen_kappa
from stub_model import stub_judge_model # swap for your real model call
cases = [json.loads(line) for line in open("labels.jsonl")]
human, model, errors = [], [], []
for case in cases:
result = judge(case, stub_judge_model)
if result is None:
errors.append(case["id"])
continue
human.append(case["human"])
model.append(result["overall"])
case["judge"] = result["overall"]
pairs = list(zip(human, model))
tp = pairs.count(("fail", "fail")) # judge caught a real failure
fn = pairs.count(("fail", "pass")) # judge missed it
fp = pairs.count(("pass", "fail")) # judge failed a good answer
tn = pairs.count(("pass", "pass"))
print(f"graded {len(pairs)} cases, {len(errors)} judge errors")
print(f" judge fail judge pass")
print(f"human fail {tp:>10} {fn:>10}")
print(f"human pass {fp:>10} {tn:>10}")
print(f"raw agreement {(tp + tn) / len(pairs):.2f}")
k = cohen_kappa(human, model)
lo, hi = bootstrap_kappa(human, model)
print(f"Cohen's kappa {k:.2f} (95% bootstrap CI {lo:.2f} to {hi:.2f})")
print(f"fail recall {tp / (tp + fn):.2f} (share of real failures the judge caught)")
print(f"fail precision {tp / (tp + fp):.2f} (share of judge fails that were real)")
print("\ndisagreements:")
for c in cases:
if c.get("judge") and c["judge"] != c["human"]:
words = len(c["answer"].split())
print(f" #{c['id']:<3} human={c['human']} judge={c['judge']} words={words:<3} {c['answer'][:48]}")
graded 30 cases, 0 judge errors
judge fail judge pass
human fail 6 8
human pass 2 14
raw agreement 0.67
Cohen's kappa 0.31 (95% bootstrap CI -0.03 to 0.63)
fail recall 0.43 (share of real failures the judge caught)
fail precision 0.75 (share of judge fails that were real)
disagreements:
#3 human=fail judge=pass words=8 Sale items can be returned within 30 days.
#7 human=fail judge=pass words=49 Thanks for asking about returns. Our policy is d
#13 human=fail judge=pass words=53 Great question. We want shopping to be easy, so
#14 human=pass judge=fail words=10 Every order over $50 ships free; smaller orders
#16 human=pass judge=fail words=9 The warranty lasts 1 year and covers manufacturi
#18 human=fail judge=pass words=8 Yes, accidental damage is covered for 12 months.
#19 human=fail judge=pass words=52 Electronics bought from us come with a 12 month
#23 human=fail judge=pass words=52 You have several ways to pay. We accept UPI, cre
#24 human=fail judge=pass words=6 Cash on delivery is not offered.
#29 human=fail judge=pass words=8 Yes, you can cancel any time before delivery.
The cross-check against scikit-learn's cohen_kappa_score on the same labels gives the same number:
sklearn kappa 0.31
Reading the numbers#
Kappa by hand, to see where 0.31 comes from. The two sides agree on 20 of 30 cases, so p_observed is 0.667. The person said fail 14 times and the judge 8 times, so the chance of both saying fail at random is (14/30)(8/30) = 0.124, and of both saying pass is (16/30)(22/30) = 0.391. p_chance is 0.516, and kappa is (0.667 - 0.516) / (1 - 0.516) = 0.31.
On the scale Landis and Koch proposed, 0.21 to 0.40 is "fair" agreement. That label matters less than two other numbers:
- Fail recall is 0.43. The judge caught 6 of 14 real failures. If this judge gated releases, more than half the bad answers would ship with a green check.
- The 95% interval runs from -0.03 to 0.63. Thirty labels cannot tell a useless judge from a decent one. They can tell you this judge is not good enough, which is enough to stop and fix it. To claim a judge is good, label more cases; with 100 or more the interval narrows a lot.
The disagreement list is where the work is. Four of the eight missed failures (#7, #13, #19, #23) are answers of around 50 words, and each hides one wrong fact inside polite, plausible text. A pass rate split by length shows the pattern:
# length_check.py
import json
from judge import judge
from stub_model import stub_judge_model
cases = [json.loads(line) for line in open("labels.jsonl")]
for name, group in [("long (>35 words)", [c for c in cases if len(c["answer"].split()) > 35]),
("short", [c for c in cases if len(c["answer"].split()) <= 35])]:
human = sum(c["human"] == "pass" for c in group) / len(group)
model = sum(judge(c, stub_judge_model)["overall"] == "pass" for c in group) / len(group)
print(f"{name:<17} n={len(group):<3} human pass rate {human:.2f} judge pass rate {model:.2f}")
long (>35 words) n=5 human pass rate 0.20 judge pass rate 1.00
short n=25 human pass rate 0.60 judge pass rate 0.68
People passed one long answer in five; the judge passed all of them. That is the planted length rule, found from the data. The other four misses are short answers whose wrong claim has no wrong number in it ("Sale items can be returned within 30 days", "Cash on delivery is not offered"), which tells you the groundedness check needs to compare claims, not just numbers. The two false fails (#14, #16) come from the judge treating "every" and "1 year" as unsupported when they mean the same as the context. Each disagreement points at a specific change to the rubric or the judge model.
The loop is: change the rubric or the judge model, re-run calibrate.py, read the new disagreements, stop when fail recall and kappa are good enough for the decision the judge supports. A judge that blocks releases needs high fail recall. A judge that only sorts cases for human review can be looser. When you change the judge's model version, re-run calibration before trusting any trend line, because every past score was produced by the old judge. The CI/CD post covers pinning the judge in the eval manifest.
Why raw agreement alone is a trap#
A mature app might pass 27 of 30 cases. A judge that passes everything matches the person 90% of the time:
# kappa_trap.py
from metrics import cohen_kappa
# 27 good answers, 3 bad ones, as in a mature app where most answers pass
human = ["pass"] * 27 + ["fail"] * 3
lazy_judge = ["pass"] * 30 # passes everything
agreement = sum(h == j for h, j in zip(human, lazy_judge)) / len(human)
print(f"agreement {agreement:.2f}, kappa {cohen_kappa(human, lazy_judge):.2f}")
agreement 0.90, kappa 0.00
90% agreement and a kappa of zero: the judge has learned nothing except the base rate. This is the most common way a judge looks good on a dashboard. Report kappa and fail recall next to agreement, and make sure the calibration set has enough failures in it; oversample bad answers when you build it.
Known biases, with the papers that measured them#
The planted length rule is a cartoon of a real effect. Three biases have been measured repeatedly.
Position bias. In pairwise grading, the judge prefers whichever answer appears first (or second, depending on the model). Wang et al. showed that rankings could be changed by "simply altering their order of appearance in the context," and proposed scoring both orders and combining them. Zheng et al., the MT-Bench paper, measured position bias along with the next two.
Verbosity bias. Longer answers get higher grades even when the extra text adds nothing. The MT-Bench paper documents it. The length-controlled AlpacaEval work corrects for output length with a regression and reports that the Spearman correlation of its leaderboard with Chatbot Arena rose from 0.94 to 0.98. In your own evals, the length-split check above is the cheap test.
Self-preference. A judge tends to rate answers from its own model higher than humans do. The MT-Bench paper calls this self-enhancement; Panickssery et al. found that evaluators can recognise their own outputs and favour them. The practical rule is to use a judge from a different model family than the app's generator where you can, and to treat any judge-versus-human gap on your own model's outputs as suspect until calibration rules it out.
The same MT-Bench paper also reports that strong judges reached over 80% agreement with human preferences, about the same as humans agree with each other. That is the reason to use judges at all. It is a result on their benchmark, though, and your rubric and your data are different, which is why you calibrate.
Pairwise or pointwise#
Pointwise grading looks at one answer and returns a verdict. Pairwise grading shows two answers to the same question and asks which is better.
| Pointwise (pass/fail) | Pairwise (A vs B) | |
|---|---|---|
| Question it answers | Is this answer acceptable? | Did version B get better than A? |
| Good for | Release gates, monitoring production samples | Comparing two prompts or models |
| Calls per case | 1 | 2 (both orders) |
| Main bias | Leniency, length | Position, length |
| Output you can act on | A list of failing cases | A win rate |
Pairwise is easier for a model in the same way it is easier for a person: "which is better?" needs no absolute standard. But a win rate does not tell you whether either version is good enough to ship, so most app evals use pointwise pass/fail as the gate and pairwise for choosing between candidates.
If you grade pairwise, always run both orders and keep only verdicts that survive the swap:
# pairwise.py
import json
import re
PAIR_PROMPT = """Which answer is better supported by the CONTEXT?
QUESTION: {question}
CONTEXT: {context}
ANSWER A: {a}
ANSWER B: {b}
Reply with JSON only: {{"reason": "...", "winner": "A" or "B" or "tie"}}"""
def stub_pair_model(prompt):
"""Stand-in model. When it cannot separate two answers it picks A: position bias."""
get = lambda k: re.search(rf"^{k}: (.*)$", prompt, re.MULTILINE).group(1).lower()
context = get("CONTEXT")
def support(ans):
return len(set(re.findall(r"\d+", ans)) - set(re.findall(r"\d+", context))) == 0
a, b = support(get("ANSWER A")), support(get("ANSWER B"))
winner = "A" if a == b else ("A" if a else "B")
return json.dumps({"reason": "compared numbers with context", "winner": winner})
def compare(case, a, b, call_model):
raw = call_model(PAIR_PROMPT.format(question=case["question"], context=case["context"], a=a, b=b))
return json.loads(raw)["winner"]
def compare_both_orders(case, x, y, call_model):
"""Ask twice with the answers swapped. Keep the verdict only if it survives the swap."""
first = compare(case, x, y, call_model) # x in slot A
second = compare(case, y, x, call_model) # x in slot B
second = {"A": "B", "B": "A", "tie": "tie"}[second] # map back to x/y slots
return first if first == second else "inconsistent"
cases = {c["id"]: c for c in map(json.loads, open("labels.jsonl"))}
PAIRS = [(1, 6), (1, 7), (2, 3), (4, 5), (8, 9), (8, 14), (10, 11), (12, 13), (15, 16),
(15, 20), (17, 18), (17, 19), (21, 25), (22, 23), (22, 24), (26, 27), (28, 29)]
flips = 0
for i, j in PAIRS:
verdict = compare_both_orders(cases[i], cases[i]["answer"], cases[j]["answer"], stub_pair_model)
if verdict == "inconsistent":
flips += 1
print(f" pair {i} vs {j}: winner changed when the order changed")
print(f"{flips} of {len(PAIRS)} pairs flipped with order ({flips / len(PAIRS):.0%})")
pair 1 vs 7: winner changed when the order changed
pair 2 vs 3: winner changed when the order changed
pair 4 vs 5: winner changed when the order changed
pair 8 vs 9: winner changed when the order changed
pair 8 vs 14: winner changed when the order changed
pair 17 vs 18: winner changed when the order changed
pair 17 vs 19: winner changed when the order changed
pair 21 vs 25: winner changed when the order changed
pair 22 vs 24: winner changed when the order changed
pair 26 vs 27: winner changed when the order changed
pair 28 vs 29: winner changed when the order changed
11 of 17 pairs flipped with order (65%)
The stub pair model breaks ties by picking slot A, so whenever two answers look equally supported, the "winner" depends on which one was shown first. Without the swap, those 11 verdicts would have gone into the win rate as real preferences. With the swap, they become "inconsistent", which you count as a tie or send to a person. Track the flip rate over time; if it climbs after a judge model change, the new judge is less reliable even if its win rates look reasonable.
When similarity metrics mislead#
Before reaching for a judge, many pipelines score answers by embedding similarity to a reference answer and fail anything under a threshold such as 0.85. The dashboard post shows character-level similarity failing on a reworded answer. Embeddings fail in a different way:
# similarity_trap.py
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
reference = "Orders over $50 ship free. Smaller orders pay a flat $5."
answers = {
"correct, reworded": "If you spend more than fifty dollars, delivery costs nothing; otherwise it is five dollars.",
"wrong threshold": "Orders over $500 ship free. Smaller orders pay a flat $5.",
"flipped meaning": "Orders under $50 ship free. Larger orders pay a flat $5.",
"always free": "Yes, shipping is always free on every order.",
}
ref_vec = model.encode(reference)
for label, text in answers.items():
score = model.similarity(ref_vec, model.encode(text)).item()
print(f"{label:<18} {score:.3f}")
correct, reworded 0.670
wrong threshold 0.962
flipped meaning 0.990
always free 0.577
With all-MiniLM-L6-v2, the answer that reverses the policy ("under $50 ship free") scores 0.990, and the one with a tenfold wrong threshold scores 0.962. The correct paraphrase scores 0.670. A 0.85 threshold passes both wrong answers and fails the right one. Embeddings measure whether two texts are about the same thing in similar words. Swapping "over" for "under" or 50 for 500 barely moves the vector, and those are exactly the errors a support bot must not make.
Similarity is still useful as a trend signal. A sudden drop across many cases means the answers changed shape and someone should look. It should not decide pass or fail on facts. Use string checks for facts you can name, and a calibrated judge for the rest.
Failure modes to watch#
- Parse failures counted as scores. Covered above. Return an error and count it.
- A calibration set with no failures. You cannot measure fail recall without fails. Oversample known bad answers.
- Judging with the context missing. A groundedness judge that never sees the context is grading plausibility, not grounding. Check the prompt it actually received.
- Labels from the same person who wrote the rubric, only. Have a second person label 10 of the 30. If the two people disagree a lot, the rubric is unclear, and no judge will fix that.
- Unpinned judge model. A silent provider upgrade changes every score at once. Pin the version and re-calibrate when you change it.
- Trusting the reason text. The judge's explanation is useful for debugging, but it is generated after the fact like any other output. The verdict is what you calibrate.
What to do next#
Write two or three yes/no criteria for your app, label 30 real answers with at least a third of them failures, and run the calibration script with your judge model in place of the stub. If fail recall is under about 0.8 for a judge that gates releases, read the disagreements and fix the rubric before wiring the judge into CI.
Sources#
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (position, verbosity and self-enhancement bias; agreement with humans)
- Wang et al., Large Language Models are not Fair Evaluators (position bias, balanced position calibration)
- Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations (self-preference)
- Dubois et al., Length-Controlled AlpacaEval (length bias correction)
- Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (form-filling rubric with chain of thought)
- Landis and Koch, The Measurement of Observer Agreement for Categorical Data, Biometrics, 1977
- scikit-learn,
cohen_kappa_score - Ollama docs, Structured outputs
- LangChain docs, Ollama LLM integration
- Sentence Transformers, all-MiniLM-L6-v2 model card