GyaanStack. Blog

Articles / ragprojectspython

Build a multi-document RAG system: metadata, routing, conflicts and comparisons

A local multi-document RAG system in Python with per-document metadata, filters, citations, conflict flags, comparison questions and an eval with real numbers.

By Raghunath · · 27 min read

Most RAG tutorials load a folder of PDFs, split everything into chunks, throw the chunks into one vector store and call it multi-document RAG. That works until two documents say different things about the same topic. Ask "what is the API rate limit?" against a folder that holds the v1 release notes, the v2 release notes and an old FAQ, and the retriever happily returns all three. The model then picks one, and nothing in the pipeline tells it which one is right.

This post builds the parts that the one-big-index approach skips:

  • a metadata record for every document (type, version, effective date, status), copied onto every chunk
  • a router that turns a question into metadata filters
  • citations that name the document and section
  • a conflict check when two sources disagree
  • comparison questions ("how does v2 differ from v1?") that retrieve from each side separately
  • dedup of near-identical chunks across versions, and the way it can silently corrupt answers

Everything runs locally with no API key. Embeddings come from nomic-embed-text through Ollama, the generator is a stub you can swap for qwen3:4b-instruct, and the only Python dependency is the standard library. At the end, a 22-question labelled set measures what metadata filtering changes compared with naive retrieval.

If you want a single-PDF bot with page citations, the PDF question-answering bot covers that. For the write side at scale (connectors, re-indexing, embedding cost), see RAG ingestion pipeline system design. This post sits between the two: many documents, small enough to run on a laptop, with the failure modes that only show up once documents overlap.

Why one big index fails#

Real document sets are full of near-copies. HR publishes a new leave policy every year and changes three numbers. An API gets a v2 and the release notes keep the same headings. Somebody's meeting notes mention a proposed change that never shipped. Embeddings encode topic, and all of these documents share a topic.

Here is the cosine similarity, with nomic-embed-text, between matching sections of the 2024 and 2025 leave policies in the test corpus below:

section          cosine   text identical?
Annual leave     1.0000   yes
Carry forward    0.9994   no  (8 days vs 12 days)
Parental leave   0.9995   no  (16 weeks vs 26 weeks)

To the vector index, "carry forward up to 8 days" and "carry forward up to 12 days" are the same chunk. No amount of tuning k fixes that. The retriever needs information that is not in the text: which document a chunk came from, which version it is, and whether that version is still in force.

The design#

  docs/ (md, csv, txt)          catalog.json (one record per file)
          |                              |
          v                              v
     split per section  ---->  chunk = text + doc metadata + vectors
                                         |
                            dedup (hash, or cosine + same numbers)
                                         |
  question --> router --> filters: [{family, version}] or [{family, status: current}]
                                         |
                    one filter  -> top-k inside it
                    two filters -> top-k/2 from each side (comparison)
                                         |
                    conflict check --> prompt with [doc > section] tags --> LLM

The catalog is the part people skip. In production the metadata comes from the source system (SharePoint version history, a Git tag, a CMS field). Here it is a JSON file.

Setup#

You need Ollama running with two models, and Python 3.10 or newer:

ollama pull nomic-embed-text
ollama pull qwen3:4b-instruct   # only for the optional real-LLM step
mkdir -p multi-doc-rag/docs && cd multi-doc-rag

nomic-embed-text produces 768-dimensional vectors and expects a task prefix on every input: search_document: for chunks and search_query: for questions. The model card says the prefix is required. The code calls Ollama's /api/embed endpoint, which takes a list of inputs in one request.

The corpus#

Eight files of four types, built so that they overlap on purpose. Two leave policies differ only in their numbers, two API release notes share headings, the FAQ is stale, and the meeting notes contain a proposal that conflicts with the travel policy.

cat > docs/leave-policy-2024.md <<'EOF'
# Leave policy (2024)

## Annual leave
Every full-time employee gets 24 days of paid annual leave per calendar year. Leave accrues monthly at 2 days per month.

## Carry forward
Unused annual leave can be carried forward to the next year up to a maximum of 8 days. Anything above 8 days lapses on 31 December.

## Sick leave
Employees get 10 days of paid sick leave per year. A medical certificate is needed for more than 2 consecutive sick days.

## Parental leave
The primary caregiver gets 16 weeks of paid parental leave. The secondary caregiver gets 2 weeks.
EOF
# 2025 policy: same text, new numbers
sed -e 's/(2024)/(2025)/' -e 's/8 days/12 days/g' -e 's/16 weeks/26 weeks/' \
    -e 's/gets 2 weeks/gets 4 weeks/' docs/leave-policy-2024.md > docs/leave-policy-2025.md
cat > docs/travel-policy-2025.md <<'EOF'
# Travel policy (2025)

## Flights
Book economy class for flights under 6 hours. Business class is allowed for flights of 6 hours or more.

## Per diem
The daily allowance for meals in Bangalore is INR 3,500. For other metro cities it is INR 3,000.

## Hotels
The hotel cap is INR 9,000 per night in metro cities, including taxes.
EOF
cat > docs/api-v1-release-notes.md <<'EOF'
# Acme API v1 release notes

## Rate limits
Each API key can make 60 requests per minute. Requests above the limit get HTTP 429.

## Authentication
Send your API key in the X-Api-Key header on every request.

## Payload size
The maximum request body is 5 MB.
EOF
cat > docs/api-v2-release-notes.md <<'EOF'
# Acme API v2 release notes

## Rate limits
Each client can make 300 requests per minute. Requests above the limit get HTTP 429 with a Retry-After header.

## Authentication
Use OAuth 2.0 client credentials. Send the bearer token in the Authorization header.

## Payload size
The maximum request body is 10 MB.
EOF
cat > docs/support-faq.md <<'EOF'
# Support FAQ

## What is the API rate limit?
The API allows 60 requests per minute on all plans.

## How do I reset my password?
Click "Forgot password" on the sign-in page and follow the link sent to your email.

## Can I change my plan later?
Yes. Upgrades take effect immediately and downgrades apply from the next billing cycle.
EOF
cat > docs/pricing.csv <<'EOF'
plan,price_inr_per_month,seats,storage_gb
Starter,999,3,50
Team,4999,20,500
Enterprise,24999,unlimited,5000
EOF
cat > docs/ops-meeting-2025-09.txt <<'EOF'
Ops meeting notes, 18 September 2025

Attendees discussed raising the Bangalore meal per diem to INR 4,000 from April 2026. This is a proposal only and is not yet approved policy.

Action: finance to share a cost estimate for the higher per diem by October.
EOF

The catalog gives each file a family (documents about the same subject), a version, an effective date and a status. Status carries the most weight: current, superseded or proposal.

{
  "leave-policy-2024.md":   {"doc_id": "leave-2024", "family": "leave-policy",  "doc_type": "policy",        "version": "2024", "effective": "2024-01-01", "status": "superseded"},
  "leave-policy-2025.md":   {"doc_id": "leave-2025", "family": "leave-policy",  "doc_type": "policy",        "version": "2025", "effective": "2025-01-01", "status": "current"},
  "travel-policy-2025.md":  {"doc_id": "travel-2025", "family": "travel-policy", "doc_type": "policy",       "version": "2025", "effective": "2025-04-01", "status": "current"},
  "api-v1-release-notes.md":{"doc_id": "api-v1",     "family": "api",           "doc_type": "release-notes", "version": "v1",   "effective": "2023-05-10", "status": "superseded"},
  "api-v2-release-notes.md":{"doc_id": "api-v2",     "family": "api",           "doc_type": "release-notes", "version": "v2",   "effective": "2025-06-02", "status": "current"},
  "support-faq.md":         {"doc_id": "faq",        "family": "support",       "doc_type": "faq",           "version": "2024", "effective": "2024-02-15", "status": "current"},
  "pricing.csv":            {"doc_id": "pricing",    "family": "pricing",       "doc_type": "table",         "version": "2025", "effective": "2025-01-01", "status": "current"},
  "ops-meeting-2025-09.txt":{"doc_id": "ops-notes",  "family": "travel-policy", "doc_type": "meeting-notes", "version": "2025", "effective": "2025-09-18", "status": "proposal"}
}

Save that as catalog.json. Putting the FAQ in its own support family is a deliberate call: a FAQ is a summary of other documents, so it should not compete with the release notes on API questions.

Ingest: one splitter per file type, metadata on every chunk#

Each file type gets its own splitter. Markdown splits on ## headings, each CSV row becomes one chunk written as column = value pairs, and plain text splits on blank lines. Section-level chunks suit documents this short; the RAG cheat sheet covers chunk sizes and overlap for longer ones.

Every chunk gets two vectors. vec_body embeds the chunk text alone. vec_headed embeds "<document title> | <section>" followed by the text. The header idea is a cheap version of Anthropic's contextual retrieval, which prepends document context to each chunk before embedding. The eval below shows how much each one does.

# ingest.py
import csv, hashlib, json, re, urllib.request
from pathlib import Path

OLLAMA = "http://localhost:11434/api/embed"

def embed(texts, kind):
    # nomic-embed-text wants a task prefix on every input
    prefix = "search_query: " if kind == "query" else "search_document: "
    body = json.dumps({"model": "nomic-embed-text", "input": [prefix + t for t in texts]})
    req = urllib.request.Request(OLLAMA, body.encode(), {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req))["embeddings"]

def split_markdown(text):
    """One chunk per '## ' section. Returns (title, [(section, body), ...])."""
    title = re.search(r"^# (.+)$", text, re.M).group(1).strip()
    parts = re.split(r"^## ", text, flags=re.M)[1:]
    return title, [(p.split("\n", 1)[0].strip(), p.split("\n", 1)[1].strip()) for p in parts]

def split_csv(path):
    rows = list(csv.DictReader(open(path)))
    return [(r["plan"], ", ".join(f"{k} = {v}" for k, v in r.items())) for r in rows]

def split_text(text):
    paras = [p.strip() for p in text.split("\n\n") if p.strip()]
    return paras[0], [(f"para {i}", p) for i, p in enumerate(paras[1:], 1)]

def load(folder, catalog):
    chunks = []
    for name, meta in catalog.items():
        path = Path(folder) / name
        if path.suffix == ".md":
            title, sections = split_markdown(path.read_text())
        elif path.suffix == ".csv":
            title, sections = "Pricing plans (2025)", split_csv(path)
        else:
            title, sections = split_text(path.read_text())
        for section, body in sections:
            chunks.append({**meta, "title": title, "section": section, "text": body,
                           "source": name})
    return chunks

META_KEYS = ("doc_id", "family", "doc_type", "version", "effective", "status")

def norm_hash(text):
    return hashlib.sha1(re.sub(r"\W+", " ", text.lower()).strip().encode()).hexdigest()

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / ((sum(x * x for x in a) ** 0.5) * (sum(y * y for y in b) ** 0.5))

def numbers(text):
    return sorted(re.findall(r"\d[\d,.]*", text))

def is_twin(a, b, threshold, check_numbers):
    if a["hash"] == b["hash"]:
        return True
    close = cosine(a["vec_body"], b["vec_body"]) >= threshold
    return close and (not check_numbers or numbers(a["text"]) == numbers(b["text"]))

def dedup(chunks, threshold=0.98, check_numbers=True):
    """Collapse exact and near-identical chunks; keep every source on the survivor."""
    kept = []
    for c in sorted(chunks, key=lambda c: c["effective"], reverse=True):  # newest survives
        twin = next((k for k in kept if is_twin(k, c, threshold, check_numbers)), None)
        if twin:
            twin["also_in"].append({k: c[k] for k in META_KEYS})
        else:
            kept.append({**c, "also_in": []})
    return kept

def build_index(folder="docs", catalog_path="catalog.json"):
    chunks = load(folder, json.load(open(catalog_path)))
    bodies = [c["text"] for c in chunks]
    headed = [f"{c['title']} | {c['section']}\n{c['text']}" for c in chunks]
    for c, vb, vh in zip(chunks, embed(bodies, "doc"), embed(headed, "doc")):
        c["vec_body"], c["vec_headed"], c["hash"] = vb, vh, norm_hash(c["text"])
    return chunks

if __name__ == "__main__":
    chunks = build_index()
    print(f"{len(chunks)} chunks from {len({c['doc_id'] for c in chunks})} documents")
    for check in (False, True):
        unique = dedup(chunks, check_numbers=check)
        print(f"check_numbers={check}: {len(unique)} chunks after dedup")
        for c in unique:
            if c["also_in"]:
                print(f"  kept {c['doc_id']} / {c['section']}, also in {[m['doc_id'] for m in c['also_in']]}")
    json.dump(chunks, open("index.json", "w"))

Run it:

$ python3 ingest.py
25 chunks from 8 documents
check_numbers=False: 20 chunks after dedup
  kept api-v2 / Payload size, also in ['api-v1']
  kept leave-2025 / Annual leave, also in ['leave-2024']
  kept leave-2025 / Carry forward, also in ['leave-2024']
  kept leave-2025 / Sick leave, also in ['leave-2024']
  kept leave-2025 / Parental leave, also in ['leave-2024']
check_numbers=True: 23 chunks after dedup
  kept leave-2025 / Annual leave, also in ['leave-2024']
  kept leave-2025 / Sick leave, also in ['leave-2024']

Dedup: the number check is not optional#

The first dedup pass uses cosine 0.98 alone, the threshold you see in plenty of examples. It merges five pairs. Three of them are wrong: carry forward (8 vs 12 days), parental leave (16 vs 26 weeks) and payload size (5 MB vs 10 MB). After that merge, the 2024 policy's carry-forward rule no longer exists in the index. A question about 2024 gets the 2025 number, with a citation that says 2024.

The fix in is_twin is to require that the two chunks contain the same numbers before treating them as one. With that check, only the two sections with identical text merge. For prose-heavy corpora where wording drifts, MinHash over word shingles is the usual tool, but it still needs this numeric guard: "8" and "12" are one token each and barely move a Jaccard score.

Two details in dedup matter. The newest chunk survives, so its doc_id is the current document. The merged chunk keeps every source's metadata in also_in, so a filter for version: 2024 still finds the shared sick-leave section, and the citation lists both documents.

Routing, filtering and citations#

The router reads a question and returns a list of filters. Name a version or year and you get one filter per version mentioned. Name only a subject and you get status: current, which is the rule "answer from what is in force today". A comparison question produces two filters, and retrieval gives each side its own share of the k slots so one document cannot crowd out the other.

The router here is a keyword table. That is the honest weak point of this build, and the place to put a small LLM: give qwen3:4b-instruct the catalog's families and versions, ask for the same JSON, and validate the output against the catalog before using it. A rule table goes a long way for internal corpora with stable names. It fails on synonyms the table has never seen ("PTO" for leave), so log every query that routes to [{}].

# retrieve.py
import re
from ingest import META_KEYS, cosine, embed

FAMILIES = {"leave": "leave-policy", "parental": "leave-policy", "carry": "leave-policy",
            "sick": "leave-policy", "travel": "travel-policy", "per diem": "travel-policy",
            "hotel": "travel-policy", "flight": "travel-policy", "api": "api",
            "rate limit": "api", "payload": "api", "plan": "pricing", "price": "pricing",
            "storage": "pricing", "seats": "pricing", "password": "support"}
COMPARE = re.compile(r"\b(differ|difference|changed?|compare|versus|vs)\b", re.I)

def route(query):
    """Turn a question into one metadata filter per document it is about.
    Rule-based stand-in; a small LLM returning the same JSON is the usual upgrade."""
    q = query.lower()
    family = next((f for word, f in FAMILIES.items() if word in q), None)
    versions = re.findall(r"\b(20\d\d|v\d)\b", q)
    if family and versions:
        return [{"family": family, "version": v} for v in versions]
    if family:   # no version named: use what is in force today
        return [{"family": family, "status": "current"}]
    return [{}]

def matches(chunk, flt):
    metas = [{k: chunk[k] for k in META_KEYS}] + chunk.get("also_in", [])
    return any(all(m.get(k) == v for k, v in flt.items()) for m in metas)

def search(chunks, qvec, k, flt=None, field="vec_headed"):
    pool = [c for c in chunks if flt is None or matches(c, flt)]
    return sorted(pool, key=lambda c: -cosine(qvec, c[field]))[:k]

def retrieve(chunks, query, k=3, use_filters=True, field="vec_headed"):
    qvec = embed([query], "query")[0]
    if not use_filters:
        return search(chunks, qvec, k, field=field)
    filters = route(query)
    if len(filters) > 1 or COMPARE.search(query):
        per_doc = max(1, k // len(filters))   # comparison: give each side its own slots
        return [c for f in filters for c in search(chunks, qvec, per_doc, f, field)]
    return search(chunks, qvec, k, filters[0], field)

def docs_of(c):
    return {c["doc_id"]} | {m["doc_id"] for m in c.get("also_in", [])}

def cite(c):
    extra = "".join(f", {m['doc_id']}" for m in c.get("also_in", []))
    return f"[{c['doc_id']}{extra} > {c['section']}]"

def quantities(text):
    """(unit, number) pairs such as ('requests', '60') or ('inr', '3,500')."""
    t = text.lower()
    after = re.findall(r"(\d[\d,]*)\s+([a-z]+)", t)
    before = re.findall(r"([a-z]+)\s+(\d[\d,]*)", t)
    return {(u, n) for n, u in after} | {(u, n) for u, n in before}

def conflicts(hits):
    """Pairs of hits from different documents that give different numbers for the same unit."""
    out = []
    for i, a in enumerate(hits):
        for b in hits[i + 1:]:
            qa, qb = quantities(a["text"]), quantities(b["text"])
            units = {u for u, _ in qa} & {u for u, _ in qb}
            if not docs_of(a) & docs_of(b) and any(
                    {n for u2, n in qa if u2 == u} != {n for u2, n in qb if u2 == u} for u in units):
                trust = max(a, b, key=lambda c: (c["status"] == "current", c["effective"]))
                out.append((a["doc_id"], b["doc_id"], trust["doc_id"]))
    return out

def build_prompt(query, hits):
    blocks = [f"{cite(c)} ({c['doc_type']}, effective {c['effective']}, {c['status']})\n{c['text']}"
              for c in hits]
    return ("Answer from the sources only. Cite every claim with its [doc > section] tag.\n"
            "If sources disagree, say so, and prefer the newest current policy.\n\n"
            + "\n\n".join(blocks) + f"\n\nQuestion: {query}")

def stub_llm(query, hits):
    """Keyless stand-in for the generator: quotes each source with its citation
    and flags disagreements. Swap in ollama_llm(build_prompt(query, hits))."""
    lines = [f"{c['text']} {cite(c)}" for c in hits]
    for a, b, trust in conflicts(hits):
        lines.append(f"CONFLICT: {a} and {b} disagree; trusting {trust}.")
    return "\n".join(lines)

def ollama_llm(prompt, model="qwen3:4b-instruct"):
    import json, urllib.request
    body = json.dumps({"model": model, "prompt": prompt, "stream": False,
                       "options": {"temperature": 0}}).encode()
    req = urllib.request.Request("http://localhost:11434/api/generate", body,
                                 {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req))["response"]

The filter runs in Python over a list here. In a real vector store you pass it to the query: Chroma takes a where dict with operators such as $eq, $in and $and (metadata filtering docs), and Qdrant has payload filters with must and should clauses (filtering docs). One catch with stores that filter on flat fields: the also_in trick needs a list field, for example doc_ids: ["leave-2025", "leave-2024"] matched with $in or a match any condition.

Conflicts: detect them, then decide who wins#

The prompt in build_prompt sends the doc type, effective date and status with every chunk, so a real model has what it needs to resolve disagreements. Relying on the model alone is risky, though, so conflicts() runs a cheap check first: it pulls (unit, number) pairs out of each chunk and flags two chunks from different documents that give different numbers for the same unit. The winner is the one with status: current, then the later effective date. A newer proposal never beats an older approved policy.

This heuristic is crude. It knows nothing about meaning, so "INR 9,000" for hotels and "INR 4,000" for meals would trip it if both landed in one context. Treat it as an alarm that adds a line to the prompt or the UI, not as the judge. The demo below shows it on three questions, naive retrieval first, then filtered:

Q: What is the API rate limit?
filters: [{'family': 'api', 'status': 'current'}]
-- naive:
The API allows 60 requests per minute on all plans. [faq > What is the API rate limit?]
Each API key can make 60 requests per minute. Requests above the limit get HTTP 429. [api-v1 > Rate limits]
Each client can make 300 requests per minute. Requests above the limit get HTTP 429 with a Retry-After header. [api-v2 > Rate limits]
CONFLICT: faq and api-v2 disagree; trusting api-v2.
CONFLICT: api-v1 and api-v2 disagree; trusting api-v2.
-- filtered:
Each client can make 300 requests per minute. Requests above the limit get HTTP 429 with a Retry-After header. [api-v2 > Rate limits]
The maximum request body is 10 MB. [api-v2 > Payload size]
Use OAuth 2.0 client credentials. Send the bearer token in the Authorization header. [api-v2 > Authentication]

Q: What is the meal per diem in Bangalore?
filters: [{'family': 'travel-policy', 'status': 'current'}]
-- naive:
The daily allowance for meals in Bangalore is INR 3,500. For other metro cities it is INR 3,000. [travel-2025 > Per diem]
Attendees discussed raising the Bangalore meal per diem to INR 4,000 from April 2026. This is a proposal only and is not yet approved policy. [ops-notes > para 1]
The hotel cap is INR 9,000 per night in metro cities, including taxes. [travel-2025 > Hotels]
CONFLICT: travel-2025 and ops-notes disagree; trusting travel-2025.
CONFLICT: ops-notes and travel-2025 disagree; trusting travel-2025.

Naive retrieval ranks the stale FAQ first for the rate-limit question. A model reading the context top-down has good odds of answering "60". With the filter, the FAQ and v1 never reach the prompt. On the per diem question, the ops-notes match on "Bangalore meal per diem" almost word for word, and the conflict line catches it. The second flag on that question is the hotel-versus-proposal false positive described above.

Filtering and conflict detection do different jobs. The filter stops known-stale documents from entering the context at all. The conflict check catches what the catalog does not know about, such as two current documents that disagree. When it fires on two current documents, the real fix is usually upstream: one of them should be marked superseded.

Comparison questions#

"How does API v2 differ from v1 on rate limits?" routes to two filters, {family: api, version: v2} and {family: api, version: v1}, and each side gets its own slot. Here is the real qwen3:4b-instruct answer with k=4 (temperature 0, about 3.5 seconds end to end on an Apple M5 laptop):

API v2 differs from v1 on rate limits in that API v2 imposes a limit of 300 requests per minute per client, whereas API v1 imposes a limit of 60 requests per minute per API key. [api-v2 > Rate limits] [api-v1 > Rate limits]

It picked up the difference in the unit of the limit (per client vs per API key), not only the number, and it cited both sections. For longer comparisons (two 40-page contracts), retrieve the top few chunks per side for each aspect of the question, then ask the model for a table with one row per aspect. Questions with no named aspect ("what changed between the 2024 and 2025 policy?") are better answered by diffing the two documents section by section at ingest time and storing the diff as its own chunk.

The eval: does metadata filtering earn its keep?#

The labelled set has 22 questions: 18 single-document questions (each with one gold document and an answer string) and 4 comparison questions (two gold documents and both answer strings). Some name a version; others ("How do I authenticate to the API?") leave it implicit, and the gold label is the current document. Every question is scored at k=3:

  • doc@1: the top chunk comes from the gold document (a merged chunk counts for every document in also_in)
  • ans@1: the top chunk contains the expected answer string
  • mixed: the top 3 hold a chunk from a non-gold document that conflicts with another hit
  • dupes: duplicate chunks in the top 3, summed across questions
  • compare: both answers appear somewhere in the retrieved context
# eval.py
import json
from ingest import dedup
from retrieve import conflicts, docs_of, retrieve

index = json.load(open("index.json"))
labels = json.load(open("labels.json"))

def score(chunks, use_filters, field, k=3):
    single = [l for l in labels if len(l["gold"]) == 1]
    compare = [l for l in labels if len(l["gold"]) > 1]
    hit1 = ans1 = mixed = dupes = 0
    for l in single:
        hits = retrieve(chunks, l["q"], k, use_filters, field)
        hit1 += l["gold"][0] in docs_of(hits[0])
        ans1 += l["answer"] in hits[0]["text"]
        # a non-gold document in the context disagrees with another hit
        mixed += any(d not in l["gold"] for pair in conflicts(hits) for d in pair[:2])
        dupes += len(hits) - len({h["text"] for h in hits})
    both = 0
    for l in compare:
        text = " ".join(h["text"] for h in retrieve(chunks, l["q"], k, use_filters, field))
        both += all(a in text for a in l["answer"])
    n = len(single)
    return [f"{hit1}/{n}", f"{ans1}/{n}", f"{mixed}/{n}", str(dupes), f"{both}/{len(compare)}"]

rows = [("naive, chunk text only", index, False, "vec_body"),
        ("naive, title+section header", index, False, "vec_headed"),
        ("header + metadata filter", index, True, "vec_headed"),
        ("header + filter + dedup", dedup(index), True, "vec_headed"),
        ("same, cosine-only dedup", dedup(index, check_numbers=False), True, "vec_headed")]
cols = ["doc@1", "ans@1", "mixed", "dupes", "compare"]
print(f"{'variant':30}" + "".join(f"{c:>9}" for c in cols))
for name, chunks, filt, field in rows:
    print(f"{name:30}" + "".join(f"{v:>9}" for v in score(chunks, filt, field)))

labels.json is a list of entries like these:

{"q": "Under the 2024 leave policy, how many days can be carried forward?", "gold": ["leave-2024"], "answer": "8 days"},
{"q": "How do I authenticate to the API?", "gold": ["api-v2"], "answer": "OAuth 2.0"},
{"q": "How does API v2 differ from v1 on rate limits?", "gold": ["api-v2", "api-v1"], "answer": ["300 requests", "60 requests"]}

The output:

variant                           doc@1    ans@1    mixed    dupes  compare
naive, chunk text only             9/18    10/18    12/18        1      3/4
naive, title+section header       17/18    17/18    12/18        1      4/4
header + metadata filter          18/18    18/18     0/18        0      4/4
header + filter + dedup           18/18    18/18     0/18        0      4/4
same, cosine-only dedup           18/18    16/18     0/18        0      2/4

What the rows say:

  1. Embedding the chunk text alone gets half the questions wrong at rank 1. With near-identical sections, the year or version lives only in the document title, which the chunk text does not contain. The ranking between "2024" and "2025" chunks is close to a coin flip.
  2. Adding the title and section to the embedded text fixes most of rank 1 (9 to 17). That is the cheapest change in the whole post, and the one to make first. The remaining miss is "What is the API rate limit?", where the FAQ's heading is almost the question itself.
  3. Ranking is not the whole problem. Even with headers, 12 of 18 contexts still contain a conflicting chunk from another document. The model sees the right answer at rank 1 and a wrong one at rank 2 or 3. Metadata filtering takes that to 0 of 18.
  4. Dedup removed one duplicate slot and changed nothing else here. On a corpus with many copied sections (templates, boilerplate, mirrored wiki pages) duplicates eat a larger share of k, so expect a bigger effect there.
  5. Cosine-only dedup looks perfect on doc@1 and is wrong. doc@1 stays at 18/18 because the merged chunk still lists the 2024 document, but ans@1 drops to 16/18 and comparisons to 2/4: the 2024 carry-forward and parental-leave numbers were deleted. A metric that checks only "right document" would never catch this. Always score the answer, too.

Eighteen questions over eight hand-built documents is a sanity check, not a benchmark. The corpus was built to have conflicts, so the gaps are larger than on a typical wiki. The method carries over: write 30 to 50 questions from your own corpus, label the gold document and answer string, and include unqualified questions whose correct answer depends on which version is current. For grading generated answers rather than retrieved chunks, the agentic RAG build has an eval loop you can reuse.

Failure modes to plan for#

  • Missing metadata. A document with no status gets treated as current forever. Make status and effective required at ingest and reject files without them.
  • The router guesses wrong. "What did the old leave policy say?" names no year. Add words such as "old", "previous" and "last year" to the router, or send these to the LLM router.
  • Filters too tight. A filter that returns zero chunks should fall back to an unfiltered search with a warning, not to an empty context.
  • Tables lose their headers. A CSV row embedded without column names is just numbers. The column = value format keeps it searchable; for wide tables, a SQL tool beats embedding rows.
  • Supersession is partial. v2 replaces v1's rate limits but may not mention v1's error codes at all. Marking all of v1 superseded hides facts that are still true. Section-level status, or a "still applies" list, handles this better than document-level status.

Next steps#

Swap the keyword router for a JSON-returning qwen3:4b-instruct call validated against the catalog, move the index into Chroma with where filters, and add a reranker over the filtered pool. Keep running eval.py after each change, and add every question that a user got a wrong answer to into labels.json.

Sources#