GyaanStack. Blog

Articles / ragprojectspython

Build a PDF question-answering bot that cites its sources (local, no API key)

A 70-line Python bot that answers questions from a PDF, quotes the exact sentence with its page number, and refuses when the answer isn't there.

By Raghunath · · 16 min read

A typical "chat with your PDF" tutorial starts with a vector database, an embedding model and an API key. That's three moving parts before you have answered a single question, and when the answer comes back wrong you can't tell which part broke.

This project goes the other way. You build the retrieval half of RAG first, with no model at all, so you can see exactly which passage comes back and why. The bot reads a PDF, finds the most relevant passage, quotes the sentence that answers the question, and tells you the page. When the document doesn't contain the answer, it says so. A language model comes in at the end as an optional step.

Everything below ran on my laptop. I tested with Python 3.15.2, pypdf 7.1.1, rank-bm25 0.2.2 and fpdf2 2.9.0, plus Ollama with gemma3:12b for the optional LLM step.

What you'll build#

  • A script that splits any PDF into small chunks and keeps the page number and section heading on every chunk
  • BM25 keyword search over those chunks
  • Answers that quote one sentence and cite [page N, Section]
  • A refusal when nothing in the document matches
  • An eight-question test file, so you measure the bot instead of eyeballing it
  • An optional step that hands the retrieved passages to a local LLM and asks it to cite them

Set up#

python -m venv venv
source venv/bin/activate
pip install pypdf rank-bm25 fpdf2

pypdf reads the PDF, rank-bm25 does the search, and fpdf2 is only there to make a sample PDF.

Make a test PDF you know by heart#

To catch a wrong answer, you need to know the right one. So the first file creates a small three-page "employee handbook" with facts that are easy to check: 24 days of leave, a 6,000 rupee hotel limit, and so on.

# make_sample_pdf.py
from fpdf import FPDF

PAGES = [
    ("Leave policy",
     "Every full-time employee gets 24 days of paid leave per calendar year. "
     "Leave is credited at 2 days per month on the first working day of the month. "
     "Up to 10 unused days can be carried over to the next year. Days above 10 lapse on 31 December. "
     "Sick leave is separate: you get 12 days of sick leave a year, and a doctor's note is needed for more than 2 days in a row."),
    ("Working from home",
     "You can work from home up to 3 days a week after your manager approves it in the HR portal. "
     "New joiners work from the office for their first 30 days. "
     "The company pays an internet allowance of 1,500 rupees a month to people on a hybrid schedule."),
    ("Expenses and travel",
     "Submit expense claims within 45 days of the expense, with a scanned receipt. "
     "Claims above 25,000 rupees need approval from a director. "
     "For work travel, book economy flights at least 14 days in advance. "
     "Hotel stays are reimbursed up to 6,000 rupees a night in metro cities."),
]

pdf = FPDF()
for title, body in PAGES:
    pdf.add_page()
    pdf.set_font("Helvetica", "B", 16)
    pdf.cell(0, 10, title, new_x="LMARGIN", new_y="NEXT")
    pdf.set_font("Helvetica", size=12)
    pdf.multi_cell(0, 7, body)
pdf.output("handbook.pdf")
print("wrote handbook.pdf with", len(PAGES), "pages")
$ python make_sample_pdf.py
wrote handbook.pdf with 3 pages

Once the bot works on this file, point it at a real document you know well, such as your college notes or a policy PDF from work.

The bot#

Here is the whole thing. I'll go through the three decisions that matter after the code.

# pdf_qa.py
import re
import sys

from pypdf import PdfReader
from rank_bm25 import BM25Okapi

STOPWORDS = set("a an the is are was were be to of in on for and or how many much what when who do does i my can".split())


def tokenize(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOPWORDS]


def load_chunks(path, sentences_per_chunk=3, overlap=1):
    """Split each page into small overlapping groups of sentences. Every chunk keeps its page number."""
    chunks = []
    for page_no, page in enumerate(PdfReader(path).pages, start=1):
        lines = (page.extract_text() or "").strip().splitlines()
        # A short first line with no full stop is the page heading. Keep it apart from the body.
        heading = lines.pop(0).strip() if lines and len(lines[0]) < 60 and not lines[0].rstrip().endswith(".") else ""
        text = re.sub(r"\s+", " ", " ".join(lines)).strip()
        sentences = re.split(r"(?<=[.!?])\s+", text)
        step = sentences_per_chunk - overlap
        for i in range(0, max(1, len(sentences) - overlap), step):
            piece = " ".join(sentences[i:i + sentences_per_chunk])
            if piece:
                chunks.append({"page": page_no, "section": heading, "text": piece})
    return chunks


class PdfQA:
    def __init__(self, path, min_score=0.5):
        self.chunks = load_chunks(path)
        self.bm25 = BM25Okapi([tokenize(c["text"]) for c in self.chunks])
        self.min_score = min_score

    def retrieve(self, question, k=3):
        scores = self.bm25.get_scores(tokenize(question))
        ranked = sorted(zip(scores, self.chunks), key=lambda x: x[0], reverse=True)
        return [(round(float(s), 2), c) for s, c in ranked[:k] if s >= self.min_score]

    def best_sentence(self, question, chunk_text):
        """Quote the one sentence in the chunk that shares the most words with the question."""
        q = set(tokenize(question))
        sentences = re.split(r"(?<=[.!?])\s+", chunk_text)
        return max(sentences, key=lambda s: len(q & set(tokenize(s))))

    def answer(self, question):
        hits = self.retrieve(question)
        if not hits:
            return "I couldn't find this in the document.", []
        score, top = hits[0]
        quote = self.best_sentence(question, top["text"])
        return f'"{quote}" [page {top["page"]}, {top["section"]}]', hits


if __name__ == "__main__":
    qa = PdfQA(sys.argv[1])
    print(f"Indexed {len(qa.chunks)} chunks\n")
    for question in sys.argv[2:]:
        reply, hits = qa.answer(question)
        print("Q:", question)
        print("A:", reply)
        for score, c in hits:
            print(f"   source p.{c['page']} score={score}: {c['text'][:70]}...")
        print()

Decision 1: chunk by sentences, per page#

Many tutorials join every page into one long string and cut it every 800 characters. That loses the page number, and page numbers are what make a citation useful. Here each page is chunked on its own, so every chunk knows where it came from.

Chunks are three sentences with one sentence of overlap. The overlap means a fact that sits on a chunk boundary still lands whole in at least one chunk. Three sentences is small on purpose. For a handbook, one fact usually fits in one or two sentences, and a small chunk means the quote you show is close to the answer.

Pulling out the heading also helps. pypdf returns the page title as its own first line. Without the check, "Leave policy" gets glued onto the first sentence, and your quote starts with "Leave policy Every full-time employee...". Pulling the heading out gives you a cleaner quote and a section name for the citation.

Decision 2: BM25 before embeddings#

BM25 is the ranking formula search engines used for decades before vectors. It scores a chunk higher when it contains the question's words, gives more weight to rare words, and stops rewarding a word after it repeats a few times. It needs no model download, runs instantly, and you can explain every score.

For a first version, that matters more than accuracy. When a result is wrong, you can look at the tokens and see why.

Decision 3: a refusal threshold#

min_score is the line between "here's what I found" and "I couldn't find this". Without it, the bot always returns its top chunk, even for a question the document never mentions. That is the retrieval version of a hallucination.

Run it#

$ python pdf_qa.py handbook.pdf "How many days of paid leave do I get?" "Can I carry over unused leave?" "What is the hotel limit in metro cities?" "How many days can I work from home?" "What is the dress code?"
Indexed 5 chunks

Q: How many days of paid leave do I get?
A: "Every full-time employee gets 24 days of paid leave per calendar year." [page 1, Leave policy]
   source p.1 score=1.89: Every full-time employee gets 24 days of paid leave per calendar year....
   source p.1 score=1.87: Up to 10 unused days can be carried over to the next year. Days above ...

Q: Can I carry over unused leave?
A: "Up to 10 unused days can be carried over to the next year." [page 1, Leave policy]
   source p.1 score=1.13: Every full-time employee gets 24 days of paid leave per calendar year....
   source p.1 score=1.1: Up to 10 unused days can be carried over to the next year. Days above ...

Q: What is the hotel limit in metro cities?
A: "Hotel stays are reimbursed up to 6,000 rupees a night in metro cities." [page 3, Expenses and travel]
   source p.3 score=3.86: For work travel, book economy flights at least 14 days in advance. Hot...

Q: How many days can I work from home?
A: "You can work from home up to 3 days a week after your manager approves it in the HR portal." [page 2, Working from home]
   source p.2 score=2.03: You can work from home up to 3 days a week after your manager approves...

Q: What is the dress code?
A: I couldn't find this in the document.

The carry-over question is a good one to study. The top-ranked chunk is the first one on page 1, but the quote is still the right sentence, because the overlap put the carry-over sentence in both chunks and best_sentence picked it out.

Test it, and watch the first version fail#

A demo with five hand-picked questions proves little. Write the questions down with the page the answer is on and a word the answer must contain, and include questions the document can't answer.

# test_qa.py
from pdf_qa import PdfQA

# (question, page the answer is on, a word the quote must contain). None = should refuse.
TESTS = [
    ("How many days of paid leave do I get?", 1, "24"),
    ("How much sick leave is there?", 1, "12"),
    ("Do new joiners have to come to the office?", 2, "30"),
    ("What internet allowance do hybrid staff get?", 2, "1,500"),
    ("How soon must I submit an expense claim?", 3, "45"),
    ("How early should I book flights?", 3, "14"),
    ("What is the dress code?", None, None),
    ("Is there a gym membership?", None, None),
]

qa = PdfQA("handbook.pdf")
passed = 0
for question, page, must_have in TESTS:
    reply, hits = qa.answer(question)
    if page is None:
        ok = not hits
    else:
        ok = bool(hits) and hits[0][1]["page"] == page and must_have in reply
    passed += ok
    print("PASS" if ok else "FAIL", "|", question, "->", reply[:80])
print(f"\n{passed}/{len(TESTS)} passed")

My first version used min_score=1.0. This was the result:

FAIL | How early should I book flights? -> I couldn't find this in the document.
...
7/8 passed

Printing the raw scores showed why. The flights chunk scored 0.79. Only two words, "book" and "flights", matched the question, so it fell under a threshold I had picked by guessing. The two "should refuse" questions scored exactly 0 on every chunk. Dropping the threshold to 0.5 fixed the flights question and kept both refusals:

PASS | How many days of paid leave do I get? -> "Every full-time employee gets 24 days of paid leave per calendar year." [page 1
PASS | How much sick leave is there? -> "Sick leave is separate: you get 12 days of sick leave a year, and a doctor's no
PASS | Do new joiners have to come to the office? -> "New joiners work from the office for their first 30 days." [page 2, Working fro
PASS | What internet allowance do hybrid staff get? -> "The company pays an internet allowance of 1,500 rupees a month to people on a h
PASS | How soon must I submit an expense claim? -> "Submit expense claims within 45 days of the expense, with a scanned receipt." [
PASS | How early should I book flights? -> "For work travel, book economy flights at least 14 days in advance." [page 3, Ex
PASS | What is the dress code? -> I couldn't find this in the document.
PASS | Is there a gym membership? -> I couldn't find this in the document.

8/8 passed

BM25 scores depend on how many chunks you have and how long the question is, so no single threshold is right for every PDF. Tune it against a test file like this one whenever you change documents.

Where keyword search breaks#

Ask about "vacation" and the bot gets it wrong:

Q: How many vacation days do I get?
A: "Sick leave is separate: you get 12 days of sick leave a year, and a doctor's note is needed for more than 2 days in a row." [page 1, Leave policy]

The handbook says "paid leave" and never uses the word "vacation". Only "days" matched, so BM25 picked a chunk that happened to have "days" in it. This is the main weakness of keyword search, and it's the reason people move to embeddings.

A cheap fix for a small, known domain is a synonym list that adds the document's words to the query:

SYNONYMS = {"vacation": ["paid", "leave"], "holidays": ["paid", "leave"],
            "remote": ["work", "home"], "wfh": ["work", "home"]}


def tokenize(text):
    words = [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOPWORDS]
    return words + [s for w in words for s in SYNONYMS.get(w, [])]
Q: How many vacation days do I get?
A: "Every full-time employee gets 24 days of paid leave per calendar year." [page 1, Leave policy]

Q: How many remote days are allowed?
A: "You can work from home up to 3 days a week after your manager approves it in the HR portal." [page 2, Working from home]

The test file still passed 8/8 after this change. Synonym lists don't scale past one domain, though. For general documents, add an embedding model such as sentence-transformers and combine its scores with BM25 (hybrid search). Now that you have a test file, you can measure whether the upgrade helps.

Optional: let a local LLM write the answer#

Quoting one sentence is accurate but stiff, and it can't combine two facts. For that, give the top three chunks to a model, number them, and require a citation after every fact. This version calls a local Ollama server with only the standard library:

# llm_answer.py
import json
import sys
import urllib.request

from pdf_qa import PdfQA

PROMPT = """Answer the question using ONLY the numbered sources below.
After each fact, cite the source like [1]. If the sources don't contain the answer, say "Not in the document."

{sources}

Question: {question}
Answer:"""


def ask_ollama(prompt, model="llama3.2"):
    body = json.dumps({"model": model, "prompt": prompt, "stream": False,
                       "options": {"temperature": 0}}).encode()
    req = urllib.request.Request("http://localhost:11434/api/generate", data=body,
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.loads(r.read())["response"].strip()


def llm_answer(qa, question, model="llama3.2"):
    hits = qa.retrieve(question, k=3)
    if not hits:
        return "I couldn't find this in the document."  # don't call the model with nothing to go on
    sources = "\n".join(f"[{i}] (page {c['page']}) {c['text']}" for i, (_, c) in enumerate(hits, start=1))
    reply = ask_ollama(PROMPT.format(sources=sources, question=question), model)
    pages = ", ".join(f"[{i}] = page {c['page']}" for i, (_, c) in enumerate(hits, start=1))
    return f"{reply}\n   ({pages})"


if __name__ == "__main__":
    qa = PdfQA(sys.argv[1])
    model = sys.argv[2]
    for q in sys.argv[3:]:
        print("Q:", q)
        print("A:", llm_answer(qa, q, model))
        print()

Install Ollama from ollama.com and pull any small model (ollama pull llama3.2). I ran it with the gemma3:12b model I already had:

$ python llm_answer.py handbook.pdf gemma3:12b "Can I carry over unused leave, and what happens to the rest?" "Do I need approval for a 30,000 rupee claim?" "What is the dress code?"
Q: Can I carry over unused leave, and what happens to the rest?
A: Yes, you can carry over unused leave [1]. Up to 10 unused days can be carried over to the next year [1, 2]. Days above 10 lapse on December 31st [2].
   ([1] = page 1, [2] = page 1)

Q: Do I need approval for a 30,000 rupee claim?
A: Yes, claims above 25,000 rupees need approval from a director [1].
   ([1] = page 3, [2] = page 2)

Q: What is the dress code?
A: I couldn't find this in the document.

The 30,000 rupee question needs a small inference, because the handbook only says "above 25,000", and the model handled it. The dress code question never reached the model at all. Retrieval refused first, which is cheaper and safer than trusting a prompt to say "I don't know". To use a hosted model instead, replace ask_ollama with a call to your provider's SDK and keep the same prompt.

Make it portfolio-ready#

The working code is the smaller part of what an interviewer looks at. These four additions show them more:

  1. Run it on a real 50-page PDF and write 20 test questions, including five the document can't answer. Report the pass rate in your README.
  2. Add embeddings and show the before/after. Hybrid search should fix the "vacation" class of failures. Your test file proves whether it did.
  3. Check the LLM's citations in code. Parse the [n] markers and flag any answer that cites a source number that wasn't retrieved, or has a sentence with no citation.
  4. Handle scanned PDFs. extract_text() returns an empty string for image-only pages. Detect that and either skip the page with a warning or run OCR on it.

Copy the four files into a folder, run make_sample_pdf.py, then test_qa.py, and you should see the same 8/8.