GyaanStack. Blog

Articles / ragmultimodalpython

Multimodal RAG in Python: tables, charts and images in PDFs, with page-region citations

RAG over PDFs with tables, charts and images. Caption-then-embed vs CLIP vs ColPali, table chunking, page-region citations and a runnable pipeline.

By Raghunath · · 21 min read

Ask a text-only RAG system "which month had the most support tickets?" about a report where that answer lives in a bar chart, and it will retrieve the paragraph that says "the chart below shows ticket volume". The words match. The answer is not there. The text layer of a PDF holds no pixels, so the August spike never reached the index.

Tables fail more quietly. A PDF text extractor reads a table cell by cell, so the row for APAC comes out as APAC 3.1 4.2 35%. The column names sit 40 points higher on the page, in a different chunk. The retriever may find the row, but the model then has to guess whether 4.2 is Q2 revenue, Q3 revenue or something else.

This post covers the three ways to get tables, charts and images into retrieval, what each one costs, how to chunk tables so a row means something on its own, and how to cite a box on a page instead of a whole page. It ends with a pipeline of about 180 lines that runs with no API key, plus a small eval that compares it with text-only indexing on the same PDF.

Two related posts cover ground this one skips. Design a RAG ingestion pipeline at scale covers connectors, per-page parser routing, re-indexing and embedding cost math. Build a PDF question-answering bot that cites its sources covers the text-only bot with sentence-level citations and refusals. Here the focus is only on what changes when the answer is in a table or a picture.

Three ways to index what is not text#

                 PDF page
                    |
     +--------------+------------------+
     |              |                  |
 A. caption      B. shared          C. page image
  then embed      image-text          retrieval
     |            embedding             |
 crop figure      crop figure       render whole page
     |              |                  |
 vision model    CLIP-style         ColPali-style model
 writes text     image encoder      ~1,000 patch vectors
     |              |                  |
 text embedder   one vector          multi-vector index
     |              |                  |
     +------- query: text embedding ---+
                    |
           top-k chunks or pages
                    |
     answer model (text, or vision if images are passed)

A. Caption, then embed. Crop each figure, send it to a vision-language model with a prompt that asks for chart type, axes, labels and extreme values, and index the caption as ordinary text. The rest of the pipeline stays text-only, so your existing embedder, BM25 index and reranker all keep working. The cost is at ingestion: one vision call per figure. The caption is also lossy. If the prompt does not ask for the September value, the September value is not searchable.

B. One embedding space for images and text. CLIP trains an image encoder and a text encoder so that matching pairs land close together. It learned from 400 million image-caption pairs from the web. You embed the figure crop directly and search it with a text query. This works for photos and product images ("red valve with a hex nut"). It works poorly for charts and dense diagrams, because web captions rarely say "August is the peak at 930". The text side is short too: the openai/clip-vit-base-patch32 config caps text at 77 positions, so long questions get truncated. You also get two vector spaces to merge, because your prose chunks still need a text embedder.

C. Retrieve page images. ColPali (Faysse et al., 2024) skips parsing. It renders each page as an image and a vision-language model turns it into 1,024 patch embeddings, projected to 128 dimensions. At query time each query token is matched against every patch, ColBERT style (late interaction), and the scores are summed. The paper also introduces the ViDoRe benchmark, where ColPali beats parse-OCR-caption pipelines on page retrieval. Models and code are at illuin-tech/colpali, with later versions on Hugging Face such as vidore/colpali-v1.3 and vidore/colqwen2.5-v0.2.

The ColPali paper reports numbers that make the trade-off concrete. Its Table 5 lists per-page indexing latency on the authors' setup:

Indexing step Parse + OCR + caption pipeline ColPali
Layout detection 0.81 s n/a
OCR 2.67 s n/a
Captioning 3.71 s n/a
Page encoding 0.03 s 0.39 s
Total per page 7.22 s 0.39 s

The cost moves to storage. The paper gives 257.5 KB per page for the multi-vector index. For 100,000 pages that is about 25.75 GB of vectors. A single-vector text index with bge-small-en-v1.5 (384 dimensions, float32, 1,536 bytes per vector) and five chunks per page comes to about 0.77 GB. Captioning those same 100,000 pages at 3.71 s each is about 103 hours of serial vision-model time, which is why caption pipelines run in parallel and cache by image hash.

Page retrieval also returns a page, not an answer. You still need a vision model to read the retrieved page images at answer time, and each page image costs far more tokens than a 300-token text chunk.

Which one to pick#

You have Start with Why
Reports with a few charts and many tables A, plus table-aware chunking Keeps one text index, hybrid search and rerankers
Slide decks, scanned forms, infographics C Layout carries the meaning; parsing loses it
Product photos, diagrams with little text B Visual similarity is the question
Mixed corpus, large scale A for born-digital pages, C for visually dense ones Route per page, as in the ingestion post

My default is A. It is the cheapest to debug, because every chunk is text a human can read, and a bad caption is visible in the index. C is the one to test when your eval shows that A misses pages where layout matters, such as slides where position on the page says which number belongs to which label.

Chunking tables so each row stands alone#

Tables need two kinds of chunk.

  1. One chunk per row, with the header repeated and the table title prefixed: Table 1: Revenue by region (USD millions) | Region: APAC | Q2 revenue: 3.1 | Q3 revenue: 4.2 | Growth: 35%. A lookup question ("APAC revenue in Q3") hits one row, and that row is unambiguous without its neighbours.
  2. One chunk for the whole table, as Markdown, for questions that compare rows: "which region grew fastest", "total revenue". No single row answers those.

For long tables, group 10 to 20 rows per chunk and repeat the header in each group. For tables that span pages, carry the header from the first page into the rows on the next one, because the continuation has no header row of its own. Keep the units. "USD millions" belongs in every row chunk, or someone will ask "what was APAC revenue" and get "4.2" with no scale.

PyMuPDF's Page.find_tables() detects tables from vector lines by default (strategy="lines"). Borderless tables need strategy="text", and scanned tables need OCR or a layout model before any of this works.

Citing a region, not a page#

"Source: page 14" is weak when page 14 has three tables. Each chunk in the code below carries the bounding box it came from, in PDF points. That gives you two things:

  • A citation string like [p2 figure (50,130,470,340)] that the answer model copies after each claim.
  • A cropped PNG of exactly that region, rendered with page.get_pixmap(clip=rect), that the UI shows next to the answer. A reader can check the claim against the original pixels in one glance.

For text chunks, the box comes from page.get_text("blocks"). For table rows, from each row's cell box. For figures, from page.get_image_rects(xref), which gives where the image is drawn on the page. Extracting images by xref alone, as many tutorials do, saves the image bytes but throws away the position, and with it any way to cite the figure.

A runnable pipeline#

The pipeline uses PyMuPDF and NumPy only:

python3 -m venv .venv && source .venv/bin/activate
pip install pymupdf pillow numpy

I tested it with PyMuPDF 1.28.2 and NumPy 2.5.5 on Python 3.15. Two parts are stubs so it runs with no key and no download, and the docstrings show where real models plug in:

  • StubCaptioner returns a fixed caption for the test chart. Replace its body with a call to any vision-language model, sending the image bytes and CAPTION_PROMPT.
  • HashEmbedder is a hashed bag of words. It is lexical, so it does not know "grew" and "growth" are related. Replace it with bge-small-en-v1.5 through sentence-transformers.

First, a test PDF you know by heart. Page 1 has prose and a ruled table. Page 2 has a bar chart that exists only as pixels, with the August spike, and an incident note that explains it.

"""Build a 2-page test PDF: prose, a real table, and a bar chart that is only an image."""
import io
import pymupdf
from PIL import Image, ImageDraw

MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
TICKETS = [310, 340, 325, 360, 390, 410, 420, 930, 400, 380, 350, 330]


def chart_png() -> bytes:
    img = Image.new("RGB", (600, 300), "white")
    d = ImageDraw.Draw(img)
    for i, t in enumerate(TICKETS):
        x = 30 + i * 46
        h = int(t / 1000 * 250)
        d.rectangle([x, 270 - h, x + 30, 270], fill=(40, 90, 160))
        d.text((x + 4, 276), MONTHS[i], fill="black")
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    return buf.getvalue()


def build(path: str) -> None:
    doc = pymupdf.open()
    p1 = doc.new_page()
    p1.insert_text((50, 60), "Q3 Operations Report", fontsize=18)
    p1.insert_textbox(pymupdf.Rect(50, 80, 545, 160),
        "Revenue grew in every region this quarter. The board asked for a "
        "per-region breakdown, shown in Table 1 below. Figures are in USD millions.",
        fontsize=11)
    p1.insert_text((50, 185), "Table 1: Revenue by region (USD millions)", fontsize=11)
    rows = [["Region", "Q2 revenue", "Q3 revenue", "Growth"],
            ["India", "2.8", "3.4", "21%"],
            ["APAC", "3.1", "4.2", "35%"],
            ["EMEA", "5.0", "5.3", "6%"],
            ["Americas", "6.2", "6.6", "6%"]]
    x0, y0, cw, rh = 50, 195, 120, 22
    for r, row in enumerate(rows):
        for c, cell in enumerate(row):
            rect = pymupdf.Rect(x0 + c * cw, y0 + r * rh, x0 + (c + 1) * cw, y0 + (r + 1) * rh)
            p1.draw_rect(rect, color=(0, 0, 0), width=0.8)
            p1.insert_text((rect.x0 + 5, rect.y1 - 7), cell, fontsize=10)

    p2 = doc.new_page()
    p2.insert_textbox(pymupdf.Rect(50, 50, 545, 120),
        "Support load is tracked monthly. The chart below shows ticket volume "
        "for the year. The spike is explained in the incident review.",
        fontsize=11)
    p2.insert_image(pymupdf.Rect(50, 130, 470, 340), stream=chart_png())
    p2.insert_text((50, 360), "Figure 1: Support tickets per month", fontsize=10)
    p2.insert_textbox(pymupdf.Rect(50, 380, 545, 460),
        "Incident review: a billing migration failed for two days, which caused "
        "duplicate invoices and a wave of customer complaints.",
        fontsize=11)
    doc.save(path)


if __name__ == "__main__":
    build("report.pdf")
    print("wrote report.pdf")

Then the pipeline. extract() builds four kinds of chunk per page, each with a page number and a box. When multimodal=False it indexes only the text layer, which is the baseline for the eval.

"""Multimodal RAG over one PDF: text blocks, table rows and image captions, with page-region citations.

The vision model and the embedding model are stubs behind small interfaces, so this runs with no
API key and no model download. Swap them for real ones where the comments say so.
"""
import hashlib
import re
import sys
from dataclasses import dataclass
from typing import Protocol

import numpy as np
import pymupdf


@dataclass
class Chunk:
    id: str
    page: int                 # 1-based
    bbox: tuple               # (x0, y0, x1, y1) in PDF points
    kind: str                 # "text" | "table" | "table_row" | "figure"
    text: str                 # what gets embedded
    image_sha: str = ""       # figures only: which image the caption came from


# ---------- model interfaces ----------
class Captioner(Protocol):
    def caption(self, image_bytes: bytes, nearby_text: str) -> str: ...


class Embedder(Protocol):
    def embed(self, texts: list[str]) -> np.ndarray: ...


class StubCaptioner:
    """Returns a canned caption for the test chart, so the demo needs no model.

    Real version: send image_bytes plus nearby_text to a vision model (Claude, GPT, Gemini,
    or a local Qwen2.5-VL) with a prompt like CAPTION_PROMPT and return its reply.
    """
    CANNED = {
        "chart": "Bar chart of support tickets per month, Jan to Dec. Most months are "
                 "between 310 and 420 tickets. August is the peak at about 930 tickets.",
    }

    def caption(self, image_bytes: bytes, nearby_text: str) -> str:
        return self.CANNED["chart"] if "Figure 1" in nearby_text else "Image with no caption."


CAPTION_PROMPT = (
    "Describe this figure for a search index. Name the chart type, the axes, every labelled "
    "series, and the largest and smallest values with their labels. Copy numbers exactly. "
    "Context from the page: {nearby_text}"
)


class HashEmbedder:
    """Feature-hashed bag of words and bigrams, L2-normalised. Lexical, not semantic.

    Real version: SentenceTransformer("BAAI/bge-small-en-v1.5").encode(texts, normalize_embeddings=True)
    """
    def __init__(self, dim: int = 2048):
        self.dim = dim

    def _tokens(self, text: str) -> list[str]:
        words = re.findall(r"[a-z0-9.%]+", text.lower())
        words = [w.rstrip(".") for w in words if w not in STOP]
        return words + [a + "_" + b for a, b in zip(words, words[1:])]

    def embed(self, texts: list[str]) -> np.ndarray:
        out = np.zeros((len(texts), self.dim), dtype=np.float32)
        for i, t in enumerate(texts):
            for tok in self._tokens(t):
                h = int(hashlib.md5(tok.encode()).hexdigest(), 16)
                out[i, h % self.dim] += 1.0
        norms = np.linalg.norm(out, axis=1, keepdims=True)
        return out / np.maximum(norms, 1e-9)


STOP = set("a an the of in on for to and is was what which had by per this that below with".split())


# ---------- extraction ----------
def overlaps(a, b) -> bool:
    return not (a[2] <= b[0] or b[2] <= a[0] or a[3] <= b[1] or b[3] <= a[1])


def extract(pdf_path: str, captioner: Captioner, multimodal: bool = True) -> list[Chunk]:
    doc = pymupdf.open(pdf_path)
    chunks: list[Chunk] = []
    for page in doc:
        pno = page.number + 1
        tables = page.find_tables().tables if multimodal else []
        table_boxes = [tuple(t.bbox) for t in tables]

        # 1. Prose blocks, skipping anything inside a detected table.
        for x0, y0, x1, y1, text, bno, btype in page.get_text("blocks"):
            if btype != 0 or any(overlaps((x0, y0, x1, y1), tb) for tb in table_boxes):
                continue
            chunks.append(Chunk(f"p{pno}-b{bno}", pno, (x0, y0, x1, y1), "text",
                                " ".join(text.split())))

        # 2. Tables: one chunk per row, header repeated, title prefixed.
        for ti, t in enumerate(tables):
            rows = t.extract()
            header, body = rows[0], rows[1:]
            title = nearest_text_above(page, t.bbox)
            # Whole table too, for questions that compare rows ("which region grew fastest?").
            chunks.append(Chunk(f"p{pno}-t{ti}", pno, tuple(t.bbox), "table",
                                f"{title}\n{t.to_markdown()}"))
            for ri, (row, cells) in enumerate(zip(body, t.rows[1:])):
                pairs = " | ".join(f"{h}: {v}" for h, v in zip(header, row))
                chunks.append(Chunk(f"p{pno}-t{ti}-r{ri}", pno, tuple(cells.bbox), "table_row",
                                    f"{title} | {pairs}"))

        # 3. Images: caption with a vision model, embed the caption, keep the image box.
        if multimodal:
            for img in page.get_images(full=True):
                xref = img[0]
                data = doc.extract_image(xref)["image"]
                for rect in page.get_image_rects(xref):
                    below = pymupdf.Rect(rect.x0, rect.y1, rect.x1 + 100, rect.y1 + 30)
                    nearby = page.get_text("text", clip=below)
                    # The caption line now lives inside the figure chunk; drop its duplicate.
                    chunks = [c for c in chunks
                              if not (c.page == pno and c.kind == "text" and overlaps(c.bbox, tuple(below)))]
                    cap = captioner.caption(data, nearby)
                    chunks.append(Chunk(f"p{pno}-img{xref}", pno, tuple(rect), "figure",
                                        f"{' '.join(nearby.split())} | {cap}",
                                        hashlib.sha1(data).hexdigest()[:12]))
    return chunks


def nearest_text_above(page, bbox) -> str:
    best, best_gap = "", 1e9
    for x0, y0, x1, y1, text, *_ in page.get_text("blocks"):
        gap = bbox[1] - y1
        if 0 <= gap < best_gap:
            best, best_gap = " ".join(text.split()), gap
    return best


# ---------- index, retrieve, cite ----------
class Index:
    def __init__(self, chunks: list[Chunk], embedder: Embedder):
        self.chunks, self.embedder = chunks, embedder
        self.vectors = embedder.embed([c.text for c in chunks])

    def search(self, query: str, k: int = 3) -> list[tuple[float, Chunk]]:
        q = self.embedder.embed([query])[0]
        scores = self.vectors @ q
        order = np.argsort(-scores)[:k]
        return [(float(scores[i]), self.chunks[i]) for i in order]


def cite(c: Chunk) -> str:
    x0, y0, x1, y1 = (round(v) for v in c.bbox)
    return f"[p{c.page} {c.kind} ({x0},{y0},{x1},{y1})]"


def crop(pdf_path: str, c: Chunk, out: str, pad: float = 4) -> str:
    page = pymupdf.open(pdf_path)[c.page - 1]
    r = pymupdf.Rect(c.bbox) + (-pad, -pad, pad, pad)
    page.get_pixmap(clip=r, dpi=110).save(out)
    return out


def build_prompt(question: str, hits: list[tuple[float, Chunk]]) -> str:
    ctx = "\n".join(f"{cite(c)} {c.text}" for _, c in hits)
    return ("Answer from the context only. After each claim, copy the bracketed citation "
            "of the chunk it came from. If the context does not contain the answer, say so.\n\n"
            f"Context:\n{ctx}\n\nQuestion: {question}")


if __name__ == "__main__":
    pdf = sys.argv[1] if len(sys.argv) > 1 else "report.pdf"
    chunks = extract(pdf, StubCaptioner())
    print(f"{len(chunks)} chunks")
    for c in chunks:
        print(f"  {c.id:<12} {c.kind:<9} {c.text[:70].replace(chr(10), ' ')}")
    index = Index(chunks, HashEmbedder())
    q = "Which month had the most support tickets?"
    hits = index.search(q)
    print("\n" + build_prompt(q, hits))
    print("\ncropped:", crop(pdf, hits[0][1], "evidence.png"))

Running it:

python make_pdf.py && python mmrag.py
11 chunks
  p1-b0        text      Q3 Operations Report
  p1-b1        text      Revenue grew in every region this quarter. The board asked for a per-r
  p1-b2        text      Table 1: Revenue by region (USD millions)
  p1-t0        table     Table 1: Revenue by region (USD millions) |Region|Q2 revenue|Q3 revenu
  p1-t0-r0     table_row Table 1: Revenue by region (USD millions) | Region: India | Q2 revenue
  p1-t0-r1     table_row Table 1: Revenue by region (USD millions) | Region: APAC | Q2 revenue:
  p1-t0-r2     table_row Table 1: Revenue by region (USD millions) | Region: EMEA | Q2 revenue:
  p1-t0-r3     table_row Table 1: Revenue by region (USD millions) | Region: Americas | Q2 reve
  p2-b0        text      Support load is tracked monthly. The chart below shows ticket volume f
  p2-b2        text      Incident review: a billing migration failed for two days, which caused
  p2-img52     figure    Figure 1: Support tickets per month | Bar chart of support tickets per

Answer from the context only. After each claim, copy the bracketed citation of the chunk it came from. If the context does not contain the answer, say so.

Context:
[p2 figure (50,130,470,340)] Figure 1: Support tickets per month | Bar chart of support tickets per month, Jan to Dec. Most months are between 310 and 420 tickets. August is the peak at about 930 tickets.
[p1 text (50,173,258,188)] Table 1: Revenue by region (USD millions)
[p2 text (50,50,513,81)] Support load is tracked monthly. The chart below shows ticket volume for the year. The spike is explained in the incident review.

Question: Which month had the most support tickets?

cropped: evidence.png

The chart question lands on the figure chunk, with its box. evidence.png is the crop of that box: the bar chart itself, so the UI can show the August bar next to the answer. Without the caption, nothing in the index mentions August.

The prompt also shows a limit of caption-then-embed. The top hit holds the answer only because the caption prompt asked for the largest value. A question about the smallest month would fail with this caption, so the caption prompt should list everything you expect people to ask.

Measure it against text-only#

A pipeline change needs a number. The eval below asks six questions with known answers and counts a hit when every expected string sits in a single retrieved chunk. That rule is strict on purpose: APAC 3.1 4.2 35% without the words "Q3 revenue" does not count, because a model reading it would have to guess the column.

"""Compare text-only and multimodal indexing on questions with known answers."""
from mmrag import Index, HashEmbedder, StubCaptioner, extract

EVAL = [  # (question, strings that must all appear in one retrieved chunk)
    ("Which month had the most support tickets?", ["August"]),
    ("How many tickets were there at the peak?", ["930"]),
    ("What was APAC revenue in Q3?", ["APAC", "Q3 revenue: 4.2"]),
    ("Which region grew fastest?", ["35%", "Growth"]),
    ("What caused the spike in tickets?", ["billing migration"]),
    ("What currency are the revenue figures in?", ["USD"]),
]


def hit_at_k(index, k):
    hits = 0
    for q, needles in EVAL:
        top = index.search(q, k)
        hits += any(all(n in c.text for n in needles) for _, c in top)
    return hits / len(EVAL)


for name, mm in [("text only ", False), ("multimodal", True)]:
    idx = Index(extract("report.pdf", StubCaptioner(), multimodal=mm), HashEmbedder())
    print(f"{name}  chunks={len(idx.chunks):>2}  hit@1={hit_at_k(idx, 1):.2f}  hit@3={hit_at_k(idx, 3):.2f}")

print()
idx = Index(extract("report.pdf", StubCaptioner(), multimodal=True), HashEmbedder())
for q, needles in EVAL:
    s, c = idx.search(q, 1)[0]
    ok = all(n in c.text for n in needles)
    print(f"{'ok  ' if ok else 'MISS'} {q:<44} -> {c.id} ({s:.2f})")
text only   chunks=11  hit@1=0.00  hit@3=0.33
multimodal  chunks=11  hit@1=0.67  hit@3=1.00

ok   Which month had the most support tickets?    -> p2-img52 (0.49)
ok   How many tickets were there at the peak?     -> p2-img52 (0.23)
ok   What was APAC revenue in Q3?                 -> p1-t0-r1 (0.36)
MISS Which region grew fastest?                   -> p1-b1 (0.23)
MISS What caused the spike in tickets?            -> p2-img52 (0.21)
ok   What currency are the revenue figures in?    -> p1-t0-r0 (0.24)

On this PDF, text-only indexing finds a self-contained answer in the top 3 for two questions out of six. The multimodal index finds all six in the top 3 and four at rank 1.

The two misses at rank 1 are useful. "Which region grew fastest?" lands on the prose paragraph because the hash embedder matches "region" and cannot connect "grew" with "Growth". The whole-table chunk is in the top 3, but a semantic embedder or a hybrid search with a reranker should move it to rank 1. "What caused the spike" ranks the chart above the incident note, because the chart caption shares more words with the question. A reranker reading both would pick the note. Both misses come from the stub embedder, not from the chunking, and this eval is how you would find that out with real models too. For a fuller setup with more metrics, see the RAG evaluation pipeline post.

Six questions on one synthetic PDF prove only that the code works. Before you trust a change, build 50 to 100 questions from your own documents, with at least a fifth of them answered only by a table or a figure.

Failure modes#

Symptom Likely cause Fix
Chart questions retrieve "see the chart below" Figures never captioned Caption at ingest, or add page-image retrieval
Right row retrieved, wrong number in the answer Row chunk lost its header Repeat header and units in every row chunk
Comparison questions get one row Only row chunks indexed Add a whole-table chunk
Figure caption appears twice in the top 3 Caption line indexed as text and inside the figure chunk Drop the text duplicate, as extract() does
Caption invents a number Vision model hallucination Ask it to copy numbers exactly; spot-check against the crop; keep the crop in the UI
Logos and icons flood the index Every embedded image captioned Skip images under a size threshold or repeated on every page
Tables on scanned pages missing find_tables() needs vector lines or text OCR or a layout parser first
Index cost jumps 30x after switching to ColPali Multi-vector storage per page Budget from the 257.5 KB per page figure; compress or pool vectors

What to change for production#

  • Cache captions by image hash (the image_sha field). Re-ingesting a document should not re-caption a logo it has seen 400 times.
  • Store caption_model and caption_prompt_version with every figure chunk, so you can re-caption only what an old prompt produced.
  • At answer time, for figure chunks, pass the cropped image to a vision model along with the caption. The caption found the chunk; the pixels let the model check details the caption skipped.
  • Keep the box in the citation all the way to the UI. A highlighted rectangle on the page is the quickest way for a user to catch a wrong answer.

Sources#