Articles / multimodalpythongenai
Build a multimodal app with the Gemini API: receipt images to validated JSON, plus PDF questions
Send images and PDFs to Gemini with the google-genai SDK, get schema-bound JSON, check the arithmetic, retry on bad reads, and score it with a small eval.
Turning a photo of a receipt into rows in a database used to take three systems: OCR for the text, a layout model to find the line items, and a language model or a pile of regexes to make sense of it. A multimodal model reads the image directly. With the Gemini API that is one request: image bytes and a prompt go in, JSON that matches your schema comes out.
The request is the easy part. The work is everything around it: preparing the image so it does not fail or cost too much, telling the model what shape to return, and refusing a well-formed answer that is wrong. A JSON schema can make the model return a total field. It cannot make the total correct.
This post builds a small Python app that does two jobs:
- Reads a receipt image and returns a validated
Receiptobject, with a retry when the numbers do not add up. - Answers a question about a PDF and cites the page.
It uses the current google-genai SDK, keeps the model behind a two-method interface, and runs offline against a stub, so every output below is real output from code that ran. The one line that talks to Google is marked.
Which SDK and which model, as of November 2026#
Two Python packages have "Gemini" in their history, and tutorials mix them up.
google-generativeaiis the old one. Its PyPI page now opens with "[Deprecated] Google AI Python SDK for the Gemini API" and points to the new SDK. Its last release is 0.8.6. Code that callsgenai.configure(...)andgenai.GenerativeModel(...)is this package.google-genaiis the current one (source). Version 3.6.1 shipped on 18 November 2026, and that is the version used here. You create agenai.Client()and call methods on it.
Google's migration guide maps old calls to new ones. Note also that the Gemini docs moved from ai.google.dev to aistudio.google.com/docs; the old URLs redirect.
On the API itself, Google now has two ways to call a model. The Interactions API (client.interactions.create) is generally available and is what Google recommends for new projects; most of the doc pages now show it first. The page also says generateContent "remains fully supported" while calling it legacy, and lists features such as batch and explicit caching that only generateContent has today. This post uses client.models.generate_content, because its typed request objects (types.Part, types.GenerateContentConfig) can be built and inspected offline. The gateway pattern below makes a later switch to Interactions a change in one class.
For the model, the models page says: "For any new projects, use our latest models: Gemini 4 Hydrogen or 3.5 Flash-Lite." A receipt is a high-volume, simple extraction job, so this app uses gemini-3.5-flash-lite. If you are copying older code, gemini-2.0-flash is listed as shut down, and gemini-2.5-flash is deprecated and routes to 3.5 Flash-Lite from 1 December 2026.
Prices and free tier, from the pricing page on 22 November 2026:
| Model | Free tier | Paid input, per 1M tokens | Paid output, per 1M tokens |
|---|---|---|---|
gemini-3.5-flash-lite |
Yes, input and output free | $0.30 (text, image, video, audio) | $2.50, thinking tokens included |
gemini-4-hydrogen |
Yes, input and output free | $1.50 | $7.50, thinking tokens included |
The same page says free-tier content is used to improve Google's products; paid-tier content is not. Do not send customer receipts through a free-tier key. The rate limits page does not print free-tier RPM or daily request numbers for these models; it says to view your active limits in AI Studio. Check there before you plan a batch.
What the app does#
receipt.png ──► prepare image ──► Part.from_bytes ─┐
(any size, EXIF rotate, image/jpeg │
RGBA, HEIC) RGB, max 1536px ▼
┌─────────────────────┐
Receipt (pydantic) ──► response_schema ─► ModelGateway │
│ GeminiGateway (real)│
│ StubGateway (here) │
└──────────┬──────────┘
▼ JSON text
pydantic validate ──► arithmetic check
│ fail │ fail
└──► retry once with the error
│ still failing
▼
needs_review queue
policy.pdf ──► < 20 MB? ──yes──► Part.from_bytes(application/pdf)
│no
└──► files.upload (kept 48 h) ──► Part.from_uri
Setup, in a fresh virtual environment:
python3 -m venv .venv && source .venv/bin/activate
pip install google-genai pydantic pillow
# google-genai 3.6.1, pydantic 2.14.4, pillow 13.0.0 at the time of writing
The demo draws its own test files with Pillow: a 600x900 RGBA receipt from a made-up tea stall (three items, 5% GST, total INR 288.75) and a three-page travel policy PDF. RGBA is on purpose, since screenshots and some phone exports carry an alpha channel.
Step 1: describe the output as a schema#
The schema is a Pydantic model. The same class does two jobs: the SDK converts it into the response schema sent to Gemini, and Pydantic uses it to validate what comes back.
from decimal import Decimal
from pydantic import BaseModel, Field, ValidationError
from google import genai
from google.genai import types
MODEL = "gemini-3.5-flash-lite"
INLINE_LIMIT = 20 * 1024 * 1024 # total inline request size, per the image-understanding docs
class LineItem(BaseModel):
description: str
quantity: int = Field(ge=1)
unit_price: Decimal
amount: Decimal
class Receipt(BaseModel):
merchant: str
date: str = Field(description="ISO date, YYYY-MM-DD")
currency: str = Field(description="ISO 4217 code, e.g. INR")
items: list[LineItem]
subtotal: Decimal
tax: Decimal
total: Decimal
Money is Decimal, not float. With floats, 0.1 + 0.2 != 0.3, and the arithmetic check later would reject correct receipts by a fraction of a paisa.
Step 2: prepare the image before you send it#
import io
from PIL import Image, ImageOps
def image_part(path: str, max_side: int = 1536) -> tuple[types.Part, dict]:
img = Image.open(path)
img = ImageOps.exif_transpose(img) # phone photos carry rotation in EXIF
img = img.convert("RGB") # drop alpha so JPEG encoding works
img.thumbnail((max_side, max_side)) # cap pixels, and so tokens
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=90)
data = buf.getvalue()
w, h = img.size
return types.Part.from_bytes(data=data, mime_type="image/jpeg"), {
"size": (w, h), "bytes": len(data), "tiles_258_each": image_tiles(w, h)}
def image_tiles(w: int, h: int) -> int:
"""Gemini 2.5-style tiling estimate from the docs: 258 tokens per tile."""
if w <= 384 and h <= 384:
return 1
unit = (min(w, h) * 2) // 3 # floor(min/1.5) without float error
return -(-w // unit) * -(-h // unit) # ceil division on both sides
Each line handles a failure that shows up in real uploads:
exif_transposeapplies the rotation flag that phone cameras store in EXIF (Pillow docs). Without it, a portrait photo can arrive sideways, and the model reads sideways text worse.convert("RGB")drops the alpha channel. JPEG has no alpha, so saving an RGBA image as JPEG raises an error.thumbnailcaps the long side. A 12-megapixel photo of a receipt carries no more readable text than a 1536-pixel one, but it costs more tokens and can push you over the inline size limit.- Re-encoding to JPEG means you always send one of the supported image types (PNG, JPEG, WEBP, HEIC, HEIF), whatever the user uploaded.
image_tiles follows the rough formula on the image understanding page: 258 tokens if both sides are 384 pixels or less, otherwise 768x768 tiles at 258 tokens each, with a crop unit of floor(min(width, height) / 1.5). The docs' own example, 960x540, gives 6 tiles, and so does this function. Gemini 3 and newer models also take a media_resolution setting that caps tokens per image, and the page gives no per-level numbers for images. Treat the formula as an estimate and read usage_metadata on real responses for the true count.
Step 3: put the model behind an interface#
from typing import Protocol
class ModelGateway(Protocol):
def generate(self, contents: list, config: types.GenerateContentConfig) -> str: ...
def upload(self, path: str) -> types.Part: ...
class GeminiGateway:
"""The real thing. Needs GEMINI_API_KEY in the environment."""
def __init__(self):
self.client = genai.Client()
def generate(self, contents, config):
resp = self.client.models.generate_content(model=MODEL, contents=contents, config=config)
return resp.text
def upload(self, path):
f = self.client.files.upload(file=path) # stored 48 hours
return types.Part.from_uri(file_uri=f.uri, mime_type=f.mime_type)
class StubGateway:
"""Offline stand-in: records each request and replays canned answers."""
def __init__(self, answers: list[str]):
self.answers, self.requests = list(answers), []
def generate(self, contents, config):
self.requests.append((contents, config))
return self.answers.pop(0)
def upload(self, path):
return types.Part.from_uri(file_uri="https://generativelanguage.googleapis.com/v1beta/files/stub123",
mime_type="application/pdf")
The real network call is the single generate_content line inside GeminiGateway.generate, plus files.upload for large files. No API key was available while writing this post, so every run below uses StubGateway. The stub receives the same types.Part and types.GenerateContentConfig objects the real client would, which lets the demo print exactly what would be sent. Get a key from AI Studio, export GEMINI_API_KEY, and the demo script switches to GeminiGateway.
The interface is useful beyond testing. The extraction logic never imports anything that talks to the network, so your unit tests run without a key, and a later move to client.interactions.create touches one class.
Step 4: extract, validate, check the arithmetic, retry once#
PROMPT = ("Extract this receipt. Copy numbers exactly as printed. "
"If a field is not visible, use an empty string or 0. Do not compute missing values.")
def arithmetic_errors(r: Receipt) -> list[str]:
errs, cent = [], Decimal("0.01")
for it in r.items:
if abs(it.quantity * it.unit_price - it.amount) > cent:
errs.append(f"{it.description}: {it.quantity} x {it.unit_price} != {it.amount}")
if abs(sum(i.amount for i in r.items) - r.subtotal) > cent:
errs.append(f"items sum to {sum(i.amount for i in r.items)}, subtotal says {r.subtotal}")
if abs(r.subtotal + r.tax - r.total) > cent:
errs.append(f"subtotal + tax = {r.subtotal + r.tax}, total says {r.total}")
return errs
def extract_receipt(gw: ModelGateway, path: str, max_attempts: int = 2):
part, info = image_part(path)
config = types.GenerateContentConfig(
response_mime_type="application/json",
response_schema=Receipt,
temperature=0,
)
contents = [part, PROMPT]
for attempt in range(1, max_attempts + 1):
raw = gw.generate(contents, config)
try:
r = Receipt.model_validate_json(raw)
except ValidationError as e:
problems = [f"schema: {x['loc']} {x['msg']}" for x in e.errors()]
else:
problems = arithmetic_errors(r)
if not problems:
return {"status": "ok", "attempts": attempt, "receipt": r, "image": info}
print(f"attempt {attempt} rejected: {problems}")
contents = [part, PROMPT + " A previous read had these problems, re-read the image: " + "; ".join(problems)]
return {"status": "needs_review", "attempts": max_attempts, "problems": problems}
Setting response_mime_type="application/json" with response_schema turns on structured output. Google's structured output page says the result is syntactically valid JSON and tells you to "always validate values in your application". It also lists schema keywords that are accepted and then ignored during decoding (minLength, maxLength, minProperties, maxProperties). Validation therefore happens in two layers:
- Type layer. Pydantic rejects a field with the wrong type.
- Meaning layer. A receipt carries its own checksum: quantity times unit price equals amount, the amounts sum to the subtotal, and subtotal plus tax equals the total. A misread digit almost always breaks one of those equations.
The prompt tells the model not to compute missing values. If it fills in a total by adding the lines, the arithmetic check passes by construction and stops catching anything.
On failure, the retry sends the same image with the specific error appended. A vague "try again" tends to return the same answer; naming the equation that broke points the model at the digit to re-read. After two attempts the receipt goes to a person. Two is enough here: if two reads disagree with the printed arithmetic, the photo is probably blurred, cropped or crumpled, and a third call costs money without new information.
The stub's first answer reads the total as 268.75 instead of 288.75, an 8 read as a 6. The second answer is correct. Real output:
== receipt ==
attempt 1 rejected: ['subtotal + tax = 288.75, total says 268.75']
ok after 2 attempt(s); image {'size': (600, 900), 'bytes': 18223, 'tiles_258_each': 6}
SUNRISE TEA STALL 2026-11-20 INR 3 items, total 288.75
== what was sent (first request) ==
part mime: image/jpeg | bytes: 18223
config: {'temperature': 0.0, 'response_mime_type': 'application/json'}
schema sent for 'total': {'any_of': [{'type': 'NUMBER'}, {'type': 'STRING'}], 'title': 'Total'}
retry prompt tail: previous read had these problems, re-read the image: subtotal + tax = 288.75, total says 268.75
The schema sent line is the SDK's own conversion of the Pydantic model (via its internal _transformers.t_schema, which may change between releases). It shows something easy to miss: Pydantic describes Decimal as "number or string", so the server would accept "₹288.75" as a valid total. Pydantic rejects it locally, which is why the type layer still matters with structured output on. If you want the server to rule it out, declare those fields as float in a model-facing schema and convert to Decimal after validation.
Step 5: ask a question about a PDF#
def ask_pdf(gw: ModelGateway, path: str, question: str) -> str:
data = open(path, "rb").read()
if len(data) < INLINE_LIMIT:
doc = types.Part.from_bytes(data=data, mime_type="application/pdf")
else:
doc = gw.upload(path)
config = types.GenerateContentConfig(
system_instruction="Answer only from the document. Cite the page number. Say 'not in document' if absent.",
temperature=0)
return gw.generate([doc, question], config)
== pdf question ==
The policy says the daily meal limit is INR 1,500 in metro cities (page 2).
pdf sent inline: True | bytes: 54096
The answer text is the stub's canned reply. The second line is real: the 54 KB PDF went inline as an application/pdf part.
The limits that decide inline versus upload, from Google's pages:
- The image understanding page caps a request with inline data at 20 MB in total, counting prompt, system instruction and bytes. That is the threshold used above.
- The Files API page says to upload when the whole request is over 100 MB and gives 50 MB as the PDF limit. The two pages do not agree on the inline ceiling, so the code uses the lower number.
- The document processing page allows up to 50 MB and 1,000 pages per PDF. On Gemini 3 and 4 models a page costs about 280 tokens at
lowmedia resolution, 560 atmedium(the default) and 1,120 athigh. - Uploaded files are kept for 48 hours, then deleted. Each file can be up to 2 GB, a project can hold 20 GB, and the Files API is free. Upload once and reuse the URI when you will ask several questions about the same document within that window.
This is whole-document prompting, and it works while the PDF fits comfortably in the context window. For a folder of reports, or for answers that live in charts and tables across many files, index the documents instead; multimodal RAG over PDFs covers that build.
What one receipt costs#
Worked from the numbers above, at paid-tier prices for gemini-3.5-flash-lite:
- Image: 6 tiles x 258 = 1,548 tokens by the tiling formula. The prompt adds a few dozen. Call it 1,600 input tokens.
- Output: about 250 tokens for this JSON. This is my estimate for a three-item receipt, not a measured figure, and any thinking tokens are billed as output on top.
- Input cost: 1,600 x $0.30 / 1,000,000 = $0.00048.
- Output cost: 250 x $2.50 / 1,000,000 = $0.000625.
- About $0.0011 per receipt, or roughly $1.10 per 1,000. Every receipt that needs a retry costs double.
Output tokens cost more than the image. Long field names, a notes field the model fills with prose, or a schema with many optional fields raise the bill more than a sharper photo does. The 3-page PDF at medium is about 3 x 560 = 1,680 input tokens per question, before the answer. To log real counts per call, see the cost and latency tracker.
Test it before you trust it#
A demo that works on one receipt says little. Collect 30 to 50 real receipts, type in the correct fields by hand once, and score every prompt or model change against that set:
from receipt_app import Receipt, extract_receipt
FIELDS = ["merchant", "date", "currency", "subtotal", "tax", "total"]
def score(gw, cases):
hits, n, review = 0, 0, 0
for path, expected in cases:
out = extract_receipt(gw, path)
if out["status"] != "ok":
review += 1
n += len(FIELDS)
continue
got = out["receipt"]
for f in FIELDS:
n += 1
hits += getattr(got, f) == getattr(expected, f)
return {"field_accuracy": round(hits / n, 3), "sent_to_review": review, "cases": len(cases)}
The run below feeds three cases through the stub: one clean read, one that misreads the total twice, and one that puts a rupee sign inside the number twice.
== eval ==
attempt 1 rejected: ['subtotal + tax = 288.75, total says 268.75']
attempt 2 rejected: ['subtotal + tax = 288.75, total says 268.75']
attempt 1 rejected: ["schema: ('total',) Input should be a valid decimal"]
attempt 2 rejected: ["schema: ('total',) Input should be a valid decimal"]
{'field_accuracy': 0.333, 'sent_to_review': 2, 'cases': 3}
With a stub the score only proves the scoring code works. Against the real model, track the two numbers separately. Field accuracy says how often accepted receipts are right. The review rate says how much work lands on people. A change that raises accuracy by sending half the receipts to review has not helped. Also count how often a receipt passes the arithmetic check and is still wrong (a wrong merchant or date passes every equation); that number is your silent error rate, and only the hand-labelled set can show it. For a fuller eval setup, see building an evaluation pipeline.
Failure modes to plan for#
- Sideways or rotated photos. Fixed by
exif_transpose, but only when the EXIF flag exists. Screenshots of photos lose it. - Alpha channel crash. RGBA to JPEG fails in Pillow. Convert first.
- Request too large. An inline request over 20 MB is rejected. Downscale images; upload big PDFs.
- Right shape, wrong number. Structured output guarantees JSON, not correct digits. The arithmetic check catches most misreads; nothing catches a confident wrong merchant name except a labelled test set.
- Hallucinated fields. Asked for a GSTIN that is not printed, a model may produce one that looks plausible. Tell it to leave missing fields empty, and treat an empty field as a valid answer.
- Silently ignored constraints.
maxLengthand similar keywords are accepted by the API and not enforced. Enforce them in Pydantic. - Files that vanish. A Files API URI stops working after 48 hours. Do not store it as a permanent reference.
- Model retirement. Model names here change within months:
gemini-2.0-flashis already shut down. Keep the model ID in config, not scattered through code, and rerun the eval when you change it. - Privacy on the free tier. Free-tier content is used to improve Google's products. Receipts contain names, card digits and addresses.
Adding a UI#
Any thin front end works on top of extract_receipt. In Streamlit, read the upload with st.file_uploader, save it to a temporary path or pass the bytes, call extract_receipt, and show either the validated fields in an editable table or the needs_review problems. Keep the button disabled until a file is chosen so you do not spend quota on empty requests. The UI should hold no model logic, so the same functions serve a CLI, a batch job or an API endpoint.
For where a multimodal model call fits among the other parts of an LLM product (orchestration, retrieval, evals, serving), see the LLM tech stack for 2026.
Sources#
- PyPI, google-genai (3.6.1, released 18 Nov 2026) and python-genai on GitHub.
- PyPI, google-generativeai: deprecation notice, last release 0.8.6.
- Google, Migrate to the Google GenAI SDK.
- Google, Gemini models and Gemini 3.5 Flash-Lite: recommended models, deprecations and shutdowns, checked 22 Nov 2026.
- Google, Interactions API: recommended interface;
generateContentlegacy but fully supported. - Google, Gemini API pricing and rate limits, checked 22 Nov 2026.
- Google, Image understanding: MIME types, 20 MB inline limit, tile formula,
media_resolution. - Google, Document processing: PDF size and page limits, tokens per page.
- Google, Files API: 2 GB per file, 20 GB per project, 48-hour retention.
- Google, Structured output: JSON schema support and unenforced keywords.
- Pydantic, JSON Schema; Pillow, ImageOps.