How to dockerize an AI agent: small images, locked deps, safe tools
Containerize a Python AI agent with a multi-stage uv Dockerfile, a non-root user, a healthcheck, Ollama and Qdrant in compose, real image sizes and a tool sandbox.
An agent that runs on your laptop and nowhere else is a demo. The usual fix is a Dockerfile with four lines: FROM python, COPY . ., pip install, CMD. That image works, but it is 1.84 GB, it rebuilds every dependency when you edit one line of code, it runs as root, and its dependency versions float, so next month's build may not match this month's.
This post builds the same small agent two ways and measures both. Every command and number below came from a real run on an M-series Mac with Docker Desktop 29.4.1 and Compose v5.1.3. The agent answers questions from notes stored in a Qdrant vector database and calls a local model through Ollama, so the whole stack runs with no API key.
What we are packaging#
compose network (no host ports needed)
+-----------------+ HTTP 11434 +------------------+
| agent | ------------------> | ollama |
| uid 10001 | | qwen2.5:0.5b |
| read-only fs | HTTP 6333 +------------------+
| HEALTHCHECK | ------------------> +------------------+
+-----------------+ | qdrant |
| | volume: storage |
| docker run --network none +------------------+
v
+-----------------+
| sandbox (per | one throwaway container per tool call
| tool call) |
+-----------------+
The project layout:
app/
pyproject.toml
uv.lock
.dockerignore
Dockerfile
compose.yaml
src/agent/
__init__.py
model.py # the model interface
main.py # retrieval + answer loop
healthcheck.py # liveness probe
The model sits behind a two-method interface. With no OLLAMA_URL set, the agent uses a stub that returns a fixed string, which is what you want in unit tests and CI. Swap in a hosted API by adding another class with the same complete method; the container does not change.
"""The model sits behind one small interface so the container does not care which backend runs."""
import os
from typing import Protocol
import httpx
class Model(Protocol):
def complete(self, prompt: str) -> str: ...
class StubModel:
"""Deterministic stand-in used when no model server is configured (tests, CI)."""
def complete(self, prompt: str) -> str:
return f"[stub] I would answer: {prompt[:60]}"
class OllamaModel:
def __init__(self, base_url: str, name: str):
self.base_url, self.name = base_url.rstrip("/"), name
def complete(self, prompt: str) -> str:
r = httpx.post(
f"{self.base_url}/api/generate",
json={"model": self.name, "prompt": prompt, "stream": False},
timeout=120,
)
r.raise_for_status()
return r.json()["response"].strip()
def load_model() -> Model:
url = os.environ.get("OLLAMA_URL")
if not url:
return StubModel()
return OllamaModel(url, os.environ.get("MODEL_NAME", "qwen2.5:0.5b"))
The agent itself retrieves the closest note and asks the model to answer from it. To keep the demo free of an embedding model, it uses a toy 26-number letter-count vector. That is a deliberate weak point, and it shows up in the test section.
"""A small agent worker: answer questions using notes stored in a vector DB."""
import os
import sys
import time
from pathlib import Path
from qdrant_client import QdrantClient, models
from agent.model import load_model
HEARTBEAT = Path(os.environ.get("HEARTBEAT_FILE", "/tmp/agent-heartbeat"))
COLLECTION = "notes"
NOTES = [
"Refunds are processed within 5 business days.",
"Support hours are 9am to 6pm IST, Monday to Friday.",
"Enterprise plans include a dedicated account manager.",
]
def embed(text: str) -> list[float]:
# Toy 26-dim letter-count embedding so the demo needs no embedding model.
v = [0.0] * 26
for ch in text.lower():
if "a" <= ch <= "z":
v[ord(ch) - 97] += 1
return v
def connect() -> QdrantClient:
url = os.environ.get("QDRANT_URL")
client = QdrantClient(url=url) if url else QdrantClient(":memory:")
if not client.collection_exists(COLLECTION):
client.create_collection(
COLLECTION, vectors_config=models.VectorParams(size=26, distance=models.Distance.COSINE)
)
client.upsert(
COLLECTION,
[models.PointStruct(id=i, vector=embed(t), payload={"text": t}) for i, t in enumerate(NOTES)],
)
return client
def answer(client, model, question: str) -> str:
hits = client.query_points(COLLECTION, query=embed(question), limit=1).points
context = hits[0].payload["text"]
return model.complete(f"Context: {context}\nQuestion: {question}\nAnswer in one sentence.")
def main() -> None:
client, model = connect(), load_model()
print(f"agent ready (model={type(model).__name__}, uid={os.getuid()})", flush=True)
for question in sys.argv[1:] or ["How long do refunds take?"]:
print(f"Q: {question}\nA: {answer(client, model, question)}", flush=True)
if os.environ.get("WORKER") == "1":
while True: # long-running worker: prove liveness with a heartbeat file
HEARTBEAT.write_text(str(time.time()))
time.sleep(10)
if __name__ == "__main__":
main()
Run it on the host first with uv run python -m agent.main "What are support hours?". If it fails here, it will fail in the container too, and the error is easier to read.
agent ready (model=StubModel, uid=501)
Q: What are support hours?
A: [stub] I would answer: Context: Support hours are 9am to 6pm IST, Monday to Friday.
Lock the dependencies#
pyproject.toml lists ranges (httpx>=0.28, qdrant-client>=1.15). Running uv lock resolves those ranges to exact versions for every transitive package and writes them to uv.lock. Here it resolved 21 packages, including qdrant-client 1.20.0, numpy 2.5.5 and grpcio 1.85.3. You commit the lock file, and the image installs from it with --locked, which fails the build if the lock is out of date instead of quietly re-resolving.
If you stay on pip, the equivalent is pip-compile --generate-hashes from pip-tools or pip freeze into a requirements file. A bare transformers or torch line in requirements.txt is the most common reason an agent image that built last month fails today.
The naive Dockerfile, as a baseline#
FROM python:3.13
WORKDIR /app
COPY . /app
RUN pip install .
CMD ["python", "-m", "agent.main"]
Three problems, all measurable. The full python:3.13 base is built on Debian with compilers and headers, so the image starts at 1.63 GB before your code. COPY . /app comes before the install, so any source edit invalidates the install layer and pip runs again. And pip install . ignores uv.lock, so versions float.
A multi-stage Dockerfile with uv#
# syntax=docker/dockerfile:1
# ---- build stage: resolve the locked deps into a venv ----
FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.13.9 /uv /usr/local/bin/uv
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=never
WORKDIR /app
# Dependencies first: this layer is reused until uv.lock changes.
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-install-project --no-dev
# Then the code, which changes on every commit.
COPY src ./src
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-dev --no-editable
# ---- runtime stage: no uv, no compilers, no source tree ----
FROM python:3.13-slim
RUN groupadd --system --gid 10001 agent \
&& useradd --system --uid 10001 --gid agent --no-create-home agent
COPY --from=build --chown=agent:agent /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH" PYTHONUNBUFFERED=1 WORKER=1
USER agent
HEALTHCHECK --interval=15s --timeout=3s --start-period=20s --retries=3 \
CMD ["python", "-m", "agent.healthcheck"]
CMD ["python", "-m", "agent.main"]
Why each choice is there:
- Pinned uv image.
ghcr.io/astral-sh/uv:0.13.9copies one static binary. Pin the tag so a uv release cannot change your build. - Two
uv synccalls. The first installs only third-party packages (--no-install-project), so that layer depends onpyproject.tomlanduv.lockalone. The second adds your code. This split comes straight from the uv Docker guide. --no-editable. The project is installed into the venv as a real package, so the runtime stage needs only/app/.venv, notsrc/.UV_COMPILE_BYTECODE=1. Writes.pycfiles at build time, so the first import in production does not pay for compilation.- Cache mount.
--mount=type=cachekeeps downloaded wheels between builds on the same machine without putting them in a layer (BuildKit cache docs). - Non-root user. A fixed UID (10001) makes file ownership on mounted volumes predictable, and it means a compromised dependency is not root inside the container.
The .dockerignore keeps the host venv, git history and any .env file out of the build context:
.venv
.git
__pycache__
*.pyc
.env
Dockerfile*
compose.yaml
Image size and rebuild numbers#
Both images built from the same source and the same dependency set. "On disk" is what docker images prints. "Content size" is what docker image inspect -f '{{.Size}}' returns on Docker Desktop's image store, which is close to what a registry pull transfers.
| Image | On disk | Content size |
|---|---|---|
python:3.13 (base) |
1.63 GB | 403 MB |
agent:naive |
1.84 GB | 472 MB |
python:3.13-slim (base) |
204 MB | 43 MB |
agent:slim (multi-stage) |
349 MB | 78 MB |
The dependencies weigh about the same in both: 119 MB in the naive image's site-packages, 108 MB for the slim image's venv. Almost all of the 5x difference comes from the base image. Multi-stage builds help most when the build stage needs compilers or system headers that the runtime does not. Here they keep uv and the source tree out of the final image, which is a smaller win.
Rebuild time after editing one source file, with all caches warm:
| Image | Rebuild after a code change |
|---|---|
agent:naive |
15.1 s (pip reinstalls everything) |
agent:slim |
2.0 s (only the last uv sync reruns) |
On an agent that pulls in torch or transformers, the naive layout turns that 15 seconds into minutes, because pip reinstalls several hundred megabytes of wheels on every commit.
The slim runtime is also missing tools an attacker would want. Inside it, which uv gcc finds neither. The base image still ships pip. Removing it means switching to a base image without pip or a shell, which also takes away your debugging shell.
Secrets go in at run time#
Anything you write with ENV or ARG in a Dockerfile is stored in the image metadata. A two-line test makes the point:
$ printf 'FROM python:3.13-slim\nENV API_KEY=sk-demo-not-real\n' | docker build -q -t leaky -
$ docker image inspect -f '{{.Config.Env}}' leaky
[PATH=... PYTHON_VERSION=3.13.18 ... API_KEY=sk-demo-not-real]
$ docker history --no-trunc leaky | grep -o 'API_KEY=[^ ]*'
API_KEY=sk-demo-not-real
Anyone who can pull the image can read the key. The rules that follow:
- Pass API keys at run time:
docker run -e ANTHROPIC_API_KEY ...(no value, so it is read from your shell) or anenv_filein compose. - Keep
.envin both.gitignoreand.dockerignore. - If the build itself needs a secret, such as a token for a private package index, use a BuildKit secret mount (
RUN --mount=type=secret,id=...), which is not written to any layer. - In Kubernetes or a cloud runtime, inject from the platform's secret store. The image should be safe to publish.
A healthcheck for an agent that is not a web server#
Many agents are workers: they poll a queue, run a schedule, or loop on tasks. They have no HTTP port to probe, and a process that is still running can be stuck in a dead loop or waiting on a socket that will never answer. The worker in main.py writes a timestamp every 10 seconds, and the probe checks its age:
"""Exit 0 if the worker wrote a heartbeat in the last 30 seconds."""
import os
import sys
import time
from pathlib import Path
hb = Path(os.environ.get("HEARTBEAT_FILE", "/tmp/agent-heartbeat"))
try:
age = time.time() - float(hb.read_text())
except (OSError, ValueError):
sys.exit(1)
sys.exit(0 if age < 30 else 1)
Write the heartbeat from the main loop, after real work, not from a separate thread. A background thread keeps ticking while the main loop hangs, so the check would pass on a dead agent. Checking both cases inside the running container:
$ docker compose exec agent sh -c 'python -m agent.healthcheck; echo fresh=$?; echo 0 > /tmp/agent-heartbeat; python -m agent.healthcheck; echo stale=$?'
fresh=0
stale=1
The HEALTHCHECK options are in the Dockerfile reference. Set --start-period longer than your slowest startup (loading a model, warming a cache) so the container is not marked unhealthy while it boots.
Compose with a local model and a vector DB#
services:
agent:
build: .
environment:
OLLAMA_URL: http://ollama:11434
MODEL_NAME: qwen2.5:0.5b
QDRANT_URL: http://qdrant:6333
env_file:
- path: .env # secrets live here, never in the image
required: false
depends_on:
ollama:
condition: service_healthy
qdrant:
condition: service_started
read_only: true
tmpfs: [/tmp]
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
restart: unless-stopped
ollama:
image: ollama/ollama:0.40.17
volumes: [ollama:/root/.ollama]
healthcheck:
test: ["CMD", "ollama", "list"]
interval: 10s
retries: 10
qdrant:
image: qdrant/qdrant:v1.20.2
volumes: [qdrant:/qdrant/storage]
volumes:
ollama:
qdrant:
What the settings do:
- The agent reaches the other services by name (
http://ollama:11434) on the compose network. No ports are published to the host, because nothing outside needs them. depends_onwithcondition: service_healthywaits for Ollama's own healthcheck before starting the agent. Without a condition, compose only waits for the container to start, and the first model call can hit a server that is not listening yet (startup order docs). Qdrant has no healthcheck here, so the client should retry its first connection.read_only: trueplus atmpfsfor/tmpmeans the agent can write its heartbeat and nothing else. Tryingtouch /app/xinside it returnsRead-only file system.- Model weights and vectors live in named volumes, so
docker compose downdoes not delete them.down -vdoes.
Bring it up and pull the model once into the volume:
$ docker compose up -d ollama qdrant
$ docker compose exec ollama ollama pull qwen2.5:0.5b
$ docker compose up -d --build agent
$ docker compose ps --format '{{.Service}} {{.Status}}'
agent Up 25 seconds (healthy)
ollama Up 7 minutes (healthy)
qdrant Up 7 minutes
A one-off question through the real model took 1.6 seconds end to end, including container start:
$ docker compose run --rm -e WORKER=0 agent python -m agent.main "What are support hours?"
agent ready (model=OllamaModel, uid=10001)
Q: What are support hours?
A: Support hours are 9am to 6pm IST, Monday to Friday.
The service images are the heavy part of this stack. ollama/ollama:0.40.17 is 7.21 GB on disk (2.87 GB content size). Most of that is GPU runtime: /usr/lib/ollama inside it holds 3.6 GB, mainly CUDA 12, CUDA 13 and Jetson libraries; qdrant/qdrant:v1.20.2 is 296 MB. The Ollama Docker docs cover the NVIDIA and AMD GPU setups; on this Mac the container ran on CPU, which is fine for a 0.5B model and slow for anything above a few billion parameters. Qdrant also publishes -unprivileged tags on Docker Hub if you want it to run as non-root too.
Test the container, not only the code#
The default question in the worker run, "How long do refunds take?", came back wrong:
A: The refund process time depends on the specific terms of the enterprise plan, and it can take up to several business days to complete the refund.
The model was not the problem. Querying Qdrant directly from inside the container showed which note retrieval returned:
How long do refunds take? [(2, 0.696), (1, 0.678), (0, 0.675)]
What are support hours? [(1, 0.833), (0, 0.639), (2, 0.561)]
Note 2 is the enterprise-plan note. The refund note (id 0) came last, and the three scores sit within 0.021 of each other, which is what letter counts give you. A real embedding model fixes this. Ollama can serve one from the same container through its embed endpoint, so the change is one function.
A container that builds and reports healthy can still answer wrong, so a useful smoke test for an agent image checks three things before you push it:
- It builds from the lock:
docker buildwithuv sync --lockedfails ifuv.lockis stale. - It starts as the right user and turns healthy:
docker run --rm agent:slim idprintsuid=10001(agent), anddocker compose psshows(healthy)within the start period. - It answers a small fixed set of questions correctly against a known fixture, using the stub model for plumbing and the real model for quality.
Wiring those checks into a pipeline, with eval gates on the quality step, is covered in CI/CD for LLM applications.
Sandbox the tools, not only the agent#
Running the agent in a container protects the host from the agent process. It does nothing about a tool that executes model-written code, because that code runs with the agent's permissions, network and secrets. If your agent has a "run Python" tool, give each call its own disposable container with no network:
"""Run model-written Python in a throwaway, locked-down container."""
import subprocess
SANDBOX = [
"docker", "run", "--rm", "-i",
"--network", "none", # no exfiltration, no pip install
"--read-only", "--tmpfs", "/tmp:size=16m",
"--memory", "256m", "--cpus", "0.5", "--pids-limit", "64",
"--cap-drop", "ALL", "--security-opt", "no-new-privileges",
"--user", "65534:65534", # nobody
"python:3.13-slim", "python", "-",
]
def run_code(code: str, timeout: int = 10) -> str:
try:
p = subprocess.run(SANDBOX, input=code, capture_output=True, text=True, timeout=timeout)
except subprocess.TimeoutExpired:
return "error: timed out"
return (p.stdout + p.stderr).strip()[-2000:]
if __name__ == "__main__":
print("math ->", run_code("print(sum(range(10)))"))
print("net ->", run_code("import urllib.request as u; u.urlopen('https://example.com', timeout=3)").splitlines()[-1])
print("write ->", run_code("open('/etc/x','w')").splitlines()[-1])
print("fork ->", run_code("import os\nfor _ in range(200): os.fork() or __import__('time').sleep(5)").splitlines()[-1])
print("memory ->", run_code("x = bytearray(1024**3)") or "(killed, no output)")
Output from the host:
math -> 45
net -> urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
write -> OSError: [Errno 30] Read-only file system: '/etc/x'
fork -> BlockingIOError: [Errno 11] Resource temporarily unavailable
memory -> (killed, no output)
The memory case exits with code 137, the kernel's OOM kill. Return that exit code to the model as a tool error so it can try a smaller approach instead of seeing an empty string. All five calls together took 1.1 seconds on this machine, so a fresh container per call is affordable. The flags are documented under resource constraints and docker run.
Two limits to know about:
- Do not mount the Docker socket into the agent container so it can launch sandboxes. The Docker daemon requires root privileges unless you run rootless mode, so access to its socket is root on the host. Run the sandbox launcher as a separate small service with a fixed command list, or use a remote sandbox.
- A container shares the host kernel. For untrusted code from the internet, a user-space kernel such as gVisor (
--runtime=runsc) or a microVM adds a boundary that a kernel bug alone cannot cross.
How the permission layer and the sandbox fit together in a full coding agent is in Design an AI coding agent, and the attacks that make tool sandboxing necessary are tested in LLM red teaming with prompt injection tests.
Failure modes seen in practice#
- Model weights baked into the image. A
RUNthat downloads weights at build time makes every image several gigabytes and every push slow. Pull weights into a volume, as with Ollama above, or download on first start into a mounted cache. - Weights downloaded on every start. The opposite mistake. Without a volume, each new container pulls the model again, and a restart loop turns into a bandwidth bill.
- Healthy but useless. A healthcheck that only proves the process is alive. Tie the heartbeat to completed work.
localhostinside compose.http://localhost:11434in the agent container points at the agent itself. Use the service name.- Wrong CPU architecture. An image built on an Apple Silicon Mac is
linux/arm64. A typical x86 cloud VM needsdocker buildx build --platform linux/amd64or a multi-platform build. - Unpinned service tags.
ollama/ollama:latestmoves with every release, and Docker Hub lists release candidates next to it. Pin the tag you tested.
Sources#
- uv: using uv in Docker
- Docker: multi-stage builds
- Docker: Dockerfile reference (HEALTHCHECK, USER)
- Docker: building best practices
- Docker: build cache
- Docker: build secrets
- Docker: Compose services reference
- Docker: control startup order in Compose
- Docker: resource constraints
- Docker: docker container run reference
- Docker: engine security
- Ollama: Docker
- Ollama API: embed
- pip-tools
- Ollama library: qwen2.5
- Qdrant: installation
- Qdrant on Docker Hub
- gVisor documentation