GyaanStack. Blog

Articles / agentsllmops

How to dockerize an AI agent: small images, locked deps, safe tools

Containerize a Python AI agent with a multi-stage uv Dockerfile, a non-root user, a healthcheck, Ollama and Qdrant in compose, real image sizes and a tool sandbox.

By Raghunath · · 17 min read

An agent that runs on your laptop and nowhere else is a demo. The usual fix is a Dockerfile with four lines: FROM python, COPY . ., pip install, CMD. That image works, but it is 1.84 GB, it rebuilds every dependency when you edit one line of code, it runs as root, and its dependency versions float, so next month's build may not match this month's.

This post builds the same small agent two ways and measures both. Every command and number below came from a real run on an M-series Mac with Docker Desktop 29.4.1 and Compose v5.1.3. The agent answers questions from notes stored in a Qdrant vector database and calls a local model through Ollama, so the whole stack runs with no API key.

What we are packaging#

                 compose network (no host ports needed)
   +-----------------+      HTTP 11434     +------------------+
   |  agent          | ------------------> | ollama           |
   |  uid 10001      |                     | qwen2.5:0.5b     |
   |  read-only fs   |      HTTP 6333      +------------------+
   |  HEALTHCHECK    | ------------------> +------------------+
   +-----------------+                     | qdrant           |
            |                              | volume: storage  |
            | docker run --network none    +------------------+
            v
   +-----------------+
   | sandbox (per    |   one throwaway container per tool call
   | tool call)      |
   +-----------------+

The project layout:

app/
  pyproject.toml
  uv.lock
  .dockerignore
  Dockerfile
  compose.yaml
  src/agent/
    __init__.py
    model.py        # the model interface
    main.py         # retrieval + answer loop
    healthcheck.py  # liveness probe

The model sits behind a two-method interface. With no OLLAMA_URL set, the agent uses a stub that returns a fixed string, which is what you want in unit tests and CI. Swap in a hosted API by adding another class with the same complete method; the container does not change.

"""The model sits behind one small interface so the container does not care which backend runs."""
import os
from typing import Protocol

import httpx


class Model(Protocol):
    def complete(self, prompt: str) -> str: ...


class StubModel:
    """Deterministic stand-in used when no model server is configured (tests, CI)."""

    def complete(self, prompt: str) -> str:
        return f"[stub] I would answer: {prompt[:60]}"


class OllamaModel:
    def __init__(self, base_url: str, name: str):
        self.base_url, self.name = base_url.rstrip("/"), name

    def complete(self, prompt: str) -> str:
        r = httpx.post(
            f"{self.base_url}/api/generate",
            json={"model": self.name, "prompt": prompt, "stream": False},
            timeout=120,
        )
        r.raise_for_status()
        return r.json()["response"].strip()


def load_model() -> Model:
    url = os.environ.get("OLLAMA_URL")
    if not url:
        return StubModel()
    return OllamaModel(url, os.environ.get("MODEL_NAME", "qwen2.5:0.5b"))

The agent itself retrieves the closest note and asks the model to answer from it. To keep the demo free of an embedding model, it uses a toy 26-number letter-count vector. That is a deliberate weak point, and it shows up in the test section.

"""A small agent worker: answer questions using notes stored in a vector DB."""
import os
import sys
import time
from pathlib import Path

from qdrant_client import QdrantClient, models

from agent.model import load_model

HEARTBEAT = Path(os.environ.get("HEARTBEAT_FILE", "/tmp/agent-heartbeat"))
COLLECTION = "notes"
NOTES = [
    "Refunds are processed within 5 business days.",
    "Support hours are 9am to 6pm IST, Monday to Friday.",
    "Enterprise plans include a dedicated account manager.",
]


def embed(text: str) -> list[float]:
    # Toy 26-dim letter-count embedding so the demo needs no embedding model.
    v = [0.0] * 26
    for ch in text.lower():
        if "a" <= ch <= "z":
            v[ord(ch) - 97] += 1
    return v


def connect() -> QdrantClient:
    url = os.environ.get("QDRANT_URL")
    client = QdrantClient(url=url) if url else QdrantClient(":memory:")
    if not client.collection_exists(COLLECTION):
        client.create_collection(
            COLLECTION, vectors_config=models.VectorParams(size=26, distance=models.Distance.COSINE)
        )
        client.upsert(
            COLLECTION,
            [models.PointStruct(id=i, vector=embed(t), payload={"text": t}) for i, t in enumerate(NOTES)],
        )
    return client


def answer(client, model, question: str) -> str:
    hits = client.query_points(COLLECTION, query=embed(question), limit=1).points
    context = hits[0].payload["text"]
    return model.complete(f"Context: {context}\nQuestion: {question}\nAnswer in one sentence.")


def main() -> None:
    client, model = connect(), load_model()
    print(f"agent ready (model={type(model).__name__}, uid={os.getuid()})", flush=True)
    for question in sys.argv[1:] or ["How long do refunds take?"]:
        print(f"Q: {question}\nA: {answer(client, model, question)}", flush=True)
    if os.environ.get("WORKER") == "1":
        while True:  # long-running worker: prove liveness with a heartbeat file
            HEARTBEAT.write_text(str(time.time()))
            time.sleep(10)


if __name__ == "__main__":
    main()

Run it on the host first with uv run python -m agent.main "What are support hours?". If it fails here, it will fail in the container too, and the error is easier to read.

agent ready (model=StubModel, uid=501)
Q: What are support hours?
A: [stub] I would answer: Context: Support hours are 9am to 6pm IST, Monday to Friday.

Lock the dependencies#

pyproject.toml lists ranges (httpx>=0.28, qdrant-client>=1.15). Running uv lock resolves those ranges to exact versions for every transitive package and writes them to uv.lock. Here it resolved 21 packages, including qdrant-client 1.20.0, numpy 2.5.5 and grpcio 1.85.3. You commit the lock file, and the image installs from it with --locked, which fails the build if the lock is out of date instead of quietly re-resolving.

If you stay on pip, the equivalent is pip-compile --generate-hashes from pip-tools or pip freeze into a requirements file. A bare transformers or torch line in requirements.txt is the most common reason an agent image that built last month fails today.

The naive Dockerfile, as a baseline#

FROM python:3.13
WORKDIR /app
COPY . /app
RUN pip install .
CMD ["python", "-m", "agent.main"]

Three problems, all measurable. The full python:3.13 base is built on Debian with compilers and headers, so the image starts at 1.63 GB before your code. COPY . /app comes before the install, so any source edit invalidates the install layer and pip runs again. And pip install . ignores uv.lock, so versions float.

A multi-stage Dockerfile with uv#

# syntax=docker/dockerfile:1

# ---- build stage: resolve the locked deps into a venv ----
FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.13.9 /uv /usr/local/bin/uv
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=never
WORKDIR /app

# Dependencies first: this layer is reused until uv.lock changes.
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-install-project --no-dev

# Then the code, which changes on every commit.
COPY src ./src
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-dev --no-editable

# ---- runtime stage: no uv, no compilers, no source tree ----
FROM python:3.13-slim
RUN groupadd --system --gid 10001 agent \
 && useradd --system --uid 10001 --gid agent --no-create-home agent
COPY --from=build --chown=agent:agent /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH" PYTHONUNBUFFERED=1 WORKER=1
USER agent
HEALTHCHECK --interval=15s --timeout=3s --start-period=20s --retries=3 \
  CMD ["python", "-m", "agent.healthcheck"]
CMD ["python", "-m", "agent.main"]

Why each choice is there:

  • Pinned uv image. ghcr.io/astral-sh/uv:0.13.9 copies one static binary. Pin the tag so a uv release cannot change your build.
  • Two uv sync calls. The first installs only third-party packages (--no-install-project), so that layer depends on pyproject.toml and uv.lock alone. The second adds your code. This split comes straight from the uv Docker guide.
  • --no-editable. The project is installed into the venv as a real package, so the runtime stage needs only /app/.venv, not src/.
  • UV_COMPILE_BYTECODE=1. Writes .pyc files at build time, so the first import in production does not pay for compilation.
  • Cache mount. --mount=type=cache keeps downloaded wheels between builds on the same machine without putting them in a layer (BuildKit cache docs).
  • Non-root user. A fixed UID (10001) makes file ownership on mounted volumes predictable, and it means a compromised dependency is not root inside the container.

The .dockerignore keeps the host venv, git history and any .env file out of the build context:

.venv
.git
__pycache__
*.pyc
.env
Dockerfile*
compose.yaml

Image size and rebuild numbers#

Both images built from the same source and the same dependency set. "On disk" is what docker images prints. "Content size" is what docker image inspect -f '{{.Size}}' returns on Docker Desktop's image store, which is close to what a registry pull transfers.

Image On disk Content size
python:3.13 (base) 1.63 GB 403 MB
agent:naive 1.84 GB 472 MB
python:3.13-slim (base) 204 MB 43 MB
agent:slim (multi-stage) 349 MB 78 MB

The dependencies weigh about the same in both: 119 MB in the naive image's site-packages, 108 MB for the slim image's venv. Almost all of the 5x difference comes from the base image. Multi-stage builds help most when the build stage needs compilers or system headers that the runtime does not. Here they keep uv and the source tree out of the final image, which is a smaller win.

Rebuild time after editing one source file, with all caches warm:

Image Rebuild after a code change
agent:naive 15.1 s (pip reinstalls everything)
agent:slim 2.0 s (only the last uv sync reruns)

On an agent that pulls in torch or transformers, the naive layout turns that 15 seconds into minutes, because pip reinstalls several hundred megabytes of wheels on every commit.

The slim runtime is also missing tools an attacker would want. Inside it, which uv gcc finds neither. The base image still ships pip. Removing it means switching to a base image without pip or a shell, which also takes away your debugging shell.

Secrets go in at run time#

Anything you write with ENV or ARG in a Dockerfile is stored in the image metadata. A two-line test makes the point:

$ printf 'FROM python:3.13-slim\nENV API_KEY=sk-demo-not-real\n' | docker build -q -t leaky -
$ docker image inspect -f '{{.Config.Env}}' leaky
[PATH=... PYTHON_VERSION=3.13.18 ... API_KEY=sk-demo-not-real]
$ docker history --no-trunc leaky | grep -o 'API_KEY=[^ ]*'
API_KEY=sk-demo-not-real

Anyone who can pull the image can read the key. The rules that follow:

  • Pass API keys at run time: docker run -e ANTHROPIC_API_KEY ... (no value, so it is read from your shell) or an env_file in compose.
  • Keep .env in both .gitignore and .dockerignore.
  • If the build itself needs a secret, such as a token for a private package index, use a BuildKit secret mount (RUN --mount=type=secret,id=...), which is not written to any layer.
  • In Kubernetes or a cloud runtime, inject from the platform's secret store. The image should be safe to publish.

A healthcheck for an agent that is not a web server#

Many agents are workers: they poll a queue, run a schedule, or loop on tasks. They have no HTTP port to probe, and a process that is still running can be stuck in a dead loop or waiting on a socket that will never answer. The worker in main.py writes a timestamp every 10 seconds, and the probe checks its age:

"""Exit 0 if the worker wrote a heartbeat in the last 30 seconds."""
import os
import sys
import time
from pathlib import Path

hb = Path(os.environ.get("HEARTBEAT_FILE", "/tmp/agent-heartbeat"))
try:
    age = time.time() - float(hb.read_text())
except (OSError, ValueError):
    sys.exit(1)
sys.exit(0 if age < 30 else 1)

Write the heartbeat from the main loop, after real work, not from a separate thread. A background thread keeps ticking while the main loop hangs, so the check would pass on a dead agent. Checking both cases inside the running container:

$ docker compose exec agent sh -c 'python -m agent.healthcheck; echo fresh=$?; echo 0 > /tmp/agent-heartbeat; python -m agent.healthcheck; echo stale=$?'
fresh=0
stale=1

The HEALTHCHECK options are in the Dockerfile reference. Set --start-period longer than your slowest startup (loading a model, warming a cache) so the container is not marked unhealthy while it boots.

Compose with a local model and a vector DB#

services:
  agent:
    build: .
    environment:
      OLLAMA_URL: http://ollama:11434
      MODEL_NAME: qwen2.5:0.5b
      QDRANT_URL: http://qdrant:6333
    env_file:
      - path: .env        # secrets live here, never in the image
        required: false
    depends_on:
      ollama:
        condition: service_healthy
      qdrant:
        condition: service_started
    read_only: true
    tmpfs: [/tmp]
    cap_drop: [ALL]
    security_opt: ["no-new-privileges:true"]
    restart: unless-stopped

  ollama:
    image: ollama/ollama:0.40.17
    volumes: [ollama:/root/.ollama]
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      retries: 10

  qdrant:
    image: qdrant/qdrant:v1.20.2
    volumes: [qdrant:/qdrant/storage]

volumes:
  ollama:
  qdrant:

What the settings do:

  • The agent reaches the other services by name (http://ollama:11434) on the compose network. No ports are published to the host, because nothing outside needs them.
  • depends_on with condition: service_healthy waits for Ollama's own healthcheck before starting the agent. Without a condition, compose only waits for the container to start, and the first model call can hit a server that is not listening yet (startup order docs). Qdrant has no healthcheck here, so the client should retry its first connection.
  • read_only: true plus a tmpfs for /tmp means the agent can write its heartbeat and nothing else. Trying touch /app/x inside it returns Read-only file system.
  • Model weights and vectors live in named volumes, so docker compose down does not delete them. down -v does.

Bring it up and pull the model once into the volume:

$ docker compose up -d ollama qdrant
$ docker compose exec ollama ollama pull qwen2.5:0.5b
$ docker compose up -d --build agent
$ docker compose ps --format '{{.Service}} {{.Status}}'
agent Up 25 seconds (healthy)
ollama Up 7 minutes (healthy)
qdrant Up 7 minutes

A one-off question through the real model took 1.6 seconds end to end, including container start:

$ docker compose run --rm -e WORKER=0 agent python -m agent.main "What are support hours?"
agent ready (model=OllamaModel, uid=10001)
Q: What are support hours?
A: Support hours are 9am to 6pm IST, Monday to Friday.

The service images are the heavy part of this stack. ollama/ollama:0.40.17 is 7.21 GB on disk (2.87 GB content size). Most of that is GPU runtime: /usr/lib/ollama inside it holds 3.6 GB, mainly CUDA 12, CUDA 13 and Jetson libraries; qdrant/qdrant:v1.20.2 is 296 MB. The Ollama Docker docs cover the NVIDIA and AMD GPU setups; on this Mac the container ran on CPU, which is fine for a 0.5B model and slow for anything above a few billion parameters. Qdrant also publishes -unprivileged tags on Docker Hub if you want it to run as non-root too.

Test the container, not only the code#

The default question in the worker run, "How long do refunds take?", came back wrong:

A: The refund process time depends on the specific terms of the enterprise plan, and it can take up to several business days to complete the refund.

The model was not the problem. Querying Qdrant directly from inside the container showed which note retrieval returned:

How long do refunds take? [(2, 0.696), (1, 0.678), (0, 0.675)]
What are support hours? [(1, 0.833), (0, 0.639), (2, 0.561)]

Note 2 is the enterprise-plan note. The refund note (id 0) came last, and the three scores sit within 0.021 of each other, which is what letter counts give you. A real embedding model fixes this. Ollama can serve one from the same container through its embed endpoint, so the change is one function.

A container that builds and reports healthy can still answer wrong, so a useful smoke test for an agent image checks three things before you push it:

  1. It builds from the lock: docker build with uv sync --locked fails if uv.lock is stale.
  2. It starts as the right user and turns healthy: docker run --rm agent:slim id prints uid=10001(agent), and docker compose ps shows (healthy) within the start period.
  3. It answers a small fixed set of questions correctly against a known fixture, using the stub model for plumbing and the real model for quality.

Wiring those checks into a pipeline, with eval gates on the quality step, is covered in CI/CD for LLM applications.

Sandbox the tools, not only the agent#

Running the agent in a container protects the host from the agent process. It does nothing about a tool that executes model-written code, because that code runs with the agent's permissions, network and secrets. If your agent has a "run Python" tool, give each call its own disposable container with no network:

"""Run model-written Python in a throwaway, locked-down container."""
import subprocess

SANDBOX = [
    "docker", "run", "--rm", "-i",
    "--network", "none",          # no exfiltration, no pip install
    "--read-only", "--tmpfs", "/tmp:size=16m",
    "--memory", "256m", "--cpus", "0.5", "--pids-limit", "64",
    "--cap-drop", "ALL", "--security-opt", "no-new-privileges",
    "--user", "65534:65534",      # nobody
    "python:3.13-slim", "python", "-",
]


def run_code(code: str, timeout: int = 10) -> str:
    try:
        p = subprocess.run(SANDBOX, input=code, capture_output=True, text=True, timeout=timeout)
    except subprocess.TimeoutExpired:
        return "error: timed out"
    return (p.stdout + p.stderr).strip()[-2000:]


if __name__ == "__main__":
    print("math   ->", run_code("print(sum(range(10)))"))
    print("net    ->", run_code("import urllib.request as u; u.urlopen('https://example.com', timeout=3)").splitlines()[-1])
    print("write  ->", run_code("open('/etc/x','w')").splitlines()[-1])
    print("fork   ->", run_code("import os\nfor _ in range(200): os.fork() or __import__('time').sleep(5)").splitlines()[-1])
    print("memory ->", run_code("x = bytearray(1024**3)") or "(killed, no output)")

Output from the host:

math   -> 45
net    -> urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
write  -> OSError: [Errno 30] Read-only file system: '/etc/x'
fork   -> BlockingIOError: [Errno 11] Resource temporarily unavailable
memory -> (killed, no output)

The memory case exits with code 137, the kernel's OOM kill. Return that exit code to the model as a tool error so it can try a smaller approach instead of seeing an empty string. All five calls together took 1.1 seconds on this machine, so a fresh container per call is affordable. The flags are documented under resource constraints and docker run.

Two limits to know about:

  • Do not mount the Docker socket into the agent container so it can launch sandboxes. The Docker daemon requires root privileges unless you run rootless mode, so access to its socket is root on the host. Run the sandbox launcher as a separate small service with a fixed command list, or use a remote sandbox.
  • A container shares the host kernel. For untrusted code from the internet, a user-space kernel such as gVisor (--runtime=runsc) or a microVM adds a boundary that a kernel bug alone cannot cross.

How the permission layer and the sandbox fit together in a full coding agent is in Design an AI coding agent, and the attacks that make tool sandboxing necessary are tested in LLM red teaming with prompt injection tests.

Failure modes seen in practice#

  • Model weights baked into the image. A RUN that downloads weights at build time makes every image several gigabytes and every push slow. Pull weights into a volume, as with Ollama above, or download on first start into a mounted cache.
  • Weights downloaded on every start. The opposite mistake. Without a volume, each new container pulls the model again, and a restart loop turns into a bandwidth bill.
  • Healthy but useless. A healthcheck that only proves the process is alive. Tie the heartbeat to completed work.
  • localhost inside compose. http://localhost:11434 in the agent container points at the agent itself. Use the service name.
  • Wrong CPU architecture. An image built on an Apple Silicon Mac is linux/arm64. A typical x86 cloud VM needs docker buildx build --platform linux/amd64 or a multi-platform build.
  • Unpinned service tags. ollama/ollama:latest moves with every release, and Docker Hub lists release candidates next to it. Pin the tag you tested.

Sources#