Articles / agentsgenai-system-design
Design a long-running AI agent: plans, checkpoints, retries and budgets
How to design an AI agent that runs for hours or days: triggers, a persisted plan, checkpoints, idempotent tools, approval gates, budgets and evals.
A chat agent lives for one request. An always-on agent lives for days. It wakes up on a schedule or an event, works through a plan that might take forty tool calls, waits six hours for a human to approve a step, and has to pick up exactly where it stopped after the server it ran on was replaced.
That changes the design problem. The hard parts stop being prompts and become state: where the plan lives, what happens when the process dies between "send the email" and "record that the email was sent", how much a run is allowed to spend, and what the agent's context looks like at step 150.
Most articles on this topic stop at "use durable execution" and a framework name. This post goes further: an architecture diagram, the failure each component exists to prevent, budget math with the numbers shown, a decision table for when you need a workflow engine at all, and a small runnable loop that survives a crash without sending a duplicate email.
What makes a run "long"#
I use three tests. If any one is true, the agent needs the design in this post.
- The run outlives a process. Deploys, autoscaling and crashes will kill it at least once.
- The run waits on something outside itself. A human approval, a webhook, a rate limit reset, tomorrow's data.
- The run has side effects you cannot repeat. Emails, payments, tickets, merged pull requests.
A research agent that runs for 4 minutes in one request fails all three tests and needs none of this. A nightly agent that triages 300 support tickets, files Jira issues and emails a digest passes all three.
The architecture#
TRIGGERS CONTROL PLANE
cron / schedule ──┐ ┌──────────────────────────────┐
webhook (GitHub, ├──► queue ──► run manager │
Stripe, Slack) │ (dedupe│ - create run_id │
user request ─────┘ on │ - load or create plan │
event │ - enforce budget, step cap │
id) └──────┬───────────────▲────────┘
│ next step │ checkpoint
▼ │
┌───────────┐ ┌──────────────┐ ┌───┴───────────┐
│ planner │ │ executor │ │ state store │
│ (strong │◄─┤ (cheaper ├─►│ runs, steps, │
│ model, │ │ model or │ │ results, │
│ replans) │ │ plain code)│ │ spend, events │
└───────────┘ └──────┬───────┘ └───────────────┘
│ tool call + idempotency key
▼
tools: APIs, DB, email ── approval gate ── human
Each box has one job:
- Triggers start runs. They never do work themselves.
- Run manager owns the run lifecycle: it creates the run ID, loads state, picks the next step, checks the budget, and writes checkpoints.
- Planner turns a goal into a list of steps and revises it when a step fails or the world changes.
- Executor performs one step at a time, often with a smaller model or no model at all.
- State store is the source of truth. The model's context window is a view of that state, rebuilt on each step, never the other way round.
That last rule decides more than any other. If the only record of progress is the conversation history inside a running process, a crash erases the run.
Triggers and deduplication#
Three trigger types cover nearly everything: a schedule, an external event (webhook), and a direct request. All three should go through a queue, for two reasons.
The first is timing. GitHub's webhook best practices say your server should return a 2XX within 10 seconds, and suggest a queue for processing payloads asynchronously. An agent run takes minutes, so the webhook handler should enqueue and return.
The second is duplicates. Senders retry, and schedulers misfire. The Kubernetes CronJob docs state that in certain circumstances two Jobs might be created, or none, and that your Jobs should therefore be idempotent. GitHub's X-GitHub-Delivery header keeps the same value on a redelivery, so you can dedupe on it.
My rule: derive the run ID from the trigger, not from a random UUID at start time. digest:2026-11-21 for a daily job, gh:<delivery-id> for a webhook. Then a second trigger for the same event finds the existing run and resumes it instead of starting a twin.
Planner and executor#
Splitting planning from execution is an old idea in agent research. ReWOO separates reasoning from tool observations and reports 5x token efficiency on HotpotQA. For long runs the split matters for a different reason: it gives you a plan you can store, show to a human, and diff.
A plan step I'd store looks like this:
step(
run_id, idx,
tool, args_json, -- what to do
depends_on, -- indices that must be done first
status, -- pending | running | done | failed | waiting_approval
attempt, result_json,
idempotency_key, -- f"{run_id}:{idx}", stable across retries
plan_version -- bumps on every replan
)
The planner writes steps. The executor only flips status and fills results. When a step fails in a way retries won't fix (a 404, a validation error, a policy refusal), the run manager calls the planner again with the original goal, the completed steps and the failure, and writes a new plan version. Completed steps stay completed; the replan only rewrites pending ones.
Anthropic's post on harnesses for long-running agents uses the same idea for coding agents: a JSON feature list where later sessions may only change a passes field. The authors chose JSON over Markdown because the model was less likely to overwrite it inappropriately. A stored plan with a narrow write interface protects you from the agent quietly deleting the hard steps.
Durable execution and checkpoints#
Two families of tools solve "resume after a crash". They work differently, and the difference shows up in how you write code.
Checkpointing (LangGraph style)#
LangGraph's checkpointer docs say a checkpointer saves a snapshot of graph state at each super-step, keyed by thread_id. If one node in a super-step fails, the writes from nodes that succeeded in that super-step are kept as pending writes and are not re-run on resume. Production setups use PostgresSaver; InMemorySaver loses everything on restart.
There are three durability modes:
| Mode | When state is written | Crash mid-run |
|---|---|---|
"exit" |
Only when the graph exits (done, error, or interrupt) | No recovery of intermediate state |
"async" |
While the next step runs | Small window where the last checkpoint is lost |
"sync" |
Before the next step starts | Every completed step is recoverable |
For a run that spans hours and has side effects, I'd use "sync". My estimate is that a Postgres write adds milliseconds per step, against seconds of model latency, so the overhead rarely shows.
The unit of recovery is the node. A node that crashed halfway runs again from its first line. Keep nodes small and put one side effect in each.
Workflow engines (Temporal style)#
Temporal records every step in an event history and rebuilds a workflow's state by replaying its code against that history. Its workflow docs require workflow code to be deterministic: on replay, the commands the code emits are compared to the history, and a mismatch is a non-determinism error. LLM calls, API calls and database queries go into Activities, which run outside the replay path and are retried.
Activity retries are on by default. The retry policy docs give the defaults: 1 second initial interval, backoff coefficient 2.0, maximum interval 100 seconds, unlimited attempts. Workflows themselves do not retry by default.
Two limits matter for agents. The limits page caps event history at 51,200 events or 50 MB, with a warning at 10,240 events or 10 MB. An agent that makes thousands of tool calls in one workflow will hit that, so it should Continue-As-New: carry the latest state into a fresh execution with a fresh history. The docs note that repeating this lets a workflow run forever. Also, the planner's output must come back as an Activity result. If you branched on a model call inside workflow code, replay would call the model again and get a different answer.
Which recovery model fits#
| Checkpointing (LangGraph) | Replay (Temporal) | |
|---|---|---|
| What is stored | State snapshot per super-step | Every event, state rebuilt by replay |
| Code rule | Nodes may re-run; make them idempotent | Workflow code must be deterministic |
| Waiting days | Interrupt + resume by thread_id |
Timers and messages to the workflow |
| Retries | You configure them per node | Built in for Activities |
| Ops cost | A Postgres table | A Temporal cluster or Temporal Cloud |
Neither one makes side effects safe. Both will run a step at least once more after the wrong crash. That is the next section.
Idempotent tool calls#
Here is the crash that matters:
executor: call send_email(...) ──► provider sends it
── process dies here ──
write checkpoint "step 2 done" (never happens)
resume: step 2 is still pending ──► send_email(...) again
No checkpointing scheme closes that gap, because the side effect and the checkpoint live in different systems. Temporal's activity docs say it plainly: activities may run more than once, idempotency is enforced by the service you call, and the Workflow Run ID plus Activity ID works as an idempotency key because it is consistent across retries and unique across executions.
So every side-effecting tool gets a key derived from the run and the step, never from the attempt. Stripe's idempotent requests docs show what the receiving side should do. For API v1: keys up to 255 characters, the first response saved and returned to retries (even a 500), an error if the same key arrives with different parameters, and keys that can be pruned once they are at least 24 hours old.
That pruning window is a trap for long runs. If an agent waits two days for an approval and then retries a step, the provider may treat the old key as new. For steps that can sit for days, keep your own outbox table: write "intent to send, key K" in the same transaction as the step status, and check it before calling the provider.
When the tool has no idempotency support, you have three choices, in order of preference: check-then-act (search for the ticket by a tag containing the key before creating it), route the call through your own outbox service, or put an approval gate in front of it.
Retries and their limits#
Not every failure deserves a retry. A classification I'd hard-code in the run manager:
| Failure | Action |
|---|---|
| Timeout, 429, 5xx | Retry with exponential backoff and jitter, same idempotency key |
| 4xx validation error | No retry; send to planner with the error |
| Model output fails schema | Retry once with the validation error in the prompt, then replan |
| Policy or permission refusal | Stop; ask a human |
| Same step failed 3 times | Mark failed; replan or stop the run |
Retries multiply cost. A step that retries 3 times with a 20,000-token prompt spends 60,000 tokens to fail. Count retries against the run budget, not outside it.
Human approval gates#
Some steps should never run without a person saying yes: sending money, emailing customers, deleting data, merging to main. The run must pause for hours without holding a process.
In LangGraph that is interrupt(). The interrupts docs say the graph state is saved, the payload goes to the caller, and the run resumes when you invoke with Command(resume=value) on the same thread. On resume, the node restarts from its beginning, so code before the interrupt() call runs again; the docs ask for that code to be idempotent, or for side effects to go after the interrupt or in a separate node. They also warn against wrapping interrupt() in a bare try/except, because it works by raising.
Design rules for the gate itself:
- Show the human the exact tool call and arguments that will run, not a model summary of them.
- Bind the approval to a hash of those arguments. If a replan changes the arguments, the approval is void.
- Give every approval a deadline and a default action (usually "skip and notify").
- Log who approved what, with the run ID.
Budgets, with the math#
A loop with no budget is a cost incident waiting for a bad night. I set three caps per run: max steps, max dollars, max wall-clock time. The run manager checks them before every step and pauses (not fails) when one is hit, so a human can extend it.
The dollar cap needs real numbers. Here is a worked estimate for one run, with my assumptions labelled.
- Model: Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output, from Anthropic's pricing page on 21 Nov 2026.
- 200 steps per run.
- 6,000 tokens of fixed prefix per call (system prompt, tool definitions, plan).
- Each step adds 1,500 tokens to the history (tool result plus the model's reasoning).
- 400 output tokens per step.
Without compaction, step n sends 6,000 + 1,500n tokens. Summed over 200 steps that is 31.05 million input tokens, or $62.10, plus 80,000 output tokens at $0.80. About $63 per run. The last call carries 304,500 tokens of context.
With compaction at a 20,000-token history limit (summarise down to 2,000 tokens, counting the compaction calls themselves), the run uses about 3.77 million input tokens ($7.54) and 110,000 output tokens ($1.10). About $8.64 per run, with 15 compaction calls.
For a daily agent that is about $1,890 a month versus about $260. Prompt caching cuts the input cost further: on this pricing page, a cache hit on Sonnet 5.5 costs 0.05x the base input price. But the cache lasts 5 minutes or 1 hour, depending on which you pay for. A run that waits overnight for approval comes back to a cold cache, so plan the budget for the uncached price.
Change any of the assumptions and the gap moves, but the shape holds: without compaction, input cost grows with the square of the step count.
Context growth over long runs#
The budget math has a quality side. A 300,000-token prompt at step 199 is mostly stale tool output, and the model has to find the current state inside it. Anthropic's context engineering post lists the tools I'd use, cheapest first:
- Tool result clearing. Drop raw outputs of old tool calls once their result is recorded in state. The post calls it one of the safest, lightest forms of compaction.
- Rebuild the prompt from state. Each step's prompt is the goal, the current plan with statuses, the last few results, and a short running summary. The full history stays in the state store for audits.
- Compaction. Summarise the history when it crosses a threshold and start a fresh window with the summary.
- Sub-agents. Give a hard sub-task its own clean context and take back only a condensed result.
Option 2 is the one I'd build first, because the persisted plan already holds most of what the model needs. What the agent should remember across runs (user preferences, lessons from past failures) is a separate problem; I cover it in agent memory system design. If you split work across sub-agents, the trade-offs are in multi-agent architecture patterns.
A small working example#
This loop shows the three ideas that matter most: the plan is persisted before any work starts, each step's result and spend are checkpointed in one transaction, and the side-effecting step uses an idempotency key derived from the run and step. The "model" and the email provider are stubs, so it runs with no API key. Tested with Python 3.15.2, standard library only.
# agent_loop.py - a checkpointed task loop that survives a crash.
# Standard library only. The "model" and the "email API" are stubs.
import json, sqlite3, sys
DB = sys.argv[1] if len(sys.argv) > 1 else "agent.db"
CRASH_AFTER_SEND = "--crash" in sys.argv
BUDGET_USD = 0.50
def connect():
db = sqlite3.connect(DB)
db.executescript("""
CREATE TABLE IF NOT EXISTS steps(run_id TEXT, idx INTEGER, tool TEXT, args TEXT,
status TEXT, result TEXT, PRIMARY KEY(run_id, idx));
CREATE TABLE IF NOT EXISTS runs(run_id TEXT PRIMARY KEY, spent_usd REAL);
-- the fake external service keeps its own idempotency table, like Stripe does
CREATE TABLE IF NOT EXISTS outbox(idem_key TEXT PRIMARY KEY, body TEXT);
""")
return db
def plan(goal):
# Stub planner. A real one is an LLM call that returns this JSON.
return [{"tool": "fetch_tickets", "args": {"since": "2026-11-20"}},
{"tool": "summarise", "args": {}},
{"tool": "send_email", "args": {"to": "team@example.com"}},
{"tool": "mark_done", "args": {}}]
def send_email(db, idem_key, to, body):
# The provider dedupes on the key: a retry returns the first result.
cur = db.execute("INSERT OR IGNORE INTO outbox VALUES (?, ?)", (idem_key, body))
db.commit()
return "sent" if cur.rowcount == 1 else "duplicate ignored"
def run_tool(db, run_id, idx, tool, args, results):
if tool == "fetch_tickets":
return {"tickets": [101, 102, 103]}, 0.01
if tool == "summarise":
n = len(results[0]["tickets"])
return {"text": f"{n} new tickets"}, 0.04 # pretend LLM call cost
if tool == "send_email":
key = f"{run_id}:{idx}" # stable across retries
status = send_email(db, key, args["to"], results[1]["text"])
if CRASH_AFTER_SEND:
print(f" step {idx} {tool}: {status}, then the process dies")
raise SystemExit(1) # side effect done, checkpoint not written
return {"status": status}, 0.0
return {"ok": True}, 0.0
def main(run_id="daily-digest-2026-11-21"):
db = connect()
if not db.execute("SELECT 1 FROM runs WHERE run_id=?", (run_id,)).fetchone():
db.execute("INSERT INTO runs VALUES (?, 0)", (run_id,))
for i, s in enumerate(plan("email a ticket digest")):
db.execute("INSERT INTO steps VALUES (?,?,?,?,'pending',NULL)",
(run_id, i, s["tool"], json.dumps(s["args"])))
db.commit()
print("new run, plan saved")
else:
print("resuming existing run")
rows = db.execute("SELECT idx, tool, args, status, result FROM steps "
"WHERE run_id=? ORDER BY idx", (run_id,)).fetchall()
results = [json.loads(r[4]) if r[4] else None for r in rows]
for idx, tool, args, status, _ in rows:
if status == "done":
print(f" step {idx} {tool}: already done, skipped")
continue
spent = db.execute("SELECT spent_usd FROM runs WHERE run_id=?", (run_id,)).fetchone()[0]
if spent >= BUDGET_USD:
print(" budget exhausted, pausing for a human"); return
result, cost = run_tool(db, run_id, idx, tool, json.loads(args), results)
results[idx] = result
# checkpoint: result and spend committed in one transaction
db.execute("UPDATE steps SET status='done', result=? WHERE run_id=? AND idx=?",
(json.dumps(result), run_id, idx))
db.execute("UPDATE runs SET spent_usd = spent_usd + ? WHERE run_id=?", (cost, run_id))
db.commit()
print(f" step {idx} {tool}: {result}")
sent = db.execute("SELECT COUNT(*) FROM outbox").fetchone()[0]
print(f"run complete, emails in provider outbox: {sent}")
if __name__ == "__main__":
main()
First run with a simulated crash right after the email goes out, then a normal run against the same database:
$ python3 agent_loop.py agent.db --crash
new run, plan saved
step 0 fetch_tickets: {'tickets': [101, 102, 103]}
step 1 summarise: {'text': '3 new tickets'}
step 2 send_email: sent, then the process dies
exit code 1
$ python3 agent_loop.py agent.db
resuming existing run
step 0 fetch_tickets: already done, skipped
step 1 summarise: already done, skipped
step 2 send_email: {'status': 'duplicate ignored'}
step 3 mark_done: {'ok': True}
run complete, emails in provider outbox: 1
The resumed run skips the two finished steps, which matters because step 1 stands for a paid model call. Step 2 runs again, exactly the gap from the idempotency section, and the provider's key table turns the second send into a no-op. One email went out.
Two things in this toy would break in production. The fake provider shares a database with the agent; a real one doesn't, which is why the key matters. And results is rebuilt from stored JSON, so every tool result must be serialisable. Put large outputs in blob storage and store a pointer.
Failure modes#
| Failure | What you see | Prevention |
|---|---|---|
| Loop | Same tool, same args, again and again | Hash (tool, args) per run; stop after 3 repeats; step cap |
| Stale plan | Steps act on data that changed hours ago | Re-read inputs before side effects; replan after long waits; plan version on approvals |
| Duplicate side effect | Two emails, two refunds | Idempotency key per (run, step); outbox for long waits |
| Twin runs | Two runs for one trigger | Run ID derived from trigger; unique constraint |
| Lost run | Process died before first checkpoint | Write the run row before starting; LangGraph's docs suggest an outbox row for this |
| Non-determinism on replay | Temporal errors after a deploy | Model calls only in Activities; workflow versioning |
| Silent budget burn | Run "succeeds" at 10x expected cost | Per-run dollar cap; alert on cost per run |
| Context rot | Quality drops late in long runs | Rebuild prompt from state; compaction thresholds |
| False completion | Agent reports done with work missing | Verify against the stored plan, not the model's claim |
The last row is the one Anthropic's harness post calls out for coding agents: the model declared the project finished early. A done-check that reads the plan table, and ideally runs a test, catches it.
Observability#
Logs per request aren't enough when one run spans two days. I'd emit:
- One trace per run, with the run ID as the trace ID and a span per step, so you can open one run and see the whole timeline, including approval waits.
- Per step: tool, args hash, attempt number, latency, input and output tokens, cost, plan version.
- Per run: status, steps done out of planned, replans, retries, total cost, wall-clock time, time spent waiting on humans.
- Alerts: runs stuck in one status past a threshold, cost per run above the 95th percentile, retry rate above normal, any approval past its deadline.
A dashboard of runs by status (running, waiting, failed, done) answers most on-call questions faster than log search.
Evaluation#
Evaluating a long-running agent means testing the harness as well as the model. My test set has three parts.
Task outcomes. A fixed set of goals with known end states, run in a sandbox with fake tools. Measure task success rate, steps per task, cost per task, and replans per task. My post on evaluating an AI agent in Python shows the harness pattern.
Fault injection. Kill the worker at random steps, make tools time out, return 500s, and delay approvals past their deadline. Assert three invariants after every scenario: the run finishes or pauses cleanly, each side effect happened exactly once (count rows in the fake provider), and spend stayed under the cap. The crash test in the example above is the smallest version of this.
Long-horizon quality. Compare success on 10-step tasks with 100-step tasks. If success falls off a cliff past a step count, the compaction policy or the prompt rebuild is losing information. Inspect what the model saw at the failing step, which you can, because the prompt is rebuilt from stored state.
Run fault injection in CI on every harness change. It's cheap, because the model is stubbed.
Cron script, workflow engine or agent framework#
| Cron + script | Workflow engine (Temporal and similar) | Agent framework with checkpointer (LangGraph and similar) | |
|---|---|---|---|
| Good for | Fixed steps, one run under a few minutes | Business-critical runs, many side effects, waits of days | Model-driven control flow, human-in-the-loop, quick iteration |
| Crash recovery | Rerun from scratch | Replay from event history | Resume from last checkpoint |
| Retries | You write them | Built in for Activities | You configure them |
| Waiting on humans | Poll a table | Workflow messages and timers | interrupt() and resume |
| Main code constraint | None | Deterministic workflow code | Idempotent nodes |
| Ops cost | Lowest | Highest: a cluster or managed service | A Postgres database |
| Pick it when | The plan is the same every day and reruns are safe | Losing or duplicating a step costs money | The plan changes per run and you iterate on prompts weekly |
The columns combine. A common shape is a LangGraph agent for the reasoning, inside a Temporal Activity or behind a queue, with cron as the trigger. I'd start with the agent framework and a Postgres checkpointer, add idempotency keys from day one, and move the orchestration to a workflow engine when the number of side-effecting steps or the cost of a lost run makes the extra cluster worth running.
If you are preparing for interviews, my agentic system design interview guide covers the wider set of agent designs.
Sources#
- LangGraph, Checkpointers: super-step checkpoints, pending writes, durability modes.
- LangGraph, Interrupts: pausing,
Command(resume=...), node restart on resume. - Temporal, Workflow definition: determinism and replay.
- Temporal, Activity definition: idempotency and Run ID plus Activity ID as a key.
- Temporal, Retry policies, Workflow execution limits and Continue-As-New.
- Stripe, Idempotent requests.
- GitHub, Best practices for using webhooks.
- Kubernetes, CronJob.
- Anthropic, Effective harnesses for long-running agents, 2025.
- Anthropic, Effective context engineering for AI agents, 2025.
- Anthropic, Pricing, checked 21 Nov 2026.
- Xu et al., ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models, 2023.