The AI agent engineering cheat sheet: loop, tools, memory, evals and deployment
One page for building AI agents: the loop and its stop rules, tools, memory, planning, multi-agent patterns, evals, guardrails, cost math and deployment.
An agent is a model called in a loop. Each turn it either asks for a tool or gives an answer, your code runs the tool, and the result goes back into the next call. Everything else on this page (tool design, memory, planning, evals, guardrails) exists to make that loop stop at the right time with the right answer, at a cost you can afford.
Use this sheet as a reference while you build. Each section gives the defaults I start with and links to a longer post when a topic needs one. The symptom-to-fix table near the end is the part to come back to when an agent misbehaves.
The loop in one picture#
+-------------------------------------------+
| |
task --> [ model call ] --tool call--> [ your code runs tool ] --result--+
| |
| no tool call +-- error? return short text,
v count it, maybe stop
final answer
stop when: answer | step limit | repeated call | error limit | budget spent | human says no
The model never runs anything. It emits a structured request (tool name plus JSON arguments) and your code decides whether to run it. That boundary is where permissions, validation, logging and cost limits live.
1. Workflow or agent?#
Anthropic's Building effective agents separates workflows (your code fixes the order of LLM calls) from agents (the model picks the next step). Pick the least autonomy that solves the task.
| Situation | Use |
|---|---|
| Steps are known and fixed (extract, then classify, then write) | Prompt chain, no loop |
| Input falls into a few known categories | Router to a specialised prompt |
| Several independent checks on the same input | Parallel calls, then merge |
| Steps depend on what earlier steps found | Agent loop |
| Open-ended task, hours long, many tools | Agent with durable state and checkpoints |
Reliability compounds badly. If each step succeeds 95% of the time and the steps are independent, a 10-step run succeeds about 60% of the time (0.95^10 = 0.599). Fewer model-decided steps is usually the cheapest reliability gain you will find. For the full map of agent shapes, see AI agent architectures explained.
2. The loop and its stop rules#
Every loop needs more than one way out. A step limit alone lets an agent burn ten identical calls before it gives up.
| Stop rule | Default I start with | Catches |
|---|---|---|
| Final answer (no tool call) | Always on | Normal finish |
| Step limit | 5 to 15 for chat tasks, higher for long jobs | Runaway loops |
| Repeated identical call | Stop on the second identical (tool, args) pair | The model re-asking the same thing |
| Tool error limit | 2 to 3 errors per run | Wrong tool, bad arguments, broken dependency |
| Token or money budget | Per-run cap, logged | Cost blowups from long contexts |
| Human approval | Before any write, payment or send | Irreversible actions |
A runnable stub loop#
This runs with plain Python 3 and no API key. The model is a stub that replays scripted replies, so you can test the loop's control flow on its own. The comment inside ScriptedModel.chat marks where a real provider call goes. The script runs three cases: an agent that recovers from a bad argument, one that loops, and one that keeps calling a tool that does not exist.
import json
# ---- tools: name -> (function, JSON schema the model sees) ----
def get_order(order_id: str) -> dict:
"""Look up one order by id, for example 'A17'."""
orders = {"A17": {"status": "shipped", "eta_days": 2}}
if order_id not in orders:
raise ValueError(f"no order {order_id}; ids look like 'A17'")
return orders[order_id]
TOOLS = {
"get_order": (get_order, {
"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"],
}),
}
# ---- model stub: replays scripted replies so this runs with no API key ----
class ScriptedModel:
def __init__(self, replies):
self.replies = iter(replies)
def chat(self, messages, tools):
# Real version: call your provider here with messages + tools and
# map its tool-call block to {"tool": name, "args": {...}} or {"text": ...}.
return next(self.replies)
def run_agent(model, task, max_steps=5, max_tool_errors=2):
messages = [{"role": "user", "content": task}]
schemas = [{"name": n, "input_schema": s} for n, (_, s) in TOOLS.items()]
errors, seen = 0, set()
for step in range(1, max_steps + 1):
reply = model.chat(messages, schemas)
messages.append({"role": "assistant", "content": reply})
if "tool" not in reply: # no tool call: final answer
return {"answer": reply["text"], "steps": step}
name, args = reply["tool"], reply["args"]
key = (name, json.dumps(args, sort_keys=True))
if key in seen: # same call twice: it is looping
return {"answer": f"Stopped: repeated call {name}{args}.", "steps": step}
seen.add(key)
try:
if name not in TOOLS:
raise LookupError(f"unknown tool {name}; available: {', '.join(TOOLS)}")
result = {"ok": True, "data": TOOLS[name][0](**args)}
except Exception as e: # short error the model can act on
errors += 1
result = {"ok": False, "error": f"{type(e).__name__}: {e}"}
print(f" step {step}: {name}({json.dumps(args)}) -> {json.dumps(result)}")
messages.append({"role": "tool", "name": name, "content": json.dumps(result)})
if errors > max_tool_errors:
return {"answer": "Stopped: too many tool errors.", "steps": step}
return {"answer": "Stopped: step limit reached.", "steps": max_steps}
if __name__ == "__main__":
runs = {
"recovers": [
{"tool": "get_order", "args": {"order_id": "a17"}}, # wrong case
{"tool": "get_order", "args": {"order_id": "A17"}}, # fixes it from the error
{"text": "Order A17 has shipped and should arrive in 2 days."},
],
"loops": [{"tool": "get_order", "args": {"order_id": "A17"}}] * 10,
"bad tool": [{"tool": "refund", "args": {"order_id": f"A{i}"}} for i in range(10)],
}
for label, script in runs.items():
print(label)
out = run_agent(ScriptedModel(script), "Where is my order a17?")
print(f" -> {out['answer']} (steps={out['steps']})")
Output:
recovers
step 1: get_order({"order_id": "a17"}) -> {"ok": false, "error": "ValueError: no order a17; ids look like 'A17'"}
step 2: get_order({"order_id": "A17"}) -> {"ok": true, "data": {"status": "shipped", "eta_days": 2}}
-> Order A17 has shipped and should arrive in 2 days. (steps=3)
loops
step 1: get_order({"order_id": "A17"}) -> {"ok": true, "data": {"status": "shipped", "eta_days": 2}}
-> Stopped: repeated call get_order{'order_id': 'A17'}. (steps=2)
bad tool
step 1: refund({"order_id": "A0"}) -> {"ok": false, "error": "LookupError: unknown tool refund; available: get_order"}
step 2: refund({"order_id": "A1"}) -> {"ok": false, "error": "LookupError: unknown tool refund; available: get_order"}
step 3: refund({"order_id": "A2"}) -> {"ok": false, "error": "LookupError: unknown tool refund; available: get_order"}
-> Stopped: too many tool errors. (steps=3)
Three things in that output are worth copying into a real agent. The error message in the first case tells the model what a valid id looks like, so the second call fixes itself. The loop case stops after 2 steps instead of 5 because of the repeated-call check. The unknown-tool case lists the tools that do exist, and the error limit ends the run after three misses. Scripted models like this one also make good unit tests for the loop: they check your stop logic without spending a single token.
3. Tools#
The model only sees a tool's name, description and argument schema, so that text is your interface. Anthropic's guide on writing tools for agents and the Claude tool use docs both stress this; the OpenAI function calling guide follows the same request-and-result shape.
Rules I follow:
- Name tools by the job (
search_orders, notdb_query). Describe when to use each one and give one example argument. - Type every argument and use enums where the values are fixed. Validate arguments in code before running anything.
- Return the few fields the next step needs. A 4,000-token JSON dump costs money on every later turn because it stays in the context.
- Return errors as short text the model can act on, as in the stub above. A stack trace teaches it nothing.
- Keep the tool list small per agent. When two tools overlap, the model picks between them at random.
- Make write tools idempotent (pass a request id) so a retry does not create two refunds.
To expose tools once and reuse them across clients, use the Model Context Protocol. The spec defines stdio for local servers and Streamable HTTP for remote ones. The step-by-step builds are in function calling from scratch and MCP from zero.
4. Memory#
| Kind | What it holds | Where it lives | Watch out for |
|---|---|---|---|
| Working memory | The current message list | The context window | Grows every turn; trim or summarise old tool results |
| Session state | Task variables, plan, progress | Your app's state object or a checkpoint store | Must survive a crash if the run is long |
| Long-term facts | User preferences, past decisions | A database or vector store, read through a tool | Stale or wrong facts get repeated forever |
| Knowledge | Documents the agent searches | A retrieval index | Treat it as a tool; see the RAG cheat sheet |
Defaults: keep the last few turns verbatim, replace older tool outputs with one-line summaries, and write long-term facts only when the user states them or a tool confirms them. The design trade-offs (what to write, when to forget, how to retrieve) are in agent memory system design. For retrieval as a tool the agent decides to call, see agentic RAG in Python.
5. Planning#
| Pattern | How it works | Use when | Cost |
|---|---|---|---|
| ReAct | Think, act, observe, repeat (Yao et al., 2022) | Next step depends on the last result | One model call per step |
| Plan then execute | One call writes a plan; steps run in order; replan on failure | The route is mostly known up front | Planner call plus cheaper step calls |
| Reflection | A second pass critiques the output and the agent retries | Output can be checked (tests, schema, rubric) | Roughly doubles calls per attempt |
A practical split is a strong model for the plan and a smaller, cheaper model for the simple steps. Store the plan in state, not only in the prompt, so you can show it to a human and resume from it.
6. Multi-agent patterns#
| Pattern | Shape | Good for |
|---|---|---|
| Supervisor and workers | One agent routes subtasks to specialists | Distinct skills with separate tool sets |
| Sequential handoff | A finishes, B takes over | Pipelines like research, then write, then review |
| Parallel fan-out | Several agents work at once, one merges | Breadth-first research over many sources |
Start with one agent. Split only when one prompt carries too many tools or too many jobs, and you can measure that the split helps. Anthropic reported in its multi-agent research system write-up that agents used about 4 times the tokens of chat interactions and multi-agent systems about 15 times. Every handoff is also a place where context gets lost. The patterns with code and their failure modes are in multi-agent architecture patterns.
7. Evals#
Judge an agent on the end state and on the path it took to get there.
- Task success. Did the final state match the goal? Check it in code where you can: the right row in the database, the test passing, the file written.
- Trajectory checks. Right tools called, no forbidden tools, number of steps, number of errors.
- Reliability over repeats. Run each task several times. The tau-bench paper proposed pass^k, the chance that all k runs of a task succeed. A task that passes 90% of single runs passes all 8 of 8 runs only about 43% of the time if runs are independent (0.9^8 = 0.43). Users see that variance as "it worked yesterday."
- Cost and latency per task. Track tokens, tool calls and wall-clock time next to the pass rate, so a prompt change that wins 2 points of accuracy but doubles cost is visible.
Build the set from real tasks, include ones that should end in "I can't do that," and rerun it on every prompt, model or tool change. A runnable harness is in evaluating an AI agent in Python.
8. Guardrails#
Simon Willison's lethal trifecta is the quickest risk check I know. An agent that has access to private data, reads untrusted content (web pages, emails, uploaded files) and can send data out can be made to leak that data through prompt injection. Remove at least one of the three for any agent you cannot watch.
| Layer | Control |
|---|---|
| Input | Mark untrusted content clearly; never put it in the system prompt |
| Tool permissions | Allow-list per agent; read-only by default; scoped credentials per user |
| Arguments | Validate schema and values in code (amount limits, allowed domains, path roots) |
| Actions | Human approval for writes, sends, payments and deletes |
| Output | Check for secrets and PII before showing or sending |
| Monitoring | Log every tool call with arguments and result; alert on unusual patterns |
The OWASP Top 10 for LLM applications lists prompt injection and excessive agency among its risks. The layered design is in LLM guardrails and observability, and a test suite to attack your own agent is in LLM red teaming with prompt injection tests.
9. Cost and latency#
An agent resends its whole context on every step, so cost grows with steps times context size. Prompt caching cuts the repeated part. On the Claude prompt caching page, 5-minute cache writes cost 25% more than base input and cache reads cost 0.1 times base input for most models.
A worked example at $3 per million input tokens: a 10-step run with an 8,000-token prefix (system prompt plus tool schemas) that stays the same each step.
| Prefix tokens billed | Prefix cost | |
|---|---|---|
| No caching | 10 x 8,000 = 80,000 at $3/M | $0.240 |
| Caching | 8,000 written at $3.75/M + 72,000 read at $0.30/M | $0.052 |
That is about 78% off the prefix for this run. The growing tail of tool results is not covered, which is one more reason to keep tool outputs short. Other levers, in the order I try them:
- Fewer steps (a workflow instead of an agent, better tool errors).
- Shorter tool results and summarised history.
- A smaller model for routing and simple steps.
- Parallel tool calls when the steps do not depend on each other.
- Streaming the final answer so the user sees progress.
Log tokens and latency per step from day one. A small tracker is in LLM cost and latency tracker in Python.
10. Deployment#
- Durable state. Save state after every step so a crash resumes instead of restarting. LangGraph does this with checkpointers keyed by a thread id.
- Human in the loop. Pause before risky steps and resume with the human's decision. LangGraph calls this an interrupt; you can build the same with a status column and a queue.
- Timeouts and retries. Per-tool timeout, retry with backoff only for idempotent calls, and a wall-clock limit per run.
- Tracing. One trace per run with a span per model call and per tool call. The OpenTelemetry GenAI semantic conventions give standard attribute names, so your traces work across vendors.
- Versioning. Store the prompt version, model name and tool versions on every trace so you can explain a bad run a week later.
- Packaging. Run the agent as a service with its tools behind it; see dockerize an AI agent.
Runs that last hours, survive restarts and wait on humans need more than this list. That design is in long-running AI agent system design.
When the agent goes wrong#
| Symptom | Likely cause | Fix |
|---|---|---|
| Calls the same tool again and again | No repeated-call check; tool result unclear | Stop on repeat; make the result say what changed |
| Picks the wrong tool | Overlapping names or vague descriptions | Rename by job, add "use when" lines, cut tools |
| Invents tool arguments | Schema too loose | Enums, required fields, validate in code |
| Gives up after one error | Error text is a stack trace | Return a short, actionable message |
| Answer ignores the tool result | Result buried in a large JSON blob | Return only the needed fields |
| Gets slower and pricier each step | Context grows with raw tool outputs | Summarise old results; cache the stable prefix |
| Works once, fails on rerun | Sampling variance | Measure pass^k; lower temperature; tighten tools |
| Follows instructions found in a web page or email | Prompt injection | Remove one leg of the trifecta; approve actions |
| Restarts from zero after a crash | State only in memory | Checkpoint after each step |
| Multi-agent version is worse than one agent | Context lost at handoffs | Pass structured state; go back to one agent |
Production checklist#
- [ ] Workflow considered first, with a written reason for using an agent
- [ ] Step limit, repeated-call check, error limit and budget cap in the loop
- [ ] Tool schemas typed, arguments validated in code, write tools idempotent
- [ ] Tool results trimmed to what the next step needs
- [ ] Eval set of real tasks, with pass^k and cost per task, run on every change
- [ ] Unit tests for the loop with a scripted model
- [ ] Lethal trifecta checked; human approval on irreversible actions
- [ ] Per-user scoped credentials for every tool
- [ ] Prompt caching on the stable prefix; tokens and latency logged per step
- [ ] State checkpointed after each step; resume tested by killing a run
- [ ] One trace per run with prompt, model and tool versions attached
Build the stub loop first, add one tool, and write the eval set before adding the second.
Sources#
- Anthropic, Building effective agents
- Anthropic, Writing effective tools for agents
- Anthropic, How we built our multi-agent research system
- Claude docs, Tool use overview
- Claude docs, Prompt caching
- OpenAI docs, Function calling
- Model Context Protocol, Specification 2026-07-28
- Yao et al., ReAct: Synergizing reasoning and acting in language models
- Yao et al., tau-bench: a benchmark for tool-agent-user interaction
- OWASP, Top 10 for LLM applications
- Simon Willison, The lethal trifecta for AI agents
- LangGraph docs, Persistence and Interrupts
- OpenTelemetry, Semantic conventions for generative AI