Articles / agentsgenai-system-design
AI agent architectures explained: ReAct, plan-and-execute, reflection and routers
Single-agent architectures compared with diagrams, model calls and latency per task, a decision table and runnable code running two patterns on one task.
An agent architecture is a control-flow decision. Something has to pick the next step: your code, the model, or the model once at the start and your code after that. Each choice changes three numbers you will feel in production: how many model calls a task takes, how long the user waits, and how a run fails.
This post covers the single-agent patterns: workflows versus agents, the ReAct tool loop, plan-and-execute, reflection, and routers. Each gets a diagram and a call count. A decision table follows, then a small runnable program that solves one task two ways and prints the difference. Patterns with more than one agent (supervisor, swarm, blackboard, fan-out) have their own post, Multi-agent architecture patterns, and this one does not repeat them.
Workflows or agents: who owns the control flow#
Anthropic's "Building effective agents" draws the line that most of the field now uses. Workflows are "systems where LLMs and tools are orchestrated through predefined code paths." Agents are systems where LLMs "dynamically direct their own processes and tool usage" (Anthropic).
WORKFLOW (your code owns the order) AGENT (the model owns the order)
input -> [LLM: extract] -> [LLM: summarise] input -> [ LLM ] <-> tools
-> [check] -> output | ^
v |
observe, decide again
|
v
output
The same article lists five workflow patterns: prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer. It says agents suit "open-ended problems where it's difficult or impossible to predict the required number of steps." If you can write the steps down in advance, write them in code. A chain of three fixed calls costs exactly three calls every time, you can unit-test each stage, and it never loops.
Prompt chaining is the plain case: extractor, then summariser, then a format check, each output feeding the next input. Cost is n calls for n stages, latency is the sum of the stages, and failures are local, since you can see which stage produced the bad output. It is the right answer for document generation, ETL over text, and most back-office jobs. The rest of this post is about what to do when the steps cannot be fixed in advance.
Pattern 1: the tool loop (ReAct)#
ReAct comes from Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (ICLR 2023). The model alternates a reasoning trace with an action, reads the observation, and repeats. On ALFWorld and WebShop the paper reports absolute success-rate gains of 34 and 10 points over imitation and reinforcement learning baselines (arXiv 2210.03629).
+-----------------------------+
v |
task -> [ LLM: think + pick action ] -> tool -> observation
|
+-- "final answer" --> output
Modern tool-calling APIs make this the default agent loop. The model returns a structured tool call instead of a text "Action:" line, your code runs it, and the result goes back as a tool message. The written "Thought:" step is optional now; reasoning models do that internally.
Calls. One model call per tool step, plus one for the final answer. A task with k tool steps costs k + 1 calls, all sequential, because each call needs the previous observation.
Tokens grow with every step. Each call re-reads the whole history. Take a 2,000-token prompt (system message plus tool schemas) and 300 tokens added per step. Five calls read 2,000 + 2,300 + 2,600 + 2,900 + 3,200 = 13,000 input tokens. Ten calls read 33,500. Prompt caching cuts the price of the repeated prefix, but the latency of one model call per step stays.
Breaks when. There is no step limit and the model retries a failing tool forever. The tool list grows past what the model can choose between reliably. A long run fills the context with stale tool output, and quality drops on later steps.
Use it when the next step depends on what the last tool returned: debugging, search where the first query rarely works, anything exploratory.
Pattern 2: plan-and-execute#
A planner call writes the whole list of steps first. Your code (or a cheaper executor) runs them, and one last call writes the answer from the results.
task -> [ LLM: planner ] -> plan: [s1, s2 | s3, s4]
|
executor runs steps (s1 and s2 in parallel,
then s3 and s4), no LLM between steps
|
error? --yes--> [ LLM: replan ]
|
no
v
[ LLM: solver ] -> output
Two papers measure what this buys. ReWOO plans the tool calls up front with placeholders for their results and reports 5x token efficiency with a 4% accuracy gain on HotpotQA (Xu et al., arXiv 2305.18323). LLMCompiler plans a dependency graph and runs independent calls in parallel; against ReAct it reports up to 3.7x lower latency, up to 6.7x lower cost and up to about 9% higher accuracy (Kim et al., arXiv 2312.04511). These are best-case numbers on the authors' benchmarks, on tasks where the plan holds.
Calls. Two in the good case (planner and solver), however many tool steps there are. Add one replan call each time a step fails or returns something the plan did not expect.
Latency. Independent steps run at the same time, so tool latency becomes the number of dependency levels, not the number of steps.
Breaks when. The plan is wrong and nothing notices. A step returns an empty result, the executor carries on, and the solver writes a confident answer from bad inputs. Check every step result against what the plan expected, and replan on a mismatch. The other failure is a task whose later steps depend on reading earlier results closely, such as "open the file the search returns and fix the bug in it." A plan written before the search cannot say which file.
Use it when the steps are knowable from the request: multi-part lookups, report assembly, data gathering with several independent sources.
Pattern 3: reflection and Reflexion#
Reflection adds a check after an attempt. The agent critiques the output and retries with the critique in the prompt.
task -> [ LLM: attempt ] -> output -> [ check: tests / rubric / LLM critic ]
^ |
| pass? ----+--> done
| | fail
+---- [ LLM: write lesson ] <-------+
(stored in memory, max N rounds)
Reflexion is the named version. Shinn et al. have the agent write a short verbal reflection after a failed trial and keep it in an episodic memory buffer that later trials read. They report 91% pass@1 on HumanEval against 80% for GPT-4 at the time (arXiv 2303.11366). Most of that signal came from running the generated code against tests, which is the part to copy: the check works best when it is a program.
Inside one agent, reflection wraps whatever produces the attempt. Wrapped around a ReAct loop it multiplies that loop's cost by the number of rounds. When generator and critic are split into separate agents, the multi-agent post covers it as the critic loop, including how to evaluate the critic on its own.
Calls. For a one-call attempt with an LLM critic: 2 per round (attempt, critique), so 3 rounds cost up to 6 calls. With a programmatic check (tests, a SQL parser, a JSON schema) the critique is free and only the retries cost model calls.
Breaks when. The check has no ground truth. An LLM grading its own output with a similar prompt tends to share its mistakes, approves wrong answers, and the extra rounds only add cost. Track rounds per task; if almost every task hits the cap, the pass criteria are unclear.
Use it when you can check an answer more reliably than you can produce one: code with tests, SQL against a schema, structured output against a validator.
Pattern 4: the router#
A router classifies the request once and sends it down one of several fixed paths. Anthropic's description: routing "classifies an input and directs it to a specialized followup task," and the classifier can be an LLM or a traditional model (Anthropic).
+--> refund path (chain, 2 calls)
request -> [ router ] -+--> how-to path (RAG, 1 call)
+--> open-ended (ReAct agent, k+1 calls)
+--> unknown -> human
Calls. One cheap classification call plus whatever the chosen path costs. A small model, or a fine-tuned classifier with no LLM call at all, often does the routing well, so the routing step adds little cost or delay.
Why it pays. Each branch gets a short prompt and only the tools it needs. That fixes the two ReAct failure points of too many tools and conflicting instructions, without the coordination costs of several agents. If most traffic is simple, most requests take the cheap path and only the hard ones pay for a loop.
Breaks when. A request fits two categories, or none, and the router guesses. Always include a fallback branch, log the router's choice with every request, and test the router alone on a labelled set before testing anything downstream. A router that is 90% accurate caps the whole system at 90%.
Routing to a branch is not the same as handing off to a peer agent that keeps the conversation. Handoffs and supervisors are multi-agent patterns, covered in the other post.
Calls and latency for one task, side by side#
The task: "Which is cheaper per night in USD, a hotel in Pune or Goa?" It needs four tool calls: two price lookups, two currency conversions. The conversions depend on the lookups; the two lookups do not depend on each other.
Assume 1.5 s per model call and 0.4 s per tool call. These are round figures for illustration, not measurements; replace them with your own p50s from a tracker like the one in LLM cost and latency tracking in Python.
| Pattern | Model calls | Sequential work | Simulated latency |
|---|---|---|---|
| Prompt chain (if the steps are fixed) | 1 (final summary) | 4 tool calls in code + 1 call | 3.1 s |
| ReAct loop | 5 | 5 model calls + 4 tool calls | 9.1 s |
| Plan-and-execute, parallel waves | 2 | 2 model calls + 2 tool waves | 3.8 s |
| ReAct + reflection, 1 retry | 11 | the loop twice + 1 critique | about 20 s |
| Router -> chain branch | 1 classify + branch | classify + branch | branch + about 0.5 s |
The chain row is a reminder: if the four lookups are always the same, code calls the tools and one model call writes the sentence. The ReAct and plan-and-execute rows come from the program below.
Decision table#
| If the task... | Start with | Upgrade to | Watch for |
|---|---|---|---|
| has the same steps every time | prompt chain | nothing | stage-level tests |
| falls into a few known kinds | router + chains | a ReAct branch for the open-ended kind | router accuracy, fallback rate |
| needs steps chosen from tool results | ReAct loop with a step cap | add a router if tools exceed what one prompt handles | loops, stale context |
| has many steps you can list up front | plan-and-execute | parallel waves (LLMCompiler style) | silent bad plans, replan rate |
| has output a program can check | any of the above + reflection | Reflexion memory across attempts | rounds per task, check quality |
| needs separate contexts or parallel research | read the multi-agent post first | token cost |
The order to try them is the order of the table. Each step up adds model calls and new ways to fail, so move up only when an eval shows the simpler version failing.
Runnable code: ReAct versus plan-and-execute on one task#
The program runs the Pune and Goa task both ways. There is no API key: ScriptedModel plays back the replies a model would give, and the comment in complete() marks where the real client call goes. Time is simulated with the same 1.5 s and 0.4 s figures, so the output shows call counts and the latency structure, not real model speed. Standard library only; Python 3.9 or newer.
"""ReAct loop vs plan-and-execute on the same task, with a stub model.
No API key needed. ScriptedModel stands in for the LLM; swap complete()
for a real client call (Anthropic, OpenAI, a local model) to run it live.
Time is simulated: each model call costs LLM_S seconds, each tool call TOOL_S.
"""
import json
from concurrent.futures import ThreadPoolExecutor
LLM_S, TOOL_S = 1.5, 0.4 # assumed per-call latencies, not measurements
# ---- tools -----------------------------------------------------------
PRICES_INR = {"pune": 4200, "goa": 6100} # hotel price per night
def hotel_price(city):
if city not in PRICES_INR:
return {"error": f"no data for {city}"}
return {"city": city, "inr": PRICES_INR[city]}
def inr_to_usd(inr):
return {"usd": round(inr / 84.0, 2)} # fixed demo rate
TOOLS = {"hotel_price": hotel_price, "inr_to_usd": inr_to_usd}
# ---- the model interface ---------------------------------------------
class ScriptedModel:
"""Plays back canned replies. Replace complete() with a real call."""
def __init__(self, replies):
self.replies, self.calls = list(replies), 0
def complete(self, prompt):
# real version: return client.messages.create(...).content[0].text
self.calls += 1
return self.replies.pop(0)
# ---- pattern 1: ReAct (one model call per step) ----------------------
def react(model, task, max_steps=8):
history, clock = [f"Task: {task}"], 0.0
for _ in range(max_steps):
reply = json.loads(model.complete("\n".join(history)))
clock += LLM_S
if "final" in reply:
return reply["final"], model.calls, clock
obs = TOOLS[reply["tool"]](**reply["args"])
clock += TOOL_S
history.append(f"Action: {reply}\nObservation: {obs}")
return "stopped: step limit", model.calls, clock
# ---- pattern 2: plan-and-execute (plan once, run steps, solve once) --
def plan_and_execute(model, task):
clock = 0.0
plan = json.loads(model.complete(f"Plan tool steps for: {task}"))
clock += LLM_S
results = {}
for wave in plan["waves"]: # steps in a wave are independent
def run(step):
args = {k: results[v[1:]]["inr"] if isinstance(v, str) and v.startswith("#") else v
for k, v in step["args"].items()}
return step["id"], TOOLS[step["tool"]](**args)
with ThreadPoolExecutor() as pool:
for sid, out in pool.map(run, wave):
results[sid] = out
clock += TOOL_S # a wave costs one tool latency
if any("error" in r for r in results.values()):
return f"replan needed: {results}", model.calls, clock
answer = model.complete(f"Answer {task} using {results}")
clock += LLM_S
return answer, model.calls, clock
TASK = "Which is cheaper per night in USD, a hotel in Pune or Goa?"
react_model = ScriptedModel([
'{"tool": "hotel_price", "args": {"city": "pune"}}',
'{"tool": "hotel_price", "args": {"city": "goa"}}',
'{"tool": "inr_to_usd", "args": {"inr": 4200}}',
'{"tool": "inr_to_usd", "args": {"inr": 6100}}',
'{"final": "Pune: $50.00 vs Goa: $72.62. Pune is cheaper."}',
])
plan = {"waves": [
[{"id": "p", "tool": "hotel_price", "args": {"city": "pune"}},
{"id": "g", "tool": "hotel_price", "args": {"city": "goa"}}],
[{"id": "pu", "tool": "inr_to_usd", "args": {"inr": "#p"}},
{"id": "gu", "tool": "inr_to_usd", "args": {"inr": "#g"}}],
]}
pe_model = ScriptedModel([json.dumps(plan),
"Pune: $50.00 vs Goa: $72.62. Pune is cheaper."])
for name, fn, m in [("ReAct", react, react_model),
("Plan-and-execute", plan_and_execute, pe_model)]:
answer, calls, secs = fn(m, TASK)
print(f"{name:17} calls={calls} sim_latency={secs:.1f}s -> {answer}")
# failure case: the plan names a city the tool has no data for
bad = {"waves": [[{"id": "p", "tool": "hotel_price", "args": {"city": "pune"}},
{"id": "k", "tool": "hotel_price", "args": {"city": "kochi"}}]]}
answer, calls, secs = plan_and_execute(ScriptedModel([json.dumps(bad)]), TASK)
print(f"{'P&E, bad plan':17} calls={calls} sim_latency={secs:.1f}s -> {answer}")
Output:
ReAct calls=5 sim_latency=9.1s -> Pune: $50.00 vs Goa: $72.62. Pune is cheaper.
Plan-and-execute calls=2 sim_latency=3.8s -> Pune: $50.00 vs Goa: $72.62. Pune is cheaper.
P&E, bad plan calls=1 sim_latency=1.9s -> replan needed: {'p': {'city': 'pune', 'inr': 4200}, 'k': {'error': 'no data for kochi'}}
Same answer, 5 calls against 2, and 9.1 s against 3.8 s. The third line is the part demos skip. The plan asked for a city the tool cannot answer, and the executor stopped at the first wave with a "replan needed" result instead of passing an error to the solver. In a real system that result goes back to the planner with the error text, which costs one more call. A ReAct loop handles the same surprise with no special code, because it reads every observation before choosing the next step. That is the trade in one example: plan-and-execute is cheaper when the plan holds, and ReAct copes better when it does not.
Two shortcuts in the stub would not survive production. The #p placeholder lookup hard-codes the inr field; a real executor needs a declared output schema per tool. And the scripted replies cannot go wrong, which real model output does, so parse tool calls defensively and count malformed replies as failures.
How to test whichever pattern you pick#
Build a set of 30 to 50 tasks with known answers before choosing a pattern, then run every candidate on the same set. For each run, log:
- final-answer correctness (exact match or a rubric)
- model calls, input tokens and wall-clock time per task
- for ReAct: step-limit hits and repeated identical tool calls
- for plan-and-execute: replan rate and steps whose result failed validation
- for reflection: rounds per task and the pass rate of the check itself
- for routers: accuracy on a labelled set and the fallback rate
Pick the cheapest pattern that clears your accuracy bar. Evaluating an AI agent in Python has a harness for the tool-call and final-answer checks. For how these choices come up in interviews, with five full designs, see Agentic system design interviews.
Sources#
- Anthropic, Building effective agents (December 2024)
- Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, ICLR 2023
- Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning
- Xu et al., ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models
- Kim et al., An LLM Compiler for Parallel Function Calling