GyaanStack. Blog

Articles / agentsgenai-system-design

Multi-agent architecture patterns: 8 designs, their cost, and how each one breaks

Supervisor, pipeline, hierarchical, fan-out, handoff, critic loop and blackboard, each with a diagram, LLM-call cost, failure modes and how to test it.

By Raghunath · · 13 min read

Many guides to multi-agent patterns give you a list of names and a diagram for each. That helps you recognise a pattern. It doesn't help you choose one, because the choice comes down to three questions the lists skip: how many LLM calls does a task cost, where does information get lost, and how do you test it.

This article covers eight patterns, starting from the one you should try first, a single agent. For each one I give a diagram, how control and state move, when it fits, a rough cost in LLM calls, latency, the ways it fails, and what to check in an eval. The cost numbers are my own estimates from the structure of each pattern, and I show the reasoning so you can redo them for your task.

Two numbers to keep in mind#

Tokens multiply. Anthropic's write-up of its multi-agent research system reports that, in their data, agents use about 4 times the tokens of a chat interaction and multi-agent systems about 15 times (Anthropic, "How we built our multi-agent research system"). Every agent re-reads its own context on every call, and an orchestrator re-reads the results of all its workers.

Errors compound. If each step in a chain succeeds independently with probability p, the whole chain succeeds with pⁿ. With p = 0.95:

Steps (n) 0.95ⁿ Chance the run finishes clean
5 0.774 about 3 in 4
10 0.599 about 3 in 5
20 0.358 about 1 in 3

Real steps aren't independent and some errors get corrected later, so treat this as a direction rather than a forecast. Every extra agent and every handoff adds steps, and a design that adds steps needs a way to catch errors, or reliability drops fast.

A Berkeley study of seven open-source multi-agent frameworks annotated 200 conversation traces and grouped the failures into 14 modes in three groups: system design or specification problems, misalignment between agents, and weak task verification (Cemri et al., "Why Do Multi-Agent LLM Systems Fail?"). Their point is that many failures come from the architecture, not the model. That's why the pattern choice matters.

0. The baseline: one agent with tools#

user ──> [ agent ] <──> tools (search, db, code, APIs)
             │
             └──> answer

Control and state. One loop. The model decides the next tool call, your code runs it, the result goes back into the same context. All state lives in one message history.

When it fits. Most tasks. If the job fits in one context window and needs one set of instructions, a single agent with good tools is cheaper, faster and easier to debug than any multi-agent design. Anthropic's "Building effective agents" makes the same recommendation: start with the simplest thing that works and add structure only when it measurably helps (Anthropic).

Cost. k calls, where k is the number of tool turns. Typically 3 to 10.

Breaks when. The tool list gets long and the model starts picking the wrong tool, the instructions for different sub-jobs conflict, or the context fills with tool output from earlier steps.

Eval. Golden questions with expected tool calls and arguments, plus a final-answer check. I wrote up a harness for exactly this in Evaluating an AI agent in Python.

Every pattern below is a response to one of those three breaking points. If you can't name which one you hit, stay here.

1. Sequential pipeline#

input ──> [ extract ] ──> [ transform ] ──> [ write ] ──> [ check ] ──> output

Control and state. Your code fixes the order. Each stage gets the previous stage's output, ideally as a typed object, and does one job. This is "prompt chaining" in Anthropic's terms, and strictly it's a workflow, not a set of agents, because no model decides where to go next.

When it fits. The steps are known in advance and always run in the same order: parse a document, extract fields, validate, draft a summary.

Cost. One call per stage, so m calls for m stages. The cheapest pattern here.

Latency. The sum of all stages.

Breaks when. An early stage drops or distorts information and every later stage builds on it. This is the 0.95ⁿ problem in its purest form. A pipeline also can't handle inputs that need a different path.

Eval. Contract tests between stages: does each stage's output match the schema the next stage expects? Then score each stage on its own golden set, so a drop in the final score points to a stage.

2. Supervisor (orchestrator and workers)#

                 ┌──> [ worker: search ] ──┐
user ──> [ supervisor ] ──> [ worker: sql ] ──┼──> [ supervisor ] ──> answer
                 └──> [ worker: writer ] ──┘

Control and state. A supervisor model reads the request, decides which worker to call and with what instructions, reads the result, and repeats until it can answer. Workers usually see only the instructions the supervisor writes for them, not the full conversation. In LangGraph, the langgraph-supervisor library implements this with handoff tools (langgraph-supervisor). The OpenAI Agents SDK calls the same idea "agents as tools", where a manager agent keeps ownership of the reply (OpenAI orchestration guide).

When it fits. The request can need different specialists, you can't know which in advance, and you want one place that owns the final answer.

Cost. Supervisor turns r plus each worker's turns. With 3 workers at 4 turns each and 4 supervisor turns: 4 + 12 = 16 calls. The supervisor's context grows with every worker result it reads, so its later calls are the expensive ones.

Latency. Sequential worker calls add up. Calling workers in parallel helps, but then you have pattern 4 inside pattern 2.

Breaks when. - The supervisor routes to the wrong worker. This is the most common failure, and every later step inherits it. - The supervisor's instructions to a worker leave out context the worker needed. The worker does a reasonable job on the wrong problem. - The supervisor loops, calling the same worker again because the result didn't meet a bar it never stated.

Eval. Routing accuracy on a labelled set (request → correct worker), a cap on supervisor turns with an alert when it's hit, and a check that each worker call's instructions contain the facts the worker needs.

3. Hierarchical#

[ top supervisor ]
   ├──> [ team lead: research ] ──> workers...
   └──> [ team lead: build ]    ──> workers...

Control and state. Supervisors of supervisors. Each level summarises its team's work before passing it up.

When it fits. Rarely. When one supervisor would need so many workers that routing accuracy drops, grouping them into teams can help. Large research or software tasks are the usual examples.

Cost. Multiplies. Two levels with 2 team leads, each running the 16-call supervisor setup above, plus 4 top-level turns: 4 + 2 × 16 = 36 calls.

Breaks when. Each summary is lossy. A detail a worker found can disappear two levels up, and the top supervisor makes decisions on a summary of a summary. Debugging also gets hard, because a bad answer can come from any level.

Eval. Trace every level. Check that facts found by workers reach the final answer: a simple "fact survival" test where you plant known facts in worker results and check the top-level output.

4. Parallel fan-out and fan-in#

            ┌──> [ worker A ] ──┐
[ planner ] ├──> [ worker B ] ──┼──> [ synthesiser ] ──> output
            └──> [ worker C ] ──┘

Control and state. A planner splits the task into independent parts, workers run at the same time on their own contexts, and a synthesiser merges the results. Anthropic describes both a fixed version ("parallelization", where your code splits the work) and a dynamic one ("orchestrator-workers", where a model decides the split) (Anthropic).

When it fits. The parts don't depend on each other: researching several companies, reviewing several files, or searching several sources. It's the pattern where multi-agent has the clearest benefit, because each worker gets a clean context and the work happens in parallel.

Cost. 1 planning call, w workers × t turns, 1 synthesis call. With 4 workers at 5 turns: 1 + 20 + 1 = 22 calls.

Latency. Planner + the slowest worker + synthesiser. Much better than sequential, which is the main reason to use it.

Breaks when. - The parts weren't independent after all. Two workers make conflicting choices that the synthesiser can't reconcile. Cognition's post "Don't Build Multi-Agents" makes this argument for coding: actions carry implicit decisions, and parallel agents that can't see each other's decisions produce parts that don't fit together. - Workers duplicate effort because the planner's split overlapped. - The synthesiser gets more text than it can use well and drops findings.

Eval. Coverage (did the workers together cover what the task needed?), duplication rate between workers, and whether the final output cites each worker's key findings.

5. Handoff (swarm)#

[ triage agent ] ──handoff──> [ billing agent ] ──handoff──> [ refunds agent ]
                                      ↑                              │
                                      └─────────handoff──────────────┘

Control and state. No central controller. The active agent decides to pass control to another agent, along with the conversation or a filtered part of it. In the OpenAI Agents SDK, handoffs are exposed to the model as tools and can carry structured data or a filtered history (OpenAI Agents SDK, handoffs). LangGraph's langgraph-swarm library also remembers which agent was last active, so the next user message goes back to it (langgraph-swarm).

When it fits. Conversations where a specialist should take over the reply: customer support that moves from triage to billing to a refund specialist. The OpenAI guide draws the line this way: use a handoff when the specialist should own the next response, and agents-as-tools when it should only help behind the scenes.

Cost. The sum of the turns of each agent that handles the conversation. No supervisor overhead, so it's often cheaper than pattern 2 for the same task.

Latency. Lower than a supervisor, since there's no extra routing call between steps.

Breaks when. - Agents hand back and forth. A asks B, B decides it's A's job. Put a cap on handoffs per conversation. - The receiving agent gets too little history and asks the user to repeat themselves, or too much and gets confused by the previous agent's instructions. - Nobody owns the overall goal, so a conversation can drift.

Eval. Handoff correctness on labelled conversations (should a handoff happen here, and to whom?), handoffs per conversation, and how often the user has to repeat information after a handoff.

6. Critic loop (generator and evaluator)#

[ generator ] ──draft──> [ critic ] ──feedback──┐
      ↑                                         │
      └─────────────────────────────────────────┘
                     (until pass or N rounds)

Control and state. One model drafts, another checks the draft against criteria and returns feedback, the generator revises. Anthropic calls this "evaluator-optimizer" and says it works best when the criteria are clear and revisions give a measurable gain (Anthropic). A debate setup, where several agents argue and a judge decides, is a variant with more calls.

When it fits. Outputs you can check better than you can generate: code (run the tests), SQL (run it against a schema), translations or summaries against a rubric.

Cost. 2 calls per round × rounds. With up to 3 rounds: up to 6 calls per artifact.

Breaks when. - The critic has no ground truth and approves wrong drafts, or rejects good ones, so the loop runs to the cap every time. - Generator and critic are the same model with similar prompts and share the same blind spots.

Eval. Measure the critic on its own first: show it known-good and known-bad drafts and check agreement with a human label. If the critic is wrong 20% of the time, the loop can't be better than that. Also track rounds per task; if almost every task hits the cap, the criteria are unclear. Where you can, use a programmatic check (tests, a SQL parser, a schema validator) as the critic instead of a model. I do that in Text-to-SQL with guardrails.

7. Blackboard (shared state)#

            ┌───────────────────────────┐
[ agent A ]─┤                           ├─[ agent C ]
            │   shared board / store    │
[ agent B ]─┤  (facts, tasks, status)   ├─[ controller ]
            └───────────────────────────┘

Control and state. Agents read from and write to a shared store instead of messaging each other. A controller, or the agents themselves, decide who acts next based on what's on the board. Shared graph state in LangGraph, where every node reads and updates one state object, is a structured version of this.

When it fits. Long-running work where several agents add to a common picture over time, and where you need the full state to be inspectable and resumable.

Cost. Hard to bound, because how many times agents act depends on the board. Cap it explicitly.

Breaks when. Two agents write conflicting facts and nothing resolves the conflict. Agents act on stale reads. The board grows until no agent can read all of it.

Eval. Check the board's final state against expected facts, count conflicting writes, and replay saved boards to reproduce bugs.

The decision table#

Pattern Use when LLM calls (example) Latency Main failure First eval to write
Single agent Default 3 to 10 Low Wrong tool as tools grow Tool-call golden set
Pipeline Fixed steps, same order 1 per stage Sum of stages Early error spreads Stage contract tests
Supervisor Specialists, path unknown ~16 High Misrouting Routing accuracy
Hierarchical Too many workers for one supervisor ~36 Highest Lossy summaries Fact survival
Fan-out Independent parts ~22 Slowest worker Conflicting parts Coverage and conflicts
Handoff Specialist owns the reply Sum of agents' turns Lower Ping-pong handoffs Handoff correctness
Critic loop Output is checkable Up to 6 per artifact Rounds × 2 Bad critic Critic agreement
Blackboard Long-running shared work Unbounded without a cap Varies Conflicting writes Final-state check

The call counts use the example sizes from each section. Redo them with your own numbers of workers and turns before you commit.

When to split a single agent#

My rule: start with one agent, and split only when you can point to one of these in your eval results or traces.

  1. The tool list has grown and tool selection accuracy has dropped. Split by domain, and give each agent a smaller tool set. A supervisor or handoffs follow from that.
  2. Instructions conflict. The prompt for one sub-job hurts another. Separate agents get separate prompts.
  3. Context overflows or gets noisy. Tool output from step 2 drowns out what step 7 needs. Give sub-tasks their own context, as in fan-out.
  4. Independent parts are slow in sequence. Fan-out cuts latency.
  5. You can check the output better than you can produce it. Add a critic, preferably a programmatic one.

If none of these show up, the multi-agent version will most likely cost more, run slower and fail in more places than the agent you already have.

What to log, whatever the pattern#

  • One trace per task with a span for every model call, tool call and handoff, tagged with the agent's name.
  • Calls and tokens per task, so a routing loop shows up as a cost spike before users complain.
  • A hard cap on total model calls per task, and what happened when it was hit.
  • The exact instructions each agent received. When an agent does the wrong thing, check this first: often it was never told the fact it needed.

In CareerPilot I used a router with specialist agents and a human approval step before anything was sent. If you want to try fan-out with a reviewer loop on your own laptop, the multi-agent research assistant runs with LangGraph and Ollama. For how these patterns come up in interviews, see Agentic system design: 5 designs.

Sources#