GyaanStack. Blog

Articles / agentslanggraphpython

Multi-agent workflow automation in Python: parallel agents, QA loops and human approval

Build a LangGraph workflow where two agents run in parallel, a writer and QA agent loop with a cap, a human approves, and failures retry or resume.

By Raghunath · · 19 min read

A business workflow crammed into one prompt fails in ways you can't see. The prompt asks the model to search, pull out numbers, write a report and check its own work, all in one call. When the report has a wrong number, you can't tell which of those four jobs went wrong, and you can't retry only that job.

Splitting the work into agents fixes that, but only if the pieces between the agents are solid: what state they share, what happens when a tool times out, how many times the QA loop may run, and where a person has to sign off before anything leaves the building. This post builds one such workflow in LangGraph and spends most of its time on those pieces.

The workflow produces a market brief on a technology topic:

  • a research agent and an analyst agent run in parallel, one gathering trends and one gathering numbers,
  • a writer agent drafts the brief from both,
  • a QA agent checks the draft and sends it back to the writer, at most once,
  • a human approves or rejects, and only then does a publish step write the file.

If you want a comparison of multi-agent shapes before picking one, Multi-agent architecture patterns covers eight of them with their costs. For a planner that creates a variable number of researchers with Send, see the local research assistant post. This one uses a fixed fan-out and puts the weight on failure handling, approval and tests.

The graph#

            START
           /     \
   research_agent  analyst_agent     (parallel; retry on TimeoutError)
           \     /
         writer_agent  <-----------+
              |                    |
          qa_agent ---- fail, drafts < 2
              |
      pass, or drafts == 2 (flagged)
              |
        human_review   (interrupt: run pauses here)
          /        \
     approve      reject
        |            |
     publish        END
        |
       END

Two choices in that picture matter more than the agent prompts. A draft that fails QA twice still goes to the human, marked needs_attention, rather than being published or dropped. And publish sits after human_review, so no side effect happens until a person says yes.

No API key: the model and the search tool are stubs#

The code below runs with no model and no network. The model sits behind a one-method interface, and a scripted class answers for it. The search tool is a stub that fails its first call with a timeout, so the retry path runs every time you run the demo.

I tested this with Python 3.15.2, langgraph 1.2.24 and pytest 9.2.0.

python -m venv .venv && source .venv/bin/activate
pip install langgraph pytest

stubs.py:

from collections import Counter

SNIPPETS = {
    "news": [
        {"id": "s1", "text": "Phone makers now ship on-device models for summaries and translation."},
        {"id": "s2", "text": "Retail chains run small models on store servers to keep data local."},
    ],
    "statistics": [
        {"id": "s3", "text": "A survey of 400 firms found 38% piloting small models at the edge."},
        {"id": "s4", "text": "Median latency fell to 120 ms after moving inference on device."},
    ],
}

DRAFT_1 = """## Summary
Small models are moving onto devices.
## Trends
On-device summaries and local store servers.
## Numbers
38% of 400 firms are piloting; 60% plan to by 2027."""

DRAFT_2 = """## Summary
Small models are moving onto devices and store servers.
## Trends
Phones ship on-device summaries; retailers keep data local.
## Numbers
38% of 400 surveyed firms are piloting edge models. Median latency is 120 ms on device.
## Actions
Pilot one on-device feature and measure latency against the cloud path."""


class ScriptedModel:
    """Stands in for an LLM. Replace complete() with a real call, e.g.
    ChatOllama(model="llama3.2").invoke(prompt).content"""

    def __init__(self):
        self.calls = Counter()

    def complete(self, role: str, prompt: str) -> str:
        self.calls[role] += 1
        if role == "research":
            return "Inference is moving to phones and store servers."
        if role == "writer":
            return DRAFT_1 if self.calls["writer"] == 1 else DRAFT_2
        if role == "qa":
            return '```json\n{"pass": true, "fix": ""}\n```'
        raise ValueError(role)


class FlakySearch:
    """Fails the first call with a timeout, like a real search API on a bad day."""

    def __init__(self, failures: int = 1):
        self.failures = failures
        self.calls = 0

    def run(self, query: str) -> list[dict]:
        self.calls += 1
        if self.failures > 0:
            self.failures -= 1
            print(f"  [search] timeout on {query!r}")
            raise TimeoutError("search timed out")
        return SNIPPETS["news" if query.endswith("news") else "statistics"]

The first draft is bad on purpose. It is missing the Actions section and it contains two numbers, 60% and 2027, that appear in no search result. That is the most common real failure of a writer model: a plausible forecast with no source.

To run a real model, put the call inside complete(). With Ollama that is pip install langchain-ollama, ollama pull llama3.2, then ChatOllama(model="llama3.2").invoke(prompt).content. For real web search, note that the duckduckgo-search package has been renamed to ddgs; older tutorials still install the old name.

Shared state and the parallel branches#

All of the workflow code below goes in one file, workflow.py. It starts with the interfaces and the state:

import json
import operator
import re
from pathlib import Path
from typing import Annotated, Protocol, TypedDict

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import END, START, StateGraph
from langgraph.types import Command, RetryPolicy, interrupt

MAX_DRAFTS = 2
OUT_DIR = Path("reports")


# ---------- the two things we stub: the model and the search tool ----------

class Model(Protocol):
    def complete(self, role: str, prompt: str) -> str: ...


class Search(Protocol):
    def run(self, query: str) -> list[dict]: ...


# Set by build_app(); nodes read them from here.
MODEL: Model
SEARCH: Search


# ---------- shared state ----------

class State(TypedDict, total=False):
    topic: str
    findings: Annotated[list[dict], operator.add]  # both parallel branches append
    draft: str
    issues: list[str]
    drafts: int
    status: str            # "qa_passed" | "needs_attention" | "approved" | "rejected"
    published: str

The research and analyst agents run in the same step and both write findings. The operator.add annotation tells LangGraph to concatenate their lists. Remove it and the run fails at once. I checked with a two-node graph that both write a plain list key:

InvalidUpdateError At key 'findings': Can receive only one value per step. Use an Annotated key to handle multiple values.

An error is the better outcome here. The other possibility, one branch silently overwriting the other, would leave the writer with half the facts and no warning. The LangGraph docs say the same thing in one line: add a reducer for any key that multiple nodes write.

The two agents:

def research_agent(state: State) -> dict:
    hits = SEARCH.run(f"{state['topic']} news")
    summary = MODEL.complete("research", "Summarise the trends in these snippets:\n"
                             + "\n".join(h["text"] for h in hits))
    return {"findings": [{"kind": "trend", "text": summary, "sources": hits}]}


def analyst_agent(state: State) -> dict:
    hits = SEARCH.run(f"{state['topic']} statistics")
    # Pull numbers out with code, not the model, so none can be invented here.
    facts = [{"number": n, "src": h["id"], "text": h["text"]}
             for h in hits for n in re.findall(r"\d+(?:\.\d+)?%?", h["text"])]
    return {"findings": [{"kind": "metrics", "facts": facts, "sources": hits}]}

The analyst agent makes no model call. Asking a model to "extract three adoption metrics" from search snippets is how made-up statistics get into reports: if the snippets hold one number, the model finds three anyway. A regex can only return what is in the text. Use a model for this step when the numbers are buried in prose a regex can't parse, and then check its output against the source text the same way QA does below.

Each finding keeps its sources. That costs a few hundred tokens in the writer prompt and lets QA check the draft against what was retrieved.

The writer and a QA agent that checks with code first#

def writer_agent(state: State) -> dict:
    prompt = f"Write a brief on {state['topic']} with sections Summary, Trends, Numbers, Actions.\n"
    prompt += json.dumps(state["findings"])
    if state.get("issues"):
        prompt += "\nThe last draft was rejected. Fix: " + "; ".join(state["issues"])
        prompt += "\nLast draft:\n" + state["draft"]
    return {"draft": MODEL.complete("writer", prompt), "drafts": state.get("drafts", 0) + 1}

On a second attempt the writer gets the QA issues and its own previous draft. Without the old draft it starts from scratch and can break sections that were fine.

QA runs deterministic checks before it asks a model anything:

REQUIRED = ["## Summary", "## Trends", "## Numbers", "## Actions"]


def known_numbers(findings: list[dict]) -> set[str]:
    nums = set()
    for f in findings:
        for h in f["sources"]:
            nums.update(re.findall(r"\d+(?:\.\d+)?%?", h["text"]))
    return nums


def rule_checks(draft: str, findings: list[dict]) -> list[str]:
    issues = [f"missing section {s}" for s in REQUIRED if s not in draft]
    invented = set(re.findall(r"\d+(?:\.\d+)?%?", draft)) - known_numbers(findings)
    if invented:
        issues.append(f"numbers not in any source: {sorted(invented)}")
    return issues


def parse_verdict(raw: str) -> dict:
    """Strict: anything that is not {"pass": bool, ...} counts as a fail."""
    m = re.search(r"\{.*\}", raw, re.S)
    try:
        v = json.loads(m.group(0)) if m else {}
    except json.JSONDecodeError:
        v = {}
    if not isinstance(v.get("pass"), bool):
        return {"pass": False, "fix": "QA reply was not valid JSON"}
    return v


def qa_agent(state: State) -> dict:
    issues = rule_checks(state["draft"], state["findings"])
    if not issues:  # only spend a model call when the cheap checks pass
        v = parse_verdict(MODEL.complete("qa", 'Reply {"pass": true|false, "fix": "..."}\n'
                                         + state["draft"]))
        if not v["pass"]:
            issues = [v.get("fix", "unspecified")]
    return {"issues": issues, "status": "qa_passed" if not issues else "needs_attention"}

The number check is the grounding test. Every number in the draft must appear in a retrieved snippet. It is crude, since it compares strings, so "38 percent" in the draft would not match "38%" in a source. That is acceptable because the error points the safe way: a false alarm costs one rewrite, while a missed invented number goes to a client.

parse_verdict exists because of a common bug. Many tutorials grade the model's reply with "PASS" in verdict.upper(). A reply of FAIL: the summary does not PASS a basic accuracy bar contains "PASS", so that check approves it. The strict parser demands a JSON object with a boolean pass field and treats anything else as a fail. It also accepts JSON inside a code fence, which models add whether you ask or not.

Routing, the draft cap and human approval#

def after_qa(state: State) -> str:
    if state["status"] == "qa_passed" or state["drafts"] >= MAX_DRAFTS:
        return "human_review"   # a failed draft still goes to a human, flagged
    return "writer_agent"


def human_review(state: State) -> dict:
    # Pauses the run. Everything above this line runs again on resume.
    decision = interrupt({"status": state["status"], "issues": state["issues"],
                          "preview": state["draft"][:120]})
    return {"status": "approved" if decision.get("approve") else "rejected"}


def after_review(state: State) -> str:
    return "publish" if state["status"] == "approved" else END


def publish(state: State, config) -> dict:
    # Idempotent: the file name comes from the thread id, so a retry overwrites
    # the same report instead of creating a second one.
    OUT_DIR.mkdir(exist_ok=True)
    path = OUT_DIR / f"{config['configurable']['thread_id']}.md"
    path.write_text(state["draft"], encoding="utf-8")
    return {"published": str(path)}

MAX_DRAFTS = 2 bounds the loop. Without a cap, a strict QA agent and a writer that keeps making the same mistake loop until the recursion limit or your budget runs out. With the cap, the worst case is known in advance (see the call count below).

A common way to add human approval is interrupt_before=["publish"] at compile time, then update_state to set an approval flag. That still works, but the LangGraph docs now say static interrupts are not recommended for human-in-the-loop workflows and keep them for debugging. The interrupt() function used here sends a payload to the caller (status, issues, a preview) and returns whatever the caller resumes with.

One rule of interrupt() catches people out. When the run resumes, the node restarts from the beginning, not from the interrupt line. Anything before the interrupt call runs twice. If human_review sent a Slack message before pausing, the reviewer would get it twice. Keep nodes that pause free of side effects, and put side effects in a later node, as publish is here.

publish is idempotent for a related reason. If it fails partway and a retry runs it again, it writes the same file again instead of a second copy. In a real system the equivalent is an upsert keyed by the run id, or an idempotency key on the email or ticket API.

Retries and wiring#

def build_app(model: Model, search: Search, checkpointer=None):
    global MODEL, SEARCH
    MODEL, SEARCH = model, search
    retry = RetryPolicy(max_attempts=3, initial_interval=0.01, retry_on=TimeoutError)

    g = StateGraph(State)
    g.add_node("research_agent", research_agent, retry_policy=retry)
    g.add_node("analyst_agent", analyst_agent, retry_policy=retry)
    g.add_node("writer_agent", writer_agent)
    g.add_node("qa_agent", qa_agent)
    g.add_node("human_review", human_review)
    g.add_node("publish", publish)

    g.add_edge(START, "research_agent")                       # fan out
    g.add_edge(START, "analyst_agent")
    g.add_edge(["research_agent", "analyst_agent"], "writer_agent")  # fan in
    g.add_edge("writer_agent", "qa_agent")
    g.add_conditional_edges("qa_agent", after_qa, ["writer_agent", "human_review"])
    g.add_conditional_edges("human_review", after_review, ["publish", END])
    g.add_edge("publish", END)
    return g.compile(checkpointer=checkpointer or InMemorySaver())

The edge from the list ["research_agent", "analyst_agent"] is the fan-in: the writer waits for both. Two separate edges into the writer would also run it after the parallel step here, but the list form states the intent and stays correct if one branch grows extra nodes.

RetryPolicy retries a node when it raises. retry_on=TimeoutError limits retries to the error worth retrying, and here it is required. The default, default_retry_on, skips programming errors such as ValueError and LookupError, and it also skips most OSError subclasses. TimeoutError is one of them: in langgraph 1.2.24, default_retry_on(TimeoutError()) returns False, so without the argument this node would fail on the first timeout. The short initial_interval is for the demo; real search APIs want something closer to the default of half a second with backoff.

Many tutorials do the opposite: they catch every search exception inside the tool and return a string like "search unavailable, rely on your own knowledge". The run then completes, and the model writes the brief from its training data with no sources. That is the worst result, because it looks like a normal run. Let the error surface. Retry it a few times, then fail the run, and resume it once the tool is back (the last test below does this).

The checkpointer is what makes pausing and resuming possible. InMemorySaver keeps checkpoints in RAM, which the persistence docs say are lost when the process restarts. A reviewer who approves the next morning needs a persistent one, such as SqliteSaver from langgraph-checkpoint-sqlite for a single machine or a Postgres saver for a service.

A real run#

run.py runs to the approval point, prints what the reviewer would see, then resumes with an approval:

from langgraph.types import Command

from stubs import FlakySearch, ScriptedModel
from workflow import build_app

model, search = ScriptedModel(), FlakySearch(failures=1)
app = build_app(model, search)
cfg = {"configurable": {"thread_id": "edge-slm-2026-11"}}

# Phase 1: run until the human checkpoint.
for update in app.stream({"topic": "small language models at the edge"}, cfg,
                         stream_mode="updates"):
    for node, out in update.items():
        if node == "__interrupt__":
            print(f"[{node}] waiting for a human: {out[0].value['status']}")
        elif node == "qa_agent":
            print(f"[{node}] status={out['status']} issues={out['issues']}")
        else:
            print(f"[{node}] wrote {sorted(out)}")

print("next node:", app.get_state(cfg).next)

# Phase 2: a reviewer approves; the same thread resumes from the checkpoint.
result = app.invoke(Command(resume={"approve": True}), cfg)
print("status:", result["status"], "| file:", result["published"])
print("model calls:", dict(model.calls), "| search calls:", search.calls)

Output:

  [search] timeout on 'small language models at the edge statistics'
[research_agent] wrote ['findings']
[analyst_agent] wrote ['findings']
[writer_agent] wrote ['draft', 'drafts']
[qa_agent] status=needs_attention issues=['missing section ## Actions', "numbers not in any source: ['2027', '60%']"]
[writer_agent] wrote ['draft', 'drafts']
[qa_agent] status=qa_passed issues=[]
[__interrupt__] waiting for a human: qa_passed
next node: ('human_review',)
status: approved | file: reports/edge-slm-2026-11.md
model calls: {'research': 1, 'writer': 2, 'qa': 1} | search calls: 3

Reading it from the top: the analyst's first search timed out and the retry policy ran it again (three search calls for two agents). QA rejected the first draft on rule checks alone, without a model call, and named both invented numbers. The second draft passed the rules and the model check. The run then stopped at human_review until Command(resume=...) arrived, and only then wrote the file.

What it costs per run#

Count the model calls, because that is what you pay for. With this graph and MAX_DRAFTS = 2:

Research Writer QA (model) Total
Best case, first draft passes 1 1 1 3
This run 1 2 1 4
Worst case, both drafts pass rules but fail the model check 1 2 2 5

The single-prompt version makes one call, so the split costs three to five times as many calls. What you get for that: a draft with an invented number never reaches the model-based QA, the cheap checks cost nothing, each call has a short single-purpose prompt you can test, and a search outage retries one node instead of the whole job. The writer prompt is the large one, since it carries all the findings. If cost matters, cap the snippets per search and send the writer facts instead of raw text.

Tests#

The graph is plain Python with stubs, so pytest can drive every branch without a model. test_workflow.py:

import pytest
from langgraph.types import Command

import stubs
from stubs import FlakySearch, ScriptedModel
from workflow import build_app, parse_verdict, rule_checks


def cfg(name):
    return {"configurable": {"thread_id": name}}


def test_substring_pass_check_is_a_trap():
    raw = "FAIL: the summary does not PASS a basic accuracy bar"
    assert "PASS" in raw.upper()                 # a naive check approves this
    assert parse_verdict(raw)["pass"] is False   # the strict parser does not


def test_invented_number_is_caught():
    findings = [{"sources": [{"id": "s3", "text": "38% of 400 firms"}]}]
    draft = "## Summary\n## Trends\n## Numbers\n38% today, 60% next year\n## Actions"
    assert rule_checks(draft, findings) == ["numbers not in any source: ['60%']"]


def test_draft_cap_sends_a_flagged_draft_to_a_human(monkeypatch):
    monkeypatch.setattr(stubs, "DRAFT_2", stubs.DRAFT_1)  # writer never fixes it
    app = build_app(ScriptedModel(), FlakySearch(failures=0))
    app.invoke({"topic": "t"}, cfg("cap"))
    state = app.get_state(cfg("cap"))
    assert state.next == ("human_review",)
    assert state.values["drafts"] == 2
    assert state.values["status"] == "needs_attention"


def test_rejection_publishes_nothing(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    app = build_app(ScriptedModel(), FlakySearch(failures=0))
    app.invoke({"topic": "t"}, cfg("no"))
    out = app.invoke(Command(resume={"approve": False}), cfg("no"))
    assert out["status"] == "rejected" and "published" not in out
    assert not (tmp_path / "reports").exists()


def test_run_resumes_after_search_outage():
    search = FlakySearch(failures=10)            # more failures than retries
    app = build_app(ScriptedModel(), search)
    with pytest.raises(TimeoutError):
        app.invoke({"topic": "t"}, cfg("outage"))
    search.failures = 0                          # the API comes back
    app.invoke(None, cfg("outage"))              # resume the same thread
    assert app.get_state(cfg("outage")).next == ("human_review",)
test_workflow.py::test_substring_pass_check_is_a_trap PASSED
test_workflow.py::test_invented_number_is_caught PASSED
test_workflow.py::test_draft_cap_sends_a_flagged_draft_to_a_human PASSED
test_workflow.py::test_rejection_publishes_nothing PASSED
test_workflow.py::test_run_resumes_after_search_outage PASSED
============================== 5 passed in 1.53s ===============================

The last test is the one worth copying. The search tool fails more times than the retry policy allows, so invoke raises. The checkpoint survives. Once the tool works again, invoke(None, config) on the same thread picks the run up and carries it to the approval point, with no re-run from scratch.

These tests cover the plumbing. They don't tell you whether a real model writes good briefs. For that, run the same graph with the real model over a fixed set of topics and score the traces, as in Evaluating an AI agent in Python. Track how often the first draft fails the number check: it tells you more about the writer prompt than any single run does.

Failure modes to plan for#

  • Parallel writes without a reducer. The run raises InvalidUpdateError. Add an Annotated reducer to any key two nodes write in the same step.
  • Swallowed tool errors. A run "succeeds" with no sources behind it. Retry specific exceptions, then let the run fail and resume it.
  • Unbounded QA loops. Cap the drafts and send the last failed draft to a person with its issues attached.
  • Substring verdicts. "does not PASS" passes. Parse a structured reply and fail closed.
  • Side effects before interrupt(). They repeat on resume. Move them to a node after approval and make them idempotent.
  • In-memory checkpoints in production. A restart loses every paused run. Use a persistent saver.
  • The number check misses rewording. "Thirty-eight percent" passes it unchecked. If your writer spells numbers out, normalise them before comparing, or tell the writer to use digits and have QA reject spelled-out figures.

When the workflow runs for hours, waits on several approvals or has to survive deploys, the next set of problems is budgets, plan persistence and recovery models. Design a long-running AI agent covers those. For a first automation, replace ScriptedModel.complete() with a real model call, keep the stub search for a day while you watch how often QA rejects the first draft, and only then connect live search.

Sources#