GyaanStack. Blog

Articles / agentsevalspython

Evaluating an AI agent in Python, with no API key

A small eval harness for a tool-using agent. It checks which tools were called, with what arguments and in what order, then grades the final answer.

By Raghunath · · 12 min read

Reading an agent's output and deciding it looks fine works for a demo. It stops working the first time you change a prompt and something else breaks, three questions away from the one you were looking at.

An agent eval is a fixed list of questions with known good behaviour, plus a script that runs the agent on every question and tells you what went wrong. In this post I build one in about 150 lines of plain Python. It runs offline, with no model and no API key, so you can see every part of it before you point it at a real LLM.

Everything here was run with Python 3.15.2 and the standard library only. Python 3.10 or newer is enough.

What we check, and why the final answer is not enough#

A tool-using agent does two things: it picks actions (tool calls) and it writes an answer. If you only grade the answer, you miss a whole class of bugs:

  • The agent called the right tool with the wrong argument and got lucky.
  • It skipped the tool and made the number up.
  • It called a tool that charges money or sends an email when it should not have.

So the harness checks three things for each question:

  1. Tools: did it call the expected tools? In strict order, or in any order when order does not matter.
  2. Args: were the arguments right?
  3. Answer: does the final text contain what it should? For numbers, is the right number in there?

A case passes only when all three pass.

The toy agent#

The agent below is a set of if statements standing in for a model. It has three tools: weather, currency conversion and order status. Every tool call is recorded in a Trace, and that record is what makes the agent testable.

# agent.py
import re
from dataclasses import dataclass, field

# --- tools -----------------------------------------------------------------
RATES = {("USD", "INR"): 84.0, ("EUR", "INR"): 91.0, ("INR", "USD"): 1 / 84.0}
WEATHER = {"bangalore": "24C, cloudy", "delhi": "31C, haze", "mumbai": "29C, humid"}
ORDERS = {"A123": "shipped", "B777": "processing"}


def get_weather(city: str) -> str:
    return WEATHER.get(city.lower(), "unknown city")


def convert_currency(amount: float, src: str, dst: str) -> float:
    return round(amount * RATES[(src, dst)], 2)


def order_status(order_id: str) -> str:
    return ORDERS.get(order_id, "not found")


TOOLS = {"get_weather": get_weather, "convert_currency": convert_currency,
         "order_status": order_status}


# --- trace ------------------------------------------------------------------
@dataclass
class ToolCall:
    name: str
    args: dict


@dataclass
class Trace:
    calls: list[ToolCall] = field(default_factory=list)
    answer: str = ""


# --- a rule-based "agent" standing in for an LLM ----------------------------
CURRENCY = {"dollars": "USD", "usd": "USD", "euros": "EUR", "eur": "EUR",
            "rupees": "INR", "inr": "INR"}


def rule_agent(question: str) -> Trace:
    trace = Trace()
    q = question.lower()

    def call(name, **args):
        trace.calls.append(ToolCall(name, args))
        return TOOLS[name](**args)

    if "weather" in q:
        cities = [c for c in WEATHER if c in q]
        results = [f"{c.title()}: {call('get_weather', city=c)}" for c in cities]
        trace.answer = "; ".join(results) or "Which city?"
    elif m := re.search(r"([\d.]+)\s+(\w+)\s+(?:to|in)\s+(\w+)", q):
        amount, src, dst = float(m[1]), CURRENCY[m[2]], CURRENCY[m[3]]
        value = call("convert_currency", amount=amount, src=src, dst=dst)
        trace.answer = f"{amount:g} {src} is about {value} {dst}"
    elif m := re.search(r"order\s+(\w+)", question):  # bug: case-sensitive id
        status = call("order_status", order_id=m[1])
        trace.answer = f"Order {m[1]} is {status}."
    else:
        trace.answer = "Sorry, I can only do weather, currency and orders."
    return trace

I left one bug in on purpose. The order branch reads the ID from the original question, so a user who types b777 in lower case gets an ID that does not exist. It is the kind of mistake a real agent makes all the time, and the eval should catch it.

The golden set#

The golden set is a list of questions with the behaviour you expect. Keep it in code or JSON, under version control, next to the agent.

# golden.py
GOLDEN = [
    {"id": "weather-1", "q": "What's the weather in Bangalore?",
     "calls": [("get_weather", {"city": "bangalore"})],
     "answer_has": ["24C"]},
    {"id": "weather-2", "q": "Weather in Delhi and Mumbai please",
     "calls": [("get_weather", {"city": "delhi"}), ("get_weather", {"city": "mumbai"})],
     "order": "any", "answer_has": ["31C", "29C"]},
    {"id": "fx-1", "q": "Convert 100 dollars to rupees",
     "calls": [("convert_currency", {"amount": 100, "src": "USD", "dst": "INR"})],
     "answer_number": 8400.0},
    {"id": "fx-2", "q": "How much is 50 euros in INR?",
     "calls": [("convert_currency", {"amount": 50, "src": "EUR", "dst": "INR"})],
     "answer_number": 4550.0},
    {"id": "order-1", "q": "Where is my order A123?",
     "calls": [("order_status", {"order_id": "A123"})],
     "answer_has": ["shipped"]},
    {"id": "order-2", "q": "status of order b777",
     "calls": [("order_status", {"order_id": "B777"})],
     "answer_has": ["processing"]},
    {"id": "refuse-1", "q": "Book me a flight to Goa",
     "calls": [], "answer_has": ["only do"]},
]

A few choices here are deliberate:

  • weather-2 uses "order": "any". Two weather lookups can happen in either order, and failing the agent for that would be noise.
  • The currency cases check the number, not the wording. "8400.0 INR" and "about ₹8,400" should both be fine one day. For now a regex pulls numbers out of the answer.
  • refuse-1 expects no tool calls. Testing what an agent should not do is as useful as testing what it should do.

The harness#

Each check is a function that takes the trace and the case and returns (passed, reason). The runner calls every check on every case, counts passes and prints the failures with their reasons.

# evaluate.py
import re
from collections import Counter

from agent import rule_agent
from golden import GOLDEN


def check_tools(trace, case):
    got = [c.name for c in trace.calls]
    want = [name for name, _ in case["calls"]]
    if case.get("order", "strict") == "any":
        ok = Counter(got) == Counter(want)
    else:
        ok = got == want
    return ok, f"tools {got} != {want}"


def check_args(trace, case):
    got = sorted((c.name, sorted(c.args.items())) for c in trace.calls)
    want = sorted((n, sorted(a.items())) for n, a in case["calls"])
    return got == want, f"args {got} != {want}"


def check_answer(trace, case):
    if "answer_number" in case:
        nums = [float(x) for x in re.findall(r"\d+(?:\.\d+)?", trace.answer)]
        ok = any(abs(n - case["answer_number"]) < 0.01 for n in nums)
        return ok, f"no {case['answer_number']} in {trace.answer!r}"
    missing = [s for s in case["answer_has"] if s.lower() not in trace.answer.lower()]
    return not missing, f"missing {missing} in {trace.answer!r}"


CHECKS = {"tools": check_tools, "args": check_args, "answer": check_answer}


def run(agent, golden):
    tally = Counter()
    failures = []
    for case in golden:
        try:
            trace = agent(case["q"])
        except Exception as e:  # a crash fails every check
            failures.append((case["id"], "crash", repr(e)))
            continue
        case_ok = True
        for name, check in CHECKS.items():
            ok, why = check(trace, case)
            tally[name] += ok
            if not ok:
                case_ok = False
                failures.append((case["id"], name, why))
        tally["case"] += case_ok

    n = len(golden)
    print(f"{n} cases")
    for name in [*CHECKS, "case"]:
        print(f"  {name:<7} {tally[name]}/{n}  ({tally[name] / n:.0%})")
    for case_id, check, why in failures:
        print(f"FAIL {case_id} [{check}] {why}")
    return tally["case"] / n


if __name__ == "__main__":
    run(rule_agent, GOLDEN)

The argument check sorts the calls before comparing them, so it ignores order. Order is the tool check's job, and I keep the two separate so one failure does not show up twice under different names.

Real agents crash on bad tool output. The try/except turns a crash into a failed case, so one bad question does not stop the whole run.

Running it#

python evaluate.py

Output:

7 cases
  tools   7/7  (100%)
  args    6/7  (86%)
  answer  6/7  (86%)
  case    6/7  (86%)
FAIL order-2 [args] args [('order_status', [('order_id', 'b777')])] != [('order_status', [('order_id', 'B777')])]
FAIL order-2 [answer] missing ['processing'] in 'Order b777 is not found.'

The tool check passes, because the agent did call order_status. The argument check says why things went wrong: it passed b777 instead of B777. If I graded only the final answer, I would know order-2 failed but not where.

The fix is two lines: match on the lower-cased question and upper-case the ID before calling the tool.

    elif m := re.search(r"order\s+(\w+)", q):
        order_id = m[1].upper()
        status = call("order_status", order_id=order_id)
        trace.answer = f"Order {order_id} is {status}."

Run it again:

7 cases
  tools   7/7  (100%)
  args    7/7  (100%)
  answer  7/7  (100%)
  case    7/7  (100%)

The point of keeping the golden set is that this run happens again after every later change, not only today.

Swapping in a real LLM#

The harness only needs one thing from the agent: a function that takes a question and returns a Trace. An LLM agent is a loop. Ask the model what to do, run the tool it picked, feed the result back, and stop when it gives an answer. Here is that loop with the model passed in as a plain function:

# llm_agent.py
from agent import TOOLS, ToolCall, Trace


def make_llm_agent(llm, max_steps=5):
    """llm(messages) returns {"tool": name, "args": {...}} or {"answer": text}."""
    def agent(question: str) -> Trace:
        trace = Trace()
        messages = [{"role": "user", "content": question}]
        for _ in range(max_steps):
            step = llm(messages)
            if "answer" in step:
                trace.answer = step["answer"]
                return trace
            result = TOOLS[step["tool"]](**step["args"])
            trace.calls.append(ToolCall(step["tool"], step["args"]))
            messages.append({"role": "tool", "name": step["tool"], "content": str(result)})
        trace.answer = "(gave up: too many steps)"
        return trace
    return agent


def fake_llm(messages):
    """Stands in for a model so the loop can be tested offline."""
    question = messages[0]["content"].lower()
    tool_results = [m for m in messages if m["role"] == "tool"]
    if "bangalore" in question and not tool_results:
        return {"tool": "get_weather", "args": {"city": "bangalore"}}
    if tool_results:
        return {"answer": f"It is {tool_results[-1]['content']} right now."}
    return {"answer": "Sorry, I can only do weather, currency and orders."}
# run_llm_eval.py
from evaluate import run
from golden import GOLDEN
from llm_agent import fake_llm, make_llm_agent

run(make_llm_agent(fake_llm), GOLDEN)

The fake model only knows the Bangalore question, so most cases fail. The first lines of the output:

7 cases
  tools   2/7  (29%)
  args    2/7  (29%)
  answer  2/7  (29%)
  case    2/7  (29%)
FAIL weather-2 [tools] tools [] != ['get_weather', 'get_weather']

That is the result you want from the plumbing test. The loop runs, the trace fills in and the harness reports every gap. The other failures are the five questions the fake model does not know how to answer.

To use a real model, write an llm(messages) function that sends the messages and the tool schemas to your provider's tool-calling API, then turns the reply into {"tool": ..., "args": ...} or {"answer": ...}. Nothing in evaluate.py or golden.py changes. I have not shown provider code here because I only include code I have run for this post, and every SDK names these fields a little differently.

Things that change once a model is involved#

Run each case more than once. A model can pass a case on one run and fail it on the next. Run the set several times and look at the mean and the worst run:

# repeat.py
import contextlib
import io

from agent import rule_agent
from evaluate import run
from golden import GOLDEN


def run_many(agent, golden, n=5):
    scores = []
    for _ in range(n):
        with contextlib.redirect_stdout(io.StringIO()):  # keep it quiet
            scores.append(run(agent, golden))
    print(f"{n} runs: mean {sum(scores) / n:.0%}, worst {min(scores):.0%}")


run_many(rule_agent, GOLDEN)
5 runs: mean 86%, worst 86%

The rule agent with the bug gives the same score every time. With a model, the gap between mean and worst tells you how flaky the agent is.

Loosen argument checks where it makes sense. A model might pass "Bangalore" or "bengaluru". Normalise before comparing, or check only the arguments that matter (the amount and currencies, not a free-text note field).

Keep string checks for facts, and add a judge for tone. answer_has is cheap and never wrong about whether "shipped" is in the text. Use an LLM judge only for things a string match cannot see, like whether a refusal was polite, and spot-check the judge's grades by hand.

Grow the golden set from real failures. Each time a user hits a bug, add that exact question as a case before you fix it. After a few weeks the set covers the questions people actually ask, which is more useful than a set I invent at my desk.

What to take from this#

You can write an agent eval before you have an agent worth evaluating. Record a trace, keep a golden set and check tools, arguments and answers separately. The lower-case order ID bug here is small, but bugs like it show up in every tool-using agent, and this harness points straight at the argument that went wrong.

Start with ten real questions, run them on every change, and add a case each time something breaks.