An Agent With Tools

In the Real World · AI · Core · 2 days

An agent with four tools answers 80.0% of sixty operations questions correctly. It picks the right tool on 96.7% of them. Those two numbers are not in tension: it chooses correctly and then passes arguments that are wrong, which is a failure nobody benchmarks and every tool returns a confident answer to.

Section 1Problem Definition

An agent that calls tools has four separate ways to be wrong and one number that hides all of them. It can pick the wrong tool. It can pick the right tool and pass the wrong arguments. It can get both right and summarise the result badly. And it can answer a question that no tool it holds can answer, which is the only one of the four that produces a confident, plausible, entirely invented reply.

So this project scores four things separately, and the headline accuracy is the least useful of them.

Section 2Data Collection

A small deterministic world: 40 consignments, 6 depots, a duty table and a pricing formula. Four tools read it, and every one is a pure function, so the same call always returns the same thing.

# The entire tool description the model ever sees, verbatim.
#
#   track(consignment_id)                status, location, eta_days,
#                                        weight_kg, commodity, origin
#   duty(commodity, origin, weight_kg)   customs duty in pounds
#   depot(city, day)                     opening hours
#   quote(destination, weight_kg)        shipping price in pounds

That block is not a summary of the tools. It is the entire description the model ever sees, which is deliberate: an agent that needs a page of instructions per tool does not scale to the forty tools this desk would eventually want.

Sixty questions, five kinds

import pandas as pd

q = pd.read_csv("data/marlowe-questions.csv")
print(q.kind.value_counts().to_dict())
print(q[["kind", "question"]].head(4).to_string(index=False, max_colwidth=52))

{'single': 20, 'trap': 12, 'two_tool': 12, 'units': 8, 'unanswerable': 8}
  kind                                question
single      Any idea where CN-4800 has got to?
single   Whereabouts is CN-4807 at the moment?
single Where is consignment CN-4814 right now?
single      Any idea where CN-4821 has got to?

# Four questions from the file, one of each kind:
#
#   Any idea where CN-4800 has got to?
#   What do we pay to clear machine parts from Brazil, 87 kilos?
#   Duty on 0.469 t of porcelain tiles out of India?
#   Saturday hours for wherever CN-5045 is sitting?

Note what that does to the arguments. The duty tool takes a commodity from a fixed list, and the questions say machine parts and porcelain tiles. It takes kilogrammes, and the questions say tonnes. Extracting arguments here requires understanding rather than copying, which is the point.

Section 3Data Preprocessing

3aDuplicates and schema checks

assert q.question_id.is_unique
assert set(q.correct_tool) <= {"track", "duty", "depot", "quote", "none",
                               "track+duty", "track+depot"}
assert (q.kind == "unanswerable").sum() == 8

print("questions", len(q), "| answerable", int((q.kind != "unanswerable").sum()))

questions 60 | answerable 52

3bHandling categorical mess

The tools take enumerated values and the questions do not use them. That mismatch is the preprocessing problem in this project, and it is not one you can solve in preprocessing, because it happens inside the agent at runtime.

SYNONYMS = {"machine parts": "machinery", "porcelain tiles": "ceramics",
            "coffee beans": "coffee", "fabric rolls": "textiles"}

# This table exists in the write up and not in the agent. Building it is the
# maintenance cost that the whole comparison is about: every new supplier
# vocabulary means another line here, forever.

3cDealing with outliers

No numeric outliers. There is one structural check: every question that needs two tools has to start with the same one, because nothing except track turns a consignment id into a commodity and a weight.

import numpy as np

q["first_tool"] = np.where(q.kind == "two_tool", "track", q.correct_tool)
q.loc[q.kind == "unanswerable", "first_tool"] = "none"

for kind, tools in q.groupby("kind").first_tool.unique().items():
    print("%-13s %s" % (kind, list(tools)))

single        ['track', 'duty', 'depot', 'quote']
trap          ['duty', 'track', 'quote']
two_tool      ['track']
unanswerable  ['none']
units         ['duty', 'quote']

3dHandling missing values

A tool called with a missing argument returns an error rather than a guess, which matters: it means a wrong argument is visible in the trace instead of silently becoming a default.

from tools import duty

print(duty(commodity="machine parts", origin="Brazil", weight_kg=87))
print(duty(commodity="machinery", origin="Brazil", weight_kg=None))
print(duty(commodity="machinery", origin="Brazil", weight_kg=87))

{'error': 'unknown commodity or origin'}
{'error': 'weight_kg must be a number of kilogrammes'}
{'duty_pounds': 217.5}

3eHandling skewed data

Nothing here is skewed in the statistical sense. What is skewed is the cost of the five kinds of question: the eight unanswerable ones are a seventh of the set and carry most of the risk, because a wrong answer to those is invented rather than merely incorrect.

3fData types and normalisation

Arguments are compared the way the tool reads them, not as raw JSON, so a weight given as a string and the same weight as a number both count as correct.

def norm(args):
    out = {}
    for k, v in (args or {}).items():
        if v is None:
            continue
        out[k] = round(float(v), 2) if k == "weight_kg" else str(v).strip().lower()
    return out

print(norm({"weight_kg": "87", "commodity": "Machinery", "origin": None}))

{'weight_kg': 87.0, 'commodity': 'machinery'}

Section 4Exploratory Data Analysis

4aTarget variable analysis

There are four targets and they disagree with each other, which is the reason the project exists. The tool a question needs, the arguments it needs, the answer it should produce and whether it should be answered at all are four different labels on the same sixty rows.

4bNumerical variables

The only numeric argument is weight, and it appears in four surface forms across the corpus: kilogrammes with a space, kilogrammes without, kilos, and tonnes. Two of those four need no conversion and two do.

4cCategorical variables

Six commodities, five origins, six cities. Each commodity is written three ways, so a lookup on the exact enumerated value succeeds on roughly a third of mentions.

4dRelationships between variables

The relationship that turns out to matter is between the presence of a recognisable entity in a question and whether the agent calls a tool at all. It is measured in section 7 and it is the strongest single effect in the project.

4eTesting our hypotheses

4fSubgroups

Every number below is reported by question kind as well as overall, because the five kinds behave so differently that the average describes none of them.

Section 5Feature Engineering

5aThe leakage trap

5bNew features

The router needs three things out of a question: an identifier, a city and a weight. All three are regular expressions.

ID_RE    = re.compile(r"CN-\d{4}", re.I)
CITY_RE  = re.compile("|".join(WORLD["depots"]), re.I)
KG_RE    = re.compile(r"([\d,]+(?:\.\d+)?)\s*(kg|kilo)", re.I)
TONNE_RE = re.compile(r"([\d,]+(?:\.\d+)?)\s*(tonne|ton)", re.I)

# tonnes are converted, because anyone writing this would do that much
weight = float(kg.group(1)) if kg else float(tonne.group(1)) * 1000

5cEncoding

Tool choice is a five way decision: four tools or none. The router makes it with ordered rules, most specific vocabulary first.

def router(question):
    q = question.lower()
    if re.search(r"\bduty|customs|tariff\b", q):
        return "duty", {...}
    if re.search(r"\bopen|hours|saturday|sunday|depot\b", q) and city:
        return "depot", {...}
    if ident:
        return "track", {"consignment_id": ident.group(0).upper()}
    if re.search(r"\bquote|cost|charge|price|ship|move|send\b", q) and city:
        return "quote", {...}
    return "none", {}

5dFeature selection

The model agents get no features at all. They get the four line tool description, the question, and a schema that forces a tool name and an arguments object. That asymmetry is the comparison: the router encodes what somebody knew about the domain, and the agent has to work it out.

Section 6Model Selection

Three agents, and the two model agents are run on both local models, giving five columns in every table below.

CHOICE = {"type": "object",
          "properties": {"tool": {"type": "string"},
                         "args": {"type": "object"},
                         "reason": {"type": "string"}},
          "required": ["tool", "args"]}

# The schema forces a tool name and an args object. It does not force the
# tool to exist, and it cannot force the arguments to be right, which is
# where the rest of this article ends up.

cache = Cache("agent_cache.json")

got = cache.ask_json(CHOOSE_PROMPT.format(q=question, history=history),
                     CHOICE, model="qwen3.6:latest", num_predict=300)

cache: {'calls': 587, 'hits': 587, 'misses': 0, 'empty_responses': 0, 'unparsed_responses': 0}

Section 7Model Training

7aBaselines

The router is the baseline and it is not a bad one. It picks the right tool on 80.0% of the sixty questions and answers 43.3% of them correctly. The gap between those two numbers is the whole subject of this article, and it appears in the baseline before any model is involved.

7bComparing candidates

rows = []
for name, out in runs.items():
    d = pd.DataFrame(out)
    d["tool_ok"]   = d.first_tool.values == q.first_tool.values
    d["args_ok"]   = [args_match(e, g) for e, g in zip(q["args"], d["args"])]
    d["answer_ok"] = [answer_matches(c, a) for c, a in zip(q.correct_answer, d.answer)]
    rows.append({"method": name,
                 "tool":   round(100 * d.tool_ok.mean(), 1),
                 "args":   round(100 * d[answerable].args_ok.mean(), 1),
                 "answer": round(100 * d.answer_ok.mean(), 1),
                 "steps":  round(d.steps.mean(), 2)})

print(pd.DataFrame(rows).to_string(index=False))

               method   tool   args  answer  steps
       Keyword router   80.0   76.9    43.3   0.87
    One call, Gemma 4   86.7   71.2    50.0   0.88
        Loop, Gemma 4   86.7   71.2    60.0   0.98
   One call, Qwen 3.6   96.7   82.7    55.0   0.90
       Loop, Qwen 3.6   96.7   82.7    80.0   1.30

The best agent answers 80.0% of the questions. That is the number a demo shows you.

The two numbers that should not both be true

The best agent picks the right tool on 96.7% of all sixty questions, and on every single one of the 52 that are answerable. Its arguments are right on 82.7%.

answerable = q.kind != "unanswerable"
for name, d in per_q.items():
    a = d[answerable]
    print("%-22s tool wrong %2d | tool right args wrong %2d | both right %2d"
          % (name, (~a.tool_ok).sum(),
             (a.tool_ok & ~a.args_ok).sum(), (a.tool_ok & a.args_ok).sum()))

Keyword router         tool wrong  6 | tool right args wrong  6 | both right 40
One call, Gemma 4      tool wrong  6 | tool right args wrong  9 | both right 37
Loop, Gemma 4          tool wrong  6 | tool right args wrong  9 | both right 37
One call, Qwen 3.6     tool wrong  0 | tool right args wrong  9 | both right 43
Loop, Qwen 3.6         tool wrong  0 | tool right args wrong  9 | both right 43

Choosing the tool is the part that works. The arguments are where the agent quietly goes wrong, and a tool with wrong arguments still answers.

For Loop, Qwen 3.6 the wrong tool column is 0. Every failure it has on an answerable question is an argument it filled in wrongly, and a tool handed the wrong arguments does not complain. It returns the duty on the wrong commodity, or the price for the wrong weight, correctly and to two decimal places.

Where the arguments break

print(pd.DataFrame({m["method"]: m["args_by_kind"]
                    for m in methods}).to_string())

          Keyword r  One call,  Loop, Gem  One call,  Loop, Qwe
single         95.0       85.0       85.0       85.0       85.0
trap           50.0       66.7       66.7       75.0       75.0
two_tool      100.0       58.3       58.3      100.0      100.0
units          37.5       62.5       62.5       62.5       62.5

Plainly worded questions are fine for everyone. The wrong unit costs the router 62.5% of those questions and the models about a third, and the misleading wording costs everyone. Note the router’s 100.0% on two-tool questions next to its 0.0% on answering them: it extracts the consignment id perfectly and then has nowhere to go with it.

7cHyperparameter tuning

The only setting worth tuning is how many tool calls the agent may make, and the answer is not the same for every question.

for model, e in loop_effect.items():
    print("%-10s one call %5.1f%%  loop %5.1f%%  delta %+.1f  extra steps %+.2f"
          % (model, e["one_call"], e["loop"], e["delta"], e["extra_steps"]))
    print("           two tool %5.1f -> %5.1f   single %5.1f -> %5.1f"
          % (e["two_tool_one_call"], e["two_tool_loop"],
             e["single_one_call"], e["single_loop"]))

Gemma 4    one call  50.0%  loop  60.0%  delta +10.0  extra steps +0.10
           two tool   0.0 ->  50.0   single  70.0 ->  70.0
Qwen 3.6   one call  55.0%  loop  80.0%  delta +25.0  extra steps +0.40
           two tool   0.0 -> 100.0   single  85.0 ->  90.0

The loop is worth +25.0 points for the stronger model and +10.0 for the weaker one, and essentially all of it comes from one kind of question. Two-tool questions go from 0.0% to 100.0%. Plain single-tool questions go from 85.0% to 90.0%, which is one question.

The loop is not generally better. It is better at the thing it is for, and neutral everywhere else.

7dFinal evaluation

The eight questions with no answer

Every agent refuses some of them, and none of them refuses on the right basis.

for r in unanswerable_detail:
    print("%d/%d  %s" % (r["refused_count"], 5, r["question"]))

2/5  Who is the driver assigned to CN-4800?
5/5  What was the euro to sterling rate last Tuesday?
4/5  Has the customer paid the invoice for CN-4807?
4/5  Which insurer covers the Rotterdam depot?
2/5  What is the carbon footprint of CN-4814?
0/5  Can we reroute CN-4821 to Leeds tonight?
4/5  How many staff work at the Felixstowe depot?
5/5  What did we quote this customer last year?

Read the bottom of that list. Can we reroute CN-4821 to Leeds tonight? is refused by none of the five. It asks the agent to do something, not to look something up, and there is no tool that changes anything in the world. Every agent calls track on it and answers from whatever comes back.

The pattern across all eight is mechanical, and it has nothing to do with whether a tool can answer.

named = [r for r in detail if re.search(r"CN-\d{4}|" + cities, r["question"])]
none  = [r for r in detail if r not in named]

print("names a consignment or depot : refused by %.2f of 5 on average" %
      np.mean([r["refused_count"] for r in named]))
print("names neither                : refused by %.2f of 5 on average" %
      np.mean([r["refused_count"] for r in none]))

names a consignment or depot : refused by 2.67 of 5 on average
names neither                : refused by 5.00 of 5 on average

Refusal tracks whether the agent recognises something in the question, not whether it holds a tool that answers it.

Every unanswerable question that names nothing the agent recognises is refused by all 5 agents, without exception. Every one that names a consignment or a depot is refused by 2.67 of 5 on average. A familiar identifier is enough to start a tool call, and once a tool has returned something, the agent writes an answer out of it.

Section 8Documentation and Handoff

What ships is the harness and the four scores, not a single accuracy figure.

The checks that run before any of this is believed

• Every figure is recomputed from the cached model responses by a second route, and no model is called during the gate.
• The router is reimplemented in the gate and has to produce the same tool and arguments on all sixty questions.
• Every tool is called independently and checked against the world file.
• The claim that a named entity drives refusal is asserted, not just printed: if the questions naming nothing are not all refused, the build fails.
• Every code block in this article is extracted and executed in document order.

One number in this article is deliberately unflattering to the method the project recommends. The best agent still answers 12 of the sixty questions wrongly. This is not a solved problem, and an operations desk that deploys it without the refusal list is going to tell somebody the wrong duty on a real shipment.