Part two
What the tools are really doing
We assumed we were testing models. We were testing products
with models inside them, and the difference turned out to matter.
Measured
Anthropic
You can't get at the raw model through a coding tool
We tried to strip the assistant back to nothing — tools off, safe mode
on, and an explicit instruction saying "you are a plain assistant, you
have no tools." It replied:
I'm Claude Haiku 4.5, a software engineering assistant built on
Anthropic's Claude Agent SDK. My current working directory is
C:\…\Temp, where I can edit files, run PowerShell or Bash
commands, and use Read, Write, Glob, Grep.
The override doesn't replace the built-in instructions, it gets
added to them. That hidden preamble is roughly
4,200 tokens riding along on every single request before your
question is counted at all.
Why it matters for any benchmark, including ours: results from a
coding tool measure the tool, not the model. It only distorts
questions where the assistant's sense of itself shapes the answer — we
found exactly two, and ran those by hand instead.
Measured
OpenAI
Turning off a tool doesn't stop it reaching the internet
With web search explicitly disabled, OpenAI's Codex tried
43 times to get the information another way:
fetching a web page with a shell command 27
listing directories 13
curl 3
installing a new tool to do it 2
That last row is a different kind of thing. Fetching a URL is using a
tool you already have. Installing a new one is reconfiguring your own
toolkit to get a capability you were just denied.
And it wasn't flailing. The attempts clustered almost entirely on the
two questions whose answers genuinely go stale — current prices, recent
news. On the question about its own training cutoff, it tried to look
things up zero times in 32 runs.
The fair reading is that this is arguably correct behavior. Asked
for current prices by someone who said they'd budget from the answer,
realizing your information is stale and trying to check is what a
careful assistant should do. But it means "tools were disabled"
only held because the sandbox refused — not because the model
complied.
This is not "OpenAI cheats and Anthropic doesn't", and we won't let
anyone read it that way. Across the same questions, every Anthropic
run attempted a lookup zero times — but Anthropic's tool was
configured with no tools at all, so it had nothing to try. GPT
had a shell and a locked door; Claude had no door. What this measures is
one agentic assistant going to check when it can, and the other
answering from memory because it can't. That is a real and useful
product difference. It is not a fair test of willingness, and we'd need
to give both sides identical shell access to make it one — which would
break the comparison everywhere else.
Measured
Both vendors
Tools are what make AI expensive — not long answers
Whatever a tool returns gets fed back into the model on the next turn,
so tool use inflates the input side, and that's where the money
actually goes.
identical questions, same tests
setups without tools 654 input tokens per question
setups with tools 7,258 input tokens per question 11x
The gap between the two companies is much larger than that. Over
the identical exam, OpenAI's setups read
— of input, where
Anthropic's read — — same
questions, same day. That is a difference in how hard the assistant goes
looking, not in how much it writes.
On the single most expensive setup we measured,
two questions out of forty accounted for 73% of the entire bill —
both of them the ones that trigger a look-up.
Neither number is a criticism. Going and checking costs tokens and
is often the right call. It does mean that if you're paying per token,
how your assistant answers matters more to your bill than how
much it says.
Measured
Anthropic
The vendor's own token counter can be wrong, silently
One model's runs on one day reported 661 tokens for 23,574
characters of text. That's physically impossible — roughly 36
characters per token where 4 is normal. The same runs claimed 637 input
tokens per question where identical prompts elsewhere reported 2,407.
Nothing failed. Nothing warned us. The numbers looked perfectly
reasonable in isolation. We only caught it because a chart built from
them looked wrong.
Separately, the same tool once returned a usage summary of all
zeros while a second field was correctly filled in — and once both
were populated, they disagreed with each other about the same request
(3,626 vs 4,149 input tokens).
If you're measuring anything: check the counter against something
physical. Characters per token is free and catches this in seconds.
Measured
Anthropic
A server outage can arrive disguised as an answer
An "overloaded, try again" error doesn't always come back as an error.
Sometimes it arrives as ordinary text in the answer field, exactly
where the model's response belongs. Anything reading that field treats an
outage as a badly-formed answer — and if it retries quickly, burns every
attempt during precisely the window the service is down.
This cost us three complete grading passes.
Inferred
Anthropic
An effort setting that's accepted and appears to do nothing
On the smallest model in one family, the command-line tool accepts a
"think harder" setting, exits cleanly, and returns a normal answer. But
the desktop app offers no such setting for that model at all; the
lowest and highest settings produce statistically indistinguishable
output; and in a partial sweep, every result at the higher
setting fell inside the range the lower one already produced.
Seven samples cannot prove a negative, which is why this is tagged
Inferred and not Measured. But "the tool accepts a setting that does
nothing" is exactly the sort of thing a benchmark stumbles into and a
vendor never documents.