Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions

Chubin Zhang; Zhenglin Wan; Xingrui Yu; Jingxuan Wu; Yaxin Zhou; Ivor Tsang; Bo An

cs.AIOct 5, 2026

Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions

Authors: Chubin Zhang, Zhenglin Wan, Xingrui Yu, Jingxuan Wu, Yaxin Zhou, Ivor Tsang, Bo An

Organizations: Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore · IHPC, Agency for Science, Technology and Research, Singapore · Department of Statistics and Operations Research, UNC-Chapel Hill, United States · Carnegie Mellon University, United States

Abstract

An agent whose tool keeps returning nothing useful should stop relying on it. In a retrieval environment with controlled source failures, we separate how agents judge results from what they do. We compare stopping at the same step after longer and shorter runs of results the agent judged useless; this contrast is zero for clock- or deadline-driven stopping. Where we record their judgments, the seven agents we test call a failing source's results useless 97-100% of the time, yet most of them rarely stop on that judgment. Prompt cues change when they stop but not what they stop on. Permission to answer from memory and a reasoning mode can bring early stops regardless of evidence, a stated budget moves the 7-8B models' stops to the deadline, and a stopping rule or call cost in the prompt is followed at most partly. Stopping follows the evidence only when the harness enforces an integration step that makes the agent answer after five consecutive results it judged useless. This step raises failing-source success for every model, keeps the stopping point fixed when the budget doubles, and needs no extra judgment call when the agent states its judgments. A pre-registered replication on 300 fresh questions confirms the dissociation and the rule's effect.

Figures & tables

Figure 1: Left: a real held-out trajectory of Claude Haiku 4.5 with a persistently failing source (selected by a fixed rule, Appendix B.3 ). It calls every result irrelevant and recalls at step 6 that Barlow is in a band, yet searches until its budget runs out. With the enforced rule, it answers correctly. Right: share of side-channel judgments that call a failing source’s results useless (red), and share of questions with five such judgments in a row on which the unaided agent answers (gray). Judgments are replayed on the unaided agent’s own trajectories, except for Claude Haiku 4.5 (*), whose judgments come from its enforced-rule run on the same questions.

Figure 2: What

\Delta

compares at a fixed step. The enforced rule answers in (a) and continues in (b).

Condition	What changes	Who stops
none	base prompt	agent
permit	may answer from memory	agent
budget	permit + budget, counter	agent
stated rule	permit + rule in words	agent
call cost	budget + price per call	agent
decide	running count shown	agent

Table 1: Conditions (top) and stopping measures (bottom). Both the stated and the enforced rule prescribe answering after five consecutive results judged useless.

\Delta

and

I

are pre-registered; prompts in Appendix B.1 .

Figure 3: Only when the harness stops (blue labels) does stopping follow the agent’s judgments for every model. Cells show

\Delta

from the agents’ own judgments in test300 (blue

>0

, red

<0

), with the fresh300 replication in parentheses. Decide is exploratory; intervals are in Appendix C.1 .

Condition	Qwen2.5 7B	Llama-3.1 8B	Qwen3 8B	Qwen3 32B
none	.192	.263	.448	.509
permit	.213	.269	.423	.517
budget	.328	.312	.464	.581
stated rule	.206	.267	.439	.526
call cost	.326	.317	.458	.575
decide	.360	.266	.426	–

Table 2: The budget condition matches or exceeds the enforced rule’s success on several models, so success alone cannot show what stopping responds to. Mean6 success (test300, raw; bold: best per model; –: not run). Bottom: agents run only unaided and with the enforced rule ( † first 100 questions; ‡ RL-trained, own prompt). Intervals and per-regime results: Appendix D.1 .

Figure 4: Budget-cued 7–8B models answer at whatever the deadline is, whereas the enforced rule and the combination answer at the same action under both budgets. Bars: median action at which agents answer on a persistently failing source (among answered questions) with an 8-action (light) and a 16-action (dark) budget. Qwen2.5-7B answers early under the enforced rule because even unaided it stops after one useless result.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Claim (main text)	Pre-registered tests (outcome)	Exploratory	Appendix
Agents judge a failing source’s results useless (§ 3 )	16b: stated and side-channel judgments agree (pass)	judgment rates; specificity on real results; accuracy of stated judgments	B.2 , C.3 , D.4
Unaided agents rarely answer after five useless judgments (§ 3 )	J1 (reported); F5, fresh300 (pass 3/3); T2, stated judgments (pass 2/2); R3, Search-R1 (fail)	whether stopping would have paid	C.1 , D.4 , D.2
Without enforcement, stopping does not follow the useless run (§§ 3 – 4 )	J2: none, permit, budget (pass 12/12), stated rule (fail for Llama-3.1-8B, Qwen3-32B); C, call cost (pass 3/4); PP1, stated rule (partial for Llama-3.1-8B only); H1 and H3 of A20, Claude Haiku 4.5 (partial; not on the evidence); F1–F2, fresh300 (pass 8/8); T1 of A18, stated judgments (pass 4/4)	hazard model; decide	C.1 , C.2 , D.4 , D.3
A stated budget moves stopping to the deadline (§ 4 )	H9, H10, F-H9, F-H10, D1, D3, D4, Rp3 (pass); L-H9 (fail) and A9 (mixed), Qwen3-32B		D.1 , D.6
The enforced rule raises success (§ 5 )	H1, F-H1, L-H1, Rp1, F4, R2, S-H2 (pass); S-H1, Claude Sonnet 5 (fail, $p=.051$ )		D.1 , B.4 , D.4 , D.6
without loss where the source recovers	H3 (14/16; 18/20 with Qwen3-32B’s four under L-H1), F-H3 (12/12); 13c, recovery after four or five failures (mixed)		D.1 , D.5

Appendix

Table 3: Where each claim of the main text is tested. Test IDs and decision rules as in Tables 4 and 5 ; exploratory analyses were not pre-registered.

Doc	ID	Prediction; models; decision rule	Outcome
Orig.	H1	Rule $>$ none (mean6); 7–8B, Claude Haiku 4.5; one-sided, Holm	pass 4/4: $+.045$ / $+.064$ / $+.026$ ; Claude Haiku 4.5 $+.097$
	H2	Rule $>$ rule fed a rate-matched random signal (A1); 7–8B; one-sided, Holm	pass 3/3: $+.011$ ( $p{=}.0285$ ) / $+.029$ / $+.021$
	H3	Rule $\geq$ none $-0.05$ on rec1, rec2, rec3, clean; 7–8B, Claude Haiku 4.5; 95% LB	14/16; fail Qwen3-8B rec1 (LB $-.057$ ), rec3 (LB $-.060$ )
	H4	Decide $<$ rule (mean6); 7–8B; one-sided	rule $-$ decide $-.123$ / $+.062$ / $+.048$ : fail for Qwen2.5-7B
	H5 e	Rule vs. rule fed a lexical detector; 7–8B	$+.012$ / $-.001$ / $-.011$
	H6 e	Thinking vs. rule, incl. rec3; Qwen3-8B	$+.003$ [ $-.032$ , $+.039$ ]; rec3: .437 vs. rule .470

Appendix

Table 4: Every pre-registered test, part 1 (original pre-registration and amendments A3–A9); notation as in Table 5 .

Doc	ID	Prediction; models; decision rule	Outcome
A10	Rp1	Replicate (new seeds, new draw of failed observations): rule $>$ none (mean6); 7–8B; one-sided, Holm	pass 3/3: $+.048$ / $+.061$ / $+.042$
	Rp2	Replicate: rule $I$ LB $>+.30$ ; none $I$ $\leq+.05$ ; 7–8B	rule $I$ .19 [.14, .24] / .46 / .46: fail for Qwen2.5-7B; none $-.31$ / $.00$ / $.00$ : pass
	Rp3	Replicate: budget arm’s median answering action on persistent $=8$ and $I$ $\leq+.10$ ; 7–8B	pass 3/3: median 8 / 8 / 8; $I$ $-.01$ / $+.04$ / $-.02$
	Rp4	Replicate: combo $\geq$ rule $-0.05$ and $\geq$ budget $-0.05$ (95% LB); 7–8B	5/6: vs. rule $+.108$ / $+.007$ / $-.024$ [LB $-.054$ ], fail for Qwen3-8B (repaired: pass); vs. budget LB $+.033$ / $+.008$ / $+.003$
A11	PP1	Stopping rule stated in the prompt: executed on the evidence if $\Delta$ LB $>0$ , persistent median $\leq 7$ and non-inferior to the rule; not executed if $\Delta$ CI includes or lies below 0 or persistent answer rate $<.5$ ; else partial; 7–8B, Qwen3-32B	not executed / partial / not executed; Qwen3-32B not executed (design-label $\Delta$ $-.05$ / $+.07$ / $-.10$ ; $-.10$ )
	PP2	Stated rule vs. external rule (non-inferiority, margin $-0.05$ ) and vs. permit (mean6)	fail 4/4 vs. rule: $-.031$ / $-.061$ / $-.034$ ; Qwen3-32B $-.047$ (LB $-.056$ / $-.089$ / $-.061$ / $-.073$ ; all upper bounds $<0$ ; repaired: Qwen3-8B $-.003$ [ $-.026$ , $+.021$ ], others unchanged in sign); vs. permit $-.007$ / $-.002$ / $+.016$ ; $+.008$

Appendix

Table 5: Every pre-registered test (part 2, amendments A10–A20; part 1 in Table 4 ), by the document that fixed it (IDs and condition names as registered: policy = stated rule, rule or external rule = enforced rule, cost = call cost; Orig.: original pre-registration; A3–A20: amendments; A1 and A2 are described in the text). Test IDs are those of each document; where a document names none (A12’s C, A15’s T1) or its names would clash (A17’s S1–S3 and validity check), we label them here (R1–R3, V). Paired differences in mean6 unless stated, as Qwen2.5-7B / Llama-3.1-8B / Qwen3-8B where three appear; brackets: 95% intervals; LB: lower bound; 7–8B: Qwen2.5-7B, Llama-3.1-8B, Qwen3-8B; rec

c

: recover_after_

c

. Raw analysis. e Exploratory by design. ‡ A8 adopted the local models’ criteria without naming H3; the main text counts these four tests together with H3. A9’s predictions are unnamed in the pre-registration.

Doc	ID	Prediction; models; decision rule	Outcome
A16	F1–F2	fresh300: own-judgment $\Delta$ UB $<+.06$ for none and budget; 7–8B (Qwen3-32B if time allows)	pass 8/8: none $-.09$ / $-.03$ / $-.05$ , Qwen3-32B $-.06$ ; budget $-.17$ / $-.14$ / $-.19$ , Qwen3-32B $-.05$ (UB $+.01$ ; others $<0$ )
	F3	fresh300: rule $\Delta$ LB $>+.20$	pass 4/4: $+.31$ [LB $+.22$ ] / $+.35$ [ $+.33$ ] / $+.34$ [ $+.31$ ]; Qwen3-32B $+.30$ [ $+.26$ ]
	F4	fresh300: rule $>$ none (mean6), one-sided, Holm	pass 4/4: $+.030$ / $+.061$ / $+.036$ ; Qwen3-32B $+.057$ (all $p<.001$ )
	F5	fresh300: unaided agent answers on $<10\%$ of questions after five own useless judgments (persistent)	pass 3/3: 3/125, 2/283, 0/289
	16b	Stated vs. side-channel judgments: annotator binary $\kappa\geq.80$ on the adjudicated labels to proceed; agreement $\geq.85$ expected	$\kappa$ .874 [.798, .939]; agreement .87 / .94 / .94; Qwen3-32B .96
A17	V	Search-R1: judgment validity, USELESS after failed $\geq.80$ and after real $\leq.50$ , else $\Delta$ only descriptive	fail : 1.00 and .89

Appendix

Table 5: (continued) Amendments A16–A20.

Figure 5: Stopping hazard: probability of answering at a decision point after

j

consecutive failed observations (final-action decisions excluded; 8-action budget; test300, Claude Sonnet 5 on 100 questions). Unaided agents (gray) do not rise with

j

, except for a partial rise in Claude Sonnet 5. Prompt cues (purple) stay flat or fall in the 7–8B models and rise only in Qwen3-32B. The enforced rule (blue) steps up at

j=5

; the reasoning mode (brown, Qwen3-8B) and the budget-cued Qwen3-32B rise with

j

, but not with the evidence at a fixed step (Qwen3-32B:

\Delta=0.00

, Figure 3 ; thinking: design-label contrast

-0.02

, Appendix C.1 ).

Model	none	permit	budget	stated	cost	decide	enforced	combo
Qwen2.5-7B	${-}.32/{-}.11$	${-}.29/{-}.16$	${-}.02/{-}.18$	${-}.18/{-}.31$	$.00/{-}.18$	${-}.33/{-}.24$	$.18/.33$	$.47/.21$
Llama-3.1-8B	$.00/{-}.02$	$.03/{-}.19$	$.05/{-}.10$	$.31/.09$	$.02/{-}.17$	$.05/{-}.14$	$.48/.37$	$.49/.27$
Qwen3-8B	${-}.01/{-}.06$	${-}.23/{-}.32$	$.00/{-}.20$	$.03/{-}.18$	$.00/{-}.19$	${-}.29/{-}.15$	$.47/.33$	$.43/.20$
Qwen3-32B	$.01/{-}.07$	$.17/.01$	$.32/.00$	$.54/.08$	$.41/.09$	–	$.49/.33$	$.52/.19$
Claude Haiku 4.5	$.00/\text{--}$	–	–	–	–	–	$.50/.40$	–
Claude Sonnet 5 †	$.18/\text{--}$	–	–	–	–	–	$.49/\text{--}$	–

Appendix

Table 9: Integration index

I

(pre-registered for some arms, Appendix A ) and time-matched contrast

\Delta

on the agent’s own judgments, test300, raw analysis. † First 100 questions; its per-step judgments were not saved (Appendix B.4 ).

Figure 6: The three models err in different directions, and where a model’s own report favors answering, its action does not follow. Held-out persistent regime, placeholder answers repaired; means over decision points, by the number of failed results so far. Left: self-reported value of continuing against the ideal observer (the realized value is the success on this source, 0.00–0.04). Middle: self-reported value of answering now against the realized value of the answer-now branch. Right: share of decision points at which the agent answers, and at which its own report favors answering.

HotpotQA test300

FEVER

raw

repaired

raw

repaired

Contrast (mean6)

Q2.5

rule

-

none (H1, F-H1)

+.045

+.064

+.026

+.046

+.069

+.040

+.129

+.201

+.142

+.134

+.203

+.142

rule

-

random (H2, F-H2)

+.011

+.029

+.021

+.011

+.032

+.033

+.044

+.123

+.096

+.045

+.124

+.096

rule

-

decide (H4)

-.123

+.062

+.048

-.123

+.059

+.062

-.084

+.350

+.005

-.079

+.180

+.005

rule

-

permit (H7)

+.023

+.059

+.051

+.024

+.061

+.054

+.138

+.138

+.064

+.142

+.108

+.062

Appendix

Table 12: Key contrasts, raw vs. repaired (IDs refer to HotpotQA; FEVER H4/H7/H8 counterparts are exploratory). Q2.5: Qwen2.5-7B; L: Llama-3.1-8B; Q3: Qwen3-8B. † Significance (95% interval excluding zero) differs between analyses. Also unchanged in decision: H1 for Claude Haiku 4.5 (

+.097

both), L-H1 (

+.063

both), Qwen3-8B’s H3 lower bounds on rec1/rec3 (

-.057

-.060

raw,

-.057

-.057

repaired), H6 (

+.003

raw,

-.011

repaired, both n.s.). Realized answer-now success on persistent:

.12

.15

.14

raw,

.14

.17

.19

repaired.

		success per regime								integration index
Model	Arm	pers.	rec1	rec2	rec3	late	clean	plaus.	mean6	$I$	95% CI	$h_{4\text{-}5}$	$h_{1\text{-}2}$
Qwen2.5-7B	none	.043	.160	.153	.147	.220	.427	.077	.192	$-.32$	$[-.37,-.27]$	.02	.34
	permit	.033	.170	.207	.137	.303	.430	.143	.213	$-.29$	$[-.34,-.24]$	.03	.32
	budget	.090	.407	.377	.327	.333	.433	.183	.328	$-.02$	$[-.04,.00]$	.02	.04
	decide	.183	.370	.300	.277	.517	.513	.230	.360	$-.33$	$[-.37,-.29]$	.03	.36
	random	.080	.187	.183	.163	.300	.443	.143	.226	$-.11$	$[-.17,-.05]$	.23	.33

Appendix

Table 13: HotpotQA (test300, 8-action budget), raw analysis: success per regime, mean6, and the integration index with its two components, for every model and arm.

n=300

questions per arm (Claude Sonnet 5: the first 100; its plausible regime was not run). pers.: persistent; rec

c

: recover_after_

c

; late: late_onset_from_3; plaus.: plausible (not part of mean6).

I

=h_{4\text{-}5}-h_{1\text{-}2}

, where

h_{a\text{-}b}

is the probability of answering at a decision point just after

a

b

consecutive failed observations, pooled over the five regimes with failures, decisions at the final action excluded; its 95% CI is from a question-cluster bootstrap.

		success per regime							integration index
Model	Arm	pers.	rec1	rec2	rec3	late	clean	mean6	$I$	95% CI	$h_{4\text{-}5}$	$h_{1\text{-}2}$
Qwen2.5-7B	none	.127	.675	.664	.644	.572	.716	.566	$-.12$	$[-.15,-.10]$	.01	.13
	permit	.151	.579	.541	.538	.729	.805	.557	$-.20$	$[-.25,-.16]$	.02	.23
	budget	.384	.818	.764	.798	.750	.801	.719	.00	$[-.02,.02]$	.03	.03
	decide	.668	.822	.733	.729	.870	.856	.780	$-.26$	$[-.35,-.15]$	.17	.43
	random	.377	.723	.695	.682	.699	.733	.651	.12	$[.08,.16]$	.25	.14

Appendix

Table 14: FEVER (292 claims, 8-action budget, every parameter carried over from HotpotQA), raw analysis: success per regime, mean6 and integration index for every arm.

n=292

claims per arm; no plausible regime. Columns as in Table 13 .

		HotpotQA			FEVER
Model	Contrast	$\Delta$ mean6	95% CI	$p$	$\Delta$ mean6	95% CI	$p$
Qwen2.5-7B	rule $-$ none	$+.045$	$[+.029,+.062]$	$<$ .001	$+.129$	$[+.104,+.154]$	$<$ .001
	rule $-$ permit	$+.023$	$[-.005,+.051]$	.055	$+.138$	$[+.104,+.172]$	$<$ .001
	rule $-$ budget	$-.091$	$[-.124,-.060]$	1.000	$-.024$	$[-.055,+.007]$	.938
	rule $-$ random	$+.011$	$[.000,+.022]$	.029	$+.044$	$[+.032,+.057]$	$<$ .001
	rule $-$ lexical	$+.012$	$[-.002,+.026]$	.043	$+.031$	$[+.019,+.043]$	$<$ .001

Appendix

Table 15: Paired contrasts in mean6 between arms, raw analysis: difference (first arm minus second), 95% interval over 2,000 question-level bootstrap resamples, and one-sided bootstrap

p

for a positive difference (unadjusted; the main text applies Holm correction across models where pre-registered).

n=300

paired questions on HotpotQA (Claude Sonnet 5: 100) and 292 on FEVER; Qwen3-32B, Claude Haiku 4.5 and Claude Sonnet 5 were not run on FEVER.

		8-action budget							16-action budget
Model	Arm	mean3	pers.	ans.	med.	final	calls	$I$	mean3	pers.	ans.	med.	final	calls	$I$
Qwen2.5-7B	none	.208	.043	156	2	.00	3.9	$-.35$	.219	.043	158	2	.01	7.1	$-.35$
	budget	.300	.090	175	8	.75	6.8	$-.02$	.312	.093	232	16	.75	13.4	$-.03$
	rule	.246	.100	262	2	.00	2.8	.09	.249	.093	260	2	.00	2.8	.10
	combo	.357	.163	282	6	.00	4.7	.45	.357	.130	267	6	.00	4.7	.44
Llama-3.1-8B	none	.241	.000	4	4.5	.25	7.9	.00	.298	.010	16	13	.06	15.6	.00

Appendix

Table 16: Budget scaling on HotpotQA (test300, raw analysis): each arm with the 8- and the 16-action budget. The 16-action runs cover three regimes (persistent, recover_after_2, clean), so mean3 averages these three at both budgets in place of mean6. On the persistent regime: success (pers.), number of the 300 questions on which the agent answers rather than exhausting its budget or failing to produce a well-formed action (ans.), median action at which it answers, among those (med., 1-based), share of those answers given at the final action (final), and mean tool calls (search and lookup) per question (calls).

I

is computed as in Table 13 but over the persistent and recover_after_2 regimes only, with the final action of the respective budget excluded; its 8-action values therefore differ from Table 13 .

n=300

per cell; Qwen3-32B without intervention was not run with 16 actions.

Model	budget	rule	combo
Qwen2.5-7B	5.09 $\to$ 8.69	2.88 $\to$ 2.91	4.06 $\to$ 4.10
Llama-3.1-8B	5.45 $\to$ 8.81	5.11 $\to$ 5.39	4.74 $\to$ 4.99
Qwen3-8B	4.11 $\to$ 6.62	4.10 $\to$ 4.27	3.54 $\to$ 3.62

Appendix

Table 18: Tool calls per trajectory with an 8- and a 16-action budget (persistent, recover_after_2 and clean, the regimes run at both budgets; exploratory).

Model	Condition	mean6	tool calls	extra calls	utility ( $\lambda=0.05$ )
Qwen2.5-7B	none	0.192	3.63	0.00	$+0.010$
	budget	0.328	4.90	0.00	$+0.083$
	enforced rule	0.237	2.94	2.86	$-0.053$
	combo	0.375	4.03	4.00	$-0.027$
	stated-judgment rule	0.209	3.46	0.08	$+0.033$
	budget + stated-judgment rule	0.343	4.42	0.24	$+0.110$

Appendix

Table 19: Stopping on the judgment stated in the agent’s reasoning (amendment 19; test300, raw, per question averaged over the six mean6 regimes). Extra calls: side-channel judgments for the enforced rule and the combination, discarded turns for the stated-judgment conditions.

	E1	E2	E3	E4
Model	success vs none	success vs enforced rule	utility vs enforced rule	combination utility vs budget	fires (pers.)
Qwen2.5-7B	$+0.018^{*}$ [ $+0.011,+0.024$ ]	$-0.027$ [ $-0.041,-0.012$ ]	$+0.086^{*}$ [ $+0.070,+0.102$ ]	$+0.027^{*}$ [ $+0.018,+0.037$ ]	0.05
Llama-3.1-8B	$+0.064^{*}$ [ $+0.049,+0.079$ ]	$+0.000$ [ $-0.017,+0.018$ ]	$+0.236^{*}$ [ $+0.216,+0.257$ ]	$+0.045^{*}$ [ $+0.032,+0.059$ ]	0.91
Qwen3-8B	$+0.036^{*}$ [ $+0.022,+0.051$ ]	$+0.010$ [ $-0.002,+0.021$ ]	$+0.196^{*}$ [ $+0.182,+0.210$ ]	$+0.022^{*}$ [ $+0.009,+0.034$ ]	0.97
Qwen3-32B	$+0.064^{*}$ [ $+0.047,+0.081$ ]	$+0.001$ [ $-0.014,+0.016$ ]	$+0.190^{*}$ [ $+0.173,+0.205$ ]	$0.000$ [ $-0.007,+0.006$ ]	0.89

Appendix

Table 20: Pre-registered tests of amendment 19 for the stated-judgment rule (E1–E3) and its combination with the budget (E4); mean6 or utility at

\lambda=0.05

; question-paired 95% intervals; ∗ one-sided Holm-corrected

p<0.05

; E2 is a non-inferiority test with margin

-0.05

, passed by all four models. By the pre-registered applicability rule, a model whose stated-judgment rule fires on fewer than 20% of persistently failing questions does not count as support.

Claude Haiku 4.5	mean6	$-$ unaided	$-$ enforced rule	design-label contrast	answered (pers.)	median action
stated rule	0.561	$+0.043$ [ $+0.007,+0.077$ ]	$-0.050$ [ $-0.086,-0.017$ ]	$+0.05$ [ $+0.02,+0.09$ ]	0.565	7
call cost	0.663	$+0.145$ [ $+0.113,+0.182$ ]	$+0.053$ [ $+0.023,+0.082$ ]	$+0.01$ [ $-0.02,+0.05$ ]	0.985	7

Appendix

Table 21: Claude Haiku 4.5 with the stopping rule or the call cost stated (amendment 20; first 200 test300 questions, raw; question-paired 95% intervals).

Explore similar work

Jul 6, 2026cs.CL

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

Large Language Model Agents Large Language Model Tool Use

Ask PDF

Oct 6, 2026cs.CL

From Evidence to Action: How Tool-Using Agents Fail

Tool-using agents make consequential changes to external state, yet correct outcomes do not guarantee that their actions were supported by evidence established beforehand. We study where this evidence-to-action chain breaks as agents move from deciding whether to act to executing single actions and dependent workflows. Across ten model-harness configurations, strong static action assessment can coexist with much weaker interactive execution. Failures often begin before execution: agents stop with incomplete investigation or act before required evidence is established. Once required evidence is obtained, single-action execution is usually reliable, while multi-action workflows additionally expose unresolved prerequisites and incomplete execution. For this analysis, we introduce SafeActBench, comprising 656 cases across six operational domains and five protocols that progress from static action judgment and investigated non-action to single- and multi-action workflows. A provenance-bound Evidence Ledger and deterministic trajectory evaluator track what information was established, when actions occurred, and whether downstream dependencies were satisfied. These results show that failures arise not only from missing information, but also from how agents use established evidence when deciding and executing actions.

Hongzhan Lin, Shidong Cao, Ziyang Luo +3

National University of Singapore · Hong Kong Baptist University · Amazon Web Services +1

Topic pending

Ask PDF

Jun 19, 2026cs.AI

Don't Blindly Trust It: How Unreliable Feedback Breaks Tool-Using LLM Agents

Tool-augmented agents are typically evaluated by their gains under reliable external feedback. Yet these gains leave open a key counterfactual: when feedback is unreliable, would the agent be better off receiving no task evidence? We study this question with a controlled matched-loop comparison that fixes the agent loop, prompt, action space, and decoding, while varying only the returned observation: faithful, misleading, or absent. Across question answering and fact verification, persistent misleading feedback produces a value inversion: agents that benefit from clean tools can perform worse than the matched no-feedback fallback. On HotpotQA, Qwen2.5-7B reaches 44.8 F1 with clean retrieval and 22.3 F1 with no feedback, but drops to 4.7 F1 under shuffled retrieval. The inversion persists under stronger clean retrieval and locally plausible distractors, but weakens when later clean evidence can repair the trajectory. Early trajectory signals predict many failures, yet simple repairs remain fallback-limited: rejecting bad evidence helps only when the exposed fallback is reliable. These results show that clean-tool gains can overstate tool value, and that matched no-feedback fallback controls are necessary for evaluating tool-augmented agents.

Chubin Zhang, Zhenglin Wan, Xingrui Yu +5

Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore +3

Large Language Model Agents Feedback Inverse Design

Ask PDF