Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities

Yikai Zhao; Saurabh Pandey; Pradeep Kumar Misra

cs.AIOct 2, 2026

Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities

Authors: Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra

Organizations: Amazon

Abstract

A deployed LLM agent emits tool calls, queries, and code that can be silently wrong -- by the time the error surfaces, the action has run. Frontier chat APIs hide the model's token probabilities; the agent's stated confidence barely beats chance on the mistakes that matter; and resampling does not help, since frontier models are highly repetitive, reproducing the same call across samples. We recover the missing signal from a low-cost open-weight surrogate run in parallel. It reads the same context, schema, and proposed action as the agent, then scores the call from its own log-probabilities through a family of complementary readouts: teacher forcing and request-PMI weigh the likelihood of each argument value, a discriminative verdict judges the call as a whole, and tool-choice competition tests the function against its siblings. One principle says which to trust: a generative likelihood localizes wrong argument values, while the verdict catches holistically wrong calls. When the error type is unknown, an ensemble is the low-regret default. The readout is training-free, needs no access to the agent's internals, and costs one prefill pass alongside the tool call. On difficult coding tasks it reaches AUROC 0.825 where the actor's stated confidence is near chance (0.598), and the generative readouts beat it by +0.07 to +0.28 across three further actors. Against self-consistency it gains +0.14 to +0.19 on near-deterministic actors, at 1/K the cost. The signal drives two deployment modes: a real-time gate escalating the least-trustworthy calls for review (+0.05 to +0.30 accepted-action accuracy at 50% coverage), and confidence feedback, returning the tool result with the score so the agent adapts its next step -- lifting task success on live-execution benchmarks (+0.119 and +0.137, p <= 1e-4) and beating a random-value control where step errors are silent (+0.078, p = 0.003).

Figures & tables

Figure 1: The proxy-confidence sidecar: a training-free plug-in hook. The sidecar (dashed) taps the agent’s tool-call hook, reads only

(\text{context},\text{schema},\text{action})

—no logits, weights, or fine-tuning—and emits a confidence

c

, scored in parallel with tool execution. Drawn here are the two in-loop modes: a real-time oversight gate (Mode A: high

c

commit, low

c

escalate to a human) and confidence feedback (Mode B: return the tool result together with

c

to the actor, which chooses its next move). Details in § 7 –§ 8 .

Signal	Readout	Targets
Verdict	$P(\text{YES})$ on “is this correct?”, renormalized over YES/NO	coarse tool/code correctness
Teacher forcing	per-token $\log P$ of $\mathcal{A}$ ’s exact call, localized to argument value tokens	wrong argument values
Request-PMI	$\log P(o\mid\text{full})$ ${-}\log P(o\mid\text{req.\ withheld})$ , on value tokens	groundedness of args in request
Tool-choice competition	does $\mathcal{S}$ prefer $\mathcal{A}$ ’s function over $k$ siblings?	wrong function choice

Table 1: The four proxy readouts and the failure mode each targets.

Actor (family; $n$ )	TF	PMI	verd.	self	PMI $-$ self
C-120b (family C; 775)	0.735	0.794	0.526	0.515	$+0.279$
C-20b (family C; 771)	0.743	0.793	0.552	0.534	$+0.255$
B (family B; 1261)	0.636	0.721	0.546	0.648	$+0.071$

Table 2: PMI improves on stated confidence for all three actors outside family A. Proxy signals scoring each actor’s xLAM calls through Qwen3-32B; best proxy in bold. Actors are de-identified (§ 3 ); the comparison of interest is proxy-vs-self within each row, not between rows.

Dataset	locus	verd.	TF	PMI	actor	fuse	$P^{\star}$
AppWorld	holistic	0.894	n/a	n/a	0.788	n/a	.996
BFCL base	holistic	0.733	0.633	n/a	0.703	0.720	.945
BFCL long	holistic	0.719	0.640	n/a	0.682	0.700	.968
BFCL v3	args	0.711	0.549	0.714	0.636	0.700	.950
parallel	args	0.661	0.653	0.662	0.577	0.695	.980
NESTFUL	args	0.594	0.643	0.554	0.620	0.655	$.840$

Table 3: The champion signal tracks the error locus. Per-signal AUROC (32B scorer) vs. the actor’s self-assessment, best single signal in bold.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: The gain is access, not scale. On one Qwen3-32B: the verdict token’s log-probability (blue) vs. the same model’s verbalized confidence (grey) and the actor’s own self-assessment (orange, A1), across six datasets. Capability is held constant between blue and grey.

Fixed policy	worst-case regret (where)	mean regret
always verdict	$+0.180$ (miss_param)	$0.046$
always teacher-forcing	$+0.165$ (BFCL v3)	$0.087$
oriented fusion	$\mathbf{+0.019}$ (BFCL long)	$\mathbf{0.007}$

Appendix

Table 4: Oriented fusion has no catastrophic regime. Worst-case regret over the six single-call regimes where verdict and TF are both defined; lower is more robust.

Dataset	$N$	base	AUROC	acc@50% ( $\Delta$ )	AURC $\downarrow$	rand
BFCL (verdict+TF)	2388	0.692	0.596	0.743 ( $+0.051$ )	0.241	0.308
xLAM (verdict)	412	0.367	0.713	0.524 ( $+0.158$ )	0.484	0.633
xLAM (needle)	725	0.542	0.890	0.843 ( $+0.300$ )	0.179	0.458
NESTFUL (needle)	238	0.613	0.788	0.832 ( $+0.218$ )	0.178	0.387
tau2 (trajectory)	149	0.772	0.679	0.867 ( $+0.095$ )	0.149	0.228

Appendix

Table 5: Ranking by proxy confidence beats random abstention everywhere. Accept the top-confidence fraction and abstain on the rest.

Figure 3: Self-consistency collapses as the actor becomes deterministic. SC AUROC vs. sample unanimity (xLAM, $K{=}5$ , execution-free; $r{=}{-}0.96$ ); proxy TF/PMI (shaded) is entropy-invariant.

Dataset	Actor	Qwen	Gemma	self	$P$ (Q/G)
AppWorld	A1	0.894	0.836	0.788	1.00/.66
AppWorld	A2	0.919	0.884	0.914	.57/.14
BFCL base	A1	0.733	0.728	0.703	.95/.90
BFCL base	A2	0.800	0.823	0.732	1.00/1.00
BFCL long	A1	0.719	0.723	0.682	.97/.98
BFCL long	A2	0.820	0.821	0.759	1.00/1.00

Appendix

Table 6: The verdict is not one actor

\times

scorer artifact. Verdict AUROC across actors and scorers.

P

=

paired-bootstrap exceedance vs. that actor’s self-assessment; italic self

=

actor not beaten (self-assesses well).

Dataset	verd.	TF/PMI	used by (actors; scorers)
AppWorld	✓	—	grid, hero, Mode B (A+B+C; Q,Gm)
BFCL base/long	✓	—	grid, hero, cap14 (A+B+C; Q,q,Gm)
BFCL parallel/v3	✓	✓	hero, cap14, battery (A; Q,q)
BFCL miss_func/param	✓	✓ †	hero, cap14 (A; Q,q)
xLAM	✓	✓	hero, cap14, selective, active, SC, § 6 (A+B+C; Q,q)
NESTFUL	✓	✓	hero, cap14, selective, active (A; Q,q)

Appendix

Table 7: Which datasets support which signal, and where each result is used.

Figure 4: Teacher forcing is size-invariant; the verdict is not. Each dumbbell joins 14B (orange) and 32B (blue) AUROC for one cell, same items and prefill. Top: teacher forcing. Middle: verdict. Bottom: AppWorld exception. Chance $=0.5$ (dotted).

Dataset	Signal	14B	32B	$\Delta$
Teacher forcing—access-bound ( $\|\Delta\|{\leq}0.015$ , TOST-equivalent)
NESTFUL/A1	TF ${}_{\text{val}}$	0.636	0.650	$-.014$
NESTFUL/A2	TF ${}_{\text{val}}$	0.604	0.611	$-.006$
xLAM/A1	TF ${}_{\text{val}}$	0.674	0.670	$+.004$
xLAM/A2	TF ${}_{\text{val}}$	0.702	0.690	$+.011$
parallel/A1	TF ${}_{\text{val}}$	0.628	0.633	$-.005$

Appendix

Table 8: 14B matches 32B on teacher forcing, trails on request-PMI and the verdict. Qwen3-14B vs. 32B, matched items per signal.

\Delta{=}

14B

-

32B; verdict uses matched YES/NO teacher-forcing. Full tally in § 6 .

Readout	$p50$	$p90$	$p99$
Teacher forcing	$159$ ms	$166$ ms	$179$ ms
Verdict	$123$ ms	$132$ ms	$137$ ms

Appendix

Table 9: A scored call is a single prefill of

{\approx}160

ms. Measured sidecar latency at realistic full-context length (

{\approx}1.5

k tokens; Qwen3-14B fp8, one L40S GPU).

	nl2bash		AppWorld test_challenge
Arm	success	vs. control	success	vs. control
control (no number)	$0.399$	—	$0.595$	—
vigilance (no number)	$0.389$	$-0.010$ n.s.	—	—
random $U(0,1)$	$0.440$	$+0.041$ n.s.	$0.727$	$+0.132$ ( $p{<}10^{-4}$ )
verdict only	$0.430$	$+0.031$ n.s.	—	—
request-PMI	$0.472$	$+0.073$ ( $p{=}.029$ )	—	—

Appendix

Table 10: Inline readout comparison, and why a random-value arm is required. A2 actor; each readout appended to the real tool result. Contrasts are paired against the unannotated control; nl2bash arms all cover the same 193 tasks, AppWorld all 417. Vigilance announces the audit but shows no number; random shows an independent

U(0,1)

draw. The two benchmarks diverge on the random arm: on nl2bash it is flat and fusion beats it, so the annotation’s content matters; on AppWorld it matches fusion, so there any scalar suffices. Against the unannotated control alone both benchmarks would have looked alike.

Dataset	Self-verif.?	blind	all	all vs. blind / placebo (Holm $p$ )
xLAM, A1 ( $n{=}196$ )	no (schema)	$0.066$	$\mathbf{0.148}$	$0.004$ / $0.013$
xLAM, A2 ( $n{=}196$ )	no (schema)	$0.097$	$0.138$	$0.46$ / $0.006$
NESTFUL ( $n{=}184$ )	yes (recomp.)	$0.844$	$0.853$	n.s. / n.s.
BFCL-v4 ( $n{=}1499$ )	yes (real res.)	$0.892$	$0.892$	n.s. (all $\approx$ )
Selective review at fixed budget $k{=}1$ call (signal- vs. random-gated):
xLAM ( $n{=}192$ )	—	sig $0.92$ / rand $0.42$		$p{<}10^{-4}$ (95/1)

Appendix

Table 11: Feedback helps only when the actor cannot self-verify. Signal-guided self-review vs. deranged-placebo, all -signal arm. Self-verif.

=

can the actor check the call at review time. Bottom: repair rate at a one-call audit budget, signal- vs. random-gated.

Figure 5: Failure-mode map (same data as Table 3 ). Signal $\times$ dataset AUROC; black box $=$ best proxy signal per dataset; grey $=$ signal undefined for that action type.

Figure 6: Teacher forcing tracks argument-value accuracy; the verdict does not. Spearman $\rho$ vs. deterministic per-parameter accuracy (no LLM judge) as the target sharpens to literal argument values (NESTFUL, A2). TF rises to $\rho{=}0.41$ ; verdict $0.02$ .

Signal	AUROC	type	note
Ensemble disagreement	0.70–0.80	epistemic	orthogonal, near-free
Self-consistency ( $k{=}5$ )	0.71–0.75	epistemic	good on easy calls
Semantic entropy	0.68–0.71	epistemic	no gain over above
Schema-satisfaction	0.50–0.51	determ.	null (frontier actors)
verdict variants (vs. plain verdict, same rows)
Debiased verdict	0.51–0.67	discrim.	no gain on tool calls

Appendix

Table 12: Extra readouts an open scorer enables, and verdict variants.

Explore similar work

Apr 17, 2026

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated against human annotation. We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains, nine production LLMs, and a 100-label human-validated subset. We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation. Substring-based judging agrees with human annotation at kappa=0.049 (chance-level); a three-LLM ensemble reaches kappa=0.432 (moderate) with a conservative bias. Under validated evaluation, a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62 (range 0.46-0.73 across models). Rejection (catching bad parameters) and recovery (correcting after acceptance) are independent model capabilities (Spearman rho=0.126, p=0.747). A tuned runtime interceptor reduces hallucination on GPT-4o-mini by 23.0 percentage points under a concurrent n=600 control, but shows no significant effect on Gemini-2.0-Flash, whose aggressive parameter rejection eliminates the target failure mode. All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.

Bhaskar Gurram

Zasti Inc., Ashburn, VA, USA

AI Agent Reliability Tool-Augmented Language Model Agents LLM-as-a-Judge LLM Agent Evaluation AI Agent Benchmarks LLM Reliability+2

Ask PDF

Sep 4, 2026

Agents' Overreliance on Unreliable Tools

LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the right tools, assuming that tool returns are reliable. However, tools can return plausible but incorrect outputs. We evaluate 14 models with three tools (web search, an LLM sub-agent, and a code executor), corrupting their returns to examine whether agents overrely on unreliable tools. Agents adopt corrupted returns at high rates, with mean adoption exceeding one third for every tool and reaching 68.0% for web search. Corrupted returns also frequently override correct answers that agents give without tools, and this reliance persists in more complex tasks and in tasks that combine multiple tools. Agents often make more tool calls under corrupted returns, and reasoning traces show them questioning a return and even stating the correct answer. Nevertheless, they still pass the corrupted content to users, rarely warning them of the conflict. We further test interventions spanning prompts, tool metadata, post-training, and activation steering. Verification prompts reduce adoption across all three tools while largely preserving accuracy with correct returns, and reliability labels provide partial mitigation in web search. The tested post-training methods offer limited gains, and the effect of activation steering depends on the model and intervention strength. Overall, our findings provide a basis for diagnosing tool overreliance and for developing agents that assess tool returns rather than assuming their reliability.

Hoyeol Yang, Woojung Song, Taewon Kim +3

Seoul National University, Seoul, Republic of Korea

LLM Tool Use LLM Agent Reliability LLM Agent Evaluation

Ask PDF

Jul 6, 2026

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability

Ask PDF