Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents

Qisai Liu; Qingyue Jiao; Bangwei Guo; Wenxi Dai; Changyi Yang; Jiahe Liu; Dimitris N. Metaxas; Yiyu Shi

cs.AIOct 16, 2026

Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents

Authors: Qisai Liu, Qingyue Jiao, Bangwei Guo, Wenxi Dai, Changyi Yang, Jiahe Liu, Dimitris N. Metaxas, Yiyu Shi,

Organizations: Iowa State University · University of Notre Dame · Rutgers University · Liquid AI ·

Abstract

Tool-using LLM agents repeatedly face decision points under uncertainty, choosing between autonomous action, additional information gathering, and human escalation. These decisions require balancing task success against interaction cost and the risk of irreversible errors. Learning to make these trade-offs is challenging because existing interaction trajectories record only the outcomes of actions taken, leaving the consequences of alternative actions unobserved. To address this gap, we construct a counterfactual dataset containing 3,115 action outcomes from 1,036 decision points in ToolSandbox. At each decision point, we execute each legal action and evaluate its downstream outcomes based on task completion, interaction cost, and irreversible errors. Using this dataset, we train SURE-Act, a controller that selects actions for a frozen backbone agent at inference time. Our experiments show that learning to rank alternative actions improves offline action-selection utility by 0.0309 over a regression-only baseline. We further show that adding SURE-Act enables end-to-end gains of up to 0.0575 in ToolSandbox's Overall Score with the Qwen3-4B agent. Moreover, SURE-Act outperforms adapted decision rules from SAGE-Agent and Calibrate-Then-Act by 0.1506 and 0.1438, respectively. This work establishes the first counterfactual dataset for ToolSandbox and provides actionable insights into decision-making under uncertainty for tool-using agents.

Figures & tables

Figure 1: Overview of SURE-Act. Counterfactual rollouts from an environment checkpoint evaluate each legal action using a backbone LLM policy, producing success, interaction-cost, and critical-error outcomes. An LLM encoder with LoRA and a three-output prediction head estimates those outcomes from the checkpoint and the candidate action. Training combines an outcome regression loss with a gap-weighted, within-checkpoint preference loss derived from realized rollout utilities. At inference, the trained controller scores the legal actions, selects the one with the highest predicted utility, and executes it using the backbone LLM policy.

Method	Frozen	Executed	Declines	Signal
Value-of-information clarify	✓	$\times$	✓	prior
Budgeted env. queries	✓	$\times$	✓	budget
Calibrate then act	✓	$\times$	✓	threshold
Abstention and escalation	✓	$\times$	✓	heuristic
Uncertainty-shaped RL	$\times$	$\times$	$\times$	reward
Replay for blame	✓	✓	$\times$	branches

Table 1: Where this work sits among methods that act under interaction uncertainty. Columns are defined in the text.

Meta-action	Records	Success	Cost	Critical
ask the user	1028	0.441	0.396	0.043
search, read only	1008	0.574	0.356	0.048
act reversibly	861	0.564	0.364	0.034
commit irreversibly	218	0.471	0.373	0.037
all	3115

Table 2: The derived resource. Every legal action at every checkpoint was executed and scored, over 1036 checkpoints from 378 task IDs and 82 families.

Population	Gain	Better	Floor
Development pool, 8B	+0.0344 $\pm$ 0.0082	20/22	+0.0068
Unseen families, 8B	+0.0216 $\pm$ 0.0093	18/21	+0.0011
Unseen families, 4B	+0.0575 $\pm$ 0.0095	20/22	-0.0012
Unseen families, 14B	+0.0252 $\pm$ 0.0101	17/20	+0.0013

Table 3: The repaired controller across four evaluation settings. Each row includes the serving-noise floor from its evaluation job.

Rule	score	vs agent	better	rate	ours over it
SURE-Act	0.451	+0.0299 $\pm$ 0.0104	17/22	0.1917
SAGE-Agent	0.300	-0.1224 $\pm$ 0.0142	1/22	1.0000	+0.1506 22/22
Calibrate-Then-Act	0.307	-0.1152 $\pm$ 0.0143	0/22	1.0000	+0.1438 21/22
agent run twice	0.418	-0.0051 $\pm$ 0.0033	11/22	0.0000

Table 4: End to end on ToolSandbox’s Overall Score, as the difference from the unmodified agent. Every rule was scored in one job and paired per scenario.

Rule	Rate	Turns	Cost
SURE-Act	0.192	+0.133	-0.068
SAGE-Agent	1.000	+0.543	+0.080
Calibrate-Then-Act	1.000	+0.492	+0.077
agent run twice	0.000	-0.004	-0.000

Table 5: What each rule costs the agent it modifies, paired per scenario against the unmodified run. Only the repaired rule shortens the episode’s cost rather than adding to it.

$k$	Rate	Turns	Gain	Interval	Better
development
2	0.089	-0.091	+0.0376	[+0.0148, +0.0605]	17/21
3 (ours)	0.192	+0.281	+0.0358	[+0.0163, +0.0553]	19/21
4	0.308	+0.677	+0.0353	[+0.0165, +0.0542]	19/21
5	0.439	+1.064	+0.0119	[-0.0107, +0.0344]	12/21
floor	0.000	-0.008	-0.0008	[-0.0080, +0.0065]	11/21

Table 6: The handover threshold swept in one job per pool. Turns is the change in episode length against the unmodified agent, and the noise floor is that agent run twice.

selector	gain	better on
SURE-Act	+0.0092 $\pm$ 0.0016	93/120
cost-aware verbalised	+0.0024 $\pm$ 0.0019	71/120
multi-signal gate	+0.0023 $\pm$ 0.0019	78/120
prompted, plain	-0.0016 $\pm$ 0.0021	57/120
prompted, cost-aware	-0.0028 $\pm$ 0.0025	58/120
cost-aware entropy	-0.0057 $\pm$ 0.0015	35/120

Table 7: All 18 selectors paired against the same per-split best fixed action, over the same 120 partitions from one scoring run.

Figure 2: Closed-loop evaluation on the development pool. Results are averaged over 22 paired partitions. Top: Overall Score for each controller and the unmodified agent, with paired score differences annotated. Bottom: Relationship between rule effectiveness and intervention frequency across nine decision rules (

r=-0.77

). T

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Encoder	Qwen/Qwen3-1.7B
Epochs	4
Batch size	8
Sequence length	1,024
Learning rate	$10^{-4}$
Utility weight $\lambda$	1

Appendix

Table 8: Training configuration for the 120-partition controller-versus-baseline evaluation, read from the run artefacts. The 120 runs differ only in split seed. The ranking-loss ablation is a separate set of runs and uses uncentered targets.

Figure 3: Loss and decision quality are decoupled across 120 training runs. Above, the within-run correlation between them, median

-0.082

, with dark bars marking the runs where the two disagree. Below, how each run moved from its first evaluation to its last, with the 36 dark points improving their decision while the loss rose.

Figure 4: Within-checkpoint action discrimination. Area under the ROC curve (AUROC) for identifying the uniquely best action at each checkpoint. Seen and unseen correspond to 82 training families and 89 held-out families. Bars show family-clustered confidence intervals, with dark indicating intervals entirely above chance and amber indicating intervals containing chance. Three of four actions exceed chance on seen families, while none do so on unseen families.

Figure 5: Offline gain over the per-split best fixed action. Results for six of the eighteen selectors: the three that exceed the baseline and the three weakest of those that do not. SURE-Act achieves the largest gain, 3.8 times that of the next selector. Dots represent paired means, and whiskers indicate two standard errors. Table 7 reports results for all eighteen selectors.

Figure 6: Training-objective ablations. Each bar represents the paired difference for one partition. Left: Within-decision ranking versus regression-only objective across 330 partitions (median

+0.0257

; one unchanged). Right: Three outcome heads versus a single scalar across 312 partitions (median

+0.0033

; 12 unchanged). Dark and amber bars indicate gains and losses from the full objective, while dashed lines mark the medians.

Design choices
Change	Paired difference	$n$
within-checkpoint ranking term	+0.0309 $\pm$ 0.0028	330
three outcome heads over a scalar	+0.0082 $\pm$ 0.0016	312
Insensitive to
a larger encoder	+0.0067 $\pm$ 0.0035	85
batch size	-0.0028 $\pm$ 0.0034	90

Appendix

Table 9: Paired ablations of the controller. Each row is a separate comparison on its own partitions.

	gain	better on	$p$
over best fixed action	+0.0092 $\pm$ 0.0016	93/120	$<$ 0.0001
SURE-Act vs cost-aware verbalised	+0.0068 $\pm$ 0.0023	80/120	0.0003
SURE-Act vs best prompted	+0.0107 $\pm$ 0.0022	86/120	$<$ 0.0001

Appendix

Table 10: Controller against the per-split best fixed action over 120 family-held-out partitions.

Figure 7: Decision-rule behavior across decision points. Distribution of five action outcomes for each decision rule. Both ported rules override at nearly every decision point, and most overrides involve asking questions. In contrast, SURE-Act overrides at approximately one-sixth of decision points, primarily through lookups. Outcomes accounting for less than 1% are labeled above the corresponding bars.

Figure 8: Partition-weighted vs. family-weighted effects. Faint circles and solid diamonds represent partition-weighted and family-weighted effects, with connecting lines indicating the magnitude of the weighting-induced change. Dark indicates rules that remain positive under family weighting, while amber indicates those that do not. Deferral, the repetition cap, and their combination reverse from positive to negative under family weighting (thicker lines), whereas SURE-Act is the only repair rule that retains a positive effect.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 277 papers

Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning
Jun 5, 2026Yijin Zhou, Linqian Zeng, Xiaoya Lu +4Reinforcement Learning LLM Uncertainty Estimation
Shanghai Jiao Tong University, China · Shanghai Artificial Intelligence Laboratory, China · Shanghai Innovation Institute, China
Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
Sep 14, 2026Yiwei Yang, Haoxiang Zhang, Bingbing Wen +6LLM Agent Reliability RL for Tool Use
University of Washington · University of California San Diego · Stanford University
Utility-Guided Agent Orchestration for Efficient LLM Tool Use
Mar 20, 2026Boyan Liu, Gongming Zhao, Hongli XuLLM Agent Orchestration LLM Tool Use
University of Science and Technology of China, Hefei, Anhui, China · Suzhou Institute for Advanced Research, University of Science and Technology of China

Jun 5, 2026J/K move · Enter open · S save

Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning

Yijin Zhou, Linqian Zeng, Xiaoya Lu, Wenyuan Xie+3

Shanghai Jiao Tong University, China · Shanghai Artificial Intelligence Laboratory, China · Shanghai Innovation Institute, China

Reinforcement Learning LLM Uncertainty Estimation LLM Agent Reliability RL for Tool Use Uncertainty Estimation in RL

Open paper AskPDF arXiv

Large language model (LLM)-based agents often make suboptimal tool-use decisions, including unsupported tool invocation and hallucinated direct responses, which may accumulate errors throughout multi-step interactions. Existing approaches mainly improve these behaviors through inference-time correction or coarse-grained reward signals based on decision outcomes and structured checklists, leaving the uncertainty characteristics of agent decisions underexplored. We observe that decision-oriented reinforcement learning tends to weaken the uncertainty separation between correct and incorrect actions, resulting in overconfident mistakes and weaker exploration signals. Therefore, we propose TRUST, which incorporates uncertainty quantification into reward design as a repulsive force for maintaining uncertainty separation, and labels lightweight key-turn annotations for unified post-training of multi-turn trajectories. Experimental results across diverse tool-use benchmarks show that TRUST consistently enhances both decision quality and agent performance while maintaining more reliable uncertainty estimates during optimization.

Abstract

Tool-Using Agents

Explore similar work

Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

Utility-Guided Agent Orchestration for Efficient LLM Tool Use