Training Advisors for LLM Agents from Task Outcomes

Sergei Polezhaev; Barys Liskavets; Ori Press; Alexander Golubev

cs.CLOct 7, 2026

Training Advisors for LLM Agents from Task Outcomes

Authors: Sergei Polezhaev, Barys Liskavets, Ori Press, Alexander Golubev

Organizations: Nebius AI

Abstract

Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including $τ^3$ and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.

Figures & tables

Figure 1: Training and using Caddie . (a) Training: the frozen agent $\pi_{b}$ rolls out a task and a step $k$ is chosen randomly; at the prefix $s_{k}$ the critic $\pi_{c}^{\theta}$ samples $G$ critiques; each critique is then appended to the prefix and the frozen agent rolls out $N$ continuations until the end of the episode. The terminal rewards are normalized within the group and update only the critic with DAPO. (b) Inference: the agent decides when to consult the critic through the call_critic tool and receives the critique as the tool output.

Critic	Attached	Wiki	ALFWorld
	MuSiQue
No critic	$21.56\pm 2.07$	$10.38\pm 1.59$	$23.79\pm 3.01$
Prompted Qwen3-4B	$23.25\pm 2.37$	$16.06\pm 2.19$	$20.24\pm 2.89$
Prompted Kimi K3	$29.13\pm 2.46$	$\textit{19.19}\pm 2.23$	$30.22\pm 2.80$
Caddie (ours)	$\mathbf{46.88}\pm 2.68$	$\mathbf{21.13}\pm 2.38$	$\mathbf{49.72}\pm 3.42$

Table 1: In-domain success rates (%) with Qwen3-4B as the base model. Each Caddie critic is trained in the corresponding task setting. Entries report mean

\pm

SEM across tasks, with eight rollouts per task. Bold marks the highest mean; italics mark other means whose

\pm

SEM intervals overlap the leader’s (not a significance test).

Attached passages
		Prompted critic
Base model	No critic	Qwen3-4B	Kimi K3	Caddie (ours)
Qwen3-30B-A3B	$28.44\pm 2.45$	$34.75\pm 2.82$	$\textit{36.00}\pm 2.86$	$\mathbf{41.63}\pm 2.83$
Qwen3.8-27B	$46.63\pm 2.93$	$45.31\pm 2.84$	$41.13\pm 2.77$	$\mathbf{55.94}\pm 2.93$
Kimi K3	$38.56\pm 2.87$	$34.44\pm 2.63$	$30.19\pm 2.36$	$\mathbf{51.00}\pm 2.76$
Wikipedia retrieval

Table 2: Transfer across base models on MuSiQue. Success rates (%) are reported as mean

\pm

SEM across tasks, with eight rollouts per task. Caddie critics are trained with Qwen3-4B and evaluated with each base model without further adaptation. Bold marks the highest mean; italics mark other means whose

\pm

SEM intervals overlap the leader’s (not a significance test).

Base model: Qwen3-4B
		$\tau^{3}$
Critic	DeepDive	Retail	Airline
No critic	$26.95\pm 2.38$	$44.69\pm 1.73$	$31.25\pm 1.57$
Prompted Qwen3-4B critic	$28.91\pm 1.39$	$42.50\pm 0.98$	$31.88\pm 2.49$
Prompted Kimi K3 critic	$27.34\pm 0.72$	$40.31\pm 1.53$	$27.50\pm 2.83$
Agent-RRM	$29.30\pm 0.75$	$40.63\pm 2.15$	$\textit{33.75}\pm 2.63$

Table 3: Transfer of MuSiQue-trained critics to unseen tasks. Entries report success rates (%): the mean over all tasks and seeds

\pm

the SEM across the eight seed-level aggregate accuracies. Shading marks our critics. Bold marks the highest mean; italics mark other means whose

\pm

SEM intervals overlap the leader’s (not a significance test).

Panel of Figure 2 — Figure 2: Solve rates on tasks where Caddie delivered feedback. Navy: feedback-bearing rollouts; lavender: no-critic rollouts on the same tasks. This generator-selected subset gives a descriptive comparison, not a causal estimate. Q4/Q38 denote Qwen3-4B/Qwen3.8-27B; DD denotes DeepDive.

Figure 3: Matched $\tau^{3}$ Retail trajectories after the same change in user intent. Each path summarizes the critic advice and the frozen-generator continuation.

Panel of Figure 4 — Figure 4: DeepDive solve rate by advice type. Lavender (retained baseline): no-critic rollouts on the tasks where that advice type was given; navy: rollouts that received it.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: MuSiQue validation success on a 20-task subset for four reward variants, shown up to 2,000 training steps. All curves use an exponential moving average with weight $0.5$ on the current evaluation. Confidence bands are unavailable; these smoothed traces illustrate training dynamics but do not establish differences in performance, convergence speed, or reward variance.

Parameter	Value
Objective	Token-level DAPO; lower/upper clip $0.20/0.28$
Optimizer	AdamW; learning rate $10^{-6}$ ; $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $\epsilon=10^{-8}$
Schedule	8-update linear warmup from zero, then constant learning rate
Regularization	Weight decay $0.1$ , excluding normalization parameters; global gradient-norm clip $0.2$
Auxiliary losses	No KL or entropy penalty
Batch size	128

Appendix

Table 4: Optimization settings for MuSiQue-attached, MuSiQue-wiki, and ALFWorld.

Qwen3-4B before GRPO
Critic	Attached	Wiki
No critic	$21.56\pm 2.07$	$10.38\pm 1.59$
Qwen3-4B after GRPO
No critic	$49.69\pm 2.98$	$\textit{19.69}\pm 2.36$
Caddie trained with original generator	$51.69\pm 2.87$	$\mathbf{24.44}\pm 2.64$
Caddie trained with GRPO generator	$\mathbf{63.56}\pm 2.98$	$\textit{23.38}\pm 2.63$

Appendix

Table 5: Critic feedback on the GRPO-trained Qwen3-4B generator (step 1,120), with the original instruction-tuned generator as a reference. All evaluations with the GRPO-trained generator require at least three searches. Success rates (%) are mean

\pm

SEM across 200 tasks, with eight rollouts per task. Both critic rows use self-call with at most three critiques per trajectory. Bold marks the highest mean; italics mark other means whose

\pm

SEM intervals overlap the leader’s (not a significance test).

Critic training			Inference protocol
Retrieval	Base model	$k$	Fixed step	Self-call
Evaluation: Qwen3-4B, attached
Wiki	Qwen3-4B	$4$	$\mathbf{28.00}\pm 2.47$	$\textit{26.88}\pm 2.46$
Attached	Qwen3.8-27B	$5$	$\mathbf{30.50}\pm 2.32$	$\textit{28.88}\pm 2.60$
Evaluation: Qwen3.8-27B, wiki
Wiki	Qwen3-4B	$4$	$\mathbf{25.94}\pm 2.63$	$19.06\pm 2.13$

Appendix

Table 6: Selected MuSiQue comparisons where fixed-step intervention has a higher mean than self-call. Both critics are Qwen3-4B models; their training retrieval settings and base models are listed below. Success rates (%) are mean

\pm

SEM across 200 tasks, with eight rollouts per task. Self-call permits at most one critique. Bold marks the highest mean; italics mark other means whose

\pm

SEM intervals overlap the leader’s.

Terminal-outcome critic, $N=1$
Base model	$k=2$	$k=3$	$k=4$	$k=5$
Qwen3-4B	$22.12\pm 1.92$	$17.75\pm 1.66$	$\textit{30.88}\pm 2.18$	$\mathbf{33.69}\pm 2.35$
Qwen3-30B-A3B	$36.44\pm 2.42$	$35.44\pm 2.19$	$\mathbf{41.50}\pm 2.46$	$36.38\pm 2.29$
Terminal-outcome critic, $N=5$
Qwen3-4B	$23.06\pm 1.96$	$13.81\pm 1.37$	$27.88\pm 2.14$	$\mathbf{32.56}\pm 2.30$
Qwen3-30B-A3B	$\textit{34.38}\pm 2.36$	$25.19\pm 1.88$	$\mathbf{35.94}\pm 2.17$	$\textit{34.75}\pm 2.32$

Appendix

Table 7: Sensitivity to neighboring intervention steps on MuSiQue-attached. Each row keeps the base model and critic checkpoint fixed. Entries are success rates (%; mean

\pm

SEM across 200 tasks, with eight rollouts per task). Bold marks the highest mean; italics mark other means whose

\pm

SEM intervals overlap the leader’s.

Panel of Figure 6 — Figure 6: Critic requests per called rollout.

Panel of Figure 7 — Figure 7: DeepDive accuracy by the number of critic requests.

Panel of Figure 8 — Figure 8: How generators use critic feedback and how the advice classes vary across domains.

Panel of Figure 9 — Figure 9: $\tau^{3}$ solve rate by advice type.

Panel of Figure 10 — Figure 10: Qwen3.8-27B on DeepDive with the MuSiQue-wiki Caddie critic. Left: paired solve-rate change (percentage points) on the tasks where the critic was called, by task family. Right: share of called rollouts that improved or degraded after the generator either committed to an answer or performed another search.

Panel of Figure 11 — Figure 11: Qwen3.8-27B solve rate by $\tau^{3}$ task family. Each group uses the same tasks and replicates for every critic; A/W denote the MuSiQue-attached/wiki Caddie checkpoints.

Explore similar work

CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
Aug 31, 2026Amir Saeidi, Zehua Zhang, Rishitosh Singh +6LLM Agents LLM Agent Training
Arizona State University · Cisco Research
Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents
Jul 14, 2026Yaopei Zeng, Congchao Wang, JianHang Chen +3AI Agent Reliability LLM Agent Evaluation
Pennsylvania State University · Virginia Tech · Purdue University +1
Where Do CoT Training Gains Land in LLM based Agents?
Jun 25, 2026Jingyu Liu, Zhiwen Wang, Yuxin Jing +2CoT Reasoning LLM Agents
Gaoling School of Artificial Intelligence Renmin University of China, Beijing, China · ByteDance · Beijing Key Laboratory of Research on Large Models and Intelligent Governance +1

Aug 31, 2026, cs.CLJ/K move · Enter open · S save

CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents

Amir Saeidi, Zehua Zhang, Rishitosh Singh, Naman Ahuja+5

Arizona State University · Cisco Research

LLM Agents LLM Agent Training Long-Horizon LLM Agents LLM Training LLM Agent Reliability

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments. In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution. Such failures may not appear in every single run, but can emerge across repeated trials, making reliability across steps and trials critical. However, ensuring agentic reliability is challenging: even frontier LLMs struggle to explain why an action may be wrong, especially in long, intertwined trajectories governed by domain-specific policies. Much recent work relies on prompt-based critique agents, while optimization-based methods lack a systematic way to produce rich verification rationales for training. We address this gap with CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization. CAST analyzes agent trajectories to synthesize structured rationales explaining action validity under partial observability. The resulting critique model is used to construct critique-aware training data for optimizing the policy model. Fine-tuning Qwen3-family models on dynamic tool-calling benchmarks, CAST improves reliability across domains, outperforming GPT-OSS-120B by over 10% pass^4 on Retail tasks and yielding an additional 9% improvement on Telehealth in an out-of-domain setting. These results demonstrate that critique-aware training improves the robustness of LLM agents in realistic dynamic environments.

Abstract

Explore similar work

CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

Where Do CoT Training Gains Land in LLM based Agents?