Beyond Corrected Actions: Training LLM Agents on Textual Critiques

Thomas Zeng; Zelei Cheng; Mingyang Zhou; Sambit Sahu; William Campbell; Genta Indra Winata

cs.CLOct 14, 2026

Beyond Corrected Actions: Training LLM Agents on Textual Critiques

Authors: Thomas Zeng, Zelei Cheng, Mingyang Zhou, Sambit Sahu, William Campbell, Genta Indra Winata

Organizations: University of Wisconsin–Madison · AI Foundations, Capital One

Abstract

Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake--which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains $τ^2$ -Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.

Figures & tables

Figure 1: Overview of CAFT. (a) The student model is rolled out and an expert labels every agent turn, writing a critique

c_{t}

and a corrected action

\tilde{a}_{t}

for each incorrect turn. (b) Action distillation replaces the incorrect step with

\tilde{a}_{t}

while CAFT keeps the incorrect step in context and trains on the critique followed by the corrected action. (c) Task success rate of Gemma 4 E2B on

\tau^{2}

Airline+Retail and WebShop.

Environment	Student Model	Base	Action	CAFT
			Distillation	Correction	Critique	Full
$\tau^{2}$ AR	Gemma 4 E2B	24.5	35.9	37.0	42.6	46.6
	Gemma 4 E4B	42.4	49.7	49.3	53.2	52.1
	Gemma 4 31B	65.1	66.7	67.0	69.4	69.4
	Nemotron 3 Nano 4B	47.8	52.1	57.6	59.3	60.3
WebShop	Gemma 4 E2B	43.1	54.6	55.9	60.3	65.3

Table 1: Comparison of CAFT and its ablated variants against action distillation. Task success (%) on

\tau^{2}

AR and WebShop for the base policies, the action-distillation baseline, and CAFT and its ablations. CAFT-Critique and CAFT consistently outperform all other methods in all student–environment pairs.

Detailed	Privileged	Gemma 4	Gemma 4	Nemotron 3
prompt	info	E2B	E4B	Nano 4B
✓	✓	46.6	52.1	60.3
✓	✗	34.2	44.7	57.7
✗	✓	40.1	50.0	60.7
✗	✗	32.9	43.3	57.2

Table 2: Ablating privileged information and detailed guidance in the expert prompt.

\tau^{2}

AR task success (%) of CAFT trained on critiques from Gemma 4 31B, varying whether the expert receives privileged task information and a detailed critique prompt. Privileged information has a larger impact than the detailed prompt.

Environment	Student	Expert Distill.	CAFT	CAFT +BC
$\tau^{2}$ AR	Gemma 4 E2B	50.4	46.6	53.7
	Gemma 4 E4B	53.8	52.1	60.1
	Nemotron 3 Nano 4B	53.3	60.3	56.0
WebShop	Gemma 4 E2B	19.1	65.3	—
	Gemma 4 E4B	16.4	62.6	—
	Nemotron 3 Nano 4B	15.0	58.9	—

Table 3: Comparison of CAFT with expert-trajectory distillation. Task success (%). Expert Distillation behavior-clones trajectories from Gemma 4 31B; CAFT +BC mixes 50% expert trajectories with 50% CAFT data. Expert distillation collapses on WebShop, so CAFT +BC is not run there.

Environment	Student Model	iteration 1	iteration 2
$\tau^{2}$ AR	Gemma 4 E2B	46.6	48.2
	Gemma 4 E4B	52.1	55.2
	Gemma 4 31B	69.4	65.4
	Nemotron 3 Nano 4B	60.3	58.8
WebShop	Gemma 4 E2B	65.3	72.1
	Gemma 4 E4B	62.6	64.9

Table 4: Using CAFT for a second iteration. Task success (%) after one and two iterations of CAFT. A second iteration helps every student on WebShop and the smaller Gemma students on

\tau^{2}

AR.

Figure 2: Using CAFT as a warm start for RL. GRPO on

\tau^{2}

AR with Nemotron 3 Nano 4B from three initializations. (a) Validation task success, averaged over the airline and retail validation splits. (b) Mean reward of on-policy rollouts during training.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: $\tau^{2}$ -Bench expert annotation prompt: system message . The per-domain note of Figure 4 is inserted immediately before item 1, and the agent’s full domain policy and tool schema are appended to it verbatim.

Panel of Figure 4 — Figure 4: $\tau^{2}$ -Bench domain notes , one per domain, inserted into the expert system message of Figure 3 .

Panel of Figure 5 — Figure 5: $\tau^{2}$ -Bench expert annotation prompt: per-step user message. Under the no-privilege condition only the <conversation_so_far> and <step_the_agent_just_took> blocks are kept.

Panel of Figure 6 — Figure 6: WebShop expert annotation prompt: system message. The expert sees the shopping instruction, the privileged goal specification, the buttons on the agent’s current page, and the episode so far.

Panel of Figure 7 — Figure 7: WebShop expert annotation prompt: per-step user message.

Panel of Figure 8 — Figure 8: WebShop student system prompt used by our ReAct harness. The CAFT system addendum of Figure 11 is appended to it during fine-tuning.

Panel of Figure 9 — Figure 9: No-privilege opening paragraph , which replaces the first two paragraphs of the expert system message of Figure 3 .

Panel of Figure 10 — Figure 10: Minimal critique guidance , which replaces criticality and correctness clauses 1 and 2 of Figure 3 ; the output format and the first-person directive are unchanged.

Panel of Figure 11 — Figure 11: CAFT (full) : student system addendum and interjection $\rho$ .

Panel of Figure 12 — Figure 12: CAFT-Correction : student system addendum and interjection $\rho$ .

Panel of Figure 13 — Figure 13: CAFT-Critique : student system addendum and interjection $\rho$ .

Parameter	Value
Advantage estimator	GRPO
Rollouts per prompt ( $n$ )	16
Prompts per step	32
Mini-batch size	16
Micro-batch size per GPU	1
Learning rate	$5\times 10^{-7}$

Appendix

Table 5: GRPO hyperparameters for the

\tau^{2}

AR warm-start experiments using Nemotron 3 Nano 4B.

Student Model	$\lambda=0.01$	$\lambda=0.1$	$\lambda=0.5$
Gemma 4 E2B	45.8	46.6	39.4
Nemotron 3 Nano 4B	53.0	60.3	54.7

Appendix

Table 6:

\tau^{2}

AR task success (%) as a function of the correct-turn loss weight

\lambda

(CAFT uses

\lambda=0.1

everywhere else in the paper). All arms share the same student trajectories and expert annotations and differ only in

\lambda

Student	$0\%$ (CAFT)	$25\%$ BC	$50\%$ BC	$75\%$ BC
Gemma 4 E2B	46.6	47.2	53.7	51.5
Gemma 4 E4B	52.1	56.3	60.1	52.8
Nemotron 3 Nano 4B	60.3	56.7	56.0	56.5

Appendix

Table 7:

\tau^{2}

AR task success (%) as a function of the expert-trajectory fraction in CAFT +BC. The

0\%

column is pure CAFT and the reference for the sweep.

		Base		CAFT		CAFT +BC
Environment	Student	Succ.	Corr./ep.	Succ.	Corr./ep.	Succ.	Corr./ep.
$\tau^{2}$ AR	Gemma 4 E2B	66.7	4.19	70.3	3.20	70.4	1.19
	Gemma 4 E4B	71.4	2.51	76.1	2.08	71.4	1.11
	Nemotron 3 Nano 4B	77.9	2.55	81.1	2.15	69.9	1.32
WebShop	Gemma 4 E2B	80.0	4.77	72.0	3.73	—	—
	Gemma 4 E4B	80.0	5.25	68.0	2.82	—	—

Appendix

Table 8: Expert Interventions. Task success (%) when the expert corrects flagged actions at inference time, alongside the average number of expert corrections per episode (Corr./ep.).

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 564 papers

Training Advisors for LLM Agents from Task Outcomes
Oct 7, 2026Sergei Polezhaev, Barys Liskavets, Ori Press +1LLM Agent Training RL for LLM Agents
Nebius AI
CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
Aug 31, 2026Amir Saeidi, Zehua Zhang, Rishitosh Singh +6LLM Agents LLM Agent Training
Arizona State University · Cisco Research
Asymmetric Actor-Critic for Multi-turn LLM Agents
Mar 31, 2026Shuli Jiang, Zhaoyang Zhang, Yi Zhang +3Asynchronous Advantage Actor-Critic LLM Agent Reliability
AWS Agentic AI

Oct 7, 2026J/K move · Enter open · S save

Training Advisors for LLM Agents from Task Outcomes

Sergei Polezhaev, Barys Liskavets, Ori Press, Alexander Golubev

Nebius AI

LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including

τ^3

and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.

Abstract

LLM Agent Training

Explore similar work

Training Advisors for LLM Agents from Task Outcomes

CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents

Asymmetric Actor-Critic for Multi-turn LLM Agents