Large language models (LLMs) exhibit strong reasoning and conversational abilities, but ensuring reliable behavior in multi-turn interactions remains challenging. In many real-world applications, agents must succeed in one-shot settings where retries are impossible. Existing approaches either rely on reflection or post-hoc evaluation, which require additional attempts, or assume fully trainable models that cannot leverage proprietary LLMs. We propose an asymmetric actor-critic framework for reliable conversational agents. A powerful proprietary LLM acts as the actor, while a smaller open-source critic provides runtime supervision, monitoring the actor's actions and intervening within the same interaction trajectory. Unlike training-based actor-critic methods, our framework supervises a fixed actor operating in open-ended conversational environments. The design leverages a generation-verification asymmetry: while high-quality generation requires large models, effective oversight can often be achieved by smaller ones. We further introduce a data generation pipeline that produces supervision signals for critic fine-tuning without modifying the actor. Experiments on τ-bench and UserBench show that our approach significantly improves reliability and task success over strong single-agent baselines. Moreover, lightweight open-source critics rival or surpass larger proprietary models in the critic role, and critic fine-tuning yields additional gains over several state-of-the-art methods.
Figures & tables
Figure 1: Existing actor–critic methods assume trainable actors and structured action spaces. We propose an asymmetric actor–critic framework where a lightweight open-source critic supervises a fixed proprietary actor at runtime, enabling reliable control for one-shot multi-turn conversational agents.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Refund Method
Result
Actor Only
Certificate (invalid)
Fail
Asym-AC-SFT
Gift card (correct)
Success
Appendix
Table 9: Example from τ -bench airline where the critic prevents a policy violation in refund processing.
Method
Rental Car Recommendation
Actor Only
C17 (wrong, reward = 0.0)
Asym-AC-SFT
C1 (optimal, reward = 1.0)
Appendix
Table 10: Example from UserBench T-22 where the critic corrects a suboptimal rental car recommendation.
Method
Flight Recommendation
Actor Only
F4 (wrong, reward = 0.0)
Asym-AC-SFT
F16 (optimal, reward = 1.0)
Appendix
Table 11: Example from UserBench T-33 where the critic corrects an incorrect flight recommendation.
Method
Apartment Recommendation
Actor Only
A1 (wrong, reward = 0.0)
Asym-AC-SFT
A13 (optimal, reward = 1.0)
Appendix
Table 12: Example from UserBench T-44 where the critic corrects an incorrect apartment recommendation.
Benchmark
Domain/Env
# Trajectories
# Samples
# Positive
# Negative
τ -bench
Airline
403
2363
713
1650
τ -bench
Retail
171
1184
259
925
UserBench
T-22
424
848
701
147
UserBench
T-33
345
690
548
142
UserBench
T-44
292
584
460
124
UserBench
Total
1061
2122
1709
413
Appendix
Table 13: Statistics of the generated critic training data. Each critic evaluation point in a trajectory corresponds to one training sample. Positive samples denote critic rejections, while negative samples denote approvals. For UserBench, the critic is trained on data aggregated across all environments.
Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake--which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains τ2-Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.
Thomas Zeng, Zelei Cheng, Mingyang Zhou +3
University of Wisconsin–Madison · AI Foundations, Capital One
Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments. In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution. Such failures may not appear in every single run, but can emerge across repeated trials, making reliability across steps and trials critical. However, ensuring agentic reliability is challenging: even frontier LLMs struggle to explain why an action may be wrong, especially in long, intertwined trajectories governed by domain-specific policies. Much recent work relies on prompt-based critique agents, while optimization-based methods lack a systematic way to produce rich verification rationales for training. We address this gap with CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization. CAST analyzes agent trajectories to synthesize structured rationales explaining action validity under partial observability. The resulting critique model is used to construct critique-aware training data for optimizing the policy model. Fine-tuning Qwen3-family models on dynamic tool-calling benchmarks, CAST improves reliability across domains, outperforming GPT-OSS-120B by over 10% pass^4 on Retail tasks and yielding an additional 9% improvement on Telehealth in an out-of-domain setting. These results demonstrate that critique-aware training improves the robustness of LLM agents in realistic dynamic environments.
Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including τ3 and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.