Supervised fine-tuning (SFT) on expert trajectories is a standard paradigm for training LLM agents. However, it suffers from a well-known training-inference distribution mismatch, calling for target-aware data synthesis strategies that ground the training data in the target model's native state distribution. To bridge this gap, emerging approaches incorporate the expert either to guide mixed rollouts alongside the target model, or to provide dense step-wise supervision. Nonetheless, existing methods apply expert assistance blind to the target model's inherent behavior: they either disrupt the rollout using sub-optimal random intervention heuristics, or they enforce indiscriminate uniform supervision that unnecessarily overwrites the model's already-correct actions. We introduce JudgeMix, a judge-guided data synthesis framework that dynamically interleaves the target model and expert based on step-wise semantic evaluations. JudgeMix queries expert guidance precisely when the target model struggles, ensuring high trajectory success rates while keeping the state distribution aligned with the target model. It additionally uses the judge evaluation to reweight expert supervision on different steps, preserving model's inherent valid behavior while correcting its errors. On two agentic tool-use benchmarks, AppWorld and AutomationBench, JudgeMix demonstrates consistent superior performance to existing methods, with improvements up to 10 pass@1 points.
Figures & tables
Data Synthesis Strategy
Rollout Action ( atrollout )
Supervision Target ( at∗ )
Supervised Steps
Expert demonstrations
πexpert
πexpert
All steps
Target model rejection sampling
πtarget
πtarget
All steps (if trajectory succeeds)
Target model rollout w/ expert supervision
πtarget
πexpert
All steps
Mixed rollouts (Expert Correction)
πtarget then πexpert
πexpert
Expert steps only
Mixed rollouts (DAgger)
πtarget or πexpert (random heuristic)
πexpert
All steps
JudgeMix
πtarget (if accepted) / πexpert (if rejected)
πexpert
Weighted by judge’s evaluation
Table 1: Comparison of agentic SFT dataset synthesis frameworks based on how the rollout action ( atrollout ) and supervision target ( at∗ ) are assigned at step t . πtarget and πexpert denote the target and expert models, respectively. JudgeMix generates mixed rollouts based on a semantic signal and applies judge-weighted supervision.
Figure 1: Overview of the JudgeMix framework: Given the history at each step, target model first proposes an action followed by the evaluation of the judge that determines if the action makes reasonable progress of solving the task or not. When accepted by the judge, the target model’s action is kept and the rollout continues; when the target model’s action is deemed wrong by the judge, expert intervenes and provides a correct action to guide the rollout ( Sec. 3.1 ). At training time, each state in the rollout is paired with the expert’s action for supervision, where JudgeMix further uses the judge results to downweight steps where the target model already succeeds ( Sec. 3.2 ). Bottom: Additionally, JudgeMix enables collection of erroneous states at steps where the target model fails. These states allow us to obtain corresponding error-recovery supervision.
Method
AppWorld
AutomationBench
Qwen3.5-2B
Qwen3.5-9B
Qwen3.5-9B
Test-N
Test-C
Test-N
Test-C
Test-N
Test-C
TGC
SGC
TGC
SGC
TGC
SGC
TGC
SGC
TGC
TGC
Target Base Model (Zero-shot)
0.1
0.0
0.2
0.1
25.2
14.1
15.4
8.0
4.0
1.0
Expert Demonstrations
32.1
22.8
19.2
12.1
77.4
65.9
65.0
52.7
17.8
8.6
RFT
–
–
–
–
52.4
38.0
37.1
24.8
7.6
4.5
Table 2: Downstream performance (pass@1 averaged over 8 runs) of target models fine-tuned using various data synthesis frameworks on AppWorld and AutomationBench. Compared to existing data synthesis strategies, JudgeMix consistently achieves the highest performance across all target model sizes. Note that RFT and Step-level RFT are not applicable on AppWorld with Qwen3.5-2B model given its near-zero untrained model performance.
Method
AppWorld
AutomationBench
Qwen3.5-2B
Qwen3.5-9B
Qwen3.5-9B
Test-Normal
Test-Challenge
Test-Normal
Test-Challenge
Test-Normal
Test-Challenge
TGC
TGC
TGC
TGC
TGC
TGC
DAgger
35.6
19.2
77.0
66.0
21.0
15.0
JudgeMix ( w=1 )
38.3
21.9
77.5
66.2
28.5
15.1
Table 3: Ablation study on the effect of judge-guided expert intervention. By comparing JudgeMix (without weighted supervision, i.e., w=1 in Eq. 3.3 ) against DAgger, we demonstrate the performance gains of intervening based on semantic correctness rather than random mixing heuristics.
Figure 2: Ablation study on the effect of weighted supervision as w varies in Eq. 3.3 . We plot performance on AppWorld Test-N and Test-C with Qwen3.5-2B and Qwen3.5-9B models. Dashed lines mark the w=1 baseline. We observe that down-weighting the supervision signal on the target model’s already-successful states improves downstream performance.
Method
AppWorld
Qwen3.5-2B
Qwen3.5-9B
Test-N
Test-C
Test-N
Test-C
TGC
TGC
TGC
TGC
JudgeMix w/o error-recovery
33.4
18.3
77.5
68.2
JudgeMix
40.9
23.3
78.7
69.8
Table 4: Ablation study on the effect of error-recovery data. Including this supplementary data in JudgeMix yields significant improvements, particularly for the weaker Qwen3.5-2B model. This suggests the value of explicitly teaching the target models how to correct from their native failure states.
Method
AppWorld
Qwen3.5-2B
Qwen3.5-9B
Test-N
Test-C
Test-N
Test-C
TGC
TGC
TGC
TGC
Expert Demonstrations
32.1
17.8
77.9
60.7
Step-level RFT
0.0
0.1
49.0
36.5
Step-level OPD
17.2
8.0
77.8
67.2
Table 5: Downstream performance (pass@1) of target models fine-tuned using various data synthesis frameworks on the AppWorld benchmark, assuming no task verifier for trajectory filtration.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Table 8
Method
AppWorld
AutomationBench
Qwen3.5-2B
Qwen3.5-9B
Qwen3.5-9B
Total
Traj.
Avg.
Total
Traj.
Avg.
Total
Traj.
Avg.
Trajs
pass@1
Steps
Trajs
pass@1
Steps
Trajs
pass@1
Steps
Expert Demonstrations
720
89.7
15.0
720
89.7
15.0
3,200
19.7
27.5
Step-level RFT
720
0.0
31.6
720
32.8
18.4
3,200
4.0
36.8
Step-level OPD
720
0.0
31.6
720
32.8
18.4
3,200
4.0
36.8
Appendix
Table 8: Dataset statistics for the training trajectories synthesized by different data generation frameworks. We report the total number of trajectories, the trajectory success rate (pass@1) prior to any filtration, and the average number of steps per trajectory.
Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake--which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains τ2-Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.
Thomas Zeng, Zelei Cheng, Mingyang Zhou +3
University of Wisconsin–Madison · AI Foundations, Capital One
As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.
Sashank Pisupati, Henry Broomfield, Eujeong Choi +5
Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including τ3 and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.