Supervised fine-tuning (SFT) on expert trajectories is a standard paradigm for training LLM agents. However, it suffers from a well-known training-inference distribution mismatch, calling for target-aware data synthesis strategies that ground the training data in the target model's native state distribution. To bridge this gap, emerging approaches incorporate the expert either to guide mixed rollouts alongside the target model, or to provide dense step-wise supervision. Nonetheless, existing methods apply expert assistance blind to the target model's inherent behavior: they either disrupt the rollout using sub-optimal random intervention heuristics, or they enforce indiscriminate uniform supervision that unnecessarily overwrites the model's already-correct actions. We introduce JudgeMix, a judge-guided data synthesis framework that dynamically interleaves the target model and expert based on step-wise semantic evaluations. JudgeMix queries expert guidance precisely when the target model struggles, ensuring high trajectory success rates while keeping the state distribution aligned with the target model. It additionally uses the judge evaluation to reweight expert supervision on different steps, preserving model's inherent valid behavior while correcting its errors. On two agentic tool-use benchmarks, AppWorld and AutomationBench, JudgeMix demonstrates consistent superior performance to existing methods, with improvements up to 10 pass@1 points.
Figures & tables
Data Synthesis Strategy
Rollout Action ( atrollout )
Supervision Target ( at∗ )
Supervised Steps
Expert demonstrations
πexpert
πexpert
All steps
Target model rejection sampling
πtarget
πtarget
All steps (if trajectory succeeds)
Target model rollout w/ expert supervision
πtarget
πexpert
All steps
Mixed rollouts (Expert Correction)
πtarget then πexpert
πexpert
Expert steps only
Mixed rollouts (DAgger)
πtarget or πexpert (random heuristic)
πexpert
All steps
JudgeMix
πtarget (if accepted) / πexpert (if rejected)
πexpert
Weighted by judge’s evaluation
Table 1: Comparison of agentic SFT dataset synthesis frameworks based on how the rollout action ( atrollout ) and supervision target ( at∗ ) are assigned at step t . πtarget and πexpert denote the target and expert models, respectively. JudgeMix generates mixed rollouts based on a semantic signal and applies judge-weighted supervision.
Figure 1: Overview of the JudgeMix framework: Given the history at each step, target model first proposes an action followed by the evaluation of the judge that determines if the action makes reasonable progress of solving the task or not. When accepted by the judge, the target model’s action is kept and the rollout continues; when the target model’s action is deemed wrong by the judge, expert intervenes and provides a correct action to guide the rollout ( Sec. 3.1 ). At training time, each state in the rollout is paired with the expert’s action for supervision, where JudgeMix further uses the judge results to downweight steps where the target model already succeeds ( Sec. 3.2 ). Bottom: Additionally, JudgeMix enables collection of erroneous states at steps where the target model fails. These states allow us to obtain corresponding error-recovery supervision.
Method
AppWorld
AutomationBench
Qwen3.5-2B
Qwen3.5-9B
Qwen3.5-9B
Test-N
Test-C
Test-N
Test-C
Test-N
Test-C
TGC
SGC
TGC
SGC
TGC
SGC
TGC
SGC
TGC
TGC
Target Base Model (Zero-shot)
0.1
0.0
0.2
0.1
25.2
14.1
15.4
8.0
4.0
1.0
Expert Demonstrations
32.1
22.8
19.2
12.1
77.4
65.9
65.0
52.7
17.8
8.6
RFT
–
–
–
–
52.4
38.0
37.1
24.8
7.6
4.5
Table 2: Downstream performance (pass@1 averaged over 8 runs) of target models fine-tuned using various data synthesis frameworks on AppWorld and AutomationBench. Compared to existing data synthesis strategies, JudgeMix consistently achieves the highest performance across all target model sizes. Note that RFT and Step-level RFT are not applicable on AppWorld with Qwen3.5-2B model given its near-zero untrained model performance.
Method
AppWorld
AutomationBench
Qwen3.5-2B
Qwen3.5-9B
Qwen3.5-9B
Test-Normal
Test-Challenge
Test-Normal
Test-Challenge
Test-Normal
Test-Challenge
TGC
TGC
TGC
TGC
TGC
TGC
DAgger
35.6
19.2
77.0
66.0
21.0
15.0
JudgeMix ( w=1 )
38.3
21.9
77.5
66.2
28.5
15.1
Table 3: Ablation study on the effect of judge-guided expert intervention. By comparing JudgeMix (without weighted supervision, i.e., w=1 in Eq. 3.3 ) against DAgger, we demonstrate the performance gains of intervening based on semantic correctness rather than random mixing heuristics.
Figure 2: Ablation study on the effect of weighted supervision as w varies in Eq. 3.3 . We plot performance on AppWorld Test-N and Test-C with Qwen3.5-2B and Qwen3.5-9B models. Dashed lines mark the w=1 baseline. We observe that down-weighting the supervision signal on the target model’s already-successful states improves downstream performance.
Method
AppWorld
Qwen3.5-2B
Qwen3.5-9B
Test-N
Test-C
Test-N
Test-C
TGC
TGC
TGC
TGC
JudgeMix w/o error-recovery
33.4
18.3
77.5
68.2
JudgeMix
40.9
23.3
78.7
69.8
Table 4: Ablation study on the effect of error-recovery data. Including this supplementary data in JudgeMix yields significant improvements, particularly for the weaker Qwen3.5-2B model. This suggests the value of explicitly teaching the target models how to correct from their native failure states.
Method
AppWorld
Qwen3.5-2B
Qwen3.5-9B
Test-N
Test-C
Test-N
Test-C
TGC
TGC
TGC
TGC
Expert Demonstrations
32.1
17.8
77.9
60.7
Step-level RFT
0.0
0.1
49.0
36.5
Step-level OPD
17.2
8.0
77.8
67.2
Table 5: Downstream performance (pass@1) of target models fine-tuned using various data synthesis frameworks on the AppWorld benchmark, assuming no task verifier for trajectory filtration.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Table 8
Method
AppWorld
AutomationBench
Qwen3.5-2B
Qwen3.5-9B
Qwen3.5-9B
Total
Traj.
Avg.
Total
Traj.
Avg.
Total
Traj.
Avg.
Trajs
pass@1
Steps
Trajs
pass@1
Steps
Trajs
pass@1
Steps
Expert Demonstrations
720
89.7
15.0
720
89.7
15.0
3,200
19.7
27.5
Step-level RFT
720
0.0
31.6
720
32.8
18.4
3,200
4.0
36.8
Step-level OPD
720
0.0
31.6
720
32.8
18.4
3,200
4.0
36.8
Appendix
Table 8: Dataset statistics for the training trajectories synthesized by different data generation frameworks. We report the total number of trajectories, the trajectory success rate (pass@1) prior to any filtration, and the average number of steps per trajectory.