cs.LGNov 12, 2026

JudgeMix: Judge-Guided Mixed-Rollout Data Synthesis for LLM Agents

Authors: Cheng-Yu Hsieh, Ting-Yao Hu, Sanjoy Chowdhury, Fatih Ilhan, Oncel Tuzel, Raviteja Vemulapalli

Organizations: Apple

Abstract

Supervised fine-tuning (SFT) on expert trajectories is a standard paradigm for training LLM agents. However, it suffers from a well-known training-inference distribution mismatch, calling for target-aware data synthesis strategies that ground the training data in the target model's native state distribution. To bridge this gap, emerging approaches incorporate the expert either to guide mixed rollouts alongside the target model, or to provide dense step-wise supervision. Nonetheless, existing methods apply expert assistance blind to the target model's inherent behavior: they either disrupt the rollout using sub-optimal random intervention heuristics, or they enforce indiscriminate uniform supervision that unnecessarily overwrites the model's already-correct actions. We introduce JudgeMix, a judge-guided data synthesis framework that dynamically interleaves the target model and expert based on step-wise semantic evaluations. JudgeMix queries expert guidance precisely when the target model struggles, ensuring high trajectory success rates while keeping the state distribution aligned with the target model. It additionally uses the judge evaluation to reweight expert supervision on different steps, preserving model's inherent valid behavior while correcting its errors. On two agentic tool-use benchmarks, AppWorld and AutomationBench, JudgeMix demonstrates consistent superior performance to existing methods, with improvements up to 10 pass@1 points.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Beyond Corrected Actions: Training LLM Agents on Textual Critiques

    Oct 14, 2026Thomas Zeng, Zelei Cheng, Mingyang Zhou +3LLM Agent TrainingLanguage Model Distillation

  2. Counsel: A Meta-Evaluation Dataset for Agentic Tasks

    Jun 19, 2026Sashank Pisupati, Henry Broomfield, Eujeong Choi +5LLM-as-a-JudgeLLM Agent Evaluation

  3. Training Advisors for LLM Agents from Task Outcomes

    Oct 7, 2026Sergei Polezhaev, Barys Liskavets, Ori Press +1LLM Agent TrainingRL for LLM Agents