Organizations: The Chinese University of Hong Kong, Shenzhen, China · Fudan University, China · University of Illinois Urbana-Champaign, USA · Tsinghua University, China · National University of Singapore, Singapore
In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R2Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R2Orch consistently outperforms baselines across question answering, math, code, and interactive tasks. Our code is available at https://github.com/beita6969/R2Orch.
Figures & tables
Figure 1: A recursion needs two channels: the outer loop sets the inner loop’s objective, and the inner loop, Proposer–Director self-play on a randomized rollout group, returns the evidence for withdrawing skills.
Figure 2: Paradigm comparison. (a) Learned orchestration trains on a hand-designed static curriculum . (b) Self-improving agents measure the frontier with skills present, a skill-anchored frontier where advantages collapse once skills leave. (c) Skill scaffolds withdraw guidance through success-rate gates , unreliable evaluation signals pinned by self-play. (d) R 2 Orch nests two loops joined by downward (objective) and upward (evidence) channels.
Figure 3: Overview of R 2 Orch. Top: Proposer–Director self-play (inner loop) and skill withdrawal (outer loop) share one randomized rollout group. Bottom left: Recursive Learnability scores a task with the candidate skill hidden. Bottom right: Skill Withdrawal withdraws or admits each skill by the posterior of its paired contrast.
Question answering
Math
Code
Interactive
Average
(a) In-distribution
HotpotQA
NQ-Open
MedQA
AIME 2026
MBPP+
ALFWorld
IID
Type
Method
EM
F1
EM
F1
Acc.
Acc.
Pass@1
SR
EM
F1
Acc.
Backbone
Qwen3.5-9B
57.81 ± 0.00
73.75 ± 0.22
23.59 ± 0.86
32.89 ± 1.39
70.94 ± 0.86
48.67 ± 2.98
78.59 ± 0.70
47.19 ± 0.70
40.70 ± 0.43
53.32 ± 0.70
61.35 ± 0.84
DeepSeek-V4-Flash
71.25 ± 1.28
82.50 ± 0.85
38.44 ± 0.86
49.77 ± 0.89
84.38 ± 0.00
49.33 ± 3.65
84.69 ± 0.70
60.16 ± 1.10
54.84 ± 0.71
66.14 ± 0.62
69.64 ± 1.07
Training
SFT
63.28 ± 1.10
77.58 ± 0.89
25.78 ± 0.00
37.50 ± 0.19
73.59 ± 0.86
34.67 ± 1.83
78.28 ± 0.86
40.94 ± 0.43
44.53 ± 0.55
57.54 ± 0.46
56.87 ± 0.59
GRPO
69.22 ± 0.43
81.33 ± 0.43
24.53 ± 0.70
32.27 ± 1.41
77.03 ± 0.70
34.00 ± 4.94
82.19 ± 0.65
50.94 ± 1.02
46.88 ± 0.48
56.80 ± 0.74
61.04 ± 1.60
Table 1: Main results (five-run mean ± std, %), with benchmarks grouped by task domain. All methods share the frozen Workers, the training tasks, and the compute budget, each in its best configuration, and R 2 Orch is tested with zero skills. Each test set has 128 items (AIME 2026: 30); bold marks the best score in each benchmark column.
Variant
IID
OOD
HotpotQA
NQ-Open
MedQA
AIME
MBPP+
ALFWorld
TriviaQA
MuSiQue
GPQA
MATH
SWE
WebShop
Ans EM
Ans EM
Acc.
Acc.
Pass@1
SR
Ans EM
Ans EM
Acc.
Acc.
Resolved
SR
Qwen3.5-9B (direct prompting)
57.81
23.59
70.94
48.67
78.59
47.19
44.53
39.22
61.41
89.38
15.78
32.81
Recursive Self-Play Orchestration (Sec. 4.1)
− Recursive Self-Play Orchestration (GRPO only)
86.09
82.97
88.44
59.33
86.88
73.44
91.41
78.91
80.78
92.03
35.00
71.72
− Skills (no scaffold)
86.88
83.59
89.22
64.00
88.13
77.81
92.34
80.63
82.66
92.34
37.66
76.09
Table 2: Ablation of R 2 Orch, tested with zero skills (five-run mean, %). The first row of each group removes the whole component and the other rows change one part (Appendix D ); free-form QA benchmarks report Ans EM, and the Qwen3.5-9B and R 2 Orch rows repeat their scores in Table 1 .
Figure 4: Training dynamics and backbone transfer. (a) Held-out accuracy with skills and at zero skills. (b) Graph types over the six IID datasets per Proposer objective (final counts at the curve ends). (c) The Proposer’s probability mass on unseen, unsolved ( p=0 ), learnable ( 0<p<1 ), and solved ( p=1 ) candidates by previous success rate p (power-scaled axis). (d) Probability of choosing a solved candidate as a moving average over 10 cycles (dots: single cycles) for the trained Proposer, the untrained model with the same prompt, and random choice. (e) Backbone transfer: the Director trained with Qwen3.5-9B Workers is used unchanged at zero skills with six other backbones. Radars: IID scores with R 2 Orch (solid) and without it (dashed), the radius linear over 20–80 (inner half) and 80–100 (outer half). Bars: OOD scores per task domain, the frozen backbone in blue and R 2 Orch’s gain in pink (Code: SWE-Bench; Interactive: WebShop; QA: mean of TriviaQA, MuSiQue, and GPQA; Math: MATH-Hard).
Figure 5: Proposer objectives, selection cost, and skill withdrawals (RQ4, RQ5); only the Proposer’s objective changes in (a)–(d) (AZR: Absolute Zero). (a) IID scores. (b) Row minus column of the mean OOD score. (c) Rollouts and tokens per effective group; bubble area: A100 hours. (d) Training (pink) and selection (blue) rollouts per cycle. (e) Seen-task picks over cycles 50–199 by previous success rate p : realized success and share of mixed new groups. (f) The seed cards’ held-out gain over no cards, before training and around each withdrawal (95% CI).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
πθ , ρϕ , ρ0
Director, Proposer, Proposer reference
ρ , Ct
evaluation distribution, candidate list
St , Kx , kx
scaffold, skills retrieved for x , skill under test
Compound AI systems promise capabilities beyond those of individual models, yet their success depends critically on effective orchestration. Existing routing approaches face two limitations: (1) input-level routers make coarse query-level decisions that ignore evolving task requirements; (2) RL-trained orchestrators are expensive to adapt and often suffer from routing collapse, repeatedly invoking one strong but costly option in multi-turn scenarios. We introduce SkillOrchestra, a framework for skill-aware orchestration. Instead of directly learning a routing policy end-to-end, SkillOrchestra learns fine-grained skills from execution experience and models agent-specific competence and cost under those skills. At deployment, the orchestrator infers the skill demands of the current interaction and selects agents that best satisfy them under an explicit performance-cost trade-off. Extensive experiments across ten benchmarks demonstrate that SkillOrchestra outperforms SoTA RL-based orchestrators by up to 22.5% with 700x and 300x learning cost reduction compared to Router-R1 and ToolOrchestra, respectively. These results show that explicit skill modeling enables scalable, interpretable, and sample-efficient orchestration, offering a principled alternative to data-intensive RL-based approaches. The code is available at: https://github.com/jiayuww/SkillOrchestra.
Jiayu Wang, Yifei Ming, Zixuan Ke +3
University of Wisconsin-Madison · Salesforce AI Research
In recent years, a variety of powerful LLM-based agentic systems have been applied to automate complex tasks through task orchestration. However, existing orchestration methods still face key challenges, including strategy collapse under reward maximization, high gradient variance with opaque credit assignment, and unguided skill evolution whose decisions are typically made by directly prompting an LLM to judge rather than derived from principled training signals. To address these challenges, we propose SkillFlow, a flow-based framework that takes a trainable Supervisor as the agent and a structured environment with dynamic skill library and frozen executor, automating task orchestration through multi-turn interaction. SkillFlow employs Tempered Trajectory Balance (TTB), a regression-based flow-matching loss that samples trajectories proportional to reward, preserving diverse orchestration strategies rather than collapsing to a single mode. The same flow objective yields a jointly learned backward policy that provides transparent per-step credit assignment at zero additional inference cost. Building on these flow diagnostics, a recursive skill evolution mechanism determines when to evolve, what skills to create or prune, and where decision gaps lie -- closing the loop from training signal to autonomous capability growth. Experimental results on 14 datasets show that SkillFlow significantly outperforms baselines across question answering, mathematical reasoning, code generation, and real-world interactive decision making tasks. Our code is available at https://github.com/beita6969/SkillFlow.
Mingda Zhang, Tiesunlong Shen, Haoran Luo +4
The Chinese University of Hong Kong, Shenzhen · National University of Singapore · Nanyang Technological University +1
Agentic orchestration delegates tasks to sub-agents with independent contexts, providing a key path to extending the effective execution horizon of large language model agents. However, existing orchestration methods face two limitations: in execution, current methods fix sub-agents' goals and harnesses at creation time, limiting their adaptability to dynamic real-world environments; in optimization, sparse and delayed feedback constrains effective learning of the orchestration policy. To address these limitations, we introduce POrchestra, an on-the-fly proactive agentic orchestration framework. It allows sub-agents to proactively communicate with the orchestrator at any logical point to request harness adjustments while maintaining continuity of local execution. This mechanism not only enables sub-agents to adapt seamlessly to dynamic environments but also naturally inspires On-Policy Orchestration Distillation (OPOD) based on hindsight feedback. OPOD attributes sub-agents' proactive revision requests back to preceding orchestration decisions to derive targeted improvement signals, and leverages these signals as process supervision alongside the final reward to optimize the orchestration policy. We evaluate POrchestra on three long-horizon agent benchmarks spanning deep research, dynamic digital assistance, and software engineering. POrchestra improves task success by an average of 9.1% relative to the strongest baseline; combined with OPOD, it improves small-orchestrator success by an average of 10.0% relative to existing training baselines. POrchestra also achieves a more favorable cost-performance trade-off. Our code is publicly available at https://github.com/XueyangFeng/POrchestra.
Xueyang Feng, Chaozhuo Li, Xiaohe Bo +4
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Academy of Artificial Intelligence · The Chinese University of Hong Kong Shenzhen