R$^2$Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

Mingda Zhang; Yun Huang; Qiang Huang; Yanjin Li; Rui Qian; Qichao Zhao; Tiesunlong Shen; Na Zhao

cs.AIOct 15, 2026

R $^2$ Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

Authors: Mingda Zhang, Yun Huang, Qiang Huang, Yanjin Li, Rui Qian, Qichao Zhao, Tiesunlong Shen, Na Zhao,

Organizations: The Chinese University of Hong Kong, Shenzhen, China · Fudan University, China · University of Illinois Urbana-Champaign, USA · Tsinghua University, China ·

Abstract

In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R $^2$ Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R $^2$ Orch consistently outperforms baselines across question answering, math, code, and interactive tasks. Our code is available at https://github.com/beita6969/R2Orch.

Figures & tables

Figure 1: A recursion needs two channels: the outer loop sets the inner loop’s objective, and the inner loop, Proposer–Director self-play on a randomized rollout group, returns the evidence for withdrawing skills.

Figure 2: Paradigm comparison. (a) Learned orchestration trains on a hand-designed static curriculum . (b) Self-improving agents measure the frontier with skills present, a skill-anchored frontier where advantages collapse once skills leave. (c) Skill scaffolds withdraw guidance through success-rate gates , unreliable evaluation signals pinned by self-play. (d) R 2 Orch nests two loops joined by downward (objective) and upward (evidence) channels.

Figure 3: Overview of R 2 Orch. Top: Proposer–Director self-play (inner loop) and skill withdrawal (outer loop) share one randomized rollout group. Bottom left: Recursive Learnability scores a task with the candidate skill hidden. Bottom right: Skill Withdrawal withdraws or admits each skill by the posterior of its paired contrast.

		Question answering					Math	Code	Interactive	Average
(a) In-distribution		HotpotQA		NQ-Open		MedQA	AIME 2026	MBPP+	ALFWorld	IID
Type	Method	EM	F1	EM	F1	Acc.	Acc.	Pass@1	SR	EM	F1	Acc.
Backbone	Qwen3.5-9B	57.81 $\pm$ 0.00	73.75 $\pm$ 0.22	23.59 $\pm$ 0.86	32.89 $\pm$ 1.39	70.94 $\pm$ 0.86	48.67 $\pm$ 2.98	78.59 $\pm$ 0.70	47.19 $\pm$ 0.70	40.70 $\pm$ 0.43	53.32 $\pm$ 0.70	61.35 $\pm$ 0.84
Backbone	DeepSeek-V4-Flash	71.25 $\pm$ 1.28	82.50 $\pm$ 0.85	38.44 $\pm$ 0.86	49.77 $\pm$ 0.89	84.38 $\pm$ 0.00	49.33 $\pm$ 3.65	84.69 $\pm$ 0.70	60.16 $\pm$ 1.10	54.84 $\pm$ 0.71	66.14 $\pm$ 0.62	69.64 $\pm$ 1.07
Training	SFT	63.28 $\pm$ 1.10	77.58 $\pm$ 0.89	25.78 $\pm$ 0.00	37.50 $\pm$ 0.19	73.59 $\pm$ 0.86	34.67 $\pm$ 1.83	78.28 $\pm$ 0.86	40.94 $\pm$ 0.43	44.53 $\pm$ 0.55	57.54 $\pm$ 0.46	56.87 $\pm$ 0.59
Training	GRPO	69.22 $\pm$ 0.43	81.33 $\pm$ 0.43	24.53 $\pm$ 0.70	32.27 $\pm$ 1.41	77.03 $\pm$ 0.70	34.00 $\pm$ 4.94	82.19 $\pm$ 0.65	50.94 $\pm$ 1.02	46.88 $\pm$ 0.48	56.80 $\pm$ 0.74	61.04 $\pm$ 1.60

Table 1: Main results (five-run mean

\pm

std, %), with benchmarks grouped by task domain. All methods share the frozen Workers, the training tasks, and the compute budget, each in its best configuration, and R 2 Orch is tested with zero skills. Each test set has 128 items (AIME 2026: 30); bold marks the best score in each benchmark column.

Recursive Self-Play Orchestration (Sec. 4.1)
Variant	IID						OOD
	HotpotQA	NQ-Open	MedQA	AIME	MBPP+	ALFWorld	TriviaQA	MuSiQue	GPQA	MATH	SWE	WebShop
	Ans EM	Ans EM	Acc.	Acc.	Pass@1	SR	Ans EM	Ans EM	Acc.	Acc.	Resolved	SR
Qwen3.5-9B (direct prompting)	57.81	23.59	70.94	48.67	78.59	47.19	44.53	39.22	61.41	89.38	15.78	32.81
$-$ Recursive Self-Play Orchestration (GRPO only)	86.09	82.97	88.44	59.33	86.88	73.44	91.41	78.91	80.78	92.03	35.00	71.72
$-$ Skills (no scaffold)	86.88	83.59	89.22	64.00	88.13	77.81	92.34	80.63	82.66	92.34	37.66	76.09

Table 2: Ablation of R 2 Orch, tested with zero skills (five-run mean, %). The first row of each group removes the whole component and the other rows change one part (Appendix D ); free-form QA benchmarks report Ans EM, and the Qwen3.5-9B and R 2 Orch rows repeat their scores in Table 1 .

Figure 4: Training dynamics and backbone transfer. (a) Held-out accuracy with skills and at zero skills. (b) Graph types over the six IID datasets per Proposer objective (final counts at the curve ends). (c) The Proposer’s probability mass on unseen, unsolved (

p=0

), learnable (

0<p<1

), and solved (

p=1

) candidates by previous success rate

p

(power-scaled axis). (d) Probability of choosing a solved candidate as a moving average over 10 cycles (dots: single cycles) for the trained Proposer, the untrained model with the same prompt, and random choice. (e) Backbone transfer: the Director trained with Qwen3.5-9B Workers is used unchanged at zero skills with six other backbones. Radars: IID scores with R 2 Orch (solid) and without it (dashed), the radius linear over 20–80 (inner half) and 80–100 (outer half). Bars: OOD scores per task domain, the frozen backbone in blue and R 2 Orch’s gain in pink (Code: SWE-Bench; Interactive: WebShop; QA: mean of TriviaQA, MuSiQue, and GPQA; Math: MATH-Hard).

Panel of Figure 5 — Figure 5: Proposer objectives, selection cost, and skill withdrawals (RQ4, RQ5); only the Proposer’s objective changes in (a)–(d) (AZR: Absolute Zero). (a) IID scores. (b) Row minus column of the mean OOD score. (c) Rollouts and tokens per effective group; bubble area: A100 hours. (d) Training (pink) and selection (blue) rollouts per cycle. (e) Seen-task picks over cycles 50–199 by previous success rate $p$ : realized success and share of mixed new groups. (f) The seed cards’ held-out gain over no cards, before training and around each withdrawal (95% CI).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\pi_{\theta}$ , $\rho_{\phi}$ , $\rho_{0}$	Director, Proposer, Proposer reference
$\rho$ , $\mathcal{C}_{t}$	evaluation distribution, candidate list
$\mathcal{S}_{t}$ , $K_{x}$ , $k_{x}$	scaffold, skills retrieved for $x$ , skill under test
$G=n_{h}+n_{v}$	rollouts per group (hidden, visible)
$\mathcal{H}_{x}$ , $\mathcal{V}_{x}$	parts reading $K_{x}\setminus k_{x}$ and $K_{x}$
$y_{i}$ , $u_{i}$	correctness; correctness less Worker cost

Appendix

Table 3: Notation of the main text.

Symbol	Meaning	Value
$G$ ( $n_{h}$ , $n_{v}$ )	rollouts per task (hidden, visible)	5 (3, 2)
$\epsilon$	withdrawal tolerance on the success rate	0.02
$\alpha$	admission risk	0.05
$n_{\min}$	minimum records per decision	12
$B$	Bayesian-bootstrap draws	2000
–	Thompson prior standard deviation	0.25

Appendix

Table 4: Hyperparameters of R 2 Orch.

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 458 papers

SkillOrchestra: Learning to Route Agents via Skill Transfer
Feb 23, 2026Jiayu Wang, Yifei Ming, Zixuan Ke +3LLM Agent Orchestration LLM Agent Routing
University of Wisconsin-Madison · Salesforce AI Research
SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration
May 13, 2026Mingda Zhang, Tiesunlong Shen, Haoran Luo +4LLM Agent Orchestration LLM Agent Self-Improvement
The Chinese University of Hong Kong, Shenzhen · National University of Singapore · Nanyang Technological University +1
Proactive Agentic Orchestration
Oct 9, 2026Xueyang Feng, Chaozhuo Li, Xiaohe Bo +4LLM Agent Orchestration LLM Agent Workflow Optimization
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Academy of Artificial Intelligence · The Chinese University of Hong Kong Shenzhen

Feb 23, 2026J/K move · Enter open · S save

SkillOrchestra: Learning to Route Agents via Skill Transfer

Jiayu Wang, Yifei Ming, Zixuan Ke, Shafiq Joty+2

University of Wisconsin-Madison · Salesforce AI Research

LLM Agent Orchestration LLM Agent Routing LLM Agent Skill Learning

Open paper AskPDF arXiv

Compound AI systems promise capabilities beyond those of individual models, yet their success depends critically on effective orchestration. Existing routing approaches face two limitations: (1) input-level routers make coarse query-level decisions that ignore evolving task requirements; (2) RL-trained orchestrators are expensive to adapt and often suffer from routing collapse, repeatedly invoking one strong but costly option in multi-turn scenarios. We introduce SkillOrchestra, a framework for skill-aware orchestration. Instead of directly learning a routing policy end-to-end, SkillOrchestra learns fine-grained skills from execution experience and models agent-specific competence and cost under those skills. At deployment, the orchestrator infers the skill demands of the current interaction and selects agents that best satisfy them under an explicit performance-cost trade-off. Extensive experiments across ten benchmarks demonstrate that SkillOrchestra outperforms SoTA RL-based orchestrators by up to 22.5% with 700x and 300x learning cost reduction compared to Router-R1 and ToolOrchestra, respectively. These results show that explicit skill modeling enables scalable, interpretable, and sample-efficient orchestration, offering a principled alternative to data-intensive RL-based approaches. The code is available at: https://github.com/jiayuww/SkillOrchestra.

Abstract

LLM Agent Orchestration

Explore similar work

SkillOrchestra: Learning to Route Agents via Skill Transfer

SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration

Proactive Agentic Orchestration