$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

Yaocheng Zhang; Yuanheng Zhu; Wenyue Chong; Songjun Tu; Qichao Zhang; Jiajun Chai; Xiaohan Wang; Wei Lin

cs.LGApr 15, 2026

$π$ -Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

Authors: Yaocheng Zhang, Yuanheng Zhu, Wenyue Chong, Songjun Tu, Qichao Zhang, Jiajun Chai, Xiaohan Wang, Wei Lin,

Organizations: Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Meituan

Abstract

Deep search agents have emerged as a promising paradigm for addressing complex information-seeking tasks, but their training remains challenging due to sparse rewards, weak credit assignment, and limited labeled data. Self-play offers a scalable route to reduce data dependence, but conventional self-play optimizes students only through sparse outcome rewards, leading to low learning efficiency. In this work, we observe that self-play naturally produces a question construction path (QCP) during task generation, an intermediate artifact that captures the reverse solution process. This reveals a new source of privileged information: self-play can provide high-quality privileged information for the self-distillation at low cost and at scale, without relying on human feedback or curated privileged information. Leveraging this insight, we propose Privileged Information Self-Play ( $π$ -Play), a novel multi-agent self-evolution framework combining self-play and self-distillation. In $π$ -Play, an examiner generates tasks together with QCPs, and a teacher employs QCP as privileged context to densely supervise a student via self-distillation. This design transforms sparse-reward self-play into a dense-feedback co-evolution. Extensive experiments show that data-free $π$ -Play surpasses fully supervised search agents and improves evolutionary efficiency by 2-3 $\times$ over conventional self-play. Code is available at https://github.com/zhyaoch/pi-play.

Figures & tables

Figure 1: Comparison of $\boldsymbol{\pi}$ -Play with other self-evolution frameworks. All models (examiner, teacher, and student) in $\pi$ -Play are initialized from the same base LLM and function as search agents. $\pi$ -Play uses alternating optimization to evolve multiple agents in a closed loop. Compared to self-play, it overcomes the sparse-reward of the student and enables the student to be optimized under the joint effect of outcome rewards and teacher guidance.

Figure 2: Overview of QCP-guided self-distillation in $\pi$ -Play. The examiner is equipped with search tools and interacts with the search engine to obtain factual information, ensuring the correctness of both the synthesized QA pairs and their construction paths $c$ . The teacher policy ${\pi_{\psi}^{T}}$ leverages QCP as additional context to provide token-level supervision to the student policy ${\pi_{\theta}^{S}}$ along the student’s rollout $y$ , by minimizing the per-token reverse KL divergence $\mathbb{D}_{\mathrm{Distill}}\bigl({\pi_{\theta}^{S}}\,\|\,\text{stopgrad}[{\pi_{\psi}^{T}}]\bigr)$ (Eq. 10 ).

	General QA			Multi-Hop QA
	NQ	TriviaQA	PopQA	HotpotQA	2WikiMQA	MuSiQue	Bamboogle	Total
	Qwen3-4B
ReAct	7.2	16.6	10.3	11.2	12.7	1.6	8.8	68.4
Search-R1	35.2	57.3	39.5	30.7	28.9	10.5	37.6	239.7
SQLM*	39.9	60.4	42.1	33.0	30.8	10.8	29.6	246.6
Dr.Zero	40.4	61.2	44.3	31.0	29.3	7.8	31.2	245.2

Table 1: The main results of

\boldsymbol{\pi}

-Play. Bold values indicate the best result; underline values denote the second-best. Through efficient teacher guidance,

\pi

-Play outperforms both supervised and self-play search agents.

Qwen3-4B
		General QA			Multi-Hop QA
	Methods	NQ	TriviaQA	PopQA	HotpotQA	2WikiMQA	MuSiQue	Bamboogle	Total
Iter1	Dr.Zero	36.1	58.7	42.8	28.6	25.9	6.2	21.6	219.9
Iter1	$\pi$ -Play	38.6	59.4	43.5	32.2	31.4	10.7	32.8	248.6
Iter2	Dr.Zero	40.6	60.5	44.4	30.8	29.1	8.0	28.8	242.2
Iter2	$\pi$ -Play	38.1	58.4	42.3	32.3	32.5	11.2	35.2	250.0

Table 2: Learning dynamics of

\pi

-Play with increasing iterations.

Figure 3: Iterative reward and entropy dynamics of the examiner and student in

\pi

-Play with Qwen3-4B-Instruct . Both reward and entropy reach a converged state by Iteration 3.

	General QA			Multi-Hop QA
	NQ	TriviaQA	PopQA	HotpotQA	2WikiMQA	MuSiQue	Bamboogle	Total
$\pi$ -Play w/o Distillation	41.2	62.3	43.6	37.7	29.0	13.3	38.4	265.5
$\pi$ -Play w/ GT	42.0	62.0	44.5	37.7	29.4	13.3	35.2	264.1
$\pi$ -Play w/ GT+Hop	42.1	61.9	42.9	37.7	29.1	14.0	39.2	266.9
$\pi$ -Play w/ Partial QCP	41.6	62.1	43.3	38.3	31.0	13.6	43.2	273.1
$\pi$ -Play w/ QCP (Ours)	41.8	63.2	44.0	38.5	32.3	13.4	44.0	277.2

Table 3: Ablation study on the question construction path (QCP) with Qwen3-4B-Instruct.

\pi

-Play w/o Distillation denotes the variant of

\pi

-Play without teacher-guided distillation loss. Variants of the form

\pi

-Play w/ [Privileged Info] replace QCP with different forms of privileged information for self-distillation.

	Performance of Evaluated Model
	Base Model	Student(step 50)	Student(step 100)
$\mathcal{D}_{\text{step 50}}$	27.5	57.1	58.4
$\mathcal{D}_{\text{step 100}}$	27.0	51.9	57.3
$\mathcal{D}_{\text{step 150}}$	22.9	45.1	53.3

Table 4: Co-evolutionary dynamic of the examiner. We report the student’s accuracy on the datasets generated by the examiner at different steps.

	Accuracy $\uparrow$	Search Count $\downarrow$	Query Redundancy $\downarrow$
SQLM*	35.7	2.3	0.43
Dr.Zero	38.0	2.4	0.48
$\boldsymbol{\pi}$ -Play	39.6	1.9	0.37

Table 5: Quantitative analysis of search count and query redundancy with Qwen3-4B-Instruct. We report the average number of search actions and the average query redundancy across seven QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Steps	50
Optimizer	AdamW
Optimizer Momentum	$\beta_{1}$ , $\beta_{2}$ = 0.9, 0.999
Warmup Ratio	0.03
Weight Decay	0.01
Learning Rate	5e-7, 1e-6
Max Gradient Norm	0.1, 1.0

Appendix

Table 6: Examiner hyperparameter settings.

Steps	50
Optimizer	AdamW
Optimizer Momentum	$\beta_{1}$ , $\beta_{2}$ = 0.9, 0.999
Warmup Ratio	0.03
Weight Decay	0.01
Learning Rate	1e-6
Max Gradient Norm	0.1, 1.0

Appendix

Table 7: Student hyperparameter settings.

Precision (WA)	BF16-mixed
Soft Update Weight $\tau$	0.05
Max Sequence Length	8192

Appendix

Table 8: Teacher hyperparameter settings.

	General QA			Multi-Hop QA
$\boldsymbol{\lambda}$	NQ	TriviaQA	PopQA	HotpotQA	2WikiMQA	MuSiQue	Bamboogle	Total
$\boldsymbol{\lambda}$	Iter 1
0.100	40.1	61.8	42.5	35.5	30.1	12.8	40.8	263.6
0.050	39.8	62.1	42.2	34.9	31.4	11.8	41.6	263.8
0.030	40.1	61.1	42.7	33.7	30.4	9.85	40.0	257.9
0.010	38.5	61.6	42.5	35.4	26.6	12.4	35.2	252.2

Appendix

Table 9: Ablation study of

\lambda

. Bold value indicates the top-performing result, while underline value denotes the second-best.

	General QA			Multi-Hop QA
Top $K$	NQ	TriviaQA	PopQA	HotpotQA	2WikiMQA	MuSiQue	Bamboogle	Total
Top $K$	Iter 1
200	37.8	61.7	42.1	37.1	29.0	12.9	35.2	255.8
100	39.4	61.8	42.4	36.8	28.4	12.6	38.4	259.8
50	40.7	62.1	42.9	36.7	30.1	13.2	39.2	264.9
25	40.0	61.2	41.0	35.1	29.8	11.5	36.8	255.4

Appendix

Table 10: Ablation study of

K

. Bold value indicates the top-performing result

	Iter1		Iter2		Iter3		Average
	Dr.Zero	$\pi$ -Play	Dr.Zero	$\pi$ -Play	Dr.Zero	$\pi$ -Play	Dr.Zero	$\pi$ -Play
Examiner	311.6	311.6	407.8	526.3	623.4	529.9	447.6	455.9
Student (Rollout)	88.9	81.1	146.0	159.9	176.8	149.8	137.2	130.3
Teacher (Forward)	-	21.8	-	31.8	-	30.6	-	28.1
Student (Updating)	94.0	94.1	116.2	123.0	123.2	117.8	111.1	111.6
Total Time	494.5	508.6	670.0	841.0	923.4	828.1	695.9	725.9

Appendix

Table 11: The training time. We analyze the per-step training time (in seconds) for each iteration under Qwen3-4B-Instruct. Although the teacher model requires an additional forward pass along the student’s rollout to compute token probabilities,

\pi

-Play introduces only a small per-step training overhead of 4.3% relative to Dr.Zero.

Figure 4: Side-by-side trajectories of Dr.Zero (left) and

\pi

-Play (right) on the same question. Each trajectory shows multi-turn interactions with the search engine (actions, responses, and final answer). Although both Dr.Zero and

\pi

-Play answer the query correctly, our method (right) uses fewer queries and reaches a logically structured answer with minimal redundancy.

Self-Play RL

Self-play reinforcement learning trains agents through repeated interaction with copies, opponents, or co-evolving task generators.

See all 83 papers

Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember
Jul 31, 2026Zenghuang Fu, Zhaoyang Li, Qiuyuan Ai +6Self-Play RL Agent Skill Learning
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Peking University +4
SearchMaster: Grounded and Regulated Self-Play for Search Agents
Aug 3, 2026Wentao Tan, Qiong Cao, Jiaqi Wang +1Multi-Hop QA Tool-Augmented Language Model Agents
JD Future Academy, Beijing
DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
Jul 8, 2026Xinyu Geng, Xuanhua He, Sixiang Chen +7Web Search Agents LLM Agent Self-Improvement
HKUST · Tencent · HKUST(GZ)

Jul 31, 2026J/K move · Enter open · S save

Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember

Zenghuang Fu, Zhaoyang Li, Qiuyuan Ai, Haoyu Wu+5

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Peking University+4

Self-Play RL Agent Skill Learning Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Self-play agents can generate training problems without questions from target benchmarks, but their curricula lack persistent state: failures affect gradients yet do not explicitly shape future practice. External skill memories preserve procedural experience but are typically learned from fixed task distributions. We introduce SESA (Self-Evolving Skill-Augmented Agent), which makes procedural memory an evolving state of tool-augmented search self-play. A challenger poses problems, while a separately parameterized solver alone retrieves skills. Informative failures are distilled into reusable skills and written back to memory. The updated memory changes solver behavior and success, which changes the challenger's reward and the distribution of future problems; the resulting frontier produces new failures that rewrite memory. This bidirectional loop makes task generation and skill memory co-evolve. Because retrieved skills shape on-policy training trajectories, their benefits can enter the model parameters as well as remain in the external bank, enabling memory-free deployment and optional inference-time retrieval. Across seven open-domain and multi-hop question-answering benchmarks, SESA improves average accuracy over SSP by 1.2--3.2 points across multiple backbones and surpasses the skill-augmented SkillRL baseline by 0.9 points under a unified evaluation protocol. On Qwen3 models, SESA-Off retains 1.8--2.2 points of improvement over SSP, while the final skill bank adds a further 0.5--1.0 points. These results show that evolving skill memory is not merely an inference-time plug-in: it changes policy learning and the future training distribution while retaining value as optional external memory. Our code is available at https://github.com/Zenghuang-Fu/SESA-Self-Evolving-Search-Agents.

Abstract

Self-Play RL

Explore similar work

Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember

SearchMaster: Grounded and Regulated Self-Play for Search Agents

DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment