DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

Yingping Wang; Min Cai; Chenyu Yuan; Xi Ye; Zhijiang Guo; Jiayi Huang

cs.LGOct 25, 2026

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

Authors: Yingping Wang, Min Cai, Chenyu Yuan, Xi Ye, Zhijiang Guo, Jiayi Huang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · University of Alberta

Abstract

Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$ - $5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$ - $90.1\%$ , while maintaining comparable multi-domain performance.

Figures & tables

Figure 1: Two system costs in RL-then-MOPD: repeated response generation and GPU resources reserved for online multi-teacher serving.

Figure 2: MOPD system costs. (a-c) Rollout time, teacher GPU allocation, and teacher idle time under two settings. (d) Minimum GPU demand for three resident domain teachers (BF16 weights).

Figure 3: Overview of DoubleFree. Exploration stage produces both specialist policies and historical trajectories. DoubleFree reuses the trajectories and precomputes specialist scores, removing additional generation and online teacher residency from student training.

Method	Trajectory Source	Additional Rollout	Consolidation objective	Online Teacher	Supervision Source
Standard MOPD	Current student	Online	PD	Yes	Final specialist
Mixed SFT	Final specialist	Offline	SFT	No	Generated response tokens
Offline MPD	Final specialist	Offline	PD	No	Final specialist
Lightning MOPD	Initial student	Offline	PD	No	Final specialist
DoubleFree-SFT	RL training policies	None	SFT	No	RL response tokens
DoubleFree-MPD	RL training policies	None	PD	No	Final specialist

Table 1: Comparison of multi-task training methods.

Method	GPUs	GPU-hours $\downarrow$						Time (h) $\downarrow$	Speedup $\uparrow$
Method	GPUs	Offline/Online Rollout	Offline/Online Teacher	Cache load	Update	Others	Total	Time (h) $\downarrow$	Speedup $\uparrow$
Qwen3-4B
Math RL	8	51.37	–	–	75.70	2.05	129.13	16.14	–
Code RL	8	34.62	–	–	35.05	2.21	71.88	8.98	–
IF RL	8	17.35	–	–	6.31	1.95	25.61	3.20	–
Mixed SFT	4	24.10	–	0.63	10.56	2.31	37.60	9.44	$1.40\times$

Table 2: Training efficiency across three models. Teacher costs include offline pre-scoring or online teacher residency, including idle time. Cache load for offline methods includes cached trajectory access and associated batch-processing overhead.

Method	Math					Code				Instruction Following			All AVG
Method	AIME24 mean@32	AIME25 mean@32	AMC23 mean@32	MATH500	AVG	LCB v6 pass@1	BCB pass@1	HumanEval+ pass@1	AVG	IFBench	IFEval	AVG	All AVG
Qwen3-4B
Base	21.46	20.31	65.55	80.20	46.88	26.07	8.11	76.22	36.80	26.33	79.85	53.09	45.59
Final Specialists	54.69	54.48	92.27	90.40	72.96	31.79	11.82	81.71	41.77	40.00	83.92	61.96	58.90
Mixed SFT	54.69	52.71	92.19	89.60	72.30	32.57	10.14	78.05	40.25	38.67	83.55	61.11	57.89
Offline MPD	55.52	53.13	91.80	90.40	72.71	32.00	9.54	77.44	39.66	39.33	82.62	60.98	57.78

Table 3: Multi-task accuracy (%). Bold marks the best consolidated student per model. DoubleFree-MPD reports mean

\pm

sample standard deviation over three seeds.

Figure 4: Trajectory-source coverage during Qwen3-4B consolidation. Lower values indicate less distribution mismatch or better coverage.

Selector	Math	Code	IF	AVG
Random	71.21	40.40	61.28	57.63
Reward	71.30	41.30	60.61	57.74
Balance	72.65	41.14	62.77	58.85

Table 4: Trajectory-selection ablation.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 9

Panel of Figure 5 — Figure 5: Training dynamics by trajectory source on Qwen3-4B.

Method	Math					Code				Instruction Following			All AVG
Method	AIME24 mean@32	AIME25 mean@32	AMC23 mean@32	MATH500	AVG	LCB v6 pass@1	BCB pass@1	HumanEval+ pass@1	AVG	IFBench	IFEval	AVG	All AVG
DoubleFree-SFT
Mixed SFT	54.69	52.71	92.19	89.60	72.30	32.57	10.14	78.05	40.25	38.67	83.55	61.11	57.89
Random	52.71	51.04	90.39	90.40	71.14	31.93	10.90	80.49	41.10	39.67	83.92	61.79	58.01
Reward	53.44	53.33	91.88	88.60	71.81	31.50	10.56	81.71	41.25	41.33	83.92	62.63	58.56
Balance	50.63	53.02	91.64	89.40	71.17	32.21	10.73	81.71	41.55	43.00	84.10	63.55	58.76

Appendix

Table 7: Trajectory-selection ablations for DoubleFree-SFT and DoubleFree-MPD on Qwen3-4B (%). Bold indicates the best score in each column within each block.

Multi-Teacher Knowledge Distillation

Multi-teacher knowledge distillation combines expertise from multiple models to train compact or unified students across tasks and domains.

See all 106 papers

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
Jun 29, 2026Wenhan Ma, Jianyu Wei, Liang Zhao +10Multi-Teacher Knowledge Distillation On-Policy Distillation
Peking University · Xiaomi · LLM Core, Xiaomi +2
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Sep 28, 2026Xin Li, Hao Jiang, Xin Gao +6Multi-Teacher Knowledge Distillation Policy Learning
Nanyang Technological University · Yale University · University of Manchester
D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Aug 25, 2026Zechen Sun, Zhiwei Zhang, Fei Zhao +7Adaptive Sampling Language Model Distillation
AllSpark Team

Jun 29, 2026J/K move · Enter open · S save

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang+9

Peking University · Xiaomi · LLM Core, Xiaomi+2

Multi-Teacher Knowledge Distillation On-Policy Distillation RL Post-Training LLM Post-Training Language Model Post-Training

Open paper AskPDF arXiv

Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.

Abstract

Multi-Teacher Knowledge Distillation

Explore similar work

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

D3^33-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation