Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging

Jingyuan Huang; Zuming Huang; Yucheng Shi; Zhongzhi Li; Xiaoming Zhai; Wei Chu; Ninghao Liu

cs.AISep 26, 2026

Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging

Authors: Jingyuan Huang, Zuming Huang, Yucheng Shi, Zhongzhi Li, Xiaoming Zhai, Wei Chu, Ninghao Liu

Organizations: University of Georgia · INF Tech · Tencent · Hong Kong Polytechnic University

Abstract

Domain experts trained from a shared checkpoint can transfer their specialized capabilities to a single student through on-policy distillation (OPD). Existing research primarily focuses on improving this merging process, while the algorithms used to train the experts have received limited systematic comparison. We investigate which training algorithm produces teachers better suited to OPD through controlled single-teacher comparisons of supervised fine-tuning (SFT) and reinforcement learning (RL) across Agentic, Reasoning, and Perception. Teachers and students share the same Qwen3.5-9B initialization, and the two teacher types are compared at similar task performance. Our experiments show that RL teachers yield stronger students and higher recovery of teacher performance gains across all three domains. At their best checkpoints, RL-guided students outperform SFT-guided students by 4.27, 1.50, and 0.86 percentage points, respectively. In Agentic, the best SFT-guided student recovers only 44.44% of its teacher's performance gain over the base model, whereas the best RL-guided student recovers 115.00%, surpassing its teacher. Our further analysis shows that RL teachers undergo smaller parameter displacements from the shared initialization than SFT teachers. These findings support the hypothesis that RL teachers' smaller departures from the student's starting point facilitate learning through OPD, resulting in stronger students.

Figures & tables

Domain	Evaluation benchmarks	Task metric
Agentic	MobileWorld ( Kong et al., 2025 )	Task success rate
Reasoning	Geo3K test set ( Lu et al., 2021 ) , MathVista ( Lu et al., 2024 ) , MathVision ( Wang et al., 2024 ) , MathVerse ( Zhang et al., 2024 )	Answer accuracy
Perception	ScreenSpot-v2 ( Wu et al., 2024 ) , ScreenSpot-Pro ( Li et al., 2025 ) , UI-Vision ( Nayak et al., 2025 ) , MMBench-GUI L2 ( Wang et al., 2025 ) , OSWorld-G, OSWorld-G-Refine ( Xie et al., 2025 )	Grounding accuracy

Table 1: Evaluation benchmarks and metrics.

Figure 1: MobileWorld success rates across OPD checkpoints.

Figure 2: Geo3K accuracies across OPD checkpoints.

Figure 3: Mean accuracies across the six Perception benchmarks at OPD checkpoints. The horizontal axis is expanded for the first ten steps.

Figure 4: Teacher displacement and OPD performance. (a) Comparison across domains. (b) Agentic accuracy trajectories. (c) Gain recovery at each group’s best student checkpoint.

Figure 5: Teacher–student Top-20 overlap on student rollouts. Step 0 denotes the base student before its first update. Shading marks the early steps compared in the text.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Agentic			Reasoning			Perception
Step	RL	SFT	Step	RL	SFT	Step	RL	SFT
0	0	0	0	0	0	0	0	0
8	0.209	0.197	8	0.481	0.505	10	0.374	0.384
16	0.380	0.383	16	0.715	0.812	20	0.629	0.726
24	0.505	0.588	24	0.828	0.981	30	0.760	0.864
32	0.579	0.714	32	0.901	1.099	40	0.855	0.969

Appendix

Table 2: Student parameter displacement

r_{t}

and teacher displacement

\tau_{T}

	Agentic	Reasoning	Perception
Number of targets	25,904	2,075	6,990
Mean length	217.02	2,315.89	38.55
Median length	218	1,099	39
10th–90th percentile	181–254	356–6,140	38–39
Free-form reasoning	Rationale	Chain of thought	None

Appendix

Table 3: SFT targets. Lengths are in tokens.

	Agentic		Reasoning		Perception
Parameter	SFT	RL	SFT	RL	SFT	RL
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW	AdamW
Learning rate	$10^{-6}$	$10^{-6}$	$5\!\times\!10^{-6}$	$10^{-6}$	$2.5\!\times\!10^{-6}$	$2.5\!\times\!10^{-6}$
Weight decay	0.01	0	0.01	0.01	0.01	0.01
Batch	32	$128\!\times\!8$	32	$16\!\times\!5$	112	$14\!\times\!8$
Max tokens	16,384	8,192/512	16,384	8,192/8,192	8,320	19,872/128

Appendix

Table 4: Teacher training settings. Batch: demonstrations (SFT) or prompts

\times

sampled responses (RL) per update. Max tokens: total (SFT) or prompt/response (RL). Learning rates are configured values.

Parameter	Agentic	Reasoning	Perception
Optimizer	AdamW	AdamW	AdamW
Learning rate	$10^{-6}$	$1.875\!\times\!10^{-6}$	$1.875\!\times\!10^{-6}$
Weight decay	0	0.01	0.01
Input batch	128	32	112
Responses per prompt ( $n$ )	8	1	1
Warmup (updates)	0	3	3

Appendix

Table 5: Student OPD training settings.

Checkpoint	R1	R2	R3	Mean	R1	R2	R3	Mean
	SFT				RL
Teacher	28	23	29	22.79	26	28	28	23.36
Qwen3.5-9B	22	20	20	17.66	22	20	20	17.66
Step 8	21	28	21	19.94	17	23	25	18.52
Step 16	19	19	20	16.52	25	18	23	18.80
Step 24	23	22	22	19.09	23	29	24	21.65

Appendix

Table 6: MobileWorld results across OPD checkpoints. R1–R3: successes per round; Mean: success rate (%).

Model	Step	Geo3K	MathVista	MathVision	MathVerse	Avg.
Qwen3.5-9B	–	79.70	84.00	57.24	70.18	72.78
SFT teacher	–	88.35	80.80	53.95	67.89	72.75
RL teacher	–	84.19	83.60	57.57	73.10	74.61
Student $\leftarrow$ SFT	4	86.36	83.60	51.32	71.45	73.18
	8	86.19	81.50	55.59	64.85	72.03
	12	89.02	82.10	56.91	68.15	74.04

Appendix

Table 7: Reasoning accuracy (%) across OPD checkpoints.

Model	Step	ScreenSpot- v2	ScreenSpot- Pro	UI-Vision	MMBench- GUI L2	OSWorld- G	OSWorld- G-R	Avg.
Qwen3.5-9B	–	91.75	63.00	26.92	80.41	61.35	67.73	65.19
SFT teacher	–	92.92	65.15	33.67	83.87	62.94	70.92	68.24
RL teacher	–	93.16	65.28	30.52	82.39	63.65	71.28	67.71
Student $\leftarrow$ SFT	2	91.12	62.87	26.14	79.53	61.52	68.44	64.94
	4	91.35	63.19	27.52	81.17	61.52	68.44	65.53
	6	92.61	63.38	32.94	82.66	63.30	67.91	67.13

Appendix

Table 8: Perception accuracy (%) across OPD checkpoints.

Explore similar work

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Sep 28, 2026Seonghyeon Kim, Chaeyun Jang, Noah Lee +2Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
KAIST AI · Kakao
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Sep 28, 2026Xin Li, Hao Jiang, Xin Gao +6Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
Nanyang Technological University · Yale University · University of Manchester
Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers
Oct 5, 2026Xiaoyu Chen, Bo Shao, Tiangang Zhu +5
Institute of Software, Chinese Academy of Sciences · Microsoft · Work done during an internship at Microsoft +1

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim+1

KAIST AI · Kakao

Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher Merge

Open paper AskPDF arXiv

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.

Abstract

Explore similar work

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers