A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation

Haoyu Chen; Jun Bai; Yang Liu; Jianfei Zhang; Wenge Rong

cs.LGOct 16, 2026

A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation

Authors: Haoyu Chen, Jun Bai, Yang Liu, Jianfei Zhang, Wenge Rong

Organizations: National College for Excellent Engineers, Beihang University · Beijing Institute for General Artificial Intelligence, BIGAI · Meituan Inc. · School of Computer Science and Engineering, Beihang University

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved Mixture-of-Experts (MoE) LLMs, but its outcome-level rewards provide only sparse supervision. On-Policy Distillation offers dense token-level feedback, yet requires a stronger compatible teacher, while on-policy self-distillation relies on hand-designed privileged information. We observe that the MoE architecture itself offers a way to resolve this dilemma, as increasing the activation budget per token exposes latent expert capacity and yields a potentially stronger teacher within the model. However, naive expansion degrades performance because newly co-activated experts are not optimized to work together. We therefore propose MoEvolve, a bidirectional distillation framework following a recovery-then-improvement strategy: we first restore the activation-expanded high- $k$ model and further strengthen it by exploiting its additional active capacity, then reverse the direction of distillation to transfer the enhanced capabilities back into the original low- $k$ configuration, retaining its inference-time activation budget. Across diverse MoE models and benchmarks, MoEvolve improves the initial models by up to $22.2$ domain-average points and outperforms same-budget RLVR baselines by up to $6.5$ points. Notably, the final low- $k$ models often surpass their activation-expanded teachers.

Figures & tables

Figure 1: Overview of MoEvolve. We increase the activation budget from $k$ to $k^{\prime}$ , restore expert cooperation (ECRD), further strengthen the activation-expanded model with RL (CU), and finally use it as a teacher to improve the original $k$ -activation model without changing its inference-time budget (RD).

OLMoE-1B-7B
Method	ScienceQA				Mathematics						Code generation
Method	GPQA	ARC-C	MMLU-S	Avg.	GSM8K	MATH500	Minerva	AMC23	Poly-L	Avg.	HEval	MBPP	MBPP-S	LCB	Avg.
Original $\mathcal{M}_{k}$	21.5	46.6	32.1	33.4 ±0.12	27.5	7.1	2.1	5.5	25.6	13.6 ±0.08	25.0	19.2	30.0	1.1	18.8 ±0.09
SFT	–	–	–	–	30.4	7.5	1.9	5.5	30.0	15.1 ±0.11	–	–	–	–	–
RFT	26.2	59.4	42.5	42.7 ±0.13	34.2	7.8	2.8	5.0	30.6	16.1 ±0.09	27.2	20.3	31.8	2.4	20.4 ±0.10
RL	26.5	63.3	43.9	44.6 ±0.15	34.6	7.9	2.8	4.8	30.9	16.2 ±0.10	27.4	20.5	32.0	2.3	20.6 ±0.12

Table 1: Main results on OLMoE-1B-7B and DeepSeek-V2-Lite (pass@1, %). Avg. denotes the average score within each domain. SFT covers only mathematics, so its missing entries are marked “–”. Error bars indicate the standard error across 3 seeds.

GPT-OSS-20B
Method	ScienceQA				Mathematics						Code generation
Method	GPQA	ARC-C	MMLU-S	Avg.	AMC23	Poly-M	AIME24	AIME25	AIME26	Avg.	HEval	MBPP	MBPP-S	LCB	Avg.
Original $\mathcal{M}_{k}$	43.5	82.7	70.4	65.5 ±0.16	84.8	48.5	41.3	38.0	43.0	51.1 ±0.14	93.0	52.0	67.5	47.6	65.0 ±0.15
SFT	–	–	–	–	85.8	51.0	45.3	40.0	46.3	53.7 ±0.17	–	–	–	–	–
RFT	48.6	91.5	79.4	73.2 ±0.18	86.5	52.5	46.7	39.7	47.7	54.6 ±0.17	93.4	54.7	70.1	56.6	68.7 ±0.18
RL	50.0	91.1	80.4	73.8 ±0.13	86.8	52.3	47.0	41.7	48.3	55.2 ±0.12	93.5	54.8	70.3	57.5	69.0 ±0.13

Table 2: Main results on GPT-OSS-20B and Qwen3-30B-A3B (pass@1, %). Avg. denotes the average score within each domain. SFT covers only mathematics, so its missing entries are marked “–”. Error bars indicate the standard error across 3 seeds.

Figure 2: Recovery of the activation-expanded model

\mathcal{M}_{k^{\prime}}

, shown as the gap of each recovery strategy relative to the original model

\mathcal{M}_{k}

(

y{=}0

) on the mathematics average (left) and code average (right). Error bars show standard deviation across 3 seeds.

Figure 3: Capability unlocked by the CU stage, measured as the gain from the recovered activation-expanded model after ECRD to the same model after CU. Error bars show standard deviation across 3 seeds.

Figure 4: Gate weights on Qwen3-30B-A3B during cooperation restoration. (a) Mean gate weight per expert before and after ECRD, with experts sorted by their base-model weight. The dashed line marks the uniform-weight reference

1/N

. (b) Mean gate weight of the head, mid, and tail groups over ECRD training steps in multiples of

1/N

Param	AMC23	Poly-M	AIME24	AIME25	AIME26	Avg.
All	71.8	48.2	27.3	20.0	20.7	37.6
Routers	collapse
Experts	65.0	43.8	24.3	17.7	15.7	33.3
R + E	66.0	44.6	23.7	18.7	17.0	34.0

Table 3: Mathematics pass@1 (%) of the activation-expanded Qwen3-30B-A3B after ECRD that updates only the listed components. All denotes full parameter ECRD, and R + E denotes updating both the routers and the experts. All settings are trained for 12 hours of wall-clock time.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Params (A/F)	#Exp. (A/F)
OLMoE-1B-7B	$1.3$ / $6.9$ B	$8$ / $64$
DeepSeek-V2-Lite	$2.4$ / $15.7$ B	$6$ / $64$
GPT-OSS-20B	$3.6$ / $20.9$ B	$4$ / $32$
Qwen3-30B-A3B	$3.3$ / $30.5$ B	$8$ / $128$

Appendix

Table 4: Overview of the MoE models used in this work. A and F refer to “Activated” and “Full”, and Params are in billions.

Group	Setting	Value
Optimization	Learning rate	$1\times 10^{-6}$
	Warm-up steps	0
	Optimizer	Adam
Sequence	Maximum prompt length	2000
Sequence	Maximum response length	11000
Rollout	Rollout $n$	8

Appendix

Table 5: Main training configuration.

Figure 5: Training curves of applying RL directly on the activation-expanded DeepSeek-V2-Lite, without prior cooperation restoration.

Figure 6: Math avg. pass@1 vs. wall-clock training time on Qwen3-30B-A3B. The grey region is Forward Distillation, during which

\mathcal{M}_{k}

is unchanged, and the orange region is Backward Distillation, where capability is transferred back to

\mathcal{M}_{k}

. The dashed blue line shows the RL baseline trained directly on

\mathcal{M}_{k}

for the same total time.

Figure 7: Per-layer, per-expert load of Qwen3-30B-A3B ( $k{=}8$ , $N{=}128$ , $48$ MoE layers) before (top) and after (bottom) our training, measured on the mathematics benchmarks. A cell is the actual/theoretical load of a (layer, expert) slot, so $1.0$ (white) is the uniform-load reference $k/N$ , red is overloaded, pale is starved, and $0$ means the expert is never selected. The number of such dead slots is given in each panel title. Experts are sorted per layer by their load in the base model and the same permutation is used in both panels.

Figure 8: Training dynamics (top) and final expert load (bottom) on Qwen3-30B-A3B. The top row shows mean reward, policy entropy, and response length over training. The bottom row shows per-expert utilization, load balance

H/\log N

, and the fraction of never-selected expert slots.

Objective	OLMoE-1B-7B			DeepSeek-V2-Lite			GPT-OSS-20B			Qwen3-30B-A3B
Objective	Sci.	Math	Code	Sci.	Math	Code	Sci.	Math	Code	Sci.	Math	Code
RFT	44.6	15.9	20.4	53.2	34.4	34.0	74.3	55.9	69.0	81.2	56.5	68.4
OPD	44.7	16.1	20.6	53.2	34.5	34.3	75.3	56.7	69.6	81.4	57.5	68.6
ExOPD	45.7	17.7	22.6	53.8	36.2	35.8	75.4	59.3	70.6	82.0	59.8	69.1

Appendix

Table 6: Ablation on the reverse-distillation objective (Eq. ( 3 )). Given the same recovered teacher

\mathcal{M}_{k^{\prime}}^{\star}

, the original

k

-activation model

\mathcal{M}_{k}

is distilled back using RFT, vanilla OPD, or ExOPD. We report the average score within each domain (Sci. denotes ScienceQA).

Figure 9: Effect of the intermediate activation budget

k^{\prime}

on the three backbones. Error bars indicate the standard error across 3 seeds.

Figure 10: Comparisonof training reward (top) and policy entropy (bottom) on Qwen3-30B-A3B between RL on recovered

\mathcal{M}_{k^{\prime}}

and RL on

\mathcal{M}_{k}

Figure 11: Held-out advantage of the activation-expanded teacher

\mathcal{M}_{k^{\prime}}^{\star}

over RL on the original

\mathcal{M}_{k}

, in pass@1 points. Bars report the three domain averages.

Backbone	$\mathcal{M}_{k}$	ExOPD-teacher
Backbone	$\mathcal{M}_{k}$	RL on $\mathcal{M}_{k}$	$\mathcal{M}_{k^{\prime}}^{\star}$
OLMoE-1B-7B	13.6	16.3	17.7
DeepSeek-V2-Lite	32.5	33.9	36.2
GPT-OSS-20B	51.1	55.7	59.3
Qwen3-30B-A3B	37.6	53.1	59.8

Appendix

Table 7: Teacher-choice comparison for RD on the mathematics average.

Backbone	Math avg.		Code avg.
Backbone	$\mathcal{M}_{k^{\prime}}^{\star}$	$\mathcal{M}_{k}^{\star}$	$\mathcal{M}_{k^{\prime}}^{\star}$	$\mathcal{M}_{k}^{\star}$
OLMoE-1B-7B	17.3	17.7	20.9	22.6
DeepSeek-V2-Lite	35.3	36.2	34.4	35.8
GPT-OSS-20B	57.2	59.3	69.6	70.6
Qwen3-30B-A3B	59.7	59.8	69.5	69.1

Appendix

Table 8: Comparison between the activation-expanded teacher

\mathcal{M}_{k^{\prime}}^{\star}

and the final reverse-distilled student

\mathcal{M}_{k}^{\star}

on mathematics and code averages.

Figure 12: ScienceQA prompt template.

Figure 13: Mathematics prompt template.

Figure 14: Code prompt template.

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 221 papers

Weak-to-Strong Generalization via Direct On-Policy Distillation
Jul 6, 2026Shiyuan Feng, Huan-ang Gao, Haohan Chi +7RL for Language Model Reasoning Weak-to-Strong Generalization
1SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · 4Peking University +1
On-policy Distillation with Verifiable Reward
Aug 25, 2026Wenze Lin, Jiale Zhao, Xitai Jiang +5RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +2
Expert-Space Exploration in MoE Reinforcement Learning
Sep 11, 2026Hongyi He, Zhenghao Lin, Xiao Liu +3Reinforcement Learning Expert Routing
Tsinghua University · Microsoft Research

Jul 6, 2026J/K move · Enter open · S save

Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu+6

1SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · 4Peking University+1

RL for Language Model Reasoning Weak-to-Strong Generalization Reinforcement Learning with Verifiable Rewards On-Policy Distillation Policy Learning

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

Abstract

Mixture-of-Experts Language Models

Explore similar work

Weak-to-Strong Generalization via Direct On-Policy Distillation

On-policy Distillation with Verifiable Reward

Expert-Space Exploration in MoE Reinforcement Learning