MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

Xuyang Zhong; Jiuding Yang; Yiwen Guo; Chen Liu

cs.LGOct 22, 2026

MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

Authors: Xuyang Zhong, Jiuding Yang, Yiwen Guo, Chen Liu

Organizations: City University of Hong Kong · LIGHTSPEED · Independent Researcher

Abstract

Mix-policy optimization incorporates off-policy expert data into the reinforcement learning (RL) loop to transcend the capability boundary of on-policy RL. However, existing methods either provide static full-trace guidance that suppresses self-exploration, or adopt expert prefix guidance without correctness guarantees, leading to suboptimal performance or even training collapse. In this paper, we propose MOER, a mix-policy optimization framework that harmonizes expert guidance with self-exploration. MOER injects a decaying expert chain-of-thought prefix into the prompt and lets the model complete the remaining reasoning, with the prefix length progressively annealed to encourage self-exploration. A selective rollback mechanism replaces failed completions with the original full expert trace, guaranteeing the correctness of every off-policy trajectory. Furthermore, a harmonized mix-policy objective optimizes on-policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust-region SFT loss, substantially stabilizing training. Extensive experiments show that MOER achieves the highest average scores in mathematical and general domain reasoning tasks, outperforming all basic and mix-policy baselines, and generalizes consistently across diverse tasks, model scales, and base model categories. Codes are available at https://github.com/CityU-MLO/MOER.

Figures & tables

Figure 1: Overview of MOER , which incorporates decaying expert prefix guidance with a selective rollback mechanism and harmonized mix-policy objective. Decaying expert prefix guidance encourages the model’s self-exploration capability as the training progresses. Selective rollback mechanism guarantees the correctness of off-policy traces. Harmonized mix-policy objective makes the training stable and effective.

Panel of Figure 2 — Figure 2: Training dynamics of MOER and other baselines with prefix guidance. The base model is Qwen2.5-Math-7B. (a) MOER reaches the highest validation accuracy. (b) Selective rollback mechanism guarantees the correctness of expert guidance. (c) MOER exhibits a stable increase in response length. Note that TRAPO is early-stopped due to training collapse.

Figure 3: Average output probability of pure on-policy, completion, and expert prefix tokens. The solid curves are smoothed.

Method	Math Reasoning Performance						General Domain Reasoning Performance
Method	AIME 24/25	AMC	MATH-500	Minerva	Olympiad	Avg.	ARC-c	GPQA *	MMLU-Pro	Avg.
Base	11.5/4.9	31.3	43.6	7.4	15.6	19.0	18.2	11.1	16.9	15.4
Basic Baselines
SFT	22.2/ 22.3	52.8	82.6	40.8	43.7	44.1	75.2	24.7	42.7	47.5
RL	25.1/15.3	62.0	84.4	39.3	46.8	45.5	82.3	40.4	49.3	57.3
RL w/ SFT Loss	19.5/16.4	49.7	80.4	34.9	39.4	40.1	71.2	23.7	43.2	46.0

Table 1: Main experiment results on math and general reasoning benchmarks based on Qwen2.5-Math-7B. All baselines are evaluated on the same dataset and base model. Bold and underline indicate the best and second-best results, respectively.

Method	Math Reasoning Performance						General Domain Reasoning Performance
Method	AIME 24/25	AMC	MATH-500	Minerva	Olympiad	Avg.	ARC-c	GPQA *	MMLU-Pro	Avg.
Qwen2.5-Math-1.5B
LUFFY	16.0/ 12.9	47.1	77.2	32.7	42.7	38.1	55.6	26.8	34.6	39.0
ReLIFT	14.6/9.0	41.7	72.0	29.0	38.4	34.1	55.0	25.8	30.4	37.1
MOER	16.6 /12.2	46.7	79.6	29.8	43.4	38.1	60.4	26.8	34.6	40.6
Qwen2.5-Instruct-7B

Table 2: Experiment results on math and general reasoning benchmarks based on Qwen2.5-Math-1.5B and Qwen2.5-Instruct-7B.

Method	Math Reasoning Performance						General Domain Reasoning Performance
Method	AIME 24/25	AMC	MATH-500	Minerva	Olympiad	Avg.	ARC-c	GPQA *	MMLU-Pro	Avg.
LUFFY (GRPO+ FG )	28.5/21.6	61.0	86.4	40.4	52.4	48.4	81.9	37.9	52.2	57.3
+ HO	27.8/21.5	63.1	87.6	37.9	53.8	48.6	83.2	38.9	52.9	58.3
+ HO + PG + COff	24.7/15.4	60.7	84.4	40.8	51.3	46.2	82.5	41.9	50.4	58.3
+ HO + PG + SR	27.1/22.6	63.4	87.8	38.2	54.4	48.9	83.3	39.5	53.2	58.7
+ HO - FG + SR	24.9/18.8	61.9	85.8	37.5	53.9	47.1	83.4	36.4	53.3	57.7

Table 3: Ablation study on components of MOER based on Qwen2.5-Math-7B. HO denotes harmonized objective , PG denotes decaying prefix guidance , SR denotes selective rollback , FG denotes full expert guidance , COff denotes completion as off-policy tokens .

Panel of Figure 4 — Figure 4: Training dynamics of MOER and its ablations. HO denotes harmonized objective, PG denotes decaying prefix guidance, and SR denotes selective rollback.

Panel of Figure 5 — Figure 5: Effect of selective rollback mechanism. Note that MOER w/o SR is identical to +HO+PG+COff and it is early-stopped due to training collapse. The actual prefix ratio is calculated by $\mathrm{prefix~ratio}\times(1-\mathrm{rollback~rate})+\mathrm{rollback~rate}$ .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	LUFFY	ReLIFT	CHORD	TRAPO	Prefix-RFT	MOER
Scaled	✓	$\bm{\times}$	✓	$\bm{\times}$	✓	✓
Bounded	✓	$\bm{\times}$	✓	✓	$\bm{\times}$	✓

Appendix

Table 4: Properties of gradient weights of different off-policy objectives. Scaled denotes that the gradient weight is scaled based on the SFT objective. Bounded denotes that the value of the gradient weight is bounded.

Panel of Figure 7 — Figure 7: Training dynamics of different off-policy objectives.

Panel of Figure 8 — Figure 8: Training dynamics of LUFFY and MOER under different rollout/update batch size settings. When the rollout batch size is $64$ and the update batch size is $16$ , i.e., the updates become more off-policy, LUFFY collapses in the middle of training, while MOER remains stable.

Figure 9: Accuracy of on-policy rollouts and prefix-conditioned completions along training. Note that the first

50

steps are warm-up stage, where the prefix is the full expert CoT.

Method	Pass@1	Entropy	Response Length
LUFFY	33.8	0.453	1903
MOER	38.3	0.246	668

Appendix

Table 6: Results on the LiveCodeBench benchmark. We report the pass@1, entropy, and response length of MOER and LUFFY.

Method	Math Reasoning Performance
Method	AIME 24/25	AMC	MATH-500	Minerva	Olympiad	Avg.
Qwen2.5-Instruct-14B
LUFFY	18.4/ 18.5	52.9	84.4	46.3	48.3	44.8
MOER	25.7 /14.3	54.9	85.2	43.0	52.0	45.9

Appendix

Table 7: Experiment results on mathematical reasoning benchmarks based on Qwen2.5-Instruct-14B.

Figure 11: Pass@k results of MOER and baselines on the AIME 2025 benchmark. The base model is Qwen2.5-Math-7B. We report the results with

k=2^{0},2^{1},...,2^{8}

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,087 papers

expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling
May 11, 2026Mingxiong Lin, Zhangquan Gong, Maowen Tang +6Reinforcement Learning KL-Regularized RL
OPPO AI Center
Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating
Mar 11, 2026Yuning Wu, Ke Wang, Haoran Liu +3Reinforcement Learning RL for Language Model Reasoning
Amazon
Experience Augmented Policy Optimization for LLM Reasoning
Jun 29, 2026Jinda Lu, Kexin Huang, Junkang Wu +7Reinforcement Learning RL for Language Model Reasoning
University of Science and Technology of China · 2Peking University · 3Dartmouth College +1

May 11, 2026J/K move · Enter open · S save

expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling

Mingxiong Lin, Zhangquan Gong, Maowen Tang, Qian Li+5

OPPO AI Center

Reinforcement Learning KL-Regularized RL RL for Language Model Reasoning Group Relative Policy Optimization Curriculum RL RL Exploration+2

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, where Group Relative Policy Optimization (GRPO) serves as the mainstream algorithm. We point out two understudied inefficiencies existing in GRPO. First, the fixed KL penalty coefficient overly restricts policy exploration at stages where the model requires significant deviation from the reference policy. Second, uniform sampling of training questions ignores that moderately difficult problems provide the most informative gradient signals for optimization. We propose Exploration-Prioritized Policy Optimization (EXPO) with two lightweight plug-in modules. The Accuracy-Conditioned KL Scaling (AKL) dynamically adjusts KL regularization strength through a smooth nonlinear function of batch average accuracy, relaxing the penalty when the model underperforms and strengthening it when the model achieves good results. The Gaussian Curriculum Sampling (GCS) assigns sampling weights to questions following a Gaussian distribution centered at moderate accuracy around 0.5, focusing training on the model's learning frontier. We conduct extensive experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base over six mathematical reasoning benchmarks. The results show EXPO steadily surpasses vanilla GRPO. It obtains an absolute gain of 13.34 on AIME 2025 pass@32, rising from 63.33 percent to 76.67 percent, and achieves an average pass@32 improvement of 2.66 on the 8B model. The much larger performance gains on pass@32 compared with pass@1 demonstrate that EXPO effectively enlarges the model's exploration boundary under a fixed inference cost budget.

Abstract

RL for Language Model Reasoning

Explore similar work

expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling

Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating

Experience Augmented Policy Optimization for LLM Reasoning