Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning

Yiming Huang; Zhenbo Shi; Shuzheng Gao; Cuiyun Gao; Peiyi Han; Chuanyi Liu

cs.CLApr 11, 2026

Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning

Authors: Yiming Huang, Zhenbo Shi, Shuzheng Gao, Cuiyun Gao, Peiyi Han, Chuanyi Liu

Organizations: Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is an essential paradigm that enhances the reasoning capabilities of Large Language Models (LLMs). However, existing methods typically rely on static policy optimization schemes that misalign with the model's evolving reasoning capabilities. To address this issue, we propose Adaptive Power-Mean Policy Optimization (APMPO), which comprises two main innovations: Power-Mean Policy Optimization (PMPO) and Feedback-Adaptive Clipping (FAC). Specifically, PMPO introduces a generalized power-mean objective. This enables the model to adaptively transition from the signal-amplifying behavior of the arithmetic mean to the consistency-enforcing behavior of the geometric mean. FAC adaptively adjusts clipping bounds based on real-time reward statistics to overcome the limitations of static mechanisms. Capitalizing on these innovations, APMPO improves learning dynamics and reasoning performance. Extensive experiments on nine datasets across three reasoning tasks showcase the superiority of APMPO over state-of-the-art RLVR-based baselines. For instance, APMPO boosts the average Pass@1 score on mathematical reasoning benchmarks by 3.0 points compared to GRPO when using Qwen2.5-3B-Instruct.

Figures & tables

Panel of Figure 1 — Figure 1: Illustrations of training dynamics in terms of training rewards, policy entropy, and training time using the MATH training dataset.

Figure 2: Illustration of APMPO, which consists of Power-Mean Policy Optimization (PMPO) and Feedback-Adaptive Clipping (FAC).

Method	Math500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B-Instruct
Base	74.2	10.0 / 16.7	3.3 / 10.0	47.5 / 70.0	28.7	35.2	33.2 / 32.2
GRPO	75.2 ±0.4	13.3 ±0.0 / 16.7 ±0.0	13.3 ±0.0 / 16.7 ±0.0	52.5 ±1.4 / 75.0 ±0.0	29.4 ±0.3	39.0 ±0.6	37.1 ±0.5 / 36.1 ±0.0
DAPO	77.2 ±0.5	16.7 ±1.9 / 23.3 ±0.0	16.7 ±1.9 / 23.3 ±0.0	57.5 ±0.0 / 80.0 ±1.4	29.0 ±0.4	40.4 ±0.6	39.6 ±0.9 / 42.2 ±0.5
GMPO	76.6 ±0.4	13.3 ±0.0 / 20.0 ±1.9	20.0 ±0.0 / 26.7 ±0.0	55.0 ±1.4 / 82.5 ±0.0	30.1 ±0.3	38.7 ±0.5	39.0 ±0.4 / 43.1 ±0.6
APMPO	78.0 ±0.3	20.0 ±0.0 / 30.0 ±0.0	16.7 ±0.0 / 26.7 ±0.0	62.5 ±0.0 / 85.0 ±0.0	30.5 ±0.2	42.4 ±0.3	41.7 ±0.1 / 47.2 ±0.0

Table 1: Experimental results on multiple mathematical reasoning benchmarks. The results are reported as mean and standard deviation across 3 random seeds (format:

\text{Mean}_{\pm\text{Std}}

). The best results are highlighted in bold. Note that “

\cdot/\cdot

” indicates “Pass@1/Pass@16”, and the last column indicates “Average Pass@1 / Average Pass@16”.

Panel of Figure 3 — Figure 3: Experimental results on (a) SQL generation, and (b) multi-modal reasoning tasks.

Panel of Figure 4 — Figure 4: Ablation studies and sensitivity analysis. Results are reported as average Pass@1 scores on mathematical reasoning benchmarks. (a) Contribution of each component (PMPO and FAC). (b-c) Effectiveness of different design choices for FSS and the adaptive exponent function. (d-e) Sensitivity analysis for $\gamma$ and $(\epsilon_{\text{min}},\epsilon_{\text{max}})$ .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\pi_{old},\pi$	The policy before and after the update step.
$J(\pi)$	The true expected return of policy $\pi$ in the environment.
$L_{true}(\pi)$	The theoretical surrogate objective using exact advantage values.
$\hat{L}(\pi)$	The empirical surrogate objective using sampled advantages.
$\mathcal{J}(\theta)$	Shorthand for the objective function parameterized by $\theta$ .
$\sigma_{R}$	The standard deviation of the reward estimation noise.

Appendix

Table 2: Summary of notations used in the theoretical derivation.

Figure 5: Illustration of the synergy of PMPO and FAC.

Method	Math500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B-Instruct
FSS (Adaptive Lower)	77.5	20.0	16.7	60.0	30.1	42.0	41.1
FSS (Fixed Lower)	78.0	20.0	16.7	62.5	30.5	42.4	41.7
Qwen2.5-3B-Instruct
FSS (Adaptive Lower)	68.2	10.0	10.0	42.5	27.6	32.9	31.8
FSS (Fixed Lower)	68.4	10.0	10.0	45.0	27.9	33.2	32.4

Appendix

Table 3: Ablation study on the asymmetry of FAC. We compare our default asymmetric design (Fixed Lower Bound) against a symmetric variant where both bounds are adaptive. The best results are highlighted in bold.

Method	ACS ( $\downarrow$ )	Self-BLEU ( $\downarrow$ )	GPT-Diversity ( $\uparrow$ )
GRPO	0.84	0.76	1.22
DAPO	0.79	0.71	1.35
GMPO	0.75	0.65	1.41
APMPO	0.66	0.54	1.62

Appendix

Table 4: Diversity metrics on the MATH-500 dataset using Qwen2.5-Math-1.5B-Instruct. Comparisons are made against state-of-the-art RLVR-based baselines.

\downarrow

indicates lower is better, and

\uparrow

indicates higher is better.

Method	Geometry3K	Spider	BIRD
Base	25.7	70.2	43.6
GRPO	35.6 ±0.8	73.8 ±0.5	55.7 ±0.9
DAPO	36.5 ±0.7	75.6 ±0.6	58.2 ±0.8
GMPO	37.2 ±0.6	74.8 ±0.5	57.8 ±0.7
APMPO	37.9 ±0.5	76.4 ±0.4	60.6 ±0.6

Appendix

Table 5: Generalization performance on multi-modal reasoning (Geometry3K) and SQL generation (Spider, BIRD). The results are given as mean and standard deviation across 3 random seeds.

Method	Math500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B-Instruct
GRPO	75.2	13.3	13.3	52.5	29.4	39.0	37.5
+ FAC	76.4	16.7	13.3	57.5	29.8	40.1	39.0
+ PMPO	77.2	20.0	16.7	60.0	30.1	41.4	40.9
APMPO	78.0	20.0	16.7	62.5	30.5	42.4	41.7
Qwen2.5-3B-Instruct

Appendix

Table 6: Ablation study on multiple mathematical reasoning benchmarks using different modules. The best results are highlighted in bold, and we reported Pass@1 score.

Method	Math500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B-Instruct
Only $\sigma_{R}$	77.2	16.7	16.7	60.0	29.8	41.4	40.3
Only $\mu_{R}$	77.6	20.0	16.7	60.0	30.1	42.0	41.1
FSS	78.0	20.0	16.7	62.5	30.5	42.4	41.7
Qwen2.5-3B-Instruct
Only $\sigma_{R}$	67.6	6.7	10.0	42.5	26.8	32.6	31.0

Appendix

Table 7: Ablation study on multiple mathematical reasoning benchmarks using different variants of FSS. The best results are highlighted in bold, and we reported Pass@1 score.

Method	Math500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B-Instruct
Linear $p$	77.2	16.7	16.7	57.5	29.8	41.8	40.0
Ours	78.0	20.0	16.7	62.5	30.5	42.4	41.7
Qwen2.5-3B-Instruct
Linear $p$	68.0	6.7	10.0	40.0	27.2	32.8	30.8
Ours	68.4	10.0	10.0	45.0	27.9	33.2	32.4

Appendix

Table 8: Ablation study on multiple mathematical reasoning benchmarks using different

p

formulations. The best results are highlighted in bold, and we reported Pass@1 score.

$\gamma$	Math500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B-Instruct
0.2	76.8	13.3	13.3	55.0	28.6	41.4	38.1
0.4	77.2	13.3	13.3	57.5	29.4	41.8	38.8
0.6	77.6	16.7	16.7	60.0	30.1	42.1	40.5
0.8	78.0	20.0	16.7	62.5	30.5	42.4	41.7
1.0	77.8	20.0	13.3	62.5	29.8	41.5	40.8

Appendix

Table 9: Ablation study on multiple mathematical reasoning benchmarks using different values of

\gamma

. The best results are highlighted in bold, and we reported Pass@1 score.

$(\epsilon_{\text{min}},\epsilon_{\text{max}})$	Math500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B-Instruct
$(0.1,0.3)$	77.2	16.7	13.3	55.0	29.4	41.2	38.8
$(0.1,0.4)$	77.8	20.0	13.3	60.0	30.1	42.1	40.6
$(0.2,0.3)$	77.6	16.7	13.3	60.0	29.8	41.4	39.8
$(0.2,0.4)$	78.0	20.0	16.7	62.5	30.5	42.4	41.7
Qwen2.5-3B-Instruct

Appendix

Table 10: Ablation study on multiple mathematical reasoning benchmarks using different values of

(\epsilon_{\text{min}},\epsilon_{\text{max}})

. The best results are highlighted in bold, and we reported Pass@1 score.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,057 papers

Experience Augmented Policy Optimization for LLM Reasoning
Jun 29, 2026Jinda Lu, Kexin Huang, Junkang Wu +7Reinforcement Learning RL for Language Model Reasoning
University of Science and Technology of China · 2Peking University · 3Dartmouth College +1
CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning
Aug 4, 2026Ziqi Jia, Yalu Ouyang, Bo Pang +5Numerical Reasoning in Language Models RL for Language Model Reasoning
AI Cloud Group, Baidu · Tsinghua University · University of California, San Diego
Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR
May 8, 2026Yash Ingle, Jaival Chauhan, Ankit Yadav +1Reinforcement Learning RL for Language Model Reasoning
Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India

Jun 29, 2026J/K move · Enter open · S save

Experience Augmented Policy Optimization for LLM Reasoning

Jinda Lu, Kexin Huang, Junkang Wu, Shuo Yang+6

University of Science and Technology of China · 2Peking University · 3Dartmouth College+1

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.

Abstract

RL for Language Model Reasoning

Explore similar work

Experience Augmented Policy Optimization for LLM Reasoning

CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning

Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR