MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning

Hongjun Wang; Wei Liu; Weibo Gu; Xing Sun; Kai Han

cs.LGMar 14, 2026

MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning

Authors: Hongjun Wang, Wei Liu, Weibo Gu, Xing Sun, Kai Han

Organizations: The University of Hong Kong · Tencent Youtu Lab

Abstract

Regulating the importance ratio is critical for the training stability of Group Relative Policy Optimization (GRPO) based frameworks. However, prevailing ratio control methods, such as hard clipping, suffer from non-differentiable boundaries and vanishing gradient regions, failing to maintain gradient fidelity. Furthermore, these methods lack a hazard-aware mechanism to adaptively suppress extreme deviations, leaving the optimization process vulnerable to abrupt policy shifts. To address these challenges, we propose Modulated Hazard-aware Policy Optimization (MHPO), a novel framework designed for robust and stable reinforcement learning. The proposed MHPO introduces a Log-Fidelity Modulator (LFM) to map unbounded importance ratios into a bounded, differentiable domain. This mechanism effectively prevents high-variance outlier tokens from destabilizing the loss landscape while ensuring global gradient stability. Complementarily, a Decoupled Hazard Penalty (DHP) integrates cumulative hazard functions from survival analysis to independently regulate positive and negative policy shifts. By shaping the optimization landscape with hazard-aware penalties, the proposed MHPO achieves fine-grained regulation of asymmetric policy shifts simultaneously mitigating mode collapse from over-expansion and preventing policy erosion from catastrophic contraction within a stabilized trust region. Extensive evaluations on diverse reasoning benchmarks across both text-based and vision-language tasks demonstrate that MHPO consistently outperforms existing methods, achieving superior performance while significantly enhancing training stability.

Figures & tables

Figure 1: Overview of

\mathrm{MHPO}

. (a) Performance gain over baseline (Avg@32, %) on Qwen3-4B-Base. The bar chart displays the improvement (

\Delta

) achieved by different RL methods across five benchmarks.

\mathrm{MHPO}

(navy) consistently achieves the largest gain on every benchmark. (b) Gradient norm trajectory during RL training. Baseline methods exhibit frequent gradient spikes, whereas

\mathrm{MHPO}

maintains consistently low and stable gradient magnitudes throughout training, empirically confirming the bounded gradient multiplier guaranteed by our theoretical analysis. (c) Overall reward curve during training.

\mathrm{MHPO}

attains higher reward earlier and sustains this advantage, while competing methods plateau or exhibit degradation in later stages.

Figure 2: Characteristics of the Log-Fidelity Modulator (LFM). (a) The LFM operator

\psi(r_{t}^{i}(\theta))

with different values of

c

. The transformation exhibits reciprocal antisymmetry (

\psi(1/r_{t}^{i}(\theta))=-\psi(r_{t}^{i}(\theta))

) around the on-policy anchor (

r_{t}^{i}(\theta)=1

) and smoothly bounds the output within the interval

[-c,c]

. The marked points at

r_{t}^{i}(\theta)=\tfrac{1}{2}

and

r_{t}^{i}(\theta)=2

(for

c=1.0

) confirm

\psi(\tfrac{1}{2})=-\psi(2)

. (b) The derivative of the LFM operator, demonstrating smooth gradient attenuation for extreme importance ratios while preserving local fidelity (

d\psi/dr=1

) near the on-policy anchor.

Figure 3: Analysis of the Decoupled Hazard Penalty (DHP). (a) The penalty $\zeta(\psi)$ with different $(\lambda,k)$ configurations. The default asymmetric setting ( $\lambda_{+}=1.0,k_{+}=1.5,\lambda_{-}=0.8,k_{-}=2.0$ ) applies stronger suppression for negative shifts. (b) The corresponding survival weight $w=\exp(-\zeta)\in(0,1]$ , demonstrating that the DHP can only attenuate and never amplify token contributions.

(A) Qwen2.5-7B-Instruct
Method	MATH500	HMMT25	AMC23	AIME25	AIME24	Avg.
Baseline	70.0 (+0.0)	5.6 (+0.0)	52.4 (+0.0)	3.5 (+0.0)	16.4 (+0.0)	29.6 (+0.0)
GRPO ( Shao et al., 2024 )	79.2 (+9.2)	7.8 (+2.2)	68.3 (+15.9)	8.6 (+5.1)	27.4 (+11.0)	38.3 (+8.7)
GSPO ( Zheng et al., 2025 )	80.6 (+10.6)	8.5 (+2.9)	70.1 (+17.7)	10.8 (+7.3)	26.8 (+10.4)	39.4 (+9.8)
DAPO ( Yu et al., 2025 )	81.2 (+11.2)	9.1 (+3.5)	71.3 (+18.9)	11.4 (+7.9)	28.0 (+11.6)	40.2 (+10.6)
SAPO ( Gao et al., 2025 )	80.4 (+10.4)	8.0 (+2.4)	69.2 (+16.8)	10.1 (+6.6)	27.1 (+10.7)	39.0 (+9.4)

Table 1: Main results on instruction-tuned backbones. We report Avg@32 (%) for the best checkpoint. Subtable (A) reports results on Qwen2.5-7B-Instruct (general-purpose), Subtable (B) reports text-only math reasoning on Qwen2.5-Math-7B-Instruct (domain-specialized), and Subtable (C) reports vision-language math reasoning on Qwen2.5-VL-7B-Instruct .

Method	MATH500	HMMT25	AMC23	AIME25	AIME24	Avg.
Baseline	57.2 (+0.0)	3.8 (+0.0)	41.3 (+0.0)	5.7 (+0.0)	8.8 (+0.0)	23.4 (+0.0)
GRPO ( Shao et al., 2024 )	64.8 (+7.6)	11.1 (+7.3)	81.3 (+40.0)	30.8 (+25.1)	40.4 (+31.6)	45.7 (+22.3)
GSPO ( Zheng et al., 2025 )	66.2 (+9.0)	12.4 (+8.6)	83.5 (+42.2)	34.0 (+28.3)	39.7 (+30.9)	47.2 (+23.8)
DAPO ( Yu et al., 2025 )	64.0 (+6.8)	14.9 (+11.1)	75.8 (+34.5)	29.2 (+23.5)	34.2 (+25.4)	43.6 (+20.2)
SAPO ( Gao et al., 2025 )	66.4 (+9.2)	11.5 (+7.7)	81.4 (+40.1)	32.9 (+27.2)	39.6 (+30.8)	46.4 (+23.0)
$\mathrm{MHPO}$ (Ours)	70.9 (+13.7)	19.4 (+15.6)	88.2 (+46.9)	41.4 (+35.7)	44.7 (+35.9)	52.9 (+29.5)

Table 2: Results on base model without instruction tuning. We report Avg@32 (%) for the best checkpoint on Qwen3-4B-Base. Avg. denotes the macro-average across benchmarks.

(a) Bound $c$ (hazard-free)		(b) Shape $k$ ( $c{=}1.5,\;\lambda{=}0.8$ )		(c) Scale $\lambda$ ( $c{=}1.5,\;k{=}3.0$ )
$c$	Avg@32	$k$	Avg@32	$\lambda$	Avg@32
0.5	48.8	1.0	52.2	0.5	52.9
1.0	47.5	1.5	51.4	0.8	53.0
1.5	49.7	2.5	52.3	1.0	52.2
2.0	49.6	3.0	53.4	2.0	50.0

Table 3: Hyperparameter sensitivity of

\mathrm{MHPO}

(Qwen2.5-VL-7B, Geo3K training). We report macro-average Avg@32 (%) across MathVista, MathVision, and MathVerse. Bold denotes the best value among the reported settings.

Method	MATH500		HMMT25		AMC23		AIME25		AIME24		Avg.		$\Delta$
	Best	Latest	Best	Latest	Best	Latest	Best	Latest	Best	Latest	Best	Latest
GRPO	64.8	58.9	11.1	8.8	81.3	72.5	30.8	23.3	40.4	24.2	45.7	37.5	$-$ 8.2
GSPO	66.2	59.9	12.4	8.6	83.5	75.9	34.0	25.9	39.7	27.0	47.2	39.5	$-$ 7.7
DAPO	64.0	47.5	14.9	7.0	75.8	66.3	29.2	17.9	34.2	18.4	43.6	31.4	$-$ 12.2
SAPO	66.4	59.2	11.5	7.3	81.4	77.6	32.9	23.4	39.6	24.0	46.4	38.3	$-$ 8.1
$\mathrm{MHPO}$ (Ours)	70.9	70.1	19.4	19.4	88.2	88.2	41.4	40.8	44.7	41.8	52.9	52.1	$-$ 0.8

Table 4: Training stability comparison between Best and Latest checkpoints (Qwen3-4B-Base). We report Avg@32 (%) with separate columns for the best checkpoint (Best) and the final checkpoint (Latest).

\Delta

denotes the performance degradation from Best to Latest.

\mathrm{MHPO}

exhibits minimal degradation, demonstrating superior training stability.

Explore similar work

May 26, 2026

Ratio-Variance Regularized Policy Optimization

Standard on-policy reinforcement learning relies on heuristic clipping to enforce trust regions, but this mechanism imposes a severe cost by indiscriminately truncating high-return yet high-divergence updates. We demonstrate that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints, eliminating the need for binary hard clipping. By acting as a distributional ``soft brake'', this approach preserves critical gradient signals from novel discoveries while naturally down-weighting and enabling the reuse of stale, off-policy data. We introduce

{\bf R}^2{\bf VPO}

(Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework. Extensive evaluations across

7

LLM scales, spanning both fast and slow reasoning paradigms, and

10

robotic control tasks demonstrate the generality of the proposed approach. R

^2

VPO achieves substantial performance gains on mathematical reasoning benchmarks, with particularly pronounced improvements on smaller models, while significantly improving sample efficiency. Furthermore, it consistently outperforms PPO baselines in continuous control domains, particularly in sparse-reward and dynamic environments. Together, these findings establish ratio-variance regularization as a principled foundation for stable and data-efficient policy optimization.

Yu Luo, Shuo Han, Yihan Hu +5

Department of Foundation Model, 2012 Labs, Huawei Shanghai · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Department of Computer Science and Technology, Tsinghua University +1

RL for Language Model Reasoning Policy Optimization Robotic RL Proximal Policy Optimization Sample-Efficient RL

Ask PDF

May 12, 2026

Holder Policy Optimisation

Group Relative Policy Optimisation (GRPO) enhances large language models by estimating advantages across a group of sampled trajectories. However, mapping these trajectory-level advantages to policy updates requires aggregating token-level probabilities within each sequence. Relying on a fixed aggregation mechanism for this step fundamentally limits the algorithm's adaptability. Empirically, we observe a critical trade-off: certain fixed aggregations frequently suffer from training collapse, while others fail to yield satisfactory performance. To resolve this, we propose HölderPO, a generalised policy optimisation framework unifying token-level probability aggregation via the Hölder mean. By explicitly modulating the parameter

p

, our framework provides continuous control over the trade-off between gradient concentration and variance bounds. Theoretically, we prove that a larger

p

concentrates the gradient to amplify sparse learning signals, whereas a smaller

p

strictly bounds gradient variance. Because no static configuration can universally resolve this concentration-stability trade-off, we instantiate the framework with a dynamic annealing algorithm that progressively schedules

p

across the training lifecycle. Extensive evaluations demonstrate superior stability and convergence over existing baselines. Specifically, our approach achieves a state-of-the-art average accuracy of

54.9\%

across multiple mathematical benchmarks, yielding a substantial

7.2\%

relative gain over standard GRPO and secures an exceptional

93.8\%

success rate on ALFWorld.

Yuxiang Chen, Dingli Liang, Yihang Chen +8

University College London, London, United Kingdom. · 3The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · 2Shanghai Jiao Tong University, Shanghai, China.

RL for Language Model Reasoning Group Relative Policy Optimization Policy Optimization

Ask PDF

Mar 1, 2026

Stabilizing Policy Optimization via Logits Convexity

While reinforcement learning (RL) has been central to the recent success of large language models (LLMs), RL optimization is notoriously unstable, especially when compared to supervised fine-tuning (SFT). In this work, we investigate the stability gap between SFT and RL from a gradient-based perspective, and show that the convexity of the SFT loss with respect to model logits plays a key role in enabling stable training. Our theoretical analysis demonstrates that this property induces favorable gradient directionality during optimization. In contrast, Proximal Policy Optimization (PPO), a widely adopted policy gradient algorithm utilizing a clipped surrogate objective, lacks this stabilizing property. Motivated by this observation, we propose Logits Convex Optimization (LCO), a simple yet effective policy optimization framework that aligns the learned policy with an optimal target derived from the original RL objective, thereby emulating the stabilizing effects of logits-level convexity. Extensive experiments across multiple model families show that our LCO framework consistently improves training stability and outperforms conventional RL methods on a broad range of benchmarks.

Hongzhan Chen, Tao Yang, Yuhua Zhu +3

School of Computer Science and Engineering, Sun Yat-sen University, China · Shanghai Innovation Institute · Wechat Search, Tencent Inc, China +1

Policy Optimization Reinforcement Learning RL Fine-Tuning Policy Gradient Methods Proximal Policy Optimization

Ask PDF