Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization

Xiangchi Yuan; Xingjian Diao; Zheyuan Liu; Chunhui Zhang; Dachuan Shi; Kejing Xia; Zhenbang Du; Shenglong Yao

cs.LGOct 26, 2026

Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization

Authors: Xiangchi Yuan, Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Dachuan Shi, Kejing Xia, Zhenbang Du, Shenglong Yao,

Organizations: Georgia Institute of Technology · Dartmouth College · University of Notre Dame

Abstract

LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used additive reward aggregation in RL can violate these priorities: when primary and auxiliary advantages conflict, a sufficiently large auxiliary signal can overturn the preference induced by the primary objective. We term this problem advantage priority inversion. Although reward-level gating can protect priority more directly, it still removes potentially useful auxiliary distinctions among rollout samples that fail the primary condition. To address this trade-off, we propose Primary Advantage Policy Optimization (PAPO), a priority-aware advantage aggregation method that asymmetrically combines a primary and an auxiliary advantage. PAPO retains auxiliary supervision when the objectives agree and smoothly attenuates conflicting auxiliary signals, preserving the primary sign during fusion for primary-consistent ordering. This rule applies to sequence-level advantages in GRPO and token-level advantages in actor--critic PPO. Experiments on multiple domains and models show that PAPO consistently achieves the best primary objective while maintaining best or competitive auxiliary-objective performance across both GRPO- and PPO-based optimization. Code is available.

Figures & tables

Figure 1: Advantage aggregation. (a) Additive aggregation can cause advantage priority inversion , where a dominant auxiliary signal reverses the sign induced by the primary advantage. (b) Hard gating enforces priority coarsely by discarding auxiliary distinctions for unsatisfied primary condition. (c) PAPO preserves the primary sign during fusion while retaining graded auxiliary supervision, maintaining the primary-induced preference structure.

Figure 2: Overview. (a) GRPO-based PAPO. Given multiple rollouts, primary and auxiliary rewards are converted to sequence-level advantages and combined. (b) PPO-based PAPO. A multi-head critic estimates objective-specific values for the primary and auxiliary objectives. Token-level advantages are then computed using GAE and fused by PAPO before optimizing the actor.

Figure 3: PAPO Dynamics. PAPO maintains the best accuracy

\uparrow

while having a competitive auxiliary length compliance

\uparrow

for GRPO-based (top) and PPO-based (bottom) methods.

Figure 5: Multimodal safety alignment.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Multimodal safety alignment results for the Qwen2-VL-7B model.

Figure 7: Dynamics on agentic tool use. Although baseline methods can reduce tool calls aggressively, this behavior can coincide with degraded accuracy. PAPO maintains stronger primary-objective performance while still reducing tool usage, yielding a better trade-off for GRPO-based (top) and PPO-based (bottom) methods.

Figure 8: Per-dataset evaluation dynamics on mathematical reasoning. We report accuracy and length compliance throughout training for each benchmark under GRPO- and PPO-based optimization.

Figure 9: Per-difficulty evaluation dynamics on agentic tool use. We report accuracy and the number of tool calls for easy, medium, and hard instances under GRPO- and PPO-based optimization.

Dataset	Method	Pass@1	Pass@2	Pass@3	Pass@4	Pass@5	Len. $\leq$ 8K
AIME24	Base	50.67	60.00	63.00	63.33	63.33	30.00
	PPO	41.33	48.67	53.67	57.33	60.00	84.67
	GRPO	44.67	55.00	59.67	63.33	66.67	90.67
	GDPO	46.00	53.00	57.33	60.67	63.33	96.00
	PAPO-PPO (Ours)	63.33	73.33	76.00	76.67	76.67	68.00
	PAPO-GRPO (Ours)	63.33	73.00	77.00	79.33	80.00	77.33

Appendix

Table 6: Pass@

k

and length-compliance results on mathematical reasoning benchmarks. We report Pass@

k

(%) for

k\in\{1,2,3,4,5\}

and the percentage of responses within the 8K-token length budget (Len.

\leq

8K). The best result for each metric is shown in bold , and the second-best result is underlined .

Hyperparameter	GRPO-based	PPO-based
Training batch size	512	512
Policy mini-batch size	64	64
Micro-batch / GPU	8	8
Rollouts / prompt	4	1
Maximum prompt length	1,024	1,024
Maximum response length	12,000	12,000

Appendix

Table 7: Training hyperparameters for mathematical reasoning experiments. Settings shared across methods are held fixed within each optimization family.

Hyperparameter	GRPO-based	PPO-based
Training batch size	256	256
Policy mini-batch size	32	32
Micro-batch / GPU	8	8
Rollouts / prompt	8	1
Maximum prompt length	4,096	4,096
Maximum response length	2,048	2,048

Appendix

Table 8: Training hyperparameters for the When2Tool experiments.

Dataset	# Examples	Role	Scope
DAPO-Math-17K	$\sim$ 17K	Train	Competition-style mathematics with verifiable reference answers; covers algebra, geometry, number theory, combinatorics, and related topics.
AMC23	40	Test	2023 AMC competition problems; comparatively short high-school competition mathematics across algebra, geometry, number theory, and combinatorics.
AIME 2024	30	Test	AIME I and II 2024; difficult competition problems with integer final answers.
AIME 2025	30	Test	AIME I and II 2025; difficult competition problems with integer final answers.
MATH-500	500	Test	Curated subset of MATH covering algebra, geometry, number theory, counting/probability, intermediate algebra, and precalculus.
Minerva Math	272	Test	University-level quantitative reasoning from STEM subjects, including mathematics, physics, astronomy, and chemistry.

Appendix

Table 9: Datasets used for mathematical reasoning. DAPO-Math-17K is used for policy optimization, while the remaining datasets are held out for evaluation. The evaluation suite spans standard competition mathematics, Olympiad-level problems, and university-level quantitative reasoning.

Category	Environment	Capability	Easy $\rightarrow$ Hard progression
Computational Scale	Calculator	Arithmetic evaluation	Small integer arithmetic $\rightarrow$ large multi-operation or trillion-scale expressions.
	Statistics	Statistical computation	Mean/median of short lists $\rightarrow$ variance, standard deviation, or correlation over longer lists.
	Counting	Combinatorial counting	Small combinations such as $\binom{6}{4}$ $\rightarrow$ large combinatorial quantities such as $\binom{80}{40}$ .
	Matrix	Matrix computation	Small $2\times 2$ operations $\rightarrow$ larger-matrix determinants and other matrix operations.
	Prime	Primality testing	Small familiar integers $\rightarrow$ large integers whose primality is difficult to determine reliably without computation.
Knowledge Boundaries	Retriever	Information retrieval	Common factual knowledge $\rightarrow$ obscure or synthetic entities requiring retrieval from the provided corpus.

Appendix

Table 10: Environment composition of When2Tool . Each single-hop environment contains easy, medium, and hard instances. The last column summarizes how each environment moves from tasks that are often solvable internally to tasks for which tool assistance becomes increasingly useful or necessary. The three multi-hop environments contain three dependent operations.

Safety domain	Fine-grained categories
National and public security	Terrorism or extremism; hacking or digital crime.
Health and safety risks	Harmful health content; environmental damage; dangerous behavior.
Physical and psychological harm	Violence and physical harm; psychological harm and manipulation; animal abuse.
Privacy and consent violations	Privacy invasion and surveillance; identity misuse and impersonation.
Discrimination and harassment	Discriminatory depictions; insulting and harassing behavior.
Deceptive and unethical behavior	Deception in personal relationships; financial and academic fraud.

Appendix

Table 11: Safety taxonomy of BeaverTails-V. The dataset contains 20 fine-grained categories grouped into nine broader safety domains, providing diverse supervision for multimodal safety alignment.

Dataset	# Examples	Role	Content / Annotation
BeaverTails-V	$\sim$ 30K	Train	Multimodal preference pairs across 20 safety categories; includes separate safety and helpfulness preferences, binary safety labels, image-severity levels, and graded response scores.
BeaverTails-V	$\sim$ 590	Test	Held-out examples distributed across the same 20 fine-grained categories, generally 28–30 examples per category.
MM-SafetyBench	5,040	Test	Text–image safety benchmark covering 13 safety-critical scenarios and multiple image-based attack constructions.

Appendix

Table 12: Datasets used for multimodal safety alignment and evaluation. BeaverTails-V provides safety–helpfulness preference supervision, while MM-SafetyBench provides an independent multimodal safety evaluation.

Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.

See all 143 papers

Towards Better Training Signal: Advantage Clipped Policy Optimization
Sep 29, 2026Ruichuan Huang, Jinghan Liu, Congliang ChenPolicy Optimization RL for Language Model Reasoning
Shenzhen Loop Area Institute
GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization
Oct 26, 2026Zixuan Xu, Haoxuan Chen, Jiangxin Sun +1Multi-Objective Language Model Alignment Multi-Objective Reinforcement Learning
ISEE Lab, Sun Yat-sen University · Computer Science and Technology, University of Trento
Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
Aug 17, 2026Yixuan Wang, Yifei Chen, Haichao Zhang +6Multi-Objective Reinforcement Learning RL for Language Model Reasoning
University of Florida · UC San Diego · Northeastern University +3

Sep 29, 2026J/K move · Enter open · S save

Towards Better Training Signal: Advantage Clipped Policy Optimization

Ruichuan Huang, Jinghan Liu, Congliang Chen

Shenzhen Loop Area Institute

Policy Optimization RL for Language Model Reasoning RL Post-Training Off-Policy RL

Open paper AskPDF arXiv

Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.

Abstract

Multi-Objective Reinforcement Learning

Explore similar work

Towards Better Training Signal: Advantage Clipped Policy Optimization

GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization

Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization