Multi-reward reinforcement learning offers a promising framework for aligning large language models (LLMs) with diverse objectives, but conflicting reward signals can induce ambiguous update directions and hinder stable optimization. Existing approaches address such conflicts through reward reweighting, gradient coordination, or rollout filtering, yet they primarily operate on per-response or group-level optimization signals and underexploit relational information among sampled responses. As a result, conflict-filtered rollouts may still contain useful supervision that is discarded during training. We propose Group Pareto-Dominance Policy Optimization (GPDPO), a multi-reward policy optimization method that recovers pairwise supervision from conflict-filtered rollouts. GPDPO identifies Pareto-dominance relations among sampled responses and converts them into auxiliary preference signals, which are jointly optimized with the original policy objective. This enables more effective reuse of existing rollouts without additional sampling or reward evaluation. Experiments on API-Bank across multiple model scales and reward configurations show that GPDPO consistently outperforms competitive multi-reward optimization baselines. Further analysis demonstrates that GPDPO reduces both group-relative and pairwise reward conflicts, supporting the effectiveness of exploiting relational supervision for multi-reward alignment. Code will be released.
Figures & tables
Figure 1: Different strategies for utilizing conflicting responses in multi-reward policy optimization. GDPO retains conflicting responses in policy updates, whereas GD 2 PO masks them to avoid ambiguous optimization signals. GPDPO preserves conflict-aware masking while recovering pairwise supervision from Pareto-dominance relations, i.e., one response is no worse than another on all reward dimensions and strictly better on at least one.
Figure 2: Conflict filtering is sensitive to the reference group and can discard useful learning signals. (a,b) Conflict-label flip rates for fixed responses under different group sizes and different peers at the same group size, respectively. (c) Percentage of filtered responses that match the ground-truth. (d) Consistent comparison relations between filtered responses and their peers. “Both” denotes being consistently better than some peers and consistently worse than others.
Figure 3: Overview of GPDPO. Based on conflict filtering, we identify Pareto dominance relations within filtered responses and leverage them to construct preference loss. The resulting preference loss is combined with the standard multi-reward policy-gradient objective for policy optimization.
Method
Level 1 Acc.
Level 2 Acc.
Level 3 Acc.
Correct Acc.
Length Rew.
Overall
Qwen3-1.7B
GRPO ( Shao et al., 2024 )
66.17
53.73
46.56
60.47
0.9643
1.569
GDPO ( Liu et al., 2026b )
66.67
61.19
43.51
60.97
0.9875
1.597
GD 2 PO ( Liu et al., 2026a )
69.17
61.19
42.75
62.48
0.9793
1.604
GPDPO
70.93
61.19
48.85
64.99
0.9812
1.631
Qwen3-4B
Table 1: Results on API-Bank under the two-reward (2R) setting. Accuracy metrics are reported as percentages. The best and second-best results are highlighted in bold and underlined , respectively.
Method
Level 1 Acc.
Level 2 Acc.
Level 3 Acc.
Correct Acc.
Format Acc.
Length Rew.
Overall
Qwen3-1.7B
GRPO ( Shao et al., 2024 )
65.91
62.69
39.69
59.80
98.99
0.9519
2.540
GDPO ( Liu et al., 2026b )
68.92
53.73
41.22
61.14
100.00
0.9608
2.572
GD 2 PO ( Liu et al., 2026a )
68.92
59.70
44.27
62.48
100.00
0.9756
2.600
GPDPO
70.43
61.19
42.75
63.32
99.66
0.9760
2.606
Qwen3-4B
Table 2: Results on API-Bank under the three-reward (3R) setting. Accuracy metrics are reported as percentages. The best and second-best results are highlighted in bold and underlined , respectively.
Figure 4: Training dynamics of Qwen3-1.7B under the three-reward (3R) setting.
Figure 5: Conflict dynamics on Qwen3-1.7B under 3R setting. Left: pairwise preference conflict rate over all non-tied within-group response pairs, including retained–retained pairs. Right: GD 2 PO conflict rate over valid rollouts.
Method
Level 1 Acc.
Level 2 Acc.
Level 3 Acc.
Correct Acc.
Length Rew.
GD 2 PO
69.17
61.19
42.75
62.48
0.9793
GPDPO (Strict Pareto Dominance)
68.67
62.69
48.09
63.48
0.9650
GPDPO (Scalarized Reward Ordering)
68.42
49.25
48.09
61.81
0.9675
GPDPO (Fixed Reference Model)
71.18
58.21
39.69
62.81
0.9767
GPDPO (Pareto-Guided Advantage Adjustment)
70.68
61.19
44.27
63.82
0.9762
GPDPO (Full)
70.93
61.19
48.85
64.99
0.9812
Table 3: Ablations on Qwen3-1.7B with 2R setting. Best Correct Acc. and Length Rew. are bold.
Figure 6: Sensitivity of λ on Qwen3-1.7B under 2R setting.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Training hardware
One node, 4× NVIDIA RTX 5880 Ada (48 GB each)
Base models
Qwen3-1.7B, Qwen3-4B
Training data
RLLA-4K, 3,920 records before prompt filtering
Max prompt / response length
2,048 / 1,024 tokens
Train batch / mini-batch size
512 / 128 prompts
Rollout group size G
4 (2,048 responses per rollout step)
Appendix
Table 4: Training settings for GPDPO. Batch sizes are measured in prompts.
Setting
Value
Evaluation hardware
One NVIDIA RTX A6000 (48 GB)
Dataset
Processed API-Bank, 597 questions
Level 1 / 2 / 3 questions
399 / 67 / 131
Decoding
Greedy, one response per question
Temperature / top- p
0.0 / 1.0
Max new tokens / total sequence length
4,096 / 4,096
Appendix
Table 5: Evaluation settings for the selected step-200 checkpoints.
Panel
Measurement
2R: count (%)
3R: count (%)
(a)
Expansion, 4→64 : changed
10,608 (16.19)
17,756 (27.09)
Expansion, 4→64 : unchanged
54,928 (83.81)
47,780 (72.91)
(b)
Peer replacement, G=4 : disagreement
5,999 (9.15)
10,434 (15.92)
Peer replacement, G=16 : disagreement
6,312 (9.63)
8,119 (12.39)
Peer replacement, G=64 : disagreement
3,561 (5.43)
3,994 (6.09)
(c)
GT tool-call structure matches
8,345 (23.22)
16,039 (27.33)
Appendix
Table 6: Detailed results for Figure 2 . Entries report count (percentage). The denominator is 65,536 paired label comparisons per setting for (a,b), and 35,937 / 58,689 filtered responses under 2R / 3R for (c,d). The four categories in (d) partition the filtered set.
Measurement
GD 2 PO
GPDPO ( λ=0.1 )
All-dimension strict-improvement edges
6,374
6,338
Pareto-dominance edges
52,421
52,631
Edges added by allowing ties
46,047 (87.84%)
46,293 (87.96%)
Tied correctness, improved length
29,229
29,578
Tied length, improved correctness
16,818
16,715
Appendix
Table 7: Comparison-edge statistics under 2R. All edge counts involve at least one filtered response. Percentages for added edges use the Pareto-edge count as the denominator.
Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.