Reinforcement learning for language models often combines several rewards by normalizing each one and adding them with fixed weights. When the rewards are correlated, the variation they share is counted once for every reward that carries it. In terms of the ZCA-decorrelated components of the reward signal, the weighted sum applies weights Σ1/2w rather than the chosen weights w, where Σ is the reward correlation matrix. We study WAS-PO, which whitens the normalized advantages with Σ−1/2 before adding them, so that the chosen weights apply to those components. We derive the scalarized objective whose gradient it follows when normalization statistics are held fixed, show that its weights on the raw rewards can be negative, give an exact two-reward condition under which the lighter-weighted reward is reversed, and give a loose bound on the error from estimating Σ during training. On tool calling with Qwen2.5-1.5B-Instruct, WAS-PO reached a higher mean training reward than GDPO in all seven seeds with three rewards (3.8% higher) and all three seeds with four (3.9% higher). WAS-PO also replaces GDPO's per-prompt normalization with batch-level normalization, and we did not separate the two changes with three or four rewards. The training-reward gains did not carry over to held-out accuracy, which was slightly lower than GDPO's with three rewards (not significant) and 0.03 lower in absolute terms with four (two seeds evaluated). With two rewards, WAS-PO trailed GDPO in training reward, and a three-seed control weakly suggests that the gap comes from batch-level normalization rather than whitening. In a math setting where the two rewards were nearly identical by construction, WAS-PO drove down the lighter-weighted reward, as the sign condition predicts.
Figures & tables
Setting
w
φw
v
ψ
Δ AUC
Tool, K=2 , time-averaged
(1.0, 0.5)
9.6∘
(1.00, 0.17)
0.00
−0.014
Tool, K=2 , largest
(1.0, 0.5)
(1.24, −0.14 )
0.11
Tool, K=3 , time-averaged
(1.0, 0.5, 0.3)
16.0∘
(1.05, 0.30, −0.15 )
0.15
+0.041
Tool, K=3 , largest
(1.0, 0.5, 0.3)
(1.22, 9.84, −9.95 )
1.01
Tool, K=4 , time-averaged
(1.0, 0.5, 0.3, 0.2)
20.7∘
(1.10, 0.39, −0.06 , −0.26 )
0.23
+0.049
Tool, K=4 , largest
(1.0, 0.5, 0.3, 0.2)
(1.32, 16.5, −3.25 , −13.5 )
0.82
Table 1: Diagnostics at the estimated correlations. φw is the angle between Σ1/2w and w (Proposition 1 ); v=Ww with the eigenvalue floor of Section 4 ; ψ is the inversion index ( 5 ). “Time-averaged” rows use Σˉ (Table 13 ). “Largest” rows set every pair to its largest logged value; the maxima need not occur at the same step. At K=3,4 these matrices are singular to numerical precision, so there the magnitudes of v are set by the floor and only its signs and ψ are meaningful. Δ AUC is WAS-PO minus GDPO.
Figure 1: Training reward of GRPO, GDPO and WAS-PO . Curves and ±1 SD bands are computed over a subset of the seeds (three at K=2 ). The per-seed AUCs in Tables 7 and 5 cover all seven seeds.
K
Seeds
Rewards
GDPO
WAS-PO
Δ
Relative
cond(Σˉ)
2
7
acc, fmt
0.985
0.970
−0.014
−1.5%
4.6
3
7
acc, fmt, len
1.089
1.130
+0.041
+3.8%
12.2
4
3
acc, fmt, len, struct
1.257
1.306
+0.049
+3.9%
16.2
Table 2: Training AUC by number of rewards. Δ is the mean paired difference, WAS-PO minus GDPO, and Relative is Δ divided by GDPO’s mean, both computed from unrounded per-seed values. cond(Σˉ) is the condition number of the time-averaged correlation estimate. The settings differ in which rewards they contain as well as in how many, and AUC is not comparable across rows. acc, fmt, len and struct are the accuracy, format, length and structure rewards.
Variant ( G=4 unless stated)
Seeds
GDPO, same G
AUC
Δ
WAS-PO
7
0.985
0.970
−0.014
WAS-PO -LW
7
0.985
0.968
−0.017
WAS-PO-Diag
3
0.986
0.965
−0.021
WAS-PO , G=8 ( N=64 )
3
1.003
0.984
−0.019
WAS-PO , G=16 ( N=128 )
3
1.011
0.993
−0.018
WAS-PO , β=0.5
1
0.990
0.967
−0.023
Table 3: Ablations with two rewards. Training AUC. Δ is relative to GDPO trained with the same group size G on the same seeds (seeds 42, 123 and 456 for the three-seed rows; seed 42 for the decay sweep) and is computed from unrounded values. WAS-PO-Diag keeps batch-level normalization and removes whitening. The decay sweep is a separate set of single-seed runs with its own GDPO baseline (0.990). Its β=0.99 run repeats the default configuration and scored 0.966, against 0.972 for seed 42 in Table 7 , so reruns at a fixed seed vary by about 0.006.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
One batch
EMA, stationary
EMA, t=100
K
λmin
E
K2∥w∥2/λmin3
N=32
N=128
N=32
N=128
N=32
N=128
2
0.355
0.112
111.8
3.61
0.880
0.018
0.004
0.508
0.124
3
0.196
0.251
1601.7
51.7
12.6
0.260
0.063
7.29
1.78
4
0.196
0.360
2932.5
94.6
23.1
0.475
0.116
13.3
3.26
Appendix
Table 4: Bound ( 6 ) against the size of the correction E .
Figure 2: Training AUC for two, three and four rewards (mean ± SD over seeds). Each panel has its own axis because AUC is not comparable across K .
Seed
GRPO
GDPO
WAS-PO
Δ
42
1.0303
1.0958
1.1252
+0.0294
123
0.9995
1.0862
1.1386
+0.0524
456
1.0028
1.0903
1.1213
+0.0310
789
1.0188
1.0947
1.1391
+0.0444
1011
1.0059
1.0798
1.1303
+0.0505
2024
1.0110
1.0863
1.1246
+0.0383
Appendix
Table 5: Training AUC with three rewards, seven seeds. Exact two-sided Wilcoxon p=0.016 ; dz=4.6 .
Seed
GRPO
GDPO
WAS-PO
Δ
42
1.1049
1.2633
1.3036
+0.0403
123
1.0995
1.2488
1.2987
+0.0499
456
1.1439
1.2579
1.3154
+0.0575
Mean ± SD
1.116±0.024
1.257±0.007
1.306±0.009
+0.049
Appendix
Table 6: Training AUC with four rewards, three seeds. dz=5.7 ; the smallest attainable two-sided Wilcoxon p with three seeds is 0.25.
Seed
GRPO
GDPO
WAS-PO
Δ
42
0.9903
0.9870
0.9721
−0.0149
123
0.9759
0.9826
0.9722
−0.0104
456
0.9918
0.9873
0.9713
−0.0160
789
0.9898
0.9833
0.9727
−0.0106
1011
0.9825
0.9870
0.9691
−0.0179
2024
0.9825
0.9814
0.9683
−0.0131
Appendix
Table 7: Training AUC with two rewards, seven seeds. Exact two-sided Wilcoxon p=0.016 ; dz=−4.5 .
G
Seed
GRPO
GDPO
WAS-PO
Δ
8
42
1.0021
1.0061
0.9849
−0.0212
8
123
0.9964
0.9976
0.9837
−0.0139
8
456
1.0020
1.0050
0.9830
−0.0220
16
42
1.0125
1.0111
0.9938
−0.0173
16
123
1.0084
1.0064
0.9905
−0.0159
16
456
1.0169
1.0153
0.9960
−0.0193
Appendix
Table 8: Training AUC at larger group sizes, two rewards, three seeds. Eight prompts per step, so N=64 at G=8 and N=128 at G=16 .
Setting
GRPO
GDPO
WAS-PO
Δ vs. GDPO
p
K=2 (seven seeds)
0.479±0.006
0.474±0.008
0.457±0.022
−0.017
0.09
K=3 (seven seeds)
0.397±0.076
0.446±0.012
0.438±0.011
−0.008
0.22
K=4 (two seeds)
0.521, 0.517
0.517, 0.515
0.485, 0.487
−0.030
—
Appendix
Table 9: Held-out accuracy (accuracy reward of the final checkpoint under greedy decoding on 500 held-out prompts; mean ± SD over seeds). p is from an unpaired two-sided Welch t -test computed from these summaries. At K=4 , scores were recorded for two of three seeds.
Seed
GRPO
GDPO
WAS-PO
Δ
42
0.8077
0.8041
0.8085
+0.0044
123
0.7988
0.8120
0.7943
−0.0177
456
0.8075
0.7994
0.7995
+0.0001
Mean
0.805
0.805
0.801
−0.004
Appendix
Table 10: Training AUC in the earlier math run with a constant format reward, where WAS-PO reduces to WAS-PO-Diag (Section 5.3 ). This is not the run of Section 5.4 .
Step
WAS-PO acc
WAS-PO fmt
GDPO acc
GDPO fmt
10
0.675
0.666
0.691
0.675
50
0.697
0.277
0.775
0.775
500
0.806
0.132
0.863
0.863
Appendix
Table 11: Per-reward batch-mean training reward in the math setting (three seeds).
Method
Accuracy
Format
Length
Structure
GDPO
0.521
0.833
0.528
0.831
WAS-PO
0.535
0.871
0.539
0.863
Δ
+0.014
+0.038
+0.011
+0.032
Appendix
Table 12: Per-reward training reward with four rewards, averaged over 500 steps (three seeds). Per-reward values were not logged for GRPO. The weighted sums of these means (1.262 and 1.305) agree with the AUCs in Table 6 to within 0.006.
K
Pair
Mean ρ
Range
cond(Σˉ)
E
2
acc–fmt
0.645
[0.530, 0.869]
4.6
0.112
eigenvalues
[0.355,1.645]
3
acc–fmt
0.643
[0.545, 0.828]
12.2
0.251
acc–len
0.642
[0.544, 0.827]
fmt–len
0.804
[0.698, 1.000]
eigenvalues
[0.196,0.408,2.396]
Appendix
Table 13: The correlation estimate Σ^t maintained by WAS-PO (seed 42 for K=2,3 ; three seeds for K=4 ). Mean and range are over the 500 training steps; Σˉ is the matrix of means, and its eigenvalues, cond(Σˉ) and E are listed.
Figure 3: Correlation estimate and reward difference (seed 42, logged every 10 steps). Blue: the EMA estimate of ρ(acc,fmt) . Red: training reward of WAS-PO minus GDPO, labeled “advantage” in the legend but meaning a difference in reward. Top: tool calling with two rewards; the two series have correlation 0.44, but both trend over training, so we do not interpret this. Bottom: the earlier math run with a constant format reward, where the estimate is the identity and the difference fluctuates around zero.
Figure 4: Training reward with four rewards (three seeds, mean ± SD).
Tool, K=2
Tool, K=3
Tool, K=4
Math
Model
Qwen2.5-1.5B-Instruct
Data
BFCL v3, single-turn, irrelevance excluded; first 80%
GSM8K train
Group size G
4
4
4
4
Prompts per step
8
8
8
8
Rollouts per step N
32
32
32
32
Max new tokens
128
128
128
384
Appendix
Table 14: Training hyperparameters. Settings not listed are the defaults of TRL 0.18 and Transformers.
Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across three domains, including code generation, tool calling, and agent security. Our code is available at https://github.com/HKUST-KnowComp/CorrGRPO.
Multi-reward reinforcement learning trains large language models to satisfy multiple behavioral objectives simultaneously. Reward-wise normalization, as used in GDPO, preserves reward-specific relative information within rollout groups, but different objectives can still exhibit uneven learning progress. We study this behavior through advantage energy, the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, we show that this energy is proportional to active-group density: the fraction of rollout groups in which the reward provides nonzero relative advantages. This reveals a residual batch-level signal imbalance and provides a basis for calibrating reward contributions. Based on this relation, we propose Density-Aware Reward Aggregation (DARA). We derive an inverse-square-root density correction that gives greater weight to signals from less frequently active rewards. DARA computes its weights from each rollout batch, adapting to changes in reward activity throughout training without modifying the underlying policy optimization objective. Experiments on tool calling and mathematical reasoning show that DARA learns the targeted behaviors faster than GDPO, reaching high format compliance in up to 26% fewer training steps on tool calling and near-saturated length compliance in up to 65% fewer steps on mathematical reasoning, while remaining competitive in final performance. Our code is available at https://github.com/zhaihaotian/DARA.
Tong Zheng, Skylar Zhai, Zhan Cheng +7
University of Chinese Academy of Sciences · University of Minnesota Twin Cities · University of Wisconsin–Madison +5
Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. Consequently, an objective whose rewards are already near their upper bound can retain substantial influence when its rewards still vary within rollout groups. We introduce Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This changes the relative contribution of each objective according to its observed reward headroom. We further derive an exact condition under which saturation aware reweighting reverses the sign of a rollout's aggregate advantage. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to 5% on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by 3.8% on average and up to 9.2% on AMC23, and on coding benchmarks it improves pass rate by up to 2.3%, while in all settings maintaining the easier objectives near their already satisfied levels. Additional experiments characterize the accompanying reward tradeoffs and sensitivity to corrupted rewards.
Yixuan Wang, Yifei Chen, Haichao Zhang +6
University of Florida · UC San Diego · Northeastern University +3