Reinforcement learning for language models often combines several rewards by normalizing each one and adding them with fixed weights. When the rewards are correlated, the variation they share is counted once for every reward that carries it. In terms of the ZCA-decorrelated components of the reward signal, the weighted sum applies weights Σ1/2w rather than the chosen weights w, where Σ is the reward correlation matrix. We study WAS-PO, which whitens the normalized advantages with Σ−1/2 before adding them, so that the chosen weights apply to those components. We derive the scalarized objective whose gradient it follows when normalization statistics are held fixed, show that its weights on the raw rewards can be negative, give an exact two-reward condition under which the lighter-weighted reward is reversed, and give a loose bound on the error from estimating Σ during training. On tool calling with Qwen2.5-1.5B-Instruct, WAS-PO reached a higher mean training reward than GDPO in all seven seeds with three rewards (3.8% higher) and all three seeds with four (3.9% higher). WAS-PO also replaces GDPO's per-prompt normalization with batch-level normalization, and we did not separate the two changes with three or four rewards. The training-reward gains did not carry over to held-out accuracy, which was slightly lower than GDPO's with three rewards (not significant) and 0.03 lower in absolute terms with four (two seeds evaluated). With two rewards, WAS-PO trailed GDPO in training reward, and a three-seed control weakly suggests that the gap comes from batch-level normalization rather than whitening. In a math setting where the two rewards were nearly identical by construction, WAS-PO drove down the lighter-weighted reward, as the sign condition predicts.
Figures & tables
Setting
w
φw
v
ψ
Δ AUC
Tool, K=2 , time-averaged
(1.0, 0.5)
9.6∘
(1.00, 0.17)
0.00
−0.014
Tool, K=2 , largest
(1.0, 0.5)
(1.24, −0.14 )
0.11
Tool, K=3 , time-averaged
(1.0, 0.5, 0.3)
16.0∘
(1.05, 0.30, −0.15 )
0.15
+0.041
Tool, K=3 , largest
(1.0, 0.5, 0.3)
(1.22, 9.84, −9.95 )
1.01
Tool, K=4 , time-averaged
(1.0, 0.5, 0.3, 0.2)
20.7∘
(1.10, 0.39, −0.06 , −0.26 )
0.23
+0.049
Tool, K=4 , largest
(1.0, 0.5, 0.3, 0.2)
(1.32, 16.5, −3.25 , −13.5 )
0.82
Table 1: Diagnostics at the estimated correlations. φw is the angle between Σ1/2w and w (Proposition 1 ); v=Ww with the eigenvalue floor of Section 4 ; ψ is the inversion index ( 5 ). “Time-averaged” rows use Σˉ (Table 13 ). “Largest” rows set every pair to its largest logged value; the maxima need not occur at the same step. At K=3,4 these matrices are singular to numerical precision, so there the magnitudes of v are set by the floor and only its signs and ψ are meaningful. Δ AUC is WAS-PO minus GDPO.
Figure 1: Training reward of GRPO, GDPO and WAS-PO . Curves and ±1 SD bands are computed over a subset of the seeds (three at K=2 ). The per-seed AUCs in Tables 7 and 5 cover all seven seeds.
K
Seeds
Rewards
GDPO
WAS-PO
Δ
Relative
cond(Σˉ)
2
7
acc, fmt
0.985
0.970
−0.014
−1.5%
4.6
3
7
acc, fmt, len
1.089
1.130
+0.041
+3.8%
12.2
4
3
acc, fmt, len, struct
1.257
1.306
+0.049
+3.9%
16.2
Table 2: Training AUC by number of rewards. Δ is the mean paired difference, WAS-PO minus GDPO, and Relative is Δ divided by GDPO’s mean, both computed from unrounded per-seed values. cond(Σˉ) is the condition number of the time-averaged correlation estimate. The settings differ in which rewards they contain as well as in how many, and AUC is not comparable across rows. acc, fmt, len and struct are the accuracy, format, length and structure rewards.
Variant ( G=4 unless stated)
Seeds
GDPO, same G
AUC
Δ
WAS-PO
7
0.985
0.970
−0.014
WAS-PO -LW
7
0.985
0.968
−0.017
WAS-PO-Diag
3
0.986
0.965
−0.021
WAS-PO , G=8 ( N=64 )
3
1.003
0.984
−0.019
WAS-PO , G=16 ( N=128 )
3
1.011
0.993
−0.018
WAS-PO , β=0.5
1
0.990
0.967
−0.023
Table 3: Ablations with two rewards. Training AUC. Δ is relative to GDPO trained with the same group size G on the same seeds (seeds 42, 123 and 456 for the three-seed rows; seed 42 for the decay sweep) and is computed from unrounded values. WAS-PO-Diag keeps batch-level normalization and removes whitening. The decay sweep is a separate set of single-seed runs with its own GDPO baseline (0.990). Its β=0.99 run repeats the default configuration and scored 0.966, against 0.972 for seed 42 in Table 7 , so reruns at a fixed seed vary by about 0.006.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
One batch
EMA, stationary
EMA, t=100
K
λmin
E
K2∥w∥2/λmin3
N=32
N=128
N=32
N=128
N=32
N=128
2
0.355
0.112
111.8
3.61
0.880
0.018
0.004
0.508
0.124
3
0.196
0.251
1601.7
51.7
12.6
0.260
0.063
7.29
1.78
4
0.196
0.360
2932.5
94.6
23.1
0.475
0.116
13.3
3.26
Appendix
Table 4: Bound ( 6 ) against the size of the correction E .
Figure 2: Training AUC for two, three and four rewards (mean ± SD over seeds). Each panel has its own axis because AUC is not comparable across K .
Seed
GRPO
GDPO
WAS-PO
Δ
42
1.0303
1.0958
1.1252
+0.0294
123
0.9995
1.0862
1.1386
+0.0524
456
1.0028
1.0903
1.1213
+0.0310
789
1.0188
1.0947
1.1391
+0.0444
1011
1.0059
1.0798
1.1303
+0.0505
2024
1.0110
1.0863
1.1246
+0.0383
Appendix
Table 5: Training AUC with three rewards, seven seeds. Exact two-sided Wilcoxon p=0.016 ; dz=4.6 .
Seed
GRPO
GDPO
WAS-PO
Δ
42
1.1049
1.2633
1.3036
+0.0403
123
1.0995
1.2488
1.2987
+0.0499
456
1.1439
1.2579
1.3154
+0.0575
Mean ± SD
1.116±0.024
1.257±0.007
1.306±0.009
+0.049
Appendix
Table 6: Training AUC with four rewards, three seeds. dz=5.7 ; the smallest attainable two-sided Wilcoxon p with three seeds is 0.25.
Seed
GRPO
GDPO
WAS-PO
Δ
42
0.9903
0.9870
0.9721
−0.0149
123
0.9759
0.9826
0.9722
−0.0104
456
0.9918
0.9873
0.9713
−0.0160
789
0.9898
0.9833
0.9727
−0.0106
1011
0.9825
0.9870
0.9691
−0.0179
2024
0.9825
0.9814
0.9683
−0.0131
Appendix
Table 7: Training AUC with two rewards, seven seeds. Exact two-sided Wilcoxon p=0.016 ; dz=−4.5 .
G
Seed
GRPO
GDPO
WAS-PO
Δ
8
42
1.0021
1.0061
0.9849
−0.0212
8
123
0.9964
0.9976
0.9837
−0.0139
8
456
1.0020
1.0050
0.9830
−0.0220
16
42
1.0125
1.0111
0.9938
−0.0173
16
123
1.0084
1.0064
0.9905
−0.0159
16
456
1.0169
1.0153
0.9960
−0.0193
Appendix
Table 8: Training AUC at larger group sizes, two rewards, three seeds. Eight prompts per step, so N=64 at G=8 and N=128 at G=16 .
Setting
GRPO
GDPO
WAS-PO
Δ vs. GDPO
p
K=2 (seven seeds)
0.479±0.006
0.474±0.008
0.457±0.022
−0.017
0.09
K=3 (seven seeds)
0.397±0.076
0.446±0.012
0.438±0.011
−0.008
0.22
K=4 (two seeds)
0.521, 0.517
0.517, 0.515
0.485, 0.487
−0.030
—
Appendix
Table 9: Held-out accuracy (accuracy reward of the final checkpoint under greedy decoding on 500 held-out prompts; mean ± SD over seeds). p is from an unpaired two-sided Welch t -test computed from these summaries. At K=4 , scores were recorded for two of three seeds.
Seed
GRPO
GDPO
WAS-PO
Δ
42
0.8077
0.8041
0.8085
+0.0044
123
0.7988
0.8120
0.7943
−0.0177
456
0.8075
0.7994
0.7995
+0.0001
Mean
0.805
0.805
0.801
−0.004
Appendix
Table 10: Training AUC in the earlier math run with a constant format reward, where WAS-PO reduces to WAS-PO-Diag (Section 5.3 ). This is not the run of Section 5.4 .
Step
WAS-PO acc
WAS-PO fmt
GDPO acc
GDPO fmt
10
0.675
0.666
0.691
0.675
50
0.697
0.277
0.775
0.775
500
0.806
0.132
0.863
0.863
Appendix
Table 11: Per-reward batch-mean training reward in the math setting (three seeds).
Method
Accuracy
Format
Length
Structure
GDPO
0.521
0.833
0.528
0.831
WAS-PO
0.535
0.871
0.539
0.863
Δ
+0.014
+0.038
+0.011
+0.032
Appendix
Table 12: Per-reward training reward with four rewards, averaged over 500 steps (three seeds). Per-reward values were not logged for GRPO. The weighted sums of these means (1.262 and 1.305) agree with the AUCs in Table 6 to within 0.006.
K
Pair
Mean ρ
Range
cond(Σˉ)
E
2
acc–fmt
0.645
[0.530, 0.869]
4.6
0.112
eigenvalues
[0.355,1.645]
3
acc–fmt
0.643
[0.545, 0.828]
12.2
0.251
acc–len
0.642
[0.544, 0.827]
fmt–len
0.804
[0.698, 1.000]
eigenvalues
[0.196,0.408,2.396]
Appendix
Table 13: The correlation estimate Σ^t maintained by WAS-PO (seed 42 for K=2,3 ; three seeds for K=4 ). Mean and range are over the 500 training steps; Σˉ is the matrix of means, and its eigenvalues, cond(Σˉ) and E are listed.
Figure 3: Correlation estimate and reward difference (seed 42, logged every 10 steps). Blue: the EMA estimate of ρ(acc,fmt) . Red: training reward of WAS-PO minus GDPO, labeled “advantage” in the legend but meaning a difference in reward. Top: tool calling with two rewards; the two series have correlation 0.44, but both trend over training, so we do not interpret this. Bottom: the earlier math run with a constant format reward, where the estimate is the identity and the difference fluctuates around zero.
Figure 4: Training reward with four rewards (three seeds, mean ± SD).
Tool, K=2
Tool, K=3
Tool, K=4
Math
Model
Qwen2.5-1.5B-Instruct
Data
BFCL v3, single-turn, irrelevance excluded; first 80%
GSM8K train
Group size G
4
4
4
4
Prompts per step
8
8
8
8
Rollouts per step N
32
32
32
32
Max new tokens
128
128
128
384
Appendix
Table 14: Training hyperparameters. Settings not listed are the defaults of TRL 0.18 and Transformers.