Authors: Dawei Zhang, Yubin Huangfu, Jessica Fan, Boyu Wei, Likang Wu, Hongke Zhao, Ming He, Zhu Teng
Organizations: University of Chinese Academy of Sciences, China · Institute of Software, Chinese Academy of Sciences, China · AI Lab at Lenovo Research, China · University of North Carolina at Chapel Hill, USA · Tianjin University, China · Beijing Jiaotong University, China
To keep long interaction contexts tractable, stepwise group-relative training for multi-turn LLM agents materializes each source trajectory into multiple response rows and valid tokens. This expansion creates an outer-accounting problem distinct from within-trajectory credit assignment: row-based outcome statistics can count one episode outcome multiple times, while token-level reduction can assign unequal aggregate objective mass across trajectories. We term these effects outcome-statistics multiplicity and objective-mass imbalance. We introduce Outcome-Mass Alignment (OMA), a two-component method that makes both accounting choices explicit. Outcome Alignment computes outcome statistics over unique source trajectories, while Mass Alignment controls aggregate objective mass after within-trajectory aggregation of optimizer-native token objectives. Neither redesigns the base optimizer's credit assignment, importance ratios, or clipping. Across matched ALFWorld GRPO comparisons at two model scales, where failed trajectories typically expand much more than successful ones, OMA improves matched-horizon endpoint success by 7.0-9.4 percentage points. Ordinary WebShop exhibits weak or reversed expansion-outcome coupling and correspondingly mixed effects, whereas in a success-oriented WebShop setting, WebShop-SOL, OMA improves success nAUC from 0.399 to 0.511. Component ablations and GiGPO and GSPO comparisons further support component separability and cross-optimizer compatibility. These results show that trajectory expansion always defines an implicit weighting rule, but its practical effect depends on how expansion couples to outcomes and optimizer-native updates.
Figures & tables
Figure 1: Expansion-induced trajectory reweighting under stepwise materialization. (a) Same-task, same-checkpoint ALFWorld rollouts: a 14-turn success and a 50-turn failure, each with one episode outcome. (b) One-response-row-per-turn materialization expands τi into ni rows containing Di valid response tokens. (c) Row-based statistics weight trajectory outcomes by ni , whereas token-mean reduction assigns objective mass by Di .
Figure 2: OMA controls outer accounting at the source-trajectory level. (a) Rows are grouped by source. (b) OA computes the designated outcome component once per unique trajectory and broadcasts it to the corresponding rows. (c) MA averages native token objectives within trajectories and then across trajectories, assigning each source outer mass 1/K . Native credit, ratios, detach, and clipping remain unchanged.
Method
Pk
Lk
Cl
Ht
Co
P2
All
nAUC
Qwen2.5-1.5B-Instruct
GRPO
86.7
57.1
81.2
58.3
65.0
70.0
73.4
43.5
GRPO + OMA
87.5
75.0
100.0
83.3
80.0
69.6
82.8
46.5
Qwen2.5-3B-Instruct
GRPO
94.6
77.8
72.0
46.2
57.9
72.0
74.2
55.5
GRPO + OMA
96.3
71.4
93.3
57.1
73.7
76.5
81.2
58.2
Table 1: ALFWorld main results (%). Pk/Lk/Cl/Ht/Co/P2 denote Pick, Look, Clean, Heat, Cool, and Pick2; All is endpoint success.
Figure 3: Matched-horizon GRPO success curves on ALFWorld, WebShop, and WebShop-SOL. Dotted lines mark stage boundaries at updates 50 and 110; Table 2 reports score metrics.
Env.
rtok
OA
MA
OMA
cos(OA,MA)
WS
0.99
0.94
1.02
0.96
0.996
ALF
1.14
1.19
1.26
0.94
0.261
ALF
2.17
1.93
2.72
1.30
-0.421
ALF
3.59
2.41
3.46
1.31
-0.506
ALF
8.78
2.44
5.05
1.60
-0.368
Table 4: Frozen-batch full-model gradient geometry. rtok is the ratio of mean failed-trajectory to successful-trajectory response tokens. OA, MA, and OMA report gradient-norm ratios to Raw; WS and ALF denote WebShop and ALFWorld.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Seed 1
Seed 2
Mean ± SD
Method
Endpoint
nAUC
Endpoint
nAUC
Endpoint
nAUC
Base
71.09
54.04
69.53
47.08
70.31±1.10
50.56±4.92
OA
64.06
44.86
69.53
44.64
66.80±3.87
44.75±0.16
MA
78.91
57.45
84.38
59.08
81.64±3.87
58.26±1.15
OMA
87.50
60.79
85.16
58.41
86.33±1.66
59.60±1.68
Appendix
Table 5: Additional ALFWorld-3B GRPO success results (%). Endpoint is validation success at update 150; nAUC integrates updates 0–150. SD is the sample standard deviation across the two runs. Seed 1 and Seed 2 are presentation labels.
Score
Success
Method
Run / summary
Endpoint
nAUC
Endpoint
nAUC
Base
Seed 1
54.35
41.10
47.66
28.41
Base
Seed 2
58.11
49.63
46.88
32.17
Base
Mean ± SD
56.23±2.66
45.37±6.03
47.27±0.55
30.29±2.66
OMA
Seed 1
66.39
42.22
56.25
24.49
OMA
Seed 2
76.84
45.57
60.16
33.80
Appendix
Table 6: Additional WebShop-1.5B GRPO results (%). Score is the standard task score; Success is exact success. Endpoints and nAUC use the same update-150 horizon as Table 5 . SD is the sample standard deviation across the two runs.
Figure 4: Additional ALFWorld-3B GRPO learning curves. Each panel shows one independent training seed for Base, OA, MA, and OMA. Lines connect every observed validation checkpoint without smoothing; marker shapes and line styles distinguish methods.
Figure 5: Additional WebShop-1.5B GRPO learning curves. Columns identify the two independent training seeds; rows report exact success and standard task score. Lines connect all observed checkpoints without smoothing. Endpoint gains need not imply higher nAUC.
ratio clip 0.2 ; low-variance KL loss coefficient 0.01 ; entropy coefficient 0.001
Reward and advantage controls
no KL term in reward; invalid-action penalty enabled with coefficient 0.1 ; standard GRPO normalization by group standard deviation
Training sample unit
16 task groups per update; 8 trajectories per group
Generation
temperature 1.0 for training and 0.4 for validation; top- p=1.0 ; top- k disabled
Model execution
rollout dtype bfloat16; LoRA disabled
Appendix
Final scientifically relevant hyperparameters: shared settings followed by environment and model deltas.
Setting
Variant
Early
Middle
Late
Endpoint
ALFWorld GRPO (3B)
Base
32.7
62.0
74.1
74.2
OA
33.6
44.0
60.2
61.7
MA
38.8
68.2
74.4
77.3
OMA
34.6
66.3
75.4
81.2
ALFWorld GRPO (1.5B)
Base
15.6
46.8
73.4
73.4
OA
11.0
22.9
35.4
38.3
Appendix
Complete success-rate stage results (%). Early, middle, and late are normalized AUC over updates [0,50] , [50,110] , and [110,150] ; Endpoint is success at update 150. Bold marks the best value within each setting.
Figure 6: Five-step centered moving averages of actor entropy, KL loss, mean episode length, and signed PG loss for the completed ALFWorld-3B and WebShop-SOL ablations. The dotted PG reference marks zero; diagnostics retain method-native weighting and are interpreted with task performance.
Gradient-norm ratio to Raw
Setting
Policy
rtok
OA
MA
OMA
cos(OA,MA)
cos(Raw,OMA)
WebShop
GRPO
0.99
0.94
1.02
0.96
0.996
0.996
ALFWorld
GRPO
1.14
1.19
1.26
0.94
0.261
0.998
ALFWorld
GRPO
2.17
1.93
2.72
1.30
-0.421
0.947
ALFWorld
GRPO
3.59
2.41
3.46
1.31
-0.506
0.985
ALFWorld
GRPO
8.78
2.44
5.05
1.60
-0.368
0.982
Appendix
Table 7: Frozen-batch full-model gradient geometry. rtok is the ratio of mean failed-trajectory response tokens to mean successful-trajectory response tokens. The final paired block compares the same ALFWorld task specification under GRPO- and GiGPO-trained policies.
Base
Aligned outcome component
Preserved native components
GRPO
AiOA
Token ratio and PPO clipping
GiGPO
AiOA in episode term
Anchor grouping, step advantage, and token clipping