Organizations: Fudan University · M-A-P · Peking University · Tsinghua University · Singapore University of Technology and Design · Shanghai Innovation Institute
Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving reasoning and code generation in large language models, and GRPO-style training is widely adopted for its simplicity and effectiveness. However, an important design choice remains underexplored: how token-level policy gradient terms are aggregated within each sampled group. Standard GRPO uses sequence aggregation, while recent work has advocated token aggregation as a better alternative. We show that these two rules induce different optimization biases: token aggregation introduces sign-length coupling, while sequence aggregation implicitly downweights longer responses through sequence-level equal weighting. To address this tension, we propose Balanced Aggregation (BA), a simple drop-in replacement that computes token-level means separately within the positive and negative subsets and then combines them with sequence-count-based weights. Experiments with Qwen2.5-Math-7B and Qwen3-1.7B on DAPO-17k and Polaris, evaluated on six reasoning and coding benchmarks, show that BA consistently improves training stability and final performance over standard token and sequence aggregation. Our analysis further shows that the relative effectiveness of token and sequence aggregation is largely governed by response-length variation and the positive-negative length gap, highlighting aggregation as a critical design dimension in GRPO-style RLVR.
Figures & tables
Peak Performance
Last-Step
Dataset
Model / Aggregation
Acc@8
Best@8
Acc@8
Best@8
DAPO-17k
Qwen2.5-Math-7B
seq-agg
0.3578
0.4821
0.3446
0.4598
token-agg
0.3595
0.4869
0.3364
0.4434
balanced-agg
0.3634
0.4896
0.3424
0.4627
Qwen3-1.7B
Table 1: Main experimental results on DAPO-17k and Polaris datasets using Qwen2.5-Math-7B and Qwen3-1.7B models. We report the average peak and last-step accuracy across six evaluation benchmarks.
Figure 1: Comparison of peak and last-step accuracy across evaluation benchmarks. For each benchmark, values are averaged over four training settings: Qwen2.5-Math-7B and Qwen3-1.7B, each trained on DAPO-17k and Polaris. Solid bars indicate peak performance, while dashed bars indicate last-step performance.
Figure 2: Training dynamics of the policy-gradient loss for Qwen2.5-Math-7B on DAPO-17k and Polaris. Triggered by the phenomenon that negative responses are systematically longer than positive responses, token-agg places disproportionate weight on negative samples, causing its loss to severely deviate from zero. In contrast, seq-agg and balanced-agg actively decouple this sign-length bias and remain much more stable.
Figure 3: Length-distribution statistics related to the relative behavior of token and sequence aggregation. We report the response-length coefficient of variation and the positive–negative length gap over training on DAPO-17k and Polaris.