Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

Fei Ding; Yongkang Zhang; youwei wang; Zijian Zeng

cs.LGApr 4, 2026

Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

Authors: Fei Ding, Yongkang Zhang, youwei wang, Zijian Zeng

Organizations: Alibaba Group · Tsinghua University

Abstract

Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment problem: the final feedback is propagated uniformly across all intermediate decisions. This leads to high gradient variance, unstable training, and many ineffective updates, ultimately limiting sustained model improvement. We propose a counterfactual-comparison framework for credit assignment. For each input, the framework samples multiple reasoning trajectories and treats their differences as implicit approximations to alternative decisions. This yields an implicit process-level advantage estimator that converts sparse terminal rewards into step-sensitive learning signals. Building on this framework, we introduce Implicit Behavior Policy Optimization (IBPO), which substantially improves training stability and the performance ceiling on mathematical and code-reasoning benchmarks. Our results point to a promising direction for unlocking the reasoning potential of LLMs.

Figures & tables

Figure 1: Overview of the DFPO pipeline. After sampling multiple responses for the same input, the method computes group-relative advantages and sequence-coupled importance weights, forming the effective token modulation $W$ . It then applies a within-group transformation, Min-Replace or Orth-Proj, to remove within-group asymmetry in $W$ , so that shared-token gradients cancel under symmetric aggregation ( $\sum=0$ ). This reduces the learning tax and stabilizes parameter updates. The upper-right comparison illustrates how asymmetric aggregation without the transformation leads to gradient non-cancellation ( $\sum\neq 0$ ) and probability drift.

Panel of Figure 2 — Figure 2: Comparison of cancellation behavior on shared tokens for token-factorized and sequence-coupled objectives. Blue tokens are shared tokens, orange tokens are divergent tokens, and the value below each token is the corresponding effective token modulation $W_{i}^{\star}\equiv W_{i,t^{\star}}$ . (a) GRPO : the modulation $r$ is determined only by the token at that position and is independent of the trajectory index; with zero-mean advantages $\sum_{i}\widehat{A}_{i}=0$ , the group-aggregated gradient of the shared token cancels exactly. (b) GSPO : the effective modulation $W_{i}^{\star}$ is induced by the full sequence weight; differences in the terminal divergent token make $W_{1}^{\star}\neq W_{2}^{\star}$ , so the within-group aggregation is no longer zero even at the shared token, accumulating a learning tax on reward-weak shared or template tokens.

Method	Qwen3-32B Acc avg@32 (%)			Qwen3-Next Acc avg@32 (%)			GPT-OSS-120B Acc avg@32 (%)
Method	AIME25	LiveCodeBench	HMMT25	AIME25	LiveCodeBench	HMMT25	AIME25	LiveCodeBench	HMMT25
DFPO (Min-Replace)	82.5 $\pm$ 1.1	71.6 $\pm$ 0.7	61.4 $\pm$ 1.5	93.2 $\pm$ 0.9	75.1 $\pm$ 0.8	80.1 $\pm$ 1.1	98.3 $\pm$ 0.5	84.7 $\pm$ 0.6	91.9 $\pm$ 0.9
DFPO (Orth-Proj)	82.6	71.6	61.3	93.1	75.2	80.2	98.2	84.8	91.7
GSPO	76.9 $\pm$ 1.3	64.7 $\pm$ 1.5	55.8 $\pm$ 0.9	89.8 $\pm$ 1.7	71.0 $\pm$ 1.4	75.8 $\pm$ 1.2	96.5 $\pm$ 0.5	80.3 $\pm$ 1.2	87.6 $\pm$ 1.0
GRPO	76.9 $\pm$ 1.2	64.5 $\pm$ 0.8	55.5 $\pm$ 1.5	89.7 $\pm$ 1.2	70.9 $\pm$ 0.9	75.5 $\pm$ 1.8	96.3 $\pm$ 0.7	80.1 $\pm$ 0.8	87.4 $\pm$ 1.5
GRPO-fix	80.6 $\pm$ 1.4	69.1 $\pm$ 1.2	59.6 $\pm$ 0.9	91.9 $\pm$ 1.5	73.9 $\pm$ 1.1	79.4 $\pm$ 0.7	98.0 $\pm$ 0.6	83.6 $\pm$ 0.9	90.8 $\pm$ 0.6

Table 1: Results on Qwen3-32B, Qwen3-Next-80B-A3B-Thinking, and GPT-OSS-120B. We report the mean over five random seeds and the corresponding 95% bootstrap confidence interval (mean

\pm

95% CI). Under a paired bootstrap test, the improvements over the baselines are statistically significant (

p<0.01

). Inference settings are provided in Appendix Q .

Figure 3: Training curves on Qwen3-Next-80B-A3B-Thinking. Under the compute-matched setting, DFPO achieves substantially higher training efficiency than GSPO.

Model variant	AIME25 (%)	LiveCodeBench (%)	HMMT25 (%)
DFPO(Min-Replace)	82.5	71.6	61.4
GSPO (baseline)	76.9	64.7	55.8
DFPO (no stop-grad)	75.6	63.8	54.9
DFPO (scale by 0.5)	75.1	63.2	54.1

Table 2: Ablation results on Qwen3-32B.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: DFPO method diagram. Multiple responses to the same input are used to compute group-relative advantages and sequence-coupled weights, forming the effective token modulation $W_{i,t}$ . DFPO then applies a stop-gradient within-group transformation to obtain $\widetilde{W}_{i,t}$ for the decoupled gradient estimator. Min-Replace uses a shared within-group scale at the selected position, whereas Orth-Proj decorrelates the modulation vector $\mathbf{W}_{t^{\star}}$ from the advantage vector $\widehat{\mathbf{A}}$ . The correction preserves token-gradient directions while restoring, or approximating, shared-token cancellation and reducing the learning tax.

G	2	4	8	16
avg@32	3.1	4.2	5.6	5.7

Appendix

Table 3: Group-size ablation. Values are DFPO Min-Replace scores minus GSPO scores on Qwen3-32B and AIME25.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,087 papers

Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
Apr 20, 2026Fei Ding, Yongkang Zhang, Youwei Wang +1RL for Language Model Reasoning Credit Assignment in RL
Alibaba Group · Tsinghua University
OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
May 21, 2026Yu Li, Rui Miao, Tian Lan +1Value Function Estimation Token-Level Credit Assignment
1George Washington University · 2The University of Texas at Dallas
Semifactual Credit-Augmented Policy Optimization
Sep 30, 2026Junshu Pan, Zhizhang Fu, Shulin Huang +5Prompt Sensitivity RL for Language Model Reasoning
Zhejiang University · Westlake University · Shanghai Innovation Institute +1

Apr 20, 2026J/K move · Enter open · S save

Reducing Credit Assignment Variance via Counterfactual Reasoning Paths

Fei Ding, Yongkang Zhang, Youwei Wang, Zijian Zeng

Alibaba Group · Tsinghua University

RL for Language Model Reasoning Credit Assignment in RL Step-Level Credit Assignment in RL Step-Level Credit Assignment Credit Assignment Sparse-Reward RL+3

Open paper AskPDF arXiv

Abstract

RL for Language Model Reasoning

Explore similar work

Reducing Credit Assignment Variance via Counterfactual Reasoning Paths

OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning

Semifactual Credit-Augmented Policy Optimization