We revisit advantage estimation in Group Relative Policy Optimization (GRPO) and observe that any token-level advantage admits a two-stage factorization: a sample-level baselining step that converts scalar verifier rewards into a comparable sequence-level signal, and a token-level credit assignment step that distributes that signal over the trajectory. Existing estimators typically address only one of these stages: affine baselines preserve the shape of each task's reward law, so heavy tails and binary cliffs propagate into the gradient, while token-level credit rules tie the size of each token's update to raw confidence scales and need not conserve the total credit issued by the verifier. We propose TACE (Two-stage Advantage and Credit Estimation), a two-stage estimator that addresses both stages. At stage~(S1), we replace linear standardization with a one-dimensional optimal-transport map to a standard normal, which gives a common population reward law across tasks and caps finite-sample outliers deterministically. At stage~(S2), we rank-calibrate a stop-gradient evidence contrast within each trajectory and renormalize the clipped token weights to have mean one, so credit is redistributed across tokens while its total mass and sign are conserved. We prove population-level inter-task moment equity, a deterministic tail bound, and mass-conservation, boundedness, and sign-anchoring properties that affine baselines and unnormalized token weights do not satisfy. On a multimodal reasoning suite, TACE obtains the strongest results on a majority of the benchmarks but does not dominate every setting.
Figures & tables
Figure 1: Overview of TACE. Stage (S1), ours: replaces (Ri−μG)/σG with the 1D optimal-transport map onto N(0,1) , Ai=Φ−1((rankτ(Ri)−0.5)/Nτ) , for inter-task gradient equity. Stage (S2), ours (starting from the evidence ratio of Yang et al., 2026 ): runs πθ under non-privileged ( PS ) and privileged ( PT ) contexts, rank-calibrates the per-token log-evidence within the trajectory, and applies a sign-anchored, mean-one (mass-preserving) magnitude weight wi,t . The composed token advantage A^i,t=Ai⋅wi,t plugs into the standard GRPO clipped surrogate.
Method
Sample baseline Ai (S1)
Token weight wi,t (S2)
Direction anchor
GRPO ( Shao et al., 2024 )
(Ri−μG)/σG
1
Verifier
Dr. GRPO ( Liu and others, 2025 )
Ri−μG
1
Verifier
EMA-GRPO ( Feng, 2025 )
(Ri−μG)/στ(t)
1
Verifier
OPSD ( Zhao et al., 2026 )
—
PT/PS as gradient target
Teacher
SDPO ( Hübotter et al., 2026 )
—
sign-free evidence ratio
Teacher
RLSD ( Yang et al., 2026 )
(Ri−μG)/σG
clip((PT/PS)sign(Ai))
Verifier
Table 1: Representative estimators viewed through Eq. ( 3 ). TACE contributes the distribution-aware stage-(S1) baseline and composes it with a rank-calibrated, mass-preserving version of direction-anchored credit.
Method
MMMU
MathVista
MathVision
ZeroBench
WeMath
Base VLM
62.44
73.80
47.37
19.76
54.10
GRPO
65.11
76.20
48.82
22.60
56.57
OPSD
63.82
75.10
47.53
21.06
54.95
SDPO
65.11
74.00
47.27
25.15
52.19
GRPO+OPSD
63.22
75.90
48.52
22.16
54.76
RLSD
67.22
78.10
52.73
24.85
58.00
Table 2: Accuracy after post-training Qwen3-VL-8B-Instruct. Bold marks the top score in each column. The Base, GRPO, OPSD, SDPO, GRPO+OPSD, and RLSD rows quote the numbers reported by RLSD; the TACE row is ours.
Figure 2: A WeMath example solved correctly by the trained model.
Method
MMMU
MathVista
MathVision
TACE w/o S1
64.17
76.20
50.53
TACE w/o S2
65.31
77.58
52.76
TACE
68.67
80.60
56.58
Table 3: Component ablation on the first three multimodal reasoning benchmarks. w/o S1 removes the quantile-normalized sample baseline; w/o S2 removes the rank-calibrated, mass-preserving token credit.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Arm
Estimator
ϵw
Warm-up
Teacher refresh
Group size
Seed
base_seed1
TACE
0.2
50
20
8
1
base_seed2
TACE
0.2
50
20
8
2
base_seed3
TACE
0.2
50
20
8
3
s1_only
TACE w/o S2
–
–
–
8
1
s2_only
TACE w/o S1
0.2
50
20
8
1
eps010
TACE
0.1
50
20
8
1
Appendix
Table 4: Hyperparameter configuration of each sensitivity arm.
Arm
Final-10 reward
WeMath pilot
Clip (%)
Step time (s)
base_seed1
0.3957
74.43
24.03
344.7
base_seed2
0.4159
78.63
25.50
361.4
base_seed3
0.3573
76.72
36.53
345.4
s1_only
0.4165
74.81
21.38
298.6
s2_only
0.3841
70.61
24.92
344.7
eps010
0.4199
74.43
21.38
371.0
Appendix
Table 5: Late-training sensitivity results. Reward is the mean over the final ten continuation steps. Clip is the mean response-token clipping ratio, and time is mean wall-clock time per continuation step.
Metric
Mean
Sample SD
Range
Final-10 reward
0.3896
0.0298
0.3573–0.4159
WeMath pilot
76.59
2.09
74.43–78.63
Appendix
Table 6: Variation among the three repeated default-configuration seeds. The standard deviation uses the n−1 denominator.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
GRPO shares a response-level advantage across tokens, leaving unresolved how strongly each token should contribute to learning. Updates induced by the shared advantage reinforce or suppress the sampled token at each position, but these updates do not always point toward increasing reward at the token level. This motivates a work-based view of credit: just as work depends on the component of force along a displacement, token credit should reflect the local reward change along the policy update direction. We introduce Direction-Aware Token Credit Assignment (DATC), which maps frozen reward-model gradients into policy logit space and allocates weights by evaluating the alignment between the reward direction and GRPO's signed sampled-action update direction, while preserving the sign and mean of the response-level advantage. DATC outperforms GRPO in all four settings across two model families.
Yikuan Hu, Jifeng Zhu, Minle Su +6
Nanjing University · National University of Singapore · Tsinghua University +1
Reinforcement Learning (RL) has substantially improved the reasoning ability of large language models (LLMs), but sparse outcome rewards still make token-level credit assignment difficult. Existing scalable RL methods typically assign trajectory-level rewards uniformly across tokens, while recent entropy-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non-local gradient components misaligned with sampled-token policy updates. We propose Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment framework based on a mode-local surrogate entropy. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy-gradient direction to leading order. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO.
Reinforcement learning with verifiable rewards scales multimodal reasoning, but an outcome reward says how much a trajectory is worth, not how that value should be spread over the decisions that produced it. We introduce Residual Visual Credit Optimization (RVCO), which treats token credit as a conserved routing problem. A controlled visual intervention yields a per-token evidence response; robust within-trajectory coordinates remove incidental scale; and a budgeted entropic router distributes a fixed amount of sequence utility according to perceptual dependence. A residual support path guarantees positive credit at every valid position, and an analytic correction restores the prescribed credit mass exactly. The resulting field is selective, bounded, full-support, and invariant to response-local score shifts, and recovers hard token selection as a limiting case. Across four model families and seven reasoning benchmarks, RVCO improves accuracy over strong RLVR baselines while maintaining late-stage optimization stability, corruption robustness, and competitive training cost. Rewards, rollouts, and the group-relative advantage estimator are unchanged; only the geometry of token-level credit differs.
Lin Qiu, Yao Liu, Diyi Hu +6
Meta Superintelligence Labs · University of Southern California · Meta Recommendation System +1