Distributional Reward Calibration and Budget-Neutral Credit Assignment for Multimodal RLVR

Lang Chen; Zongxin Xu; Borui Wang; Anran Liu; Shitong Chen

cs.AIOct 29, 2026

Distributional Reward Calibration and Budget-Neutral Credit Assignment for Multimodal RLVR

Authors: Lang Chen, Zongxin Xu, Borui Wang, Anran Liu, Shitong Chen

Organizations: Shenzhen University

Abstract

We revisit advantage estimation in Group Relative Policy Optimization (GRPO) and observe that any token-level advantage admits a two-stage factorization: a sample-level baselining step that converts scalar verifier rewards into a comparable sequence-level signal, and a token-level credit assignment step that distributes that signal over the trajectory. Existing estimators typically address only one of these stages: affine baselines preserve the shape of each task's reward law, so heavy tails and binary cliffs propagate into the gradient, while token-level credit rules tie the size of each token's update to raw confidence scales and need not conserve the total credit issued by the verifier. We propose TACE (Two-stage Advantage and Credit Estimation), a two-stage estimator that addresses both stages. At stage~(S1), we replace linear standardization with a one-dimensional optimal-transport map to a standard normal, which gives a common population reward law across tasks and caps finite-sample outliers deterministically. At stage~(S2), we rank-calibrate a stop-gradient evidence contrast within each trajectory and renormalize the clipped token weights to have mean one, so credit is redistributed across tokens while its total mass and sign are conserved. We prove population-level inter-task moment equity, a deterministic tail bound, and mass-conservation, boundedness, and sign-anchoring properties that affine baselines and unnormalized token weights do not satisfy. On a multimodal reasoning suite, TACE obtains the strongest results on a majority of the benchmarks but does not dominate every setting.

Figures & tables

Figure 1: Overview of TACE. Stage (S1), ours: replaces $(R_{i}-\mu_{G})/\sigma_{G}$ with the 1D optimal-transport map onto $\mathcal{N}(0,1)$ , $A_{i}=\Phi^{-1}((\mathrm{rank}_{\tau}(R_{i})-0.5)/N_{\tau})$ , for inter-task gradient equity. Stage (S2), ours (starting from the evidence ratio of Yang et al., 2026 ): runs $\pi_{\theta}$ under non-privileged ( $P_{S}$ ) and privileged ( $P_{T}$ ) contexts, rank-calibrates the per-token log-evidence within the trajectory, and applies a sign-anchored, mean-one (mass-preserving) magnitude weight $w_{i,t}$ . The composed token advantage $\hat{A}_{i,t}=A_{i}\cdot w_{i,t}$ plugs into the standard GRPO clipped surrogate.

Method	Sample baseline $A_{i}$ (S1)	Token weight $w_{i,t}$ (S2)	Direction anchor
GRPO ( Shao et al., 2024 )	$(R_{i}-\mu_{G})/\sigma_{G}$	$1$	Verifier
Dr. GRPO ( Liu and others, 2025 )	$R_{i}-\mu_{G}$	$1$	Verifier
EMA-GRPO ( Feng, 2025 )	$(R_{i}-\mu_{G})/\sigma_{\tau}^{(t)}$	$1$	Verifier
OPSD ( Zhao et al., 2026 )	—	$P_{T}/P_{S}$ as gradient target	Teacher
SDPO ( Hübotter et al., 2026 )	—	sign-free evidence ratio	Teacher
RLSD ( Yang et al., 2026 )	$(R_{i}-\mu_{G})/\sigma_{G}$	$\mathrm{clip}\!\bigl((P_{T}/P_{S})^{\mathrm{sign}(A_{i})}\bigr)$	Verifier

Table 1: Representative estimators viewed through Eq. ( 3 ). TACE contributes the distribution-aware stage-(S1) baseline and composes it with a rank-calibrated, mass-preserving version of direction-anchored credit.

Method	MMMU	MathVista	MathVision	ZeroBench	WeMath
Base VLM	62.44	73.80	47.37	19.76	54.10
GRPO	65.11	76.20	48.82	22.60	56.57
OPSD	63.82	75.10	47.53	21.06	54.95
SDPO	65.11	74.00	47.27	25.15	52.19
GRPO+OPSD	63.22	75.90	48.52	22.16	54.76
RLSD	67.22	78.10	52.73	24.85	58.00

Table 2: Accuracy after post-training Qwen3-VL-8B-Instruct. Bold marks the top score in each column. The Base, GRPO, OPSD, SDPO, GRPO+OPSD, and RLSD rows quote the numbers reported by RLSD; the TACE row is ours.

Figure 2: A WeMath example solved correctly by the trained model.

Method	MMMU	MathVista	MathVision
TACE w/o S1	64.17	76.20	50.53
TACE w/o S2	65.31	77.58	52.76
TACE	68.67	80.60	56.58

Table 3: Component ablation on the first three multimodal reasoning benchmarks. w/o S1 removes the quantile-normalized sample baseline; w/o S2 removes the rank-calibrated, mass-preserving token credit.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Arm	Estimator	$\epsilon_{w}$	Warm-up	Teacher refresh	Group size	Seed
base_seed1	TACE	0.2	50	20	8	1
base_seed2	TACE	0.2	50	20	8	2
base_seed3	TACE	0.2	50	20	8	3
s1_only	TACE w/o S2	–	–	–	8	1
s2_only	TACE w/o S1	0.2	50	20	8	1
eps010	TACE	0.1	50	20	8	1

Appendix

Table 4: Hyperparameter configuration of each sensitivity arm.

Arm	Final-10 reward	WeMath pilot	Clip (%)	Step time (s)
base_seed1	0.3957	74.43	24.03	344.7
base_seed2	0.4159	78.63	25.50	361.4
base_seed3	0.3573	76.72	36.53	345.4
s1_only	0.4165	74.81	21.38	298.6
s2_only	0.3841	70.61	24.92	344.7
eps010	0.4199	74.43	21.38	371.0

Appendix

Table 5: Late-training sensitivity results. Reward is the mean over the final ten continuation steps. Clip is the mean response-token clipping ratio, and time is mean wall-clock time per continuation step.

Metric	Mean	Sample SD	Range
Final-10 reward	0.3896	0.0298	0.3573–0.4159
WeMath pilot	76.59	2.09	74.43–78.63

Appendix

Table 6: Variation among the three repeated default-configuration seeds. The standard deviation uses the

n-1

denominator.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Direction-Aware Token Credit Assignment with Reward-Model Gradients
Oct 23, 2026Yikuan Hu, Jifeng Zhu, Minle Su +6Token-Level Credit Assignment Policy Gradient Methods
Nanjing University · National University of Singapore · Tsinghua University +1
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning
Oct 4, 2026Lin Qiu, Yao Liu, Diyi Hu +6Credit Assignment in RL Step-Level Credit Assignment in RL
Meta Superintelligence Labs · University of Southern California · Meta Recommendation System +1

Oct 23, 2026J/K move · Enter open · S save

Direction-Aware Token Credit Assignment with Reward-Model Gradients

Yikuan Hu, Jifeng Zhu, Minle Su, Jiaqin Yue+5

Nanjing University · National University of Singapore · Tsinghua University+1

Token-Level Credit Assignment Policy Gradient Methods Group Relative Policy Optimization Gradient-Based Attribution

Open paper AskPDF arXiv

GRPO shares a response-level advantage across tokens, leaving unresolved how strongly each token should contribute to learning. Updates induced by the shared advantage reinforce or suppress the sampled token at each position, but these updates do not always point toward increasing reward at the token level. This motivates a work-based view of credit: just as work depends on the component of force along a displacement, token credit should reflect the local reward change along the policy update direction. We introduce Direction-Aware Token Credit Assignment (DATC), which maps frozen reward-model gradients into policy logit space and allocates weights by evaluating the alignment between the reward direction and GRPO's signed sampled-action update direction, while preserving the sign and mean of the response-level advantage. DATC outperforms GRPO in all four settings across two model families.

Abstract

Reinforcement Learning

Explore similar work

Direction-Aware Token Credit Assignment with Reward-Model Gradients

ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning