Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL

Skylar Zhai; Haonan Wang; Zhining Liu; Linxin Song; Xuwei Ding; Jingcheng Liang; Jian Kang

cs.LGOct 15, 2026

Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL

Authors: Skylar Zhai, Haonan Wang, Zhining Liu, Linxin Song, Xuwei Ding, Jingcheng Liang, Jian Kang

Organizations: University of Minnesota · National University of Singapore · University of Illinois Urbana-Champaign · University of Southern California ·

Abstract

Language models are increasingly trained against several verifiable rewards at once, such as the correctness and the format of a tool call, or the individual constraints of a complex instruction. Reward-decoupled normalization (GDPO) normalizes each reward separately so that different reward combinations keep distinct advantages, but it still spreads every reward's advantage evenly over the whole response, although each verifier checks only a few positions. We propose MRTPO, which assigns the advantage of each reward to the tokens relevant to it, using the information its verifier checks, without auxiliary models or token-level annotations. For each reward, the policy replays its own response with and without that reward's verifier information, and the per-token Jensen-Shannon distance between the two next-token distributions measures each token's relevance and reassigns the advantage. On text-to-SQL, multi-constraint instruction following and tool calling, MRTPO converges faster and achieves better performance than the compared methods.

Figures & tables

Figure 1: Advantages assigned to the tokens of one SQL response that is well formed and executable but uses >= for > . MRTPO places format credit on the tags, executability credit on the query, and the execution penalty mostly on >= 30 .

Figure 2: Per-token advantage of a training response under GDPO and MRTPO. Both panels use the same per-reward advantages.

Figure 3: Leave-one-out replay on a text-to-SQL response. The response reads table telefone , which does not exist; the reference uses phone . Replay places the penalties of the executability and execution rewards mainly on telefone and the selected columns, and the format credit on the tags.

Qwen2.5-Coder-3B-Instruct
	Spider					BIRD
Method	Dev	Test	Syn	Realistic	DK	Dev	Average
Base	62.38	64.46	53.00	54.72	51.87	36.88	53.88
GRPO	74.24 ±1.26	75.00 ±0.31	65.92 ±1.50	69.49 ±1.94	64.04 ±1.69	43.54 ±1.03	65.37 ±1.18
GDPO	76.66 ±2.21	76.40 ±1.17	67.34 ±1.82	70.80 ±1.94	65.61 ±0.93	47.13 ±1.94	67.32 ±1.66
OPSD	68.18 ±0.61	69.34 ±0.45	59.15 ±0.39	62.14 ±1.38	56.12 ±0.96	39.21 ±0.70	59.02 ±0.49

Table 1: Text-to-SQL execution accuracy (%); subscripts are standard deviations over seeds. MRTPO is best on average with both models, with and without SAW.

Qwen3-4B-Instruct-2507
	IFEval		IFBench		MulDimIF	AdvancedIF
Method	Prompt	Inst.	Prompt	Inst.	Acc.	Overall	Rubric	Avg.
Instruct	83.77	88.82	30.95	33.13	56.50	38.21	76.41	58.26
GRPO	85.44 ±0.15	90.16 ±0.06	35.68 ±1.02	37.79 ±1.01	62.44 ±0.86	39.27 ±0.27	77.06 ±0.12	61.12 ±0.34
CoRT	85.37 ±0.50	90.06 ±0.31	34.74 ±1.90	36.85 ±1.95	63.00 ±1.35	38.88 ±0.39	76.95 ±0.44	60.84 ±0.78
GDPO	85.52 ±0.52	90.23 ±0.33	36.39 ±1.59	38.84 ±1.18	62.88 ±2.83	38.91 ±0.43	76.85 ±0.21	61.37 ±0.77

Table 2: Instruction-following accuracy (%); subscripts are standard deviations over seeds. MRTPO has the best average on both models, and its largest gains are on IFBench, whose constraints are not seen in training.

Qwen2.5-1.5B-Instruct
	Non-Live					Live
Method	Simple	Multiple	Parallel	Par. Mult.	Overall	Simple	Multiple	Parallel	Par. Mult.	Overall
Base	1.75	8.50	13.00	25.00	12.06	6.59	9.59	18.75	20.83	9.33
GRPO	70.07 ±0.69	86.40 ±0.42	81.90 ±1.08	73.80 ±2.20	78.04 ±0.84	76.43 ±0.84	62.18 ±0.75	55.00 ±2.80	60.83 ±6.32	64.80 ±0.55
GDPO	70.32 ±2.54	88.60 ±0.74	82.60 ±2.63	75.70 ±2.08	79.30 ±1.69	71.78 ±4.89	66.15 ±1.93	53.75 ±5.59	64.17 ±4.75	67.05 ±2.40
MRTPO	70.90 ±1.10	89.00 ±0.50	83.10 ±1.56	77.20 ±2.25	80.05 ±1.00	74.42 ±2.07	67.52 ±0.81	55.00 ±2.80	65.83 ±3.49	68.66 ±0.92

Table 3: BFCL-v4 single-turn AST accuracy (%); subscripts are standard deviations over seeds. MRTPO has the best overall accuracy on both splits with both models.

Figure 4: Tool calling on Qwen2.5-1.5B-Instruct over training: (a, b) training format and correctness rewards, median over seeds with interquartile range; (c, d) BFCL-v4 accuracy and format rate, mean over seeds. The binary format reward is harder to learn, while the continuous correctness reward rises early for all methods. The median number of steps to a format reward of 0.8 is 13 for MRTPO, 19 for GDPO and 34 for GRPO.

Figure 5: Training dynamics. (a) Text-to-SQL execution accuracy averaged over six benchmarks (Qwen2.5-Coder-7B-Instruct); (b) training response length (Qwen2.5-Coder-3B-Instruct); (c, d) IFBench prompt-level and MulDimIF accuracy (Qwen3-4B-Instruct-2507). Lines are seed means; bands span the seeds.

Figure 6: Where the replay distance falls. (a, b) Text-to-SQL with Qwen2.5-Coder-3B-Instruct; (c, d) tool calling with Qwen2.5-3B-Instruct. (a, c) Share of each reward’s distance held by the top tokens of a response; (b, d) share on each part of the response. Gray marks the uniform assignment of GDPO.

	Qwen2.5-7B			Qwen3-4B
Constraint	Dist.	Tok.	Ratio	Dist.	Tok.	Ratio
End phrase	39.6	4.4	9.1	41.4	4.0	10.4
Title	24.5	3.2	7.6	25.0	3.3	7.5
Quotation	12.3	1.1	11.1	19.6	1.1	17.4
Start phrase	13.9	3.8	3.7	–	–	–
Keyword	8.6	2.7	3.2	10.2	3.1	3.3

Table 4: Replay localizes each constraint. For each satisfied constraint type, Dist. is the percentage of the constraint’s replay distance on the part of the response its checker reads, Tok. is that part’s percentage of the tokens, and Ratio is Dist. over Tok.; uniform assignment gives a ratio of 1. The last two rows pool all satisfied and all violated constraints. “–”: fewer than five instances.

Method	Dev	Test	Syn	Real.	DK	BIRD	Avg.
GDPO	79.21	80.62	72.21	74.87	67.29	55.11	71.55
MRTPO	81.33	81.65	73.53	75.92	70.04	56.42	73.15
joint replay	80.91	81.09	72.95	75.92	69.54	54.94	72.56
shuffled pos.	77.92	80.41	71.21	75.26	66.54	54.85	71.03

Table 5: Ablations on text-to-SQL with Qwen2.5-Coder-7B-Instruct. Shuffling the weights within each response removes the gain over GDPO.

$\beta$	Prompt	Inst.	Prompt	Inst.	Acc.	Avg.
	IFEval		IFBench		MulDimIF
0.25	81.59	87.31	35.83	37.91	53.69	59.26
0.5	82.18	87.70	38.59	41.87	52.75	60.62
0.75	82.49	88.04	36.08	38.67	55.04	60.06

Table 6: Allocation strength on instruction following with Qwen2.5-7B-Instruct. Assigning half of each advantage by relevance gives the highest average.

$\beta$	Dev	Test	Syn	Real.	DK	BIRD	Avg.
0.25	81.24	83.05	73.69	76.18	71.54	55.87	73.59
0.5	81.82	81.46	73.02	77.56	69.85	56.46	73.36
0.75	80.85	81.97	72.73	74.41	70.04	56.92	72.82

Table 7: Allocation strength on text-to-SQL with Qwen2.5-Coder-7B-Instruct (one seed).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	What changes in the GDPO step	Token advantage $S^{(j)}_{t}$
GDPO ( Liu et al., 2026b )	nothing; the per-reward advantages are summed	$\sum_{k}A^{(j)}_{k}$ for all $t$
SAW ( He et al., 2026 )	each reward is weighted by its coefficient of variation in the batch	$\sum_{k}\lambda_{k}A^{(j)}_{k}$ for all $t$
DARA ( Zheng et al., 2026 )	rewards that vary in fewer groups get a larger, capped weight on their positive advantages	$\sum_{k}\big(A^{(j)}_{k}+(\lambda_{k}-1)[A^{(j)}_{k}]_{+}\big)$ for all $t$
DVAO ( Jiang et al., 2026 )	each reward is weighted by its standard deviation within the group, normalized across rewards	$\sum_{k}\lambda_{k}A^{(j)}_{k}$ for all $t$
SA-MRPO ( Wang et al., 2026b )	each reward is discounted by how saturated it is in the batch	$\sum_{k}\lambda_{k}A^{(j)}_{k}$ for all $t$
GD 2 PO ( Liu et al., 2026a )	responses whose per-reward advantages disagree in sign are dropped, and each prompt is scaled by its share of kept responses	$\kappa\,\delta^{(j)}\sum_{k}A^{(j)}_{k}$ for all $t$

Appendix

Table 8: Where each method changes the GDPO step. The other methods act on the response, so all tokens of a response receive the same advantage; MRTPO sets a weight per reward and token and can be combined with them.

Parameter	Text-to-SQL	Instruction following	Tool calling
Models	Qwen2.5-Coder-{3B, 7B}-Instruct	Qwen2.5-7B-Instruct; Qwen3-4B-Instruct-2507	Qwen2.5-{1.5B, 3B}-Instruct
Train prompts	Spider train, 6,915	HIR-16K, first two subsets, 6,998	RLLA, 3,920 (80 for validation)
Rewards	format, executable, execution; $\{0,1\}$	one per constraint, $\{0,1\}$ ; 5–13	format $\{0,1\}$ ; correctness $[-3,3]$
Training steps	100	300	100
Prompts per step / $G$	32 / 8	64 / 8	512 / 4
Rollout temp. / top- $p$ / top- $k$	1.1 / 1.0 / none	1.0 / 0.99 / 100	1.0 / 1.0 / none

Appendix

Table 9: Training and evaluation parameters of the three tasks.

Figure 7: Leave-one-out replay on a tool-calling response. The response sets offset to 2; the reference has 3. The layout follows Figure 3 .

Figure 8: Leave-one-out replay on an instruction-following response with 12 constraints, three of whose replays are shown. Layout and shading follow Figure 3 .

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 429 papers

Unlocking Token Rewards via Training-Free Reward Attribution
Oct 13, 2026Sitong Wu, Haoru Tan, Bin Xia +6Token-Level Credit Assignment RL for Language Model Reasoning
Chinese University of Hong Kong · University of Hong Kong · Hong Kong University of Science and Technology
VERPO: Verified Evidence Regularized Policy Optimization
Sep 5, 2026Haijiang Li, Chengyu Lv, Yi Zhang +8RL for Language Model Reasoning Credit Assignment in RL
Independent · Tongji University · Fudan University +3
Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
Apr 24, 2026Alex NikulkovReward Modeling Value Function Estimation
AI at Meta

Oct 13, 2026J/K move · Enter open · S save

Unlocking Token Rewards via Training-Free Reward Attribution

Sitong Wu, Haoru Tan, Bin Xia, Xichen Zhang+5

Chinese University of Hong Kong · University of Hong Kong · Hong Kong University of Science and Technology

Token-Level Credit Assignment RL for Language Model Reasoning Gradient-Based Attribution Process Reward Models

Open paper AskPDF arXiv

In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward model. Our core idea is to attribute the overall process reward to individual tokens by estimating each token's influence. This influence is defined as the change in the final macroscopic reward (e.g., the process reward) when a token is replaced with a semantically null token. Naively calculating this influence is computationally infeasible, requiring N forward passes through the PRM for an N-token sequence. We overcome this bottleneck by proposing a highly efficient gradient-based estimator. Specifically, we use a first-order Taylor approximation, which simplifies the influence calculation to the inner product of the difference between the token embedding and the null token embedding, and the gradient of the reward with respect to the token embedding. This requires only a single forward and backward pass. The resulting token-level rewards enable standard RL algorithms to perform precise credit assignment without requiring additional reward model training. Experiments on challenging reasoning benchmarks demonstrate that our method substantially improves policy optimization efficiency and enhances the generalization of LLM reasoning capabilities. Our P2T outperforms the outcome reward by +4.9% on MathVista for Qwen2.5-VL-7B-Instruct, and +11.5% on AIME24 for Qwen2.5-Math-7B. The code will be released via https://github.com/stonewst/P2T.

Abstract

RL for Language Models

Explore similar work

Unlocking Token Rewards via Training-Free Reward Attribution

VERPO: Verified Evidence Regularized Policy Optimization

Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling