Organizations: University of Minnesota · National University of Singapore · University of Illinois Urbana-Champaign · University of Southern California · University of Pennsylvania · MBZUAI
Language models are increasingly trained against several verifiable rewards at once, such as the correctness and the format of a tool call, or the individual constraints of a complex instruction. Reward-decoupled normalization (GDPO) normalizes each reward separately so that different reward combinations keep distinct advantages, but it still spreads every reward's advantage evenly over the whole response, although each verifier checks only a few positions. We propose MRTPO, which assigns the advantage of each reward to the tokens relevant to it, using the information its verifier checks, without auxiliary models or token-level annotations. For each reward, the policy replays its own response with and without that reward's verifier information, and the per-token Jensen-Shannon distance between the two next-token distributions measures each token's relevance and reassigns the advantage. On text-to-SQL, multi-constraint instruction following and tool calling, MRTPO converges faster and achieves better performance than the compared methods.
Figures & tables
Figure 1: Advantages assigned to the tokens of one SQL response that is well formed and executable but uses >= for > . MRTPO places format credit on the tags, executability credit on the query, and the execution penalty mostly on >= 30 .
Figure 2: Per-token advantage of a training response under GDPO and MRTPO. Both panels use the same per-reward advantages.
Figure 3: Leave-one-out replay on a text-to-SQL response. The response reads table telefone , which does not exist; the reference uses phone . Replay places the penalties of the executability and execution rewards mainly on telefone and the selected columns, and the format credit on the tags.
Spider
BIRD
Method
Dev
Test
Syn
Realistic
DK
Dev
Average
Qwen2.5-Coder-3B-Instruct
Base
62.38
64.46
53.00
54.72
51.87
36.88
53.88
GRPO
74.24 ±1.26
75.00 ±0.31
65.92 ±1.50
69.49 ±1.94
64.04 ±1.69
43.54 ±1.03
65.37 ±1.18
GDPO
76.66 ±2.21
76.40 ±1.17
67.34 ±1.82
70.80 ±1.94
65.61 ±0.93
47.13 ±1.94
67.32 ±1.66
OPSD
68.18 ±0.61
69.34 ±0.45
59.15 ±0.39
62.14 ±1.38
56.12 ±0.96
39.21 ±0.70
59.02 ±0.49
Table 1: Text-to-SQL execution accuracy (%); subscripts are standard deviations over seeds. MRTPO is best on average with both models, with and without SAW.
IFEval
IFBench
MulDimIF
AdvancedIF
Method
Prompt
Inst.
Prompt
Inst.
Acc.
Overall
Rubric
Avg.
Qwen3-4B-Instruct-2507
Instruct
83.77
88.82
30.95
33.13
56.50
38.21
76.41
58.26
GRPO
85.44 ±0.15
90.16 ±0.06
35.68 ±1.02
37.79 ±1.01
62.44 ±0.86
39.27 ±0.27
77.06 ±0.12
61.12 ±0.34
CoRT
85.37 ±0.50
90.06 ±0.31
34.74 ±1.90
36.85 ±1.95
63.00 ±1.35
38.88 ±0.39
76.95 ±0.44
60.84 ±0.78
GDPO
85.52 ±0.52
90.23 ±0.33
36.39 ±1.59
38.84 ±1.18
62.88 ±2.83
38.91 ±0.43
76.85 ±0.21
61.37 ±0.77
Table 2: Instruction-following accuracy (%); subscripts are standard deviations over seeds. MRTPO has the best average on both models, and its largest gains are on IFBench, whose constraints are not seen in training.
Non-Live
Live
Method
Simple
Multiple
Parallel
Par. Mult.
Overall
Simple
Multiple
Parallel
Par. Mult.
Overall
Qwen2.5-1.5B-Instruct
Base
1.75
8.50
13.00
25.00
12.06
6.59
9.59
18.75
20.83
9.33
GRPO
70.07 ±0.69
86.40 ±0.42
81.90 ±1.08
73.80 ±2.20
78.04 ±0.84
76.43 ±0.84
62.18 ±0.75
55.00 ±2.80
60.83 ±6.32
64.80 ±0.55
GDPO
70.32 ±2.54
88.60 ±0.74
82.60 ±2.63
75.70 ±2.08
79.30 ±1.69
71.78 ±4.89
66.15 ±1.93
53.75 ±5.59
64.17 ±4.75
67.05 ±2.40
MRTPO
70.90 ±1.10
89.00 ±0.50
83.10 ±1.56
77.20 ±2.25
80.05 ±1.00
74.42 ±2.07
67.52 ±0.81
55.00 ±2.80
65.83 ±3.49
68.66 ±0.92
Table 3: BFCL-v4 single-turn AST accuracy (%); subscripts are standard deviations over seeds. MRTPO has the best overall accuracy on both splits with both models.
Figure 4: Tool calling on Qwen2.5-1.5B-Instruct over training: (a, b) training format and correctness rewards, median over seeds with interquartile range; (c, d) BFCL-v4 accuracy and format rate, mean over seeds. The binary format reward is harder to learn, while the continuous correctness reward rises early for all methods. The median number of steps to a format reward of 0.8 is 13 for MRTPO, 19 for GDPO and 34 for GRPO.
Figure 5: Training dynamics. (a) Text-to-SQL execution accuracy averaged over six benchmarks (Qwen2.5-Coder-7B-Instruct); (b) training response length (Qwen2.5-Coder-3B-Instruct); (c, d) IFBench prompt-level and MulDimIF accuracy (Qwen3-4B-Instruct-2507). Lines are seed means; bands span the seeds.
Figure 6: Where the replay distance falls. (a, b) Text-to-SQL with Qwen2.5-Coder-3B-Instruct; (c, d) tool calling with Qwen2.5-3B-Instruct. (a, c) Share of each reward’s distance held by the top tokens of a response; (b, d) share on each part of the response. Gray marks the uniform assignment of GDPO.
Qwen2.5-7B
Qwen3-4B
Constraint
Dist.
Tok.
Ratio
Dist.
Tok.
Ratio
End phrase
39.6
4.4
9.1
41.4
4.0
10.4
Title
24.5
3.2
7.6
25.0
3.3
7.5
Quotation
12.3
1.1
11.1
19.6
1.1
17.4
Start phrase
13.9
3.8
3.7
–
–
–
Keyword
8.6
2.7
3.2
10.2
3.1
3.3
Table 4: Replay localizes each constraint. For each satisfied constraint type, Dist. is the percentage of the constraint’s replay distance on the part of the response its checker reads, Tok. is that part’s percentage of the tokens, and Ratio is Dist. over Tok.; uniform assignment gives a ratio of 1. The last two rows pool all satisfied and all violated constraints. “–”: fewer than five instances.
Method
Dev
Test
Syn
Real.
DK
BIRD
Avg.
GDPO
79.21
80.62
72.21
74.87
67.29
55.11
71.55
MRTPO
81.33
81.65
73.53
75.92
70.04
56.42
73.15
joint replay
80.91
81.09
72.95
75.92
69.54
54.94
72.56
shuffled pos.
77.92
80.41
71.21
75.26
66.54
54.85
71.03
Table 5: Ablations on text-to-SQL with Qwen2.5-Coder-7B-Instruct. Shuffling the weights within each response removes the gain over GDPO.
IFEval
IFBench
MulDimIF
β
Prompt
Inst.
Prompt
Inst.
Acc.
Avg.
0.25
81.59
87.31
35.83
37.91
53.69
59.26
0.5
82.18
87.70
38.59
41.87
52.75
60.62
0.75
82.49
88.04
36.08
38.67
55.04
60.06
Table 6: Allocation strength on instruction following with Qwen2.5-7B-Instruct. Assigning half of each advantage by relevance gives the highest average.
β
Dev
Test
Syn
Real.
DK
BIRD
Avg.
0.25
81.24
83.05
73.69
76.18
71.54
55.87
73.59
0.5
81.82
81.46
73.02
77.56
69.85
56.46
73.36
0.75
80.85
81.97
72.73
74.41
70.04
56.92
72.82
Table 7: Allocation strength on text-to-SQL with Qwen2.5-Coder-7B-Instruct (one seed).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
What changes in the GDPO step
Token advantage St(j)
GDPO ( Liu et al., 2026b )
nothing; the per-reward advantages are summed
∑kAk(j) for all t
SAW ( He et al., 2026 )
each reward is weighted by its coefficient of variation in the batch
∑kλkAk(j) for all t
DARA ( Zheng et al., 2026 )
rewards that vary in fewer groups get a larger, capped weight on their positive advantages
∑k(Ak(j)+(λk−1)[Ak(j)]+) for all t
DVAO ( Jiang et al., 2026 )
each reward is weighted by its standard deviation within the group, normalized across rewards
∑kλkAk(j) for all t
SA-MRPO ( Wang et al., 2026b )
each reward is discounted by how saturated it is in the batch
∑kλkAk(j) for all t
GD 2 PO ( Liu et al., 2026a )
responses whose per-reward advantages disagree in sign are dropped, and each prompt is scaled by its share of kept responses
κδ(j)∑kAk(j) for all t
Appendix
Table 8: Where each method changes the GDPO step. The other methods act on the response, so all tokens of a response receive the same advantage; MRTPO sets a weight per reward and token and can be combined with them.
Parameter
Text-to-SQL
Instruction following
Tool calling
Models
Qwen2.5-Coder-{3B, 7B}-Instruct
Qwen2.5-7B-Instruct; Qwen3-4B-Instruct-2507
Qwen2.5-{1.5B, 3B}-Instruct
Train prompts
Spider train, 6,915
HIR-16K, first two subsets, 6,998
RLLA, 3,920 (80 for validation)
Rewards
format, executable, execution; {0,1}
one per constraint, {0,1} ; 5–13
format {0,1} ; correctness [−3,3]
Training steps
100
300
100
Prompts per step / G
32 / 8
64 / 8
512 / 4
Rollout temp. / top- p / top- k
1.1 / 1.0 / none
1.0 / 0.99 / 100
1.0 / 1.0 / none
Appendix
Table 9: Training and evaluation parameters of the three tasks.
Figure 7: Leave-one-out replay on a tool-calling response. The response sets offset to 2; the reference has 3. The layout follows Figure 3 .
Figure 8: Leave-one-out replay on an instruction-following response with 12 constraints, three of whose replays are shown. Layout and shading follow Figure 3 .
In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward model. Our core idea is to attribute the overall process reward to individual tokens by estimating each token's influence. This influence is defined as the change in the final macroscopic reward (e.g., the process reward) when a token is replaced with a semantically null token. Naively calculating this influence is computationally infeasible, requiring N forward passes through the PRM for an N-token sequence. We overcome this bottleneck by proposing a highly efficient gradient-based estimator. Specifically, we use a first-order Taylor approximation, which simplifies the influence calculation to the inner product of the difference between the token embedding and the null token embedding, and the gradient of the reward with respect to the token embedding. This requires only a single forward and backward pass. The resulting token-level rewards enable standard RL algorithms to perform precise credit assignment without requiring additional reward model training. Experiments on challenging reasoning benchmarks demonstrate that our method substantially improves policy optimization efficiency and enhances the generalization of LLM reasoning capabilities. Our P2T outperforms the outcome reward by +4.9% on MathVista for Qwen2.5-VL-7B-Instruct, and +11.5% on AIME24 for Qwen2.5-Math-7B. The code will be released via https://github.com/stonewst/P2T.
Sitong Wu, Haoru Tan, Bin Xia +6
Chinese University of Hong Kong · University of Hong Kong · Hong Kong University of Science and Technology
Verifiable rewards improve language models through reliable task-level feedback, but methods based on Group Relative Policy Optimization (GRPO) apply a sequence-level advantage uniformly across all tokens. This coarse credit assignment reinforces or penalizes entire responses without identifying which local decisions to preserve, reinforce, or revise. Conversely, evidence-conditioned self-distillation provides denser token-level supervision, yet teacher imitation can transfer stylistic artifacts and miscalibrated confidence that destabilize training when misaligned with task success. We introduce VERPO, which converts evidence-conditioned guidance into reward-aligned token-level credit assignment while retaining the outcome objective. VERPO decomposes teacher guidance into an evidence-free reference term and signed, evidence-induced corrections at each token. A stopped controller combines selective acceptance, token-wise localization, and cost-aware scaling by balancing alignment with the local GRPO update direction against Fisher movement cost. Furthermore, we introduce Fisher Evidence Contrast (FEC), which attenuates nuisance shifts along an estimated evidence-presence direction through a regularized projection. Across five scientific reasoning and tool-use tasks, VERPO prevents optimization collapse and consistently achieves the highest multi-task average across model backbones, yielding marked improvements particularly on smaller models over strong baselines. Qualitative diagnostics confirm that token acceptance selectively targets reasoning bottlenecks consistent with local reward alignment and Fisher movement cost.
Haijiang Li, Chengyu Lv, Yi Zhang +8
Independent · Tongji University · Fudan University +3
Reward models in RLHF are trained to score only the final token of a response - a choice that discards rich signal from every intermediate position and produces models whose token-level outputs are noise. We argue this is a missed opportunity: a well-trained reward model's output at any token should represent the conditional expectation of the final reward given the response so far. We introduce Temporally Coherent Reward Modeling (TCRM), which induces this property via two regularization terms on top of the standard Bradley-Terry loss, with minimizers provably equal to conditional expectations. The regularizers correspond to Monte Carlo and TD value-learning objectives, establishing a direct connection to RL value functions. TCRM requires zero changes to architecture, data, or inference, yet unlocks three capabilities from one principle: interpretable token-level reward trajectories (middle-token pairwise accuracy improved from 50% to 88.9%, final-token accuracy preserved); state-of-the-art PRM performance on ProcessBench (44.9% average F1) among models trained only on outcome data; and unified reward/value modeling in PPO, reducing peak GPU memory by 27% and step time by 19% with matching LLM quality.