GRPO shares a response-level advantage across tokens, leaving unresolved how strongly each token should contribute to learning. Updates induced by the shared advantage reinforce or suppress the sampled token at each position, but these updates do not always point toward increasing reward at the token level. This motivates a work-based view of credit: just as work depends on the component of force along a displacement, token credit should reflect the local reward change along the policy update direction. We introduce Direction-Aware Token Credit Assignment (DATC), which maps frozen reward-model gradients into policy logit space and allocates weights by evaluating the alignment between the reward direction and GRPO's signed sampled-action update direction, while preserving the sign and mean of the response-level advantage. DATC outperforms GRPO in all four settings across two model families.
Figures & tables
Figure 1: A work-based view of token credit. DATC assigns larger weights to token updates with higher predicted local reward gains.
Figure 2: Overview of DATC. DATC combines the reward-improvement direction with the GRPO update direction to allocate token-level advantages, assigning larger weights to updates with higher predicted local reward gains.
Method
RM-Reward ↑
AlpacaEval ↑
Arena-Hard ↑
MT-Bench ↑
Qwen3-14B-Base
Initial
-2.54
6.65
8.42
6.76
SFT
3.75
7.00
3.15
6.53
GRPO
10.87
37.39
28.49
7.70
DATC
13.35 ± 0.16
49.57 ± 1.12
39.31 ± 1.79
7.99 ± 0.05
Qwen3-14B-Instruct
Table 1: Main experimental results, with DATC reported as mean ± standard deviation across three random seeds.
Figure 3: Raw RM reward (left) and mean response length (right) for Llama-3.1-8B-Base after lightweight SFT. Light-colored points indicate the mean for each rollout; solid lines show the trailing 25-rollout moving average.
Method
RM-Reward ↑
AlpacaEval ↑
Arena-Hard ↑
MT-Bench ↑
Initial
-2.54
6.65
8.42
6.76
SFT
3.75
7.00
3.15
6.53
GRPO
10.87
37.39
28.49
7.70
P2T
10.96
36.40
29.68
7.66
RED
6.14
13.21
22.30
6.93
DATC
Table 2: Method comparisons and credit assignment ablations on Qwen3-14B-Base.
Figure 4: DATC weights for excerpts from actual training responses. Warm orange denotes reinforcement and blue denotes suppression; darker shades within each color indicate larger weights.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Policy checkpoint
Reward model checkpoint
RL initialization
Qwen Base
Qwen3-14B-Base
Skywork-Reward-V2-Qwen3-8B
After lightweight SFT
Qwen Instruct
Qwen3-14B
Skywork-Reward-V2-Qwen3-8B
Official instruction-tuned model
Llama Base
Llama-3.1-8B
Skywork-Reward-Llama-3.1-8B-v0.2
After lightweight SFT
Llama Instruct
Llama-3.1-8B-Instruct
Skywork-Reward-Llama-3.1-8B-v0.2
Official instruction-tuned model
Appendix
Table C1: Policy models, reward models, and initializations.
Configuration
Value
Prompts per rollout / responses per prompt
8 / 8
Rollouts / policy epochs per rollout
250 / 1
Maximum prompt / response length
2,048 / 2,048 tokens
Sampling temperature / top- p / top- k
1.0 / 1.0 / 0 (full distribution)
Optimizer / learning rate / weight decay
AdamW / constant 5×10−5 / 0.01
Maximum gradient norm / clipping parameter ϵ
1.0 / 0.2
Appendix
Table C2: Shared training settings.
Figure C1: Raw RM reward (left) and mean response length (right) of DATC and GRPO across the four experimental settings. Curves show trailing 25-rollout moving averages.
Figure C2: Cumulative GPU time to complete training (left) and DATC’s overhead relative to GRPO (right) across the four experimental settings. End-to-end costs include sampling, scoring, credit computation, and policy updates.
Method
Win rate (%) ↑
DATC
65.00
GRPO
16.67
P2T
15.00
RED
3.33
Appendix
Table C3: Human preference evaluation on Qwen3-14B-Base. Win rates are the percentages of 120 prompts won by each method after tie-breaking.
Figure C3: Relationship between independent-logit predicted gain X and shared-parameter single-step first-order gain P . Each point represents a within-batch DATC–GRPO comparison from the same SFT starting point, for a total of 100 data points. The light-colored solid line is a linear fit with an intercept.
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance +1