Diffusion language models (DLMs) generate text through iterative denoising, enabling bidirectional context modeling and parallel token refinement. Recent studies have applied reinforcement learning (RL) to DLMs and improved their reasoning performance. However, the iterative denoising process makes credit assignment more challenging, as final outcome rewards provide limited information about how different tokens contribute to the final output. In this paper, we propose Denoising Token Credit Assignment (DTCA), a framework for token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after token commitment and groups these signals according to part-of-speech (POS) categories. These contribution weights are used to select informative denoising timesteps and assign token-level learning signals during RL training. DTCA further calibrates these learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.
Figures & tables
Figure 1: Preliminary analysis of POS-level differences in DLM denoising using rollouts generated from the GSM8K ( Cobbe et al., 2021 ) training set. (a) Commitment timing of different POS categories across denoising steps. (b) Improvement in the prediction confidence of remaining masked positions after individual token commitment, averaged within each POS category. POS tags denote NOUN (noun), NUM (numeral), PUNCT (punctuation), PROPN (proper noun), VERB (verb), ADP (adposition), DET (determiner), and ADJ (adjective).
Figure 2: Overview of Denoising Token Credit Assignment (DTCA). DTCA estimates token contributions from denoising refinement dynamics and uses the resulting contribution weights for denoising state selection and token-level credit assignment. The obtained learning signals are further calibrated according to entropy changes induced by policy updates for reinforcement learning optimization.
MATH500
GSM8K
Method
128
256
512
128
256
512
LLaDA-8B-Instruct ( Nie et al., 2025 )
30.8
36.4
39.2
75.3
83.2
82.9
LLaDA-1.5 ( Zhu et al., 2026 )
32.6
39.4
39.8
76.2
81.6
83.0
Uniform SFT
24.6
30.0
32.6
74.8
80.7
78.6
d1 ( Zhao et al., 2025 )
33.6
37.6
37.2
75.8
82.4
81.8
ESPO ( Ou et al., 2026 )
37.6
38.0
40.2
78.1
83.2
84.0
Table 1: Mathematical reasoning accuracy (%) on MATH500 and GSM8K. The last rows report DTCA, its ablation variants, and the improvement over the base model. Boldface indicates the best result in each column, and Δ denotes the improvement over LLaDA-8B-Instruct.
Figure 3: Optimization dynamics of DTCA variants during RL training. (a) Response reward, (b) gradient norm, and (c) mean KL divergence. DTCA is compared with variants without Contribution Credit and Entropy Calibration.
Figure 4: Analysis of commitment dynamics on MATH500. DTCA is compared with the base model across three denoising stages. Each cell shows the difference in (a) token commitment ratio, (b) confidence at commitment, and (c) entropy at commitment.
Figure 5: Analysis of entropy-based credit calibration. Each panel shows the target entropy change of a POS category during RL training. The blue solid and orange dashed curves represent DTCA and DTCA without entropy calibration, respectively. The vertical axis shows the difference between the mean predictive entropy over the most recent 25 updates and the mean over the first 100 updates of the same run. Percentages in parentheses show the relative change in assigned credit after entropy calibration, and n is the number of tokens counted.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
State ratio
Token cov.
Conf. cov.
Low conf. cov.
Entropy cov.
DTCA ( μ=0.40 )
16.82
19.99
17.72
40.59
37.81
RLDF k=16
23.31
21.33
20.79
26.52
25.56
RLDF (same count, μ=0.40 )
16.82
15.01
14.34
19.93
19.22
RLDF top-16
23.31
10.01
7.33
42.46
35.88
DTCA ( μ=0.70 )
31.81
36.02
33.24
62.81
59.31
RLDF k=32
45.42
42.61
41.73
49.91
48.73
Appendix
Table 2: Coverage of selected denoising states. State ratio is the selected fraction per response. RLDF (same count) matches DTCA’s count at each μ . All entries are mean percentages.
Figure 6: Current-commit token entropy changes in DTCA. Each 20-update bin subtracts the POS-specific mean over updates 1–100. The color scale matches the upper panel of Figure 7 .
Figure 7: Additional entropy and training-dynamics analyses. Top: current-commit token entropy changes in DTCA without entropy calibration, using the same early-training baseline, 20-update bins, and color scale as Figure 6 . Bottom: DTCA versus RLDF in (a) response reward, (b) policy entropy, and (c) mean KL divergence, using trailing 15-update means.
Figure 8: Predictive entropy changes for DTCA’s nine most frequent target POS categories. Blue solid: DTCA. Orange dashed: without entropy calibration. Curves show trailing 25-update token-weighted means relative to each run’s POS-specific baseline over updates 1–100.
Figure 9: Predictive entropy changes for the nine remaining POS categories, ordered by their frequency among DTCA’s retained targets. Colors, line styles, baseline, and smoothing match Figure 8 .
Figure 10: Predicted entropy-increasing (solid) and entropy-decreasing (dashed) components for DTCA (blue) and DTCA without entropy calibration (orange). Curves use trailing 25-update means. The vertical axis is broken, with values in 10−7 nats.
Diffusion large language models are a compelling alternative to autoregressive models, yet existing RL methods for diffusion treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. We identify two fundamental weaknesses: the absence of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization. To address these, we propose Denoising-Aware Credit Assignment for GRPO (DACA-GRPO), a lightweight, plug-and-play enhancement for any GRPO-style trainer. DACA-GRPO introduces two complementary mechanisms: Denoising Progress Scores, which extract per-token importance weights from intermediate predictions at no additional forward cost, and Stratified Masking Likelihood, which partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias. Applied on top of three GRPO base methods, DACA-GRPO achieves consistent improvements across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation, with gains of up to 5.6pp on math reasoning, 7.4pp on code generation, 36.3pp on constraint satisfaction, and 5.9pp on JSON schema adherence.
Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose DelTA, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.
Kaiyi Zhang, Wei Wu, Yankai Lin
Gaoling School of Artificial Intelligence, Renmin University of China · Ant International
Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce Reinforcement Learning from Denoising Feedback (RLDF), a novel training paradigm that leverages feedback obtained from rollout and training processes to facilitate accurate and efficient policy loss estimation. To balance the trade-off between computational efficiency and estimation effectiveness, RLDF optimizes the model toward the clipped clean state from intermediate noisy states, combined with weighted timestep sampling over denoising timesteps. Extensive experiments demonstrate that RLDF achieves consistent and substantial improvements in both performance and generalizability across two representative DLM architectures, LLaDA and Dream, on multiple reasoning benchmarks. Our work lays a principled foundation for scalable reinforcement learning in diffusion language models. We build Drift, a training framework for DLMs, available at https://github.com/ant-research/Drift.
Qi He, Huan Chen, Ya Guo +3
Fudan University · Ant Group · Hong Kong University of Science and Technology