DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models

Hongze Chen; Miao Pan; Lizhou Liang; Qinfeng Li; Xinyin Ma; Yongliang Shen; Weishi Wang; Daniel Dahlmeier

cs.CLOct 26, 2026

DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models

Authors: Hongze Chen, Miao Pan, Lizhou Liang, Qinfeng Li, Xinyin Ma, Yongliang Shen, Weishi Wang, Daniel Dahlmeier,

Organizations: Zhejiang University · National University of Singapore · SAP

Abstract

Diffusion language models (DLMs) generate text through iterative denoising, enabling bidirectional context modeling and parallel token refinement. Recent studies have applied reinforcement learning (RL) to DLMs and improved their reasoning performance. However, the iterative denoising process makes credit assignment more challenging, as final outcome rewards provide limited information about how different tokens contribute to the final output. In this paper, we propose Denoising Token Credit Assignment (DTCA), a framework for token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after token commitment and groups these signals according to part-of-speech (POS) categories. These contribution weights are used to select informative denoising timesteps and assign token-level learning signals during RL training. DTCA further calibrates these learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.

Figures & tables

Figure 1: Preliminary analysis of POS-level differences in DLM denoising using rollouts generated from the GSM8K ( Cobbe et al., 2021 ) training set. (a) Commitment timing of different POS categories across denoising steps. (b) Improvement in the prediction confidence of remaining masked positions after individual token commitment, averaged within each POS category. POS tags denote NOUN (noun), NUM (numeral), PUNCT (punctuation), PROPN (proper noun), VERB (verb), ADP (adposition), DET (determiner), and ADJ (adjective).

Figure 2: Overview of Denoising Token Credit Assignment (DTCA). DTCA estimates token contributions from denoising refinement dynamics and uses the resulting contribution weights for denoising state selection and token-level credit assignment. The obtained learning signals are further calibrated according to entropy changes induced by policy updates for reinforcement learning optimization.

	MATH500			GSM8K
Method	128	256	512	128	256	512
LLaDA-8B-Instruct ( Nie et al., 2025 )	30.8	36.4	39.2	75.3	83.2	82.9
LLaDA-1.5 ( Zhu et al., 2026 )	32.6	39.4	39.8	76.2	81.6	83.0
Uniform SFT	24.6	30.0	32.6	74.8	80.7	78.6
d1 ( Zhao et al., 2025 )	33.6	37.6	37.2	75.8	82.4	81.8
ESPO ( Ou et al., 2026 )	37.6	38.0	40.2	78.1	83.2	84.0

Table 1: Mathematical reasoning accuracy (%) on MATH500 and GSM8K. The last rows report DTCA, its ablation variants, and the improvement over the base model. Boldface indicates the best result in each column, and

\Delta

denotes the improvement over LLaDA-8B-Instruct.

Figure 3: Optimization dynamics of DTCA variants during RL training. (a) Response reward, (b) gradient norm, and (c) mean KL divergence. DTCA is compared with variants without Contribution Credit and Entropy Calibration.

Figure 4: Analysis of commitment dynamics on MATH500. DTCA is compared with the base model across three denoising stages. Each cell shows the difference in (a) token commitment ratio, (b) confidence at commitment, and (c) entropy at commitment.

Figure 5: Analysis of entropy-based credit calibration. Each panel shows the target entropy change of a POS category during RL training. The blue solid and orange dashed curves represent DTCA and DTCA without entropy calibration, respectively. The vertical axis shows the difference between the mean predictive entropy over the most recent 25 updates and the mean over the first 100 updates of the same run. Percentages in parentheses show the relative change in assigned credit after entropy calibration, and

n

is the number of tokens counted.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	State ratio	Token cov.	Conf. cov.	Low conf. cov.	Entropy cov.
DTCA ( $\mu=0.40$ )	16.82	19.99	17.72	40.59	37.81
RLDF $k=16$	23.31	21.33	20.79	26.52	25.56
RLDF (same count, $\mu=0.40$ )	16.82	15.01	14.34	19.93	19.22
RLDF top-16	23.31	10.01	7.33	42.46	35.88
DTCA ( $\mu=0.70$ )	31.81	36.02	33.24	62.81	59.31
RLDF $k=32$	45.42	42.61	41.73	49.91	48.73

Appendix

Table 2: Coverage of selected denoising states. State ratio is the selected fraction per response. RLDF (same count) matches DTCA’s count at each

\mu

. All entries are mean percentages.

Figure 6: Current-commit token entropy changes in DTCA. Each 20-update bin subtracts the POS-specific mean over updates 1–100. The color scale matches the upper panel of Figure 7 .

Panel of Figure 7 — Figure 7: Additional entropy and training-dynamics analyses. Top: current-commit token entropy changes in DTCA without entropy calibration, using the same early-training baseline, 20-update bins, and color scale as Figure 6 . Bottom: DTCA versus RLDF in (a) response reward, (b) policy entropy, and (c) mean KL divergence, using trailing 15-update means.

Figure 8: Predictive entropy changes for DTCA’s nine most frequent target POS categories. Blue solid: DTCA. Orange dashed: without entropy calibration. Curves show trailing 25-update token-weighted means relative to each run’s POS-specific baseline over updates 1–100.

Figure 9: Predictive entropy changes for the nine remaining POS categories, ordered by their frequency among DTCA’s retained targets. Colors, line styles, baseline, and smoothing match Figure 8 .

Figure 10: Predicted entropy-increasing (solid) and entropy-decreasing (dashed) components for DTCA (blue) and DTCA without entropy calibration (orange). Curves use trailing 25-update means. The vertical axis is broken, with values in

10^{-7}

nats.

Diffusion Language Model Training

See all 49 papers

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +4Group Relative Policy Optimization Credit Assignment in RL
The Ohio State University · Apple
DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
May 20, 2026Kaiyi Zhang, Wei Wu, Yankai LinPolicy Gradient Token-Level Credit Assignment
Gaoling School of Artificial Intelligence, Renmin University of China · Ant International
Reinforcement Learning from Denoising Feedback
May 25, 2026Qi He, Huan Chen, Ya Guo +3RL for Language Model Reasoning RL for Diffusion Models
Fudan University · Ant Group · Hong Kong University of Science and Technology

May 8, 2026J/K move · Enter open · S save

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade, Lokesh Boominathan+3

The Ohio State University · Apple

Group Relative Policy Optimization Credit Assignment in RL Step-Level Credit Assignment in RL Step-Level Credit Assignment RL for Diffusion Models Diffusion Language Models+1

Open paper AskPDF arXiv

Diffusion large language models are a compelling alternative to autoregressive models, yet existing RL methods for diffusion treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. We identify two fundamental weaknesses: the absence of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization. To address these, we propose Denoising-Aware Credit Assignment for GRPO (DACA-GRPO), a lightweight, plug-and-play enhancement for any GRPO-style trainer. DACA-GRPO introduces two complementary mechanisms: Denoising Progress Scores, which extract per-token importance weights from intermediate predictions at no additional forward cost, and Stratified Masking Likelihood, which partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias. Applied on top of three GRPO base methods, DACA-GRPO achieves consistent improvements across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation, with gains of up to 5.6pp on math reasoning, 7.4pp on code generation, 36.3pp on constraint satisfaction, and 5.9pp on JSON schema adherence.

Abstract

Diffusion Language Model Training

Explore similar work

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

Reinforcement Learning from Denoising Feedback