The decoding order of masked diffusion language models (dLLMs) can substantially affect generation quality, yet common decoders typically apply the same reveal rule throughout generation. Confidence provides a strong default, but it measures how decisively a position can be predicted now, not how useful revealing that position will be for subsequent decoding. We propose Cord, a lightweight prompt-conditioned policy that retains confidence decoding as a stable reference and learns when and how to depart from it through geometric residuals over generation direction, proximity to revealed context, and two-sided infilling. The residuals adapt to both the prompt and the evolving partial response. Rather than sampling a new policy action at every decoding step, Cord samples a single low-dimensional coefficient matrix per rollout, which induces an entire state-dependent reveal trajectory. This yields a contextual-bandit formulation trained directly from sequence-level rewards, without step-level credit assignment or backbone updates. Across three dLLMs and five reasoning, code, and constraint-satisfaction benchmarks, Cord improves average task success from 32.0% under confidence decoding to 48.2%, and further improves over the best per-domain fixed residual. Behavioral audits show that the learned residuals induce substantial, task-dependent changes in reveal locality and two-sided infilling. Cord adds only 0.53M trainable parameters, requires no additional backbone forward passes, and incurs only 5.39% inference latency overhead.
Figures & tables
Figure 1: Cord . Top: a prompt-conditioned MLP produces state-dependent geometric residuals: direction, proximity, and bridge, that modify confidence-based reveal scores while keeping the dLLM frozen. Bottom left: CORD reveals the highest-scoring masked position at each step. Bottom right: training samples one low-dimensional coefficient program per rollout and optimizes it from sequence-level rewards as a contextual bandit.
Figure 2: Oracle reveal diagnostics with frozen LLaDA. GSM8K favors a contiguous reasoning prefix, while MBPP benefits from structural anchors. Conservative uses core control-flow tokens (e.g. if , else , for ); Extended additionally includes function ( def , class ), delimiter, and assignment tokens.
Backbone
Decoding policy
GSM8K
MATH500
HumanEval
MBPP
Sudoku
Avg.
LLaDA-8B-Instruct
Confidence
58.7
18.0
27.8
38.4
17.1
32.0
Entropy
59.4
12.5
27.5
41.7
3.3
28.9
Margin ( Kim et al., 2025 )
58.4
22.2
28.8
42.1
25.8
35.5
EB-Sampler ( Ben-Hamu et al., 2025 )
61.4
12.7
27.1
42.7
33.8
35.5
Semi-AR † ( Nie et al., 2025 )
76.8
28.1
38.9
45.2
0.3
37.9
UNCODE ( Huang et al., 2026a )
79.2
34.8
42.1
47.8
29.8
46.7
Table 1: Main results across masked diffusion language models. Task success (%, higher is better). HumanEval and MBPP report pass@1; all other tasks report accuracy. The fixed residual uses the same geometric signals as CORD but keeps their coefficients constant throughout decoding.
Variant
LLaDA-Inst.
LLaDA-1.5
Dream
Avg.
Confidence only
32.0
33.3
26.4
30.6
No prompt (fixed residual)
48.0
49.8
40.5
46.1
Prompt only (static)
42.8
48.1
47.1
46.0
Time only
46.3
45.1
49.2
46.9
State only
47.1
46.4
33.1
42.2
Full CORD
48.2
50.7
52.5
50.5
Table 2: Decomposing adaptation in CORD. Mean task success (%) across all benchmarks.
Variant
GSM8K
MBPP
Sudoku
Avg.
−Direction
57.2
43.7
35.3
45.4
−Proximity
70.2
44.8
31.8
48.9
−Bridge
72.4
41.4
16.6
43.5
Full CORD
78.2
48.9
36.1
54.4
Table 3: Ablation of geometric residuals. Task success (%) on LLaDA-8B-Instruct.
Figure 3: Realized reveal behavior of CORD. ΔJ<0 indicates more local decoding than confidence, whereas ΔJ>0 indicates more non-local decoding. Positive Ebridge indicates preferential selection of two-sided masked positions beyond their availability.
Method
Extra Fwd.
Aux. Params
Latency (s)
Overhead
Confidence
0
0
6.31 ± 0.31
–
Fixed residual
0
0
6.55 ± 0.28
3.80%
CORD
0
527,512
6.65 ± 0.30
5.39%
Table 4: Inference efficiency on LLaDA-8B-Instruct. Latency (s) is measured on one H200 over 20 runs with 256 generated tokens and a maximum sequence length of 1024 ; we report mean ± standard deviation.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
config
value
math
GSM8K, MATH
code
HumanEval, MBPP
constraint satisfaction
4×4 Sudoku
training-domain sampling ratio
2:1:1 (math : code : Sudoku)
math training prompts
4000
code training prompts
500
Appendix
Table 5: Training and evaluation benchmarks.
config
value
residual channels B
3
basis functions J
8
coefficient dimension n=BJ
24
policy hidden dimension
128
policy depth
2 linear layers
activation
SiLU
Appendix
Table 6: Training configuration for CORD.
config
value
generation length L
512
number of decoding steps T
512
positions revealed per step
1
maximum sequence length
2048
decoding schedule
pure any-order
position selection
top- 1 by reveal score
Appendix
Table 7: Default decoding configuration.
Method
Remask
Backbone Update
GSM8K
MATH-500
HumanEval
MBPP
CORE ( Zhai et al., 2026 )
✓
✗
52.7
17.1
17.1
24.8
SCOPE+D3IM ( Yu et al., 2026 )
✓
✓
73.8
30.0
34.8
36.6
RemeDi (Remask SFT) ( Huang et al., 2026b )
✓
✓
86.3
52.2
71.3
57.8
CORD
✗
✗
78.2
34.9
42.7
48.9
Appendix
Table 8: Comparison with remasking-based decoding methods. Reported task performance (%) on overlapping reasoning and code benchmarks. CORD uses strictly monotonic decoding: once a token is revealed, it is never remasked or revised. Results from prior work are reproduced from their respective papers and are not directly comparable because model checkpoints, prompting, and decoding budgets differ.
Figure 4: A GSM8K rollout. Top: token-unmasking timeline for a held-out mathematical reasoning example. The realized reveal order is predominantly sequential, but with a few targeted jumps that expose useful later positions before the entire prefix has been completed. Bottom: the corresponding coefficient and state-feature trajectories. The direction coefficient remains strongly negative throughout the rollout, indicating a persistent left-to-right bias, while the bridge coefficient stays positive and the proximity coefficient gradually increases. Together, these signals produce a mostly local reasoning trajectory with occasional non-local or two-sided reveals when the evolving context makes them useful.
Figure 5: A MBPP rollout. Top: token-unmasking timeline for a held-out code-generation example. The reveal order is again largely structured and directional, but includes a small number of non-local reveals that help establish later code structure. Bottom: the corresponding coefficient and state-feature trajectories. Here the direction coefficient provides the dominant bias, while the proximity and bridge coefficients remain weaker and evolve more smoothly. The resulting trajectory reflects a coding regime in which the model largely builds the program in order, while still allowing selective departures from strict sequential decoding when they improve the surrounding context.
Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.
Masked diffusion language models (MDLMs) decode by repeatedly committing tokens to masked positions, but these commitments are usually irreversible. A token chosen under sparse, partial context is kept fixed, even when later context no longer supports it. Existing samplers mainly decide when to commit a token, but rarely check whether an already committed token should still be kept, allowing early mistakes to propagate. We trace this issue to confidence drift, where the model's confidence in a committed token drops from its sparse commit-time context to the denser context available later. Based on this signal, we propose CoDR (Confidence Drift Remasking), a training-free and sampler-agnostic refinement pass. CoDR estimates drift for all committed positions in only k forward passes via k-partition probing, then remasks and regenerates only the tokens the model no longer endorses. Across two backbones, four reasoning and coding tasks, and three base samplers, CoDR improves average accuracy across all evaluated model-sampler configurations and improves most individual task settings with modest overhead. Controlled experiments show that the gains come from targeted confidence-drift remasking rather than extra compute alone, and that CoDR uses far fewer forward passes than prior remasking methods. Code is available at https://github.com/YueWu0301/CoDR.
Yue Wu, Qinghe Zhang, Yu Zhang +1
The Hong Kong Polytechnic University, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China
Masked diffusion language models (MDLMs) can generate text efficiently by predicting multiple masked tokens in parallel, but predictions from the same forward pass are not necessarily reliable when committed together. We study when parallel commitment is reliable. Our diagnostics show that confidence alone does not determine a reliable commitment order: confident predictions near the end of the sequence can fix an answer before its supporting computations are established, and downstream predictions become less reliable as the uncertainty of their upstream context grows. At the same time, a single forward pass can already resolve several masked tokens, and predictions that remain stable across the final layers are more likely to be correct. Based on these findings, we propose Reliable Parallel Decoding (RPD), a training-free method that selects candidates by layerwise prediction stability and final confidence, and commits them under a cumulative entropy budget over their preceding masked positions. RPD defers predictions with uncertain upstream context while committing the remaining candidates in parallel, without relying on a fixed block schedule. Across mathematical reasoning and code generation benchmarks on LLaDA and Dream, RPD achieves the highest decoding throughput among the evaluated methods while maintaining or improving accuracy.
Zhenghao He, Bohan Liu, Guangzhi Xiong +1
Department of Computer Science, University of Virginia
Masked diffusion language models (MDLMs) generate text by iteratively unmasking tokens, but their standard decoder reduces each step to a binary action: a position is either committed to a single token or left fully masked, with no representation of partial belief in between. This all-or-nothing regime discards rich predictive information and forces premature, irrevocable commitments, leading to poor performance under a limited decoding budget. In this paper, we reinterpret mask prediction as clean-state prediction (x-prediction) and show that it can be used to induce a continuous flow in input embedding space. Building on this view, we propose a continuous decoding framework for MDLMs where tokens can accumulate partial progress at each diffusion step and remain revisable. To match the uneven contextual constraints across positions in language, we replace the globally synchronous schedule in image diffusion with a confidence-based asynchronous update in which the diffusion progress is token-wise accumulated. Additionally, we introduce a lightweight policy network and formulate its training as a reinforcement learning problem. Applied to pretrained LLaDA, our continuous decoder reaches 97% of its performance on the HumanEval dataset with 25% of decoding budget.
Weitian Wang, Lianlei Shan, Shubham Rai +2
Robert Bosch GmbH, Germany · University of the Chinese Academy of Sciences, China · Ruhr University Bochum, Germany