Cord: Confidence-Anchored Order Residual Decoding for Masked Diffusion Language Models

Chengwei Zhou; Zhaoyan Jia; Massoud Pedram; Gourav Datta

cs.CLOct 29, 2026

Cord: Confidence-Anchored Order Residual Decoding for Masked Diffusion Language Models

Authors: Chengwei Zhou, Zhaoyan Jia, Massoud Pedram, Gourav Datta

Organizations: Case Western Reserve University, Cleveland, USA · University of Southern California, Los Angeles, USA

Abstract

The decoding order of masked diffusion language models (dLLMs) can substantially affect generation quality, yet common decoders typically apply the same reveal rule throughout generation. Confidence provides a strong default, but it measures how decisively a position can be predicted now, not how useful revealing that position will be for subsequent decoding. We propose Cord, a lightweight prompt-conditioned policy that retains confidence decoding as a stable reference and learns when and how to depart from it through geometric residuals over generation direction, proximity to revealed context, and two-sided infilling. The residuals adapt to both the prompt and the evolving partial response. Rather than sampling a new policy action at every decoding step, Cord samples a single low-dimensional coefficient matrix per rollout, which induces an entire state-dependent reveal trajectory. This yields a contextual-bandit formulation trained directly from sequence-level rewards, without step-level credit assignment or backbone updates. Across three dLLMs and five reasoning, code, and constraint-satisfaction benchmarks, Cord improves average task success from 32.0% under confidence decoding to 48.2%, and further improves over the best per-domain fixed residual. Behavioral audits show that the learned residuals induce substantial, task-dependent changes in reveal locality and two-sided infilling. Cord adds only $0.53$ M trainable parameters, requires no additional backbone forward passes, and incurs only 5.39% inference latency overhead.

Figures & tables

Figure 1: Cord . Top: a prompt-conditioned MLP produces state-dependent geometric residuals: direction, proximity, and bridge, that modify confidence-based reveal scores while keeping the dLLM frozen. Bottom left: CORD reveals the highest-scoring masked position at each step. Bottom right: training samples one low-dimensional coefficient program per rollout and optimizes it from sequence-level rewards as a contextual bandit.

Figure 2: Oracle reveal diagnostics with frozen LLaDA. GSM8K favors a contiguous reasoning prefix, while MBPP benefits from structural anchors. Conservative uses core control-flow tokens (e.g. if , else , for ); Extended additionally includes function ( def , class ), delimiter, and assignment tokens.

Backbone	Decoding policy	GSM8K	MATH500	HumanEval	MBPP	Sudoku	Avg.
LLaDA-8B-Instruct	Confidence	58.7	18.0	27.8	38.4	17.1	32.0
	Entropy	59.4	12.5	27.5	41.7	3.3	28.9
	Margin ( Kim et al., 2025 )	58.4	22.2	28.8	42.1	25.8	35.5
	EB-Sampler ( Ben-Hamu et al., 2025 )	61.4	12.7	27.1	42.7	33.8	35.5
	Semi-AR † ( Nie et al., 2025 )	76.8	28.1	38.9	45.2	0.3	37.9
	UNCODE ( Huang et al., 2026a )	79.2	34.8	42.1	47.8	29.8	46.7

Table 1: Main results across masked diffusion language models. Task success (%, higher is better). HumanEval and MBPP report pass@1; all other tasks report accuracy. The fixed residual uses the same geometric signals as CORD but keeps their coefficients constant throughout decoding.

Variant	LLaDA-Inst.	LLaDA-1.5	Dream	Avg.
Confidence only	32.0	33.3	26.4	30.6
No prompt (fixed residual)	48.0	49.8	40.5	46.1
Prompt only (static)	42.8	48.1	47.1	46.0
Time only	46.3	45.1	49.2	46.9
State only	47.1	46.4	33.1	42.2
Full CORD	48.2	50.7	52.5	50.5

Table 2: Decomposing adaptation in CORD. Mean task success (%) across all benchmarks.

Variant	GSM8K	MBPP	Sudoku	Avg.
$-\mathrm{Direction}$	57.2	43.7	35.3	45.4
$-\mathrm{Proximity}$	70.2	44.8	31.8	48.9
$-\mathrm{Bridge}$	72.4	41.4	16.6	43.5
Full CORD	78.2	48.9	36.1	54.4

Table 3: Ablation of geometric residuals. Task success (%) on LLaDA-8B-Instruct.

Figure 3: Realized reveal behavior of CORD.

\Delta J<0

indicates more local decoding than confidence, whereas

\Delta J>0

indicates more non-local decoding. Positive

E_{\mathrm{bridge}}

indicates preferential selection of two-sided masked positions beyond their availability.

Method	Aux. Params	Latency (s)	Overhead
Confidence	0	6.31 $\pm$ 0.31	–
Fixed residual	0	6.55 $\pm$ 0.28	3.80%
CORD	527,512	6.65 $\pm$ 0.30	5.39%

Table 4: Inference efficiency on LLaDA-8B-Instruct. Latency (s) is measured on one H200 over

20

runs with

256

generated tokens and a maximum sequence length of

1024

; we report mean

\pm

standard deviation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

config	value
math	GSM8K, MATH
code	HumanEval, MBPP
constraint satisfaction	$4{\times}4$ Sudoku
training-domain sampling ratio	$2{:}1{:}1$ (math : code : Sudoku)
math training prompts	4000
code training prompts	500

Appendix

Table 5: Training and evaluation benchmarks.

config	value
residual channels $B$	$3$
basis functions $J$	$8$
coefficient dimension $n=BJ$	$24$
policy hidden dimension	$128$
policy depth	$2$ linear layers
activation	SiLU

Appendix

Table 6: Training configuration for CORD.

config	value
generation length $L$	$512$
number of decoding steps $T$	$512$
positions revealed per step	$1$
maximum sequence length	$2048$
decoding schedule	pure any-order
position selection	top- $1$ by reveal score

Appendix

Table 7: Default decoding configuration.

Method	Remask	Backbone Update	GSM8K	MATH-500	HumanEval	MBPP
CORE ( Zhai et al., 2026 )	✓	✗	52.7	17.1	17.1	24.8
SCOPE+D3IM ( Yu et al., 2026 )	✓	✓	73.8	30.0	34.8	36.6
RemeDi (Remask SFT) ( Huang et al., 2026b )	✓	✓	86.3	52.2	71.3	57.8
CORD	✗	✗	78.2	34.9	42.7	48.9

Appendix

Table 8: Comparison with remasking-based decoding methods. Reported task performance (%) on overlapping reasoning and code benchmarks. CORD uses strictly monotonic decoding: once a token is revealed, it is never remasked or revised. Results from prior work are reproduced from their respective papers and are not directly comparable because model checkpoints, prompting, and decoding budgets differ.

Figure 4: A GSM8K rollout. Top: token-unmasking timeline for a held-out mathematical reasoning example. The realized reveal order is predominantly sequential, but with a few targeted jumps that expose useful later positions before the entire prefix has been completed. Bottom: the corresponding coefficient and state-feature trajectories. The direction coefficient remains strongly negative throughout the rollout, indicating a persistent left-to-right bias, while the bridge coefficient stays positive and the proximity coefficient gradually increases. Together, these signals produce a mostly local reasoning trajectory with occasional non-local or two-sided reveals when the evolving context makes them useful.

Figure 5: A MBPP rollout. Top: token-unmasking timeline for a held-out code-generation example. The reveal order is again largely structured and directional, but includes a small number of non-local reveals that help establish later code structure. Bottom: the corresponding coefficient and state-feature trajectories. Here the direction coefficient provides the dominant bias, while the proximity and bridge coefficients remain weaker and evolve more smoothly. The resulting trajectory reflects a coding regime in which the model largely builds the program in order, while still allowing selective departures from strict sequential decoding when they improve the surrounding context.

\bm{\zeta}^{(i)}\leftarrow\operatorname{detach}\!\left(\bm{\mu}_{\psi}(x)+\sigma\bm{\epsilon}^{(i)}\right)

Appendix

Algorithm 1 Cord training in coefficient space.

Diffusion Language Model Decoding

Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.

See all 251 papers

CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models
Sep 28, 2026Yue Wu, Qinghe Zhang, Yu Zhang +1Diffusion Language Model Decoding Masked Diffusion Models
The Hong Kong Polytechnic University, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China
Reliable Parallel Decoding in Masked Diffusion Language Models
Sep 29, 2026Zhenghao He, Bohan Liu, Guangzhi Xiong +1Masked Diffusion Models Diffusion Language Model Decoding
Department of Computer Science, University of Virginia
Masked Diffusion Decoding as $x$ -Prediction Flow
Jun 27, 2026Weitian Wang, Lianlei Shan, Shubham Rai +2Masked Diffusion Models Diffusion Language Model Decoding
Robert Bosch GmbH, Germany · University of the Chinese Academy of Sciences, China · Ruhr University Bochum, Germany

Sep 28, 2026J/K move · Enter open · S save

CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models

Yue Wu, Qinghe Zhang, Yu Zhang, Jian Huang

The Hong Kong Polytechnic University, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China

Diffusion Language Model Decoding Masked Diffusion Models

Open paper AskPDF arXiv

Masked diffusion language models (MDLMs) decode by repeatedly committing tokens to masked positions, but these commitments are usually irreversible. A token chosen under sparse, partial context is kept fixed, even when later context no longer supports it. Existing samplers mainly decide when to commit a token, but rarely check whether an already committed token should still be kept, allowing early mistakes to propagate. We trace this issue to confidence drift, where the model's confidence in a committed token drops from its sparse commit-time context to the denser context available later. Based on this signal, we propose CoDR (Confidence Drift Remasking), a training-free and sampler-agnostic refinement pass. CoDR estimates drift for all committed positions in only k forward passes via k-partition probing, then remasks and regenerates only the tokens the model no longer endorses. Across two backbones, four reasoning and coding tasks, and three base samplers, CoDR improves average accuracy across all evaluated model-sampler configurations and improves most individual task settings with modest overhead. Controlled experiments show that the gains come from targeted confidence-drift remasking rather than extra compute alone, and that CoDR uses far fewer forward passes than prior remasking methods. Code is available at https://github.com/YueWu0301/CoDR.

Abstract

Diffusion Language Model Decoding

Explore similar work

CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models

Reliable Parallel Decoding in Masked Diffusion Language Models

Masked Diffusion Decoding as xxx-Prediction Flow

Masked Diffusion Decoding as $x$ -Prediction Flow