While Diffusion Language Models (DLMs) are theoretically well-suited for iterative refinement due to their non-causal structure, off-the-shelf masked diffusion language models (MDLMs) often fail to reliably revise incorrect tokens in practice. The key challenge is that the model cannot tell correct from erroneous visible tokens: standard MDLM training supervises only masked positions, undermining confidence-guided refinement. We study corrective behavior in DLMs, the ability to assign lower confidence to incorrect tokens and iteratively refine them while preserving correct content. Under matched compute and data, we show that conventional masked-diffusion fine-tuning induces much less of this capability than a correction-oriented post-training principle that explicitly supervises visible corrupted tokens, which enables discriminative confidence and targeted refinement. We further find that also supervising clean visible tokens, as in BERT- or GIDD-style training, stops remask-based refinement from improving with additional steps. To evaluate it, we introduce the Code Revision Benchmark, a controllable, executable benchmark for error localization and in-place correction. The effect transfers from a 0.5B model to LLaDA-8B and persists, with a smaller margin, when corruptions are drawn from the base model's own most plausible predictions. The same error-aware confidence also makes parallel decoding more effective. In HumanEval(+) code completion at 0.5B, mathematical reasoning at 16B, and high-entropy parallel decoding with remasking (ParallelBench), models trained with our objective outperform standard masked-diffusion fine-tuning under matched compute, and outperform the base model on HumanEval(+), MATH, and the high-entropy ParallelBench average; at 8B, neither objective improves from-scratch generation over the base model. Code and data: https://github.com/zhangshuibai/CDLM
Figures & tables
Figure 1: An illustrative example in which the model is equally confident in the correct variable ( i ) and the undefined bug ( j ). Global metrics (Average/Min) fail to detect this failure mode.
Figure 2: CRB corruption types. A canonical program (left, HumanEval/31) is corrupted by type-preserving token replacements of three kinds (right; original token in blue, replacement in red). A corrupted program enters CRB only if it fails the tests.
nreplace=1
nreplace=2
nreplace=3
nreplace=4
nreplace=5
Model
gap
top-1
top-5
gap
top-1
top-5
gap
top-1
top-5
gap
top-1
top-5
gap
top-1
top-5
Dream-7B-Base
0.578
0.094
0.590
0.536
0.084
0.614
0.528
0.084
0.597
0.528
0.088
0.600
0.529
0.076
0.571
LLaDA-8B-Base
0.235
0.178
0.770
0.209
0.198
0.801
0.199
0.172
0.811
0.185
0.159
0.784
0.191
0.162
0.833
Open-dCoder-0.5B
0.106
0.147
0.509
0.100
0.222
0.655
0.085
0.243
0.701
0.089
0.325
0.759
0.093
0.348
0.795
nreplace=1
nreplace=2
nreplace=3
nreplace=4
nreplace=5
Table 1: Evaluation of confidence quality (top block) and iterative correction performance (bottom block) under varying numbers of corrupted tokens nreplace . The top block reports the confidence gap between clean and erroneous tokens ( gap ), together with top-1 and top-5 hit rates measuring how often error tokens appear among the lowest-confidence positions. The bottom block reports Pass@1 after applying confidence-threshold-based iterative refinement for T∈{1,2,4} refinement steps. Each entry is the unweighted mean over the 12 CRB dataset × error-type cells (the gap pools body tokens within a cell), with τ=0.9 and greedy decoding. LLaDA-8B-Base values differ slightly from Table 4 , which uses a different localization protocol (Appendix H.3 ).
Figure 3: Comparison of training supervision in MDLM and CDLM and its implication for corrective behavior. Cross-marked boxes denote masked tokens, and beige boxes denote visible inputs. Standard MDLMs apply supervision only to masked tokens, leaving visible tokens unsupervised even when incorrect. CDLM explicitly supervises corrupted visible tokens and encourages lower confidence on unreliable content. This supervision difference enables error-aware confidence and targeted, in-place correction during iterative refinement.
Figure 4: Error localization on HumanEval. (a) Confidence gap between clean and corrupted tokens across error types (Id: identifier, Lit: literal, Op: operator) and replacement counts; CDLM’s gap exceeds MDLM’s in 14 of 15 cells (exception: Lit, n=3 ). (b) Top- K hit rate of identifying at least one true error among the K lowest-confidence positions at nreplace∈{1,2,4} ; CDLM exceeds MDLM at every K .
Figure 5: Pass@1 under confidence-based refinement, averaged over the four CRB datasets. Left: Pass@1 after four refinement steps ( T=4 , τ=0.9 ) across corruption levels. Right: Mean Pass@1 over all error types and corruption levels vs. refinement depth T (“Step T ”). CDLM outperforms MDLM and the base model at every depth and corruption level.
Table 7
LLaDA-8B-Base
Gap
Hit@1
Pass@1 T=1
Pass@1 T=4
Base (no fine-tuning)
0.265
0.165
0.551
0.595
+ absorbing-only (MDLM)
0.402
0.152
0.600
0.609
+ mixture (CDLM)
0.777
0.608
0.725
0.623
Table 4: Controlled comparison on LLaDA-8B-Base (LoRA, 2000 steps, identical settings except the objective). Gap and Hit@1 on the 541 HumanEval localization records ( nreplace=1 , pooled over error types; Appendix H.1 ); Pass@1 macro-averaged over the 12 dataset × error-type cells ( nreplace=1 , τ=0.9 ). Pass@1 truncates each completion at its first markdown fence, which both fine-tuned arms learned to emit from the corpus; no base completion contains a fence. Raw scores and fence counts are in Appendix H.2 .
Table 5: From-scratch generation across code, math and parallel-decoding benchmarks. Top: HumanEval(+) Pass@1/Pass@10 at 0.5B under vanilla and ReMDM ( Wang et al., 2025 ) decoding (MBPP(+) in Appendix E.1 ). Middle: MATH ( Hendrycks et al., 2021 ) suite (with the MATH-500 split of Lightman et al. (2024) ) for LLaDA2.0-mini-preview (16B) after one epoch of fine-tuning on Nemotron-Math-v2 ( Du et al., 2025 ) ; MDLM corresponds to vanilla SFT ( α=0 ) and CDLM to mixture SFT ( α=0.1 ), with all other training settings matched. Bottom: ParallelBench task-group averages, bold marking the best remask value (per-task results in Table 10 , Appendix E.2 ). On HumanEval(+) and MATH, CDLM improves over MDLM and base; on MBPP(+) both fine-tuned arms stay below base in Pass@1 (Appendix E.1 ).
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
GIDD
CDLM
Framework
Generalized interpolating diffusion
Standard MDLM
Training
From scratch (as published)
Post-training of a pretrained MDLM
Loss
Unified ELBO (KL + Itakura–Saito)
MDLM loss + separable noise term ( λnoise )
Noise schedule
Bell-shaped, varies with t
Fixed mixture probability α on visible tokens
Loss on clean tokens
Yes (all positions supervised)
No (only masked + corrupted)
Decoding
All tokens may change at every step
Remask-then-repredict, localized in-place
Appendix
Table 6: Methodological distinctions between GIDD [ von Rütte et al., 2025 ] and CDLM. While both combine absorbing and uniform noise, the two methods differ in framework, training paradigm, supervision targets, and decoding semantics.
Dataset
MDLM
CDLM
Base
HumanEval
0.157
0.257
0.168
HumanEval+
0.182
0.242
0.196
MBPP
0.114
0.175
0.114
MBPP+
0.137
0.226
0.120
Average
0.148
0.225
0.150
Appendix
Table 7: Pass@1 on refinement-based code correction across standard coding benchmarks for MDLM, CDLM, and the base model. Results are aggregated over all error types and corruption levels using 4 refinement steps with a confidence threshold of 0.9. CDLM consistently outperforms MDLM and the base model across datasets.
Model
T=1
T=2
T=3
T=4
LLaDA-8B-Base
0.550
0.571
0.577
0.580
Dream-7B-Base
0.528
0.548
0.402
0.427
Open-dCoder-0.5B
0.203
0.342
0.344
0.336
Appendix
Table 8: Self-revision under minimal corruption on HumanEval. For each model, we first generate programs and retain only those that pass all tests. A single type-preserving token corruption ( nreplace=1 ) is then applied to each correct program, and the model is evaluated on its ability to revise the corrupted program back to a passing solution using iterative refinement. Reported values correspond to Pass@1 after different numbers of refinement steps.
Method
MBPP
MBPP+
Avg
Pass@1
Pass@10
Pass@1
Pass@10
Pass@1
Pass@10
MDLM (Vanilla)
0.106
0.312
0.177
0.449
0.142
0.381
CDLM (Vanilla)
0.131
0.352
0.207
0.472
0.169
0.412
Base (Vanilla)
0.157
0.350
0.231
0.502
0.194
0.426
Δ MDLM − Base
−0.051
−0.038
−0.054
−0.053
−0.053
−0.046
Δ CDLM − Base
−0.026
+0.002
−0.024
−0.030
−0.025
−0.014
Appendix
Table 9: Pass@1 and Pass@10 on MBPP and MBPP+ under two decoding variants. Vanilla recomputes token confidence at every refinement step, while ReMDM fixes each token’s confidence at its first unmasking step [ Wang et al., 2025 ] . The Avg column averages MBPP and MBPP+. Boldface marks the higher of CDLM and MDLM under the same decoding setting (one training run per arm); Δ rows give the difference to the base model.
Task
Base
MDLM
CDLM
Fixed / Remask
Low-Entropy (6 tasks)
Copy
0.97 / 0.97
1.00 / 1.00
1.00 / 1.00
Reverse
0.60 / 0.63
1.00 / 1.00
1.00 / 1.00
Sort
0.15 / 0.19
0.61 / 0.61
0.60 / 0.64
Insert Index
0.23 / 0.24
1.00 / 1.00
0.99 / 0.99
Appendix
Table 10: Per-task accuracy on ParallelBench [ Kang et al., 2026 ] for LLaDA2.0-mini (16B) [ Bie et al., 2025 ] under confidence-based decoding. Each entry reports Fixed / Remask ; bold marks the best value within each row. Tasks are grouped by output entropy: low-entropy tasks admit one valid output (e.g., Copy), high-entropy admit many (e.g., Shuffle). Implementation details are in Appendix E.2 .
0.5B code main comparison, seeds
0.5B code α sweep, CDLM + ctw
LLaDA-8B LoRA
16B MATH
16B ParallelBench
Training
Base model
Open-dCoder-0.5B
Open-dCoder-0.5B
LLaDA-8B-Base
LLaDA2.0-mini-preview
LLaDA2.0-mini
Data
Nemotron-SFT-Code
Nemotron-SFT-Code
Nemotron-SFT-Code, 12,000 documents (3.55M tokens), each seen twice
Nemotron-Math-v2, 300K samples
50,000 ParallelBench examples
Length
2000 steps
2000 steps
2000 steps
1 epoch (9,306 steps)
6 epochs
Batch
3×4×1=12 , ×4096 ; seeds 1234, 2025: 3×2×2
3×4×1=12 , ×4096
1×4×3=12 , ×4096
4×8×1=32 , ×2048
–
Optimizer
AdamW (0.9, 0.95), wd 0.01, clip 1.0
same
AdamW (0.9, 0.95), wd 0.01 (not on 1-D), clip 1.0
AdamW (0.9, 0.999), wd 0.1, clip 1.0
–
Appendix
Table 11: Training and decoding settings per experiment. Objective weights are those of the CDLM arm; every MDLM arm has α=0 , so no position is replaced and λnoise has no effect. Batch: micro-batch × GPUs × accumulation steps = global batch, × maximum length. LR: peak; warmup; decay. “–”: not recorded. The Sudoku settings are in Appendix K .
Nemotron
HumanEval
HumanEval+
MBPP
MBPP+
Sequences
2000
164
163
499
378
Mean tokens
91.8
165.8
165.8
79.0
65.3
Base
1.341±0.670
0.771±0.338
0.838±0.330
0.834±0.398
0.889±0.373
MDLM
1.097±0.646
0.785±0.340
0.850±0.334
0.845±0.406
0.899±0.381
CDLM
1.072±0.660
0.808±0.352
0.883±0.350
0.886±0.434
0.950±0.412
MDLM − Base
−0.243±0.007
+0.013±0.004
+0.012±0.004
+0.011±0.003
+0.010±0.005
Appendix
Table 12: Masked-reconstruction loss (nats/token) on a held-out slice of the fine-tuning corpus (Nemotron) and on the reference programs of the four code benchmarks. Top: mean ± standard deviation over sequences. Middle: paired difference to the base model, mean ± half-width of the 95% confidence interval over sequences; a positive value means the model became worse at reconstructing that distribution. Bottom: two-sided one-sample t -test of the paired differences. Each sequence’s loss averages mask ratios {0.15,0.3,0.5,0.7}×3 mask seeds.
T
N
CDLM
Comparator
Δ
95% CI
b/c
p
CDLM vs. MDLM
1
9,087
0.095
0.066
+0.029
[+0.024,+0.035]
424/158
4.6×10−29
2
9,087
0.216
0.147
+0.070
[+0.062,+0.078]
1032/399
8.4×10−65
3
9,087
0.225
0.163
+0.062
[+0.054,+0.070]
1018/453
3.6×10−50
4
9,087
0.231
0.164
+0.067
[+0.059,+0.075]
1052/442
1.6×10−57
CDLM vs. Base
Appendix
Table 13: Paired tests over CRB problems, pooled over 4 datasets × 3 error types ×nreplace∈{1,…,5} ( τ=0.9 , greedy decoding). Δ is CDLM minus the comparator in Pass@1 (pooled over items); b/c are the discordant counts (only CDLM passes / only the comparator passes); CI from 10,000 paired bootstrap resamples; p from the exact McNemar test.
Comparator
nreplace
N
T=1
T=2
T=3
T=4
MDLM
1
3,180
+0.045
+0.042
+0.036
+0.045
2
2,429
+0.030
+0.089
+0.079
+0.081
3
1,478
+0.020
+0.085
+0.070
+0.076
4
1,154
(+0.011)
+0.097
+0.089
+0.092
5
846
(+0.007)
+0.053
+0.058
+0.060
Base
1
3,174
+0.039
+0.040
+0.032
+0.035
Appendix
Table 14: The 60-test family: Pass@1 difference (CDLM minus comparator) for each nreplace and refinement depth T , pooled over 4 datasets × 3 error types. N is the number of paired programs (constant across T ). CDLM is the main CDLM-0.5B checkpoint; its comparison with CDLM + ctw also reflects their different learning-rate schedules (Appendix F.3 ). Entries in parentheses do not survive Benjamini–Hochberg correction over the 60 tests at q<0.05 ; all other entries do.
Dataset
N
T=1
T=2
T=3
T=4
HumanEval
1,473
−0.020
−0.007
−0.001
+0.012
( 0.037 )
( 0.60 )
( 1.0 )
( 0.41 )
HumanEval+
1,567
−0.010
+0.202
+0.229
+0.230
( 0.25 )
( 1.3×10−56 )
( 1.5×10−69 )
( 4.9×10−69 )
MBPP
3,983
−0.018
+0.021
+0.015
+0.017
( 0.0010 )
( 0.0050 )
( 0.042 )
( 0.021 )
Appendix
Table 15: CDLM-0.5B (main checkpoint, longer schedule) versus CDLM + ctw by dataset: Pass@1 difference CDLM minus CDLM + ctw (first row) and exact McNemar p (second row, in parentheses), pooled over 3 error types ×nreplace∈{1,…,5} . Negative values favor CDLM + ctw. N is constant across T .
CDLM vs. MDLM ( N=6,204 )
CDLM vs. Base ( N=6,133 )
T
Δ
95% CI
p
Δ
95% CI
p
1
+0.027
[+0.021,+0.032]
4.4×10−22
+0.023
[+0.017,+0.028]
2.1×10−15
2
+0.068
[+0.059,+0.077]
3.9×10−46
+0.055
[+0.045,+0.065]
1.5×10−28
3
+0.058
[+0.049,+0.068]
6.3×10−33
+0.051
[+0.041,+0.060]
3.3×10−23
4
+0.065
[+0.056,+0.075]
1.4×10−40
+0.056
[+0.046,+0.066]
1.3×10−27
Appendix
Table 16: Identical-stimulus subset: only programs whose corrupted text is byte-identical in both arms, pooled over 4 datasets × 3 error types ×nreplace∈{1,…,5} . Δ is CDLM minus the comparator in Pass@1; p from the exact McNemar test.
T
N
CDLM
CDLM + ctw
Δ
95% CI
b/c
p
1
3,213
0.185
0.196
−0.012
[−0.025,+0.001]
206/244
0.081
2
3,213
0.214
0.198
+0.016
[+0.000,+0.031]
334/283
0.044
3
3,213
0.211
0.190
+0.021
[+0.006,+0.036]
335/266
0.0055
4
3,213
0.205
0.190
+0.016
[+0.001,+0.031]
323/272
0.040
Appendix
Table 17: Clean-token ablation with matched learning-rate schedules (Table 3 ; nreplace=1 , pooled over 4 datasets × 3 error types, N=3,213 paired programs). Δ is CDLM minus CDLM + ctw in Pass@1 (pooled over items, so the levels differ from the macro averages of Table 3 ); b/c are the discordant counts; CI from 10,000 paired bootstrap resamples; p from the exact McNemar test.
Pass@1
Arm
Seed (GPUs)
T=1
T=4
Gap
Hit@1
CDLM
42 (4)
0.1888
0.2853
0.1636
0.2277
1234 (2)
0.1829
0.2884
0.1497
0.2126
2025 (2)
0.1943
0.2836
0.1664
0.2295
mean ± SD
0.1887±0.0057
0.2858±0.0024
0.1599±0.0090
0.2233±0.0093
MDLM
42 (4)
0.1386
0.2445
0.0976
0.1470
Appendix
Table 18: Three training seeds per arm (0.5B, nreplace=1 , τ=0.9 , macro average over the 12 CRB cells). The number of training GPUs is given in parentheses; the global batch is 12 in every run. Values have four decimals so that the smallest SD is visible. SD is the sample standard deviation across the three training seeds, not the standard error of the mean. Pooled SD is (sCDLM2+sMDLM2)/2 ; the min./max. row is the worst CDLM seed minus the best MDLM seed. Gap and Hit@1 come from the first forward pass and do not depend on T .
Error type
Positions
Median rank
In top 5
Geo. mean prob.
Ratio
Programs
Operator
200 / 176
20.5 / 2
19.5% / 100%
9.7×10−6 / 9.5×10−3
975.5
1,465 / 997
Identifier
131 / 98
39 / 2
14.5% / 100%
7.0×10−6 / 3.5×10−3
494.5
934 / 728
Literal
155 / 200
3 / 2
81.3% / 100%
5.4×10−4 / 7.1×10−3
13.2
814 / 1,036
Appendix
Table 19: How plausible the corruptions are, under the shared base checkpoint (Open-dCoder-0.5B) with the corrupted position masked. Each entry gives random / plausible. Position counts, ranks, top-5 fractions and probabilities are measured on HumanEval (up to 200 corrupted programs per file); rank 1 is the most probable token. Ratio: plausible over random geometric-mean probability. The last column counts the test-failing programs over all four datasets that form the benchmark. Measured with the function body as the only context, as in the construction; with the full program visible, the plausible token is in the top 5 for 87.5% (operator), 75.5% (identifier) and 98.0% (literal) of positions.
Random CRB
Plausible CRB
Metric
Base
MDLM
CDLM
Δ
Base
MDLM
CDLM
Δ
Retained
Gap
0.099
0.094
0.158
+ 0.064
0.054
0.051
0.084
+ 0.033
52.1%
Hit@1
0.148
0.147
0.222
+ 0.075
0.093
0.092
0.129
+ 0.036
48.5%
Hit@3
0.337
0.343
0.452
+ 0.109
0.242
0.240
0.309
+ 0.069
63.2%
Hit@6
0.570
0.570
0.662
+ 0.092
0.456
0.434
0.512
+ 0.078
85.3%
Pass@1, T=1
0.140
0.140
0.185
+ 0.044
0.099
0.087
0.116
+ 0.029
66.2%
Appendix
Table 20: Random versus plausible CRB ( nreplace=1 , τ=0.9 ). Every entry is the macro average over the 12 dataset × error-type cells; within a regime all three arms see the same corrupted programs (3,213 random, 2,761 plausible). Gap and Hit@ K are read at the first forward pass. Δ : CDLM − MDLM, computed before rounding. Retained: plausible Δ divided by random Δ . The literal cells, where random replacement was already close to plausible (Table 19 ), are included; without them the retained fractions are 52.5% (gap), 48.6% (Hit@1), 68.2% (Hit@3), 85.9% (Hit@6), 64.5% (Pass@1, T=1 ) and 64.8% (Pass@1, T=4 ). One training seed per arm and one corruption draw.
Gap
Hit@1
Pass@1, T=1
Pass@1, T=4
Type
Arm
Rand.
Plaus.
Rand.
Plaus.
Rand.
Plaus.
Rand.
Plaus.
Operator
Base
0.086
0.034
0.140
0.083
0.143
0.087
0.280
0.194
MDLM
0.080
0.029
0.137
0.076
0.144
0.075
0.264
0.168
CDLM
0.146
0.049
0.221
0.101
0.181
0.085
0.307
0.187
retained
31.1%
30.0%
26.7%
45.4%
Identifier
Base
0.134
0.111
0.188
0.148
0.164
0.176
0.279
0.267
Appendix
Table 21: Plausible CRB by error type. Each entry is the unweighted mean over the four datasets ( nreplace=1 , τ=0.9 ); Rand. = random CRB, Plaus. = plausible CRB. Retained : CDLM − MDLM under plausible corruption divided by CDLM − MDLM under random corruption. For literal corruptions, random replacement was already close to plausible (Table 19 ), and the random-regime Pass@1 margin at T=1 is only +0.013 , so the literal retained fractions carry little information.
Pass@1
Modified
Failing set
Arm
T=1
T=4
T=16
T=1
T=4
Base-generated ( n=940 )
Base
0.000 (0)
0.000 (0)
0.000 (0)
20.6%
28.2%
MDLM
0.001 (1)
0.001 (1)
0.001 (1)
12.4%
20.3%
CDLM
0.001 (1)
0.001 (1)
0.001 (1)
16.0%
19.8%
CDLM-generated ( n=865 )
Base
0.001 (1)
0.003 (3)
n/a
23.6%
30.1%
MDLM
0.002 (2)
0.001 (1)
n/a
15.5%
22.2%
Appendix
Table 22: In-place repair of naturally occurring failures (Open-dCoder-0.5B arms, τ=0.9 ). Pass@1 is pooled over the programs of the four datasets; the number of repaired programs is in parentheses. Modified: fraction of programs whose output differs from the input after T rounds. T=16 was run on the base-generated set only.
Set
Programs
Median
Quartiles
Within 4
Mean length
Base-generated, HumanEval
136
109
94–116
0 (0.0%)
113
Base-generated, HumanEval+
136
110
96–117
0 (0.0%)
112
Base-generated, MBPP
404
41.5
26–97
0 (0.0%)
61
Base-generated, MBPP+
264
34
23–71
4 (1.5%)
55
Base-generated, all
940
59
29–110
4 (0.4%)
74
CDLM-generated, all
865
55
29–104
5 (0.6%)
68
Appendix
Table 23: Token-level Levenshtein distance from the failing (or corrupted) body to the canonical body, Open-dCoder tokenizer. Quartiles by linear interpolation. Within 4: programs at distance at most 4 tokens. CRB rows: all test-failing nreplace=1 instances of the plausible-CRB study, over the four datasets and three error types.
Setting
Value
Base model
GSAI-ML/LLaDA-8B-Base , revision 0f2787f2
Adapter
LoRA, rank 64, α=128 (scaling 2), dropout 0.05, on the 7 linear projections of each block
Frozen
all base weights, including embeddings and output layer
Trainable parameters
167.77M (2.05%)
Corpus
Nemotron-SFT-Code: 12,000 documents, 3.55M tokens, each document seen twice
Sequence length
at most 4096 tokens, one document per sequence
Appendix
Table 24: LLaDA-8B-Base post-training configuration, identical in both arms except for the last two rows.
T=1
T=4
Arm
HE
HE+
MBPP
MBPP+
HE
HE+
MBPP
MBPP+
Base
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
MDLM
5.6
78.7
0.4
0.5
9.9
88.5
1.5
1.6
CDLM
8.9
94.9
1.6
1.1
10.1
96.7
5.3
4.4
Appendix
Table 25: Percentage of refined programs that contain a code fence ( ``` ), per dataset, pooled over the three error types ( nreplace=1 ; 497, 511, 1,373 and 803 programs for HumanEval, HumanEval+, MBPP and MBPP+).
HumanEval
HumanEval+
MBPP
MBPP+
T
Arm
op
id
lit
op
id
lit
op
id
lit
op
id
lit
Macro
Raw
1
Base
0.621
0.351
0.775
0.616
0.379
0.824
0.533
0.307
0.646
0.515
0.314
0.734
0.551
1
MDLM
0.638
0.527
0.697
0.129
0.117
0.131
0.606
0.442
0.655
0.640
0.473
0.650
0.475
1
CDLM
0.763
0.756
0.704
0.016
0.029
0.026
0.708
0.609
0.622
0.749
0.714
0.667
0.530
4
Base
0.674
0.412
0.817
0.682
0.437
0.843
0.558
0.333
0.673
0.596
0.368
0.746
0.595
Appendix
Table 26: CRB Pass@1 of the three LLaDA-8B arms per (dataset, error type) cell ( nreplace=1 , τ=0.9 ), raw and de-fenced. op, id and lit denote operator, identifier and literal errors; Macro is the unweighted mean over the 12 cells.
De-fenced
Raw
Comparison
T
Difference
p
Difference
p
CDLM − MDLM
1
+0.115
1.8×10−32
+0.072
4.6×10−15
CDLM − MDLM
4
+0.038
1.1×10−4
+0.067
3.4×10−14
CDLM − Base
1
+0.177
2.0×10−59
+0.052
7.9×10−6
CDLM − Base
4
+0.064
4.9×10−9
−0.005
0.68
MDLM − Base
1
+0.062
1.2×10−10
−0.020
0.057
Appendix
Table 27: Paired comparisons of the LLaDA-8B arms over the 3,184 CRB programs ( nreplace=1 ): difference in the fraction of programs repaired (pooled over cells) and exact McNemar p -value.
pθ(zi∣z)
maxvpθ(v∣z)
Arm
Steps
Clean
Corr.
Gap
Hit@1
Hit@5
Clean
Corr.
Gap
Hit@1
Base
0
0.970
0.705
0.265
0.165
0.771
0.988
0.791
0.197
0.325
MDLM
2000
0.970
0.567
0.402
0.152
0.874
0.990
0.771
0.219
0.460
CDLM
2000
0.969
0.192
0.777
0.608
0.980
0.983
0.866
0.117
0.153
MDLM
200
0.963
0.705
0.258
0.105
0.806
0.986
0.804
0.181
0.287
CDLM
200
0.962
0.173
0.790
0.530
0.970
0.989
0.889
0.099
0.196
Appendix
Table 28: Localization on CRB HumanEval ( nreplace=1 , n=541 records, three error types pooled) under two confidence scores: the probability of the present token pθ(zi∣z) , which the remasking rule uses, and the maximum predicted probability. Clean and Corr. are mean scores at clean and corrupted positions; Gap is their difference averaged per record. The 200-step rows are separate runs with the same configuration.
Operator ( n=250 )
Identifier ( n=136 )
Literal ( n=155 )
Arm
Gap
Hit@1
Gap
Hit@1
Gap
Hit@1
Base
0.269
0.220
0.133
0.096
0.376
0.135
MDLM
0.426
0.148
0.303
0.147
0.452
0.161
CDLM
0.794
0.660
0.857
0.801
0.681
0.355
Appendix
Table 29: Confidence gap and Hit@1 under pθ(zi∣z) per error type on CRB HumanEval ( nreplace=1 , 2000-step adapters).
Hit@1
Pooled
Arm
HE
HE+
MBPP
MBPP+
Hit@1
Hit@5
Gap
Base
0.165
0.209
0.147
0.200
0.173
0.773
0.227
MDLM
0.152
0.345
0.443
0.480
0.393
0.892
0.379
CDLM
0.608
0.465
0.747
0.778
0.691
0.962
0.758
Appendix
Table 30: Localization on the CRB localization files of all four datasets ( nreplace=1 , three error types pooled, pθ(zi∣z) , 2000-step adapters). Hit@1 per dataset (541, 536, 1,573 and 909 records) and pooled over all 3,559 records.
Pass@1
All 5 passes
Leading fence
Arm
Dataset
n
T=1
T=4
(%), T=4
T=1
T=4
Base
HumanEval
497
0.582
0.634
66.6
0
0
Base
HumanEval+
511
0.607
0.655
56.4
0
0
Base
MBPP
1373
0.495
0.521
63.5
0
0
Base
MBPP+
803
0.521
0.570
53.2
0
0
MDLM
HumanEval
497
0.647
0.645
44.5
0
4
Appendix
Table 31: Refinement depth per dataset ( nreplace=1 , three error types). Pass@1 is de-fenced, averaged over the three cells of each dataset. “All 5 passes” is the share of programs that still remask a position in the fourth round at T=4 . “Leading fence” counts programs whose first fence precedes all code, which the de-fence rule empties.
MBPP+
Model
Corpus
HumanEval+
Primary
EvalPlus
Task filter
LLaDA-8B-Base
none
0.280
0.529
0.447
0.381
+ MDLM
Nemotron-SFT-Code
0.073
0.270
0.196
0.206
+ CDLM
Nemotron-SFT-Code
0.073
0.246
0.196
0.217
+ MDLM
OpenCodeInstruct
0.128
0.352
0.294
0.294
+ CDLM
OpenCodeInstruct
0.091
0.315
0.249
0.315
Appendix
Table 32: From-scratch generation on HumanEval+ (164 problems) and MBPP+ (378 problems): Pass@1 of one greedy sample per problem, 128 tokens in 128 steps. MBPP+ is scored on the three prompt asserts (primary), on the EvalPlus tests, and on the prompt asserts with the task filter alone.
MBPP+
Comparison
HumanEval+
Primary
EvalPlus
Nemotron-SFT-Code
CDLM − MDLM
0.000 ( 1 )
−0.024 ( 0.42 )
0.000 ( 1 )
MDLM − Base
−0.207 ( 5.4×10−9 )
−0.259 ( 8.2×10−18 )
−0.251 ( 8.4×10−20 )
CDLM − Base
−0.207 ( 1.1×10−9 )
−0.283 ( 1.2×10−21 )
−0.251 ( 2.1×10−19 )
OpenCodeInstruct
Appendix
Table 33: Paired differences on the generation check of Table 32 , with exact McNemar p -values in parentheses.
Figure 6: Effect of the mixture probability α (12 Open-dCoder-0.5B checkpoints, 2000 steps each; HumanEval CRB, mean over 3 error types ×nreplace∈{1,…,5} , τ=0.9 ). (a) Confidence gap. (b) Top-5 hit rate. (c) Top-5 false positive rate. (d) Pass@1 at T=1 and T=4 . Panels (a)–(c) are read at the first forward pass. The dotted line marks α=0.1 , used in all other experiments; these checkpoints use a shorter learning-rate horizon than CDLM-0.5B (Appendix I ).
α
N
Gap
Top-5 hit
Top-5 FP
Pass@1 T=1
Pass@1 T=4
0.04
1,488
0.152
0.811
0.716
0.094
0.257
0.06
1,495
0.146
0.779
0.740
0.091
0.258
0.08
1,496
0.189
0.811
0.708
0.115
0.249
0.1
1,502
0.194
0.807
0.710
0.110
0.240
0.2
1,495
0.216
0.837
0.699
0.119
0.234
0.3
1,490
0.320
0.852
0.659
0.201
0.292
Appendix
Table 34: Full mixture-probability sweep. Open-dCoder-0.5B fine-tuned for 2000 steps per α , all other settings fixed; HumanEval CRB, unweighted mean over 3 error types ×nreplace∈{1,…,5} , τ=0.9 . N : corrupted programs per α , summed over the 15 cells. Gap, Top-5 hit and Top-5 FP (false positive) rate are measured at the first forward pass and do not depend on T . Rules separate the three regions discussed in the text. One training seed per α .
Table 35: CRB for Open-dCoder-0.5B models trained on Nemotron-SFT-Code (Nemotron) and on OpenCodeInstruct (OCI). Pass@1 after T refinement rounds, confidence gap, Hit@1 and Hit@5 (at least one corrupted position among the 1 or 5 lowest-confidence positions), macro-averaged over the dataset × error-type [×nreplace] cells. All models refine the same inputs; the localization metrics do not depend on T . CDLM-0.5B’s T=4 value is 0.285 here and 0.286 in Table 3 ; the difference is one program (HumanEval/139) whose tests run close to the 8 s execution timeout.
Benchmark
Decoder
Base
MDLM-OCI
CDLM-OCI
Difference [95% CI]
HumanEval
vanilla
0.188
0.171
0.216
+0.045 [ +0.015 , +0.077 ]
HumanEval+
vanilla
0.167
0.153
0.196
+0.043 [ +0.013 , +0.073 ]
MBPP
vanilla
0.157
0.121
0.116
−0.005 [ −0.024 , +0.013 ]
MBPP+
vanilla
0.231
0.190
0.203
+0.013 [ −0.010 , +0.037 ]
HumanEval
ReMDM
0.182
0.160
0.221
+0.061 [ +0.024 , +0.099 ]
HumanEval+
ReMDM
0.164
0.144
0.198
+0.054 [ +0.018 , +0.091 ]
Appendix
Table 36: From-scratch generation Pass@1 of the Open-dCoder-0.5B models trained on OpenCodeInstruct, with the protocol of Table 5 . Base is the recorded run of the untrained model behind Tables 5 and 9 , shown to three decimals. The last column is CDLM-OCI minus MDLM-OCI with a paired bootstrap 95% CI over problems (164 for HumanEval and HumanEval+, 500 for MBPP, 378 for MBPP+).
Pass@1, de-fenced
Pass@1, raw
Model
Gap
Hit@1
Hit@5
T=1
T=4
T=1
T=4
LLaDA-8B-Base
0.265
0.165
0.771
0.551
0.595
0.551
0.595
+ MDLM-OCI
0.338
0.102
0.887
0.576
0.590
0.546
0.521
+ CDLM-OCI
0.835
0.593
0.978
0.741
0.726
0.518
0.506
Appendix
Table 37: LLaDA-8B-Base LoRA adapters trained on OpenCodeInstruct (2000 steps), with the protocol of Table 4 , which reports the Nemotron-trained adapters. Gap, Hit@1 and Hit@5 on the HumanEval localization files ( n=541 ); Pass@1 macro over 12 cells at nreplace=1 , de-fenced (primary) and raw. The Base row is the run reported in Table 4 .
Figure 7: Confidence on clean and noisy digits under uniform corruption (log scale). We report clean–noise confidence ratios (yellow annotations) rather than absolute confidence gaps, since confidence values span multiple orders of magnitude in this setting. CDLM exhibits substantially larger clean–noise separation than MDLM across noise ratios, indicating much stronger error localization than the absorbing-only objective.
Model
Type
Pass@1
NFE (median)
gpt-4.1-nano (Plan A, full output)
AR
0.802
89
gpt-4.1-nano (Plan B, localize)
AR
0.790
30
Dream-7B-Base
DLM ( T=1 )
0.776
2
LLaDA-8B-Base
DLM ( T=1 )
0.552
2
LLaDA-8B CDLM †
DLM ( T=1 )
0.725
2
Appendix
Table 38: Pass@1 and NFE on CRB ( nreplace=1 ). The AR model achieves slightly higher Pass@1 but uses 15 – 45× more sequential forward passes than the DLMs, which repair in place with two full-sequence passes ( T=1 ). All rows are macro-averaged over the 12 cells. † LoRA-trained CDLM arm of Section 6.3 , with each program truncated at its first markdown fence (Appendix H.2 ).
Pass@1
Model
T=1
T=4
Gap
Hit@1
Remask (%)
Base (Open-dCoder-0.5B)
0.143
0.251
0.106
0.147
3.09
MDLM (absorbing-only)
0.137
0.237
0.095
0.140
2.95
CDLM (ours)
0.188
0.286
0.169
0.228
2.87
PRISM head, τ=0.9
0.132
0.167
0.087
0.111
5.72
PRISM head, exact edit parity
0.107
0.134
0.087
0.111
2.87 †
Appendix
Table 39: Self-correction head (PRISM) versus CDLM on CRB with the Open-dCoder-0.5B backbone, nreplace=1 , macro-averaged over the 12 dataset × error-type cells. Gap (mean score of clean positions minus mean score of corrupted positions, pooled over the positions of each cell as in Table 1 ; averaged per program as in Tables 18 and 35 , CDLM’s gap is 0.164 ) and Hit@1 are computed from the first-round scores: ci for Base, MDLM and CDLM, and the head’s sigmoid output for PRISM. Remask is the percentage of editable tokens remasked in the first round. † Equal to CDLM’s by construction: the head remasks, in every program and round, as many tokens as CDLM.
Feature
Passes
Hit@1
Hit@5
AUROC
Single-pass confidence ci(0)
1
0.359
0.772
0.891
Flip rate
5
0.150
0.480
0.740
Convergence step
5
0.142
0.489
0.742
Confidence variance
5
0.253
0.622
0.840
Minimum confidence
5
0.296
0.708
0.893
Confidence slope (rising = suspicious)
5
0.434
0.752
0.735
Appendix
Table 40: Trajectory features versus single-pass confidence for error localization on CRB HumanEval: all nine models, three error types and nreplace∈{1,…,5} ( 13,446 programs). Each feature is shown in the better of its two orientations. Passes: forward passes needed to compute the feature.
HumanEval
HumanEval+
MBPP
MBPP+
Pooled
Feature
Hit@1
AUROC
Hit@1
AUROC
Hit@1
AUROC
Hit@1
AUROC
Hit@1
AUROC
Single-pass
0.359
0.891
0.176
0.880
0.324
0.873
0.340
0.857
0.308
0.874
Flip rate
0.150
0.740
0.169
0.765
0.135
0.714
0.161
0.704
0.149
0.725
Convergence step
0.142
0.742
0.095
0.756
0.133
0.721
0.146
0.707
0.131
0.727
Conf. variance
0.253
0.840
0.269
0.881
0.215
0.799
0.236
0.783
0.235
0.816
Min. confidence
0.296
0.893
0.189
0.895
0.279
0.868
0.297
0.856
0.270
0.874
Appendix
Table 41: Hit@1 and AUROC of the features of Table 40 on each CRB dataset and pooled over all four (all nine models, three error types, nreplace∈{1,…,5} ). Programs: 13,446 (HumanEval), 14,273 (HumanEval+), 36,922 (MBPP), 18,691 (MBPP+), 83,332 (pooled).
Hit@1
AUROC
Model
Single-pass
Best traj.
Δ
Single-pass
Best traj.
Δ
Open-dCoder-0.5B (base)
0.242
0.272 c
+0.031
0.840
0.892 m
+0.052
MDLM, 2,000 steps
0.263
0.267 c
+0.004
0.840
0.882 m
+0.042
CDLM, 2,000 steps
0.383
0.395 c
+0.012
0.873
0.906 m
+0.034
CDLM, 10,000 steps
0.433
0.478 m
+0.045
0.839
0.886 m
+0.047
CDLM + ctw
0.472
0.472 c
+0.001
0.852
0.884 m
+0.031
Appendix
Table 42: Per-model breakdown on CRB HumanEval (three error types, nreplace∈{1,…,5} , 1,484 to 1,505 programs per model). “Best traj.” is the highest of nine trajectory-derived scores, each in its primary orientation, chosen per model after the fact. Superscripts name the score: c, z-scored combination of single-pass confidence, flip rate and confidence variance; m, minimum confidence; s, convergence step.
Artifact
Repository
Revision
Open-dCoder-0.5B, full fine-tuning
Base model (third party)
fredzzp/open-dcoder-0.5B
d0d86d5b
MDLM-0.5B
Open-Dcoder-0.5B-baseline-mdm-step2000
fa5eef96
CDLM-0.5B
Open-Dcoder-0.5B-mixture-mdm-step2000
5a7170e7
MDLM-OCI
Open-Dcoder-0.5B-MDLM-OpenCodeInstruct
535b3693
CDLM-OCI
Open-Dcoder-0.5B-CDLM-OpenCodeInstruct
8eb87fe2
Appendix
Table 43: Released models and data on the Hugging Face Hub. Repositories without an owner prefix belong to Shuibai12138 . Revisions are the pinned commits of the release (first eight hexadecimal digits). The two Nemotron adapter repositories also hold the 200-step runs in step200/ . OCI: OpenCodeInstruct.
Masked diffusion language models (MDLMs) re-predict every position at each denoising step, but standard samplers commit tokens once revealed, leaving this revision capability unused. Existing approaches either add heuristic or learned mechanisms to revise committed tokens, or remask them back to [MASK] before re-predicting; a principled sampler that directly revises visible tokens without auxiliary modules remains underexplored. We introduce D3IM, a parameter-free sampler derived as a corrector-style reverse update that permits direct visible-to-visible revision without additional modules or auxiliary passes. D3IM also reveals a model-side obstacle we term preservation bias: the model tends to reproduce its own wrong committed tokens rather than correct them. We address this with SCOPE (Self-Conditioned On Prediction Errors), a lightweight post-training procedure that simulates D3IM's sampling process. On LLaDA-8B at 64 denoising steps, SCOPE+D3IM improves over the original LLaDA-8B with standard unmasking by +13.0 on GSM8K (68.3%), +4.8 on MATH-500 (23.6%), +15.3 on HumanEval (29.3%), and +10.4 on MBPP (30.8%), with gains that increase as more denoising steps are used on math and HumanEval.
Diffusion large language models (dLLMs) gain speed by committing multiple tokens in parallel at each denoising step, but any erroneous commitment persists as conditioning context and biases every subsequent prediction. LLaDA2.1 repairs such errors with Token-to-Token (T2T) editing, which re-examines previously unmasked tokens and overwrites them when an alternative becomes sufficiently confident. We argue that this replacement action is itself the limiting factor: under polluted context, a confident replacement can propagate the error, while under a multimodal posterior no alternative may be confident enough to trigger an edit. We propose Token-to-Mask (T2M) remasking, a training-free rule that revokes suspicious commitments by resetting them to [M] and lets the subsequent mask-filling steps re-predict them from a cleaner context. T2M improves accuracy by +13.33 points on AIME 2025 and +8.56 points on CMATH. These results suggest that, for parallel discrete generators, remasking suspect tokens rather than overwriting them is a more reliable self-correction primitive.
Lin Yao
School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Zhongguancun Academy, Beijing, 100097, China
Diffusion language models (DLMs) can revise tokens bidirectionally, but standard decoding procedures often adapt them to left-to-right generation by producing text block by block. We study a simple plug-and-play inference pattern: first generate a complete draft, then refine the full response using bidirectional diffusion. Using LLaDA2.1-Flash and LLaDA2.1-Mini, we evaluate two configurations. In Flash-Flash, the same Flash model serves as both drafter and refiner, testing whether an existing model can improve its own block-autoregressive output through global refinement. In Mini-Flash, inspired by speculative decoding, we introduce speculative correction: Mini drafts a full response, and Flash revises it as an editable initialization. Flash-Flash improves GSM8K-384 accuracy from 0.848 to 0.899 while running 1.20 times faster than the selected Flash block-autoregressive baseline, and improves MBPP-384 from 0.545 to 0.693. Latency-window-matched Flash-only controls indicate that these gains persist after targeted tuning of block-autoregressive decoding. Causal ablations indicate that completed drafts provide useful initializations: refinement from a fully masked span performs poorly, full global refinement provides a clear additional gain on GSM8K, and local refinement captures much of the gain on MBPP and MATH. Mini-Flash provides useful quality-latency trade-offs, including MATH-384 performance of 0.294 versus 0.300 for Flash while running 2.17 times faster. These results support a Pareto-frontier interpretation rather than the claim that the heterogeneous cascade uniformly matches Flash quality. Overall, same-model draft-and-refine provides evidence that bidirectional refinement is a useful decoding primitive for DLMs, while speculative correction demonstrates a training-free route to fast DLM generation.
Brian K Chen, Chong Wu, Kenji Kawaguchi
National University of Singapore · City University of Hong Kong