Predictive uncertainty in stochastic language-model inference can be characterized by repeatedly sampling an input and measuring the distribution of outputs. This requires multiple complete generations. Masked diffusion language models expose another source of information: the evolution of the output during denoising. We ask whether convergence within diffusion trajectories is associated with uncertainty across their final outputs, or endpoints. We study four masked diffusion language models on four reasoning benchmarks, collecting 20 generations per question and 32,000 denoising trajectories in total. Within each trajectory, we measure within-run concentration as the area under the curve describing stable resolution of the eventual output. Across repeated trajectories, we measure endpoint-distribution uncertainty. Stronger within-run concentration is associated with lower endpoint entropy after accounting for model and dataset (β=−0.222, p<10−25). This association persists for multi-token outputs and when reweighting positive-entropy cases. The relationship is asymmetric: high within-run concentration almost exclusively accompanies concentrated endpoint distributions, whereas lower concentration occurs under both concentrated and competing endpoint regimes. This relationship strengthens as denoising proceeds. Features from a single denoising run predict whether independent held-out runs will concentrate on a common endpoint or produce competing endpoints. These results connect dynamics within individual denoising runs to variability across repeated inference. Within-trajectory dynamics contain information about whether independent repetitions will concentrate on a common endpoint or produce competing endpoints, providing a route towards estimating endpoint-distribution properties with fewer complete generations.
Figures & tables
Region
n
%
within-low / across-high
1013
63.3
within-high / across-high
418
26.1
within-low / across-low
168
10.5
within-high / across-low
1
0.1
Table 1: Four-region counts at τ=0.9 ( N=1600 ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Controls
β^
SE
95% CI
p
None
-0.212
0.018
[-0.246,-0.178]
<10−25
Model
-0.197
0.020
[-0.237,-0.157]
5×10−22
Dataset
-0.229
0.018
[-0.265,-0.193]
<10−25
Both
-0.222
0.021
[-0.263,-0.180]
<10−25
Appendix
Table 2: Association between cwithin and endpoint entropy when accounting for model and dataset identity.
Predictor
β^
SE
p
S(0.25)
-0.145
0.016
<10−18
S(0.50)
-0.183
0.020
1.5×10−19
S(0.75)
-0.248
0.022
<10−25
S(0.90)
-0.275
0.023
<10−25
T50
0.223
0.021
<10−25
T90
0.106
0.018
3.0×10−9
Appendix
Table 3: Temporal association with endpoint entropy after accounting for model and dataset identity.
H>0
Mean β
Median β
95% bootstrap int
β<0
40%
-0.4094
-0.4092
[-0.4350,-0.3845]
1.000
50%
-0.4517
-0.4517
[-0.4783,-0.4254]
1.000
60%
-0.4835
-0.4836
[-0.5115,-0.4545]
1.000
Appendix
Table 4: Balanced-bootstrap robustness to zero-entropy imbalance. Each target uses 10,000 replicates while accounting for model and dataset identity. First column H>0 refers to the target for H>0 share, and β<0 refers to the P(β)<0 .
Figure 1: Held-out endpoint-multiplicity prediction as a function of the number k of observed denoising trajectories. Bands show 95% intervals across random trajectory splits.
Figure 2: Balanced-bootstrap distribution of the cwithin coefficient under 50/50 balancing of zero- and positive-entropy observations within model–dataset cells. The dashed line denotes zero and the dash-dotted line the full-sample estimate.
Diffusion language models expose an explicit denoising trajectory, making it possible to ask when different kinds of information become measurable during generation. We study three independent 32-step runs of LLaDA-8B-Base on masked WikiText-103 text, each with 1{,}000 probe-training sequences and 200 held-out evaluation sequences. From saved trajectories, we derive four temporal measurements: token commitment; linear recoverability of part-of-speech (POS), coarse semantic category, and token identity; confidence and entropy dynamics; and sensitivity under mid-trajectory re-masking. Across seeds, the same ordering recurs: content categories stabilize earlier than function-heavy categories, POS and coarse semantic labels remain substantially more linearly recoverable than exact lexical identity under our probe setup, uncertainty remains higher for tokens that ultimately resolve incorrectly even though late confidence becomes less calibrated, and perturbation sensitivity peaks in the middle of the trajectory. A direct/collateral decomposition shows that this peak is overwhelmingly local to the perturbed positions themselves. In this LLaDA+WikiText setting, denoising time is therefore a useful analysis axis: under our measurements, coarse labels are recovered earlier and more robustly than lexical identity, trajectory-level uncertainty tracks eventual correctness, and mid-trajectory states are the most intervention-sensitive.
Harry Lu
School of Mathematics, University of Minnesota, Minneapolis MN, United States.
Chain-of-thought reasoning helps autoregressive models solve complex problems by generating intermediate steps that support later predictions. Masked diffusion models (MDMs) offer a similar opportunity through arbitrary-order generation: they can ideally reveal intermediate results along logical dependencies. In practice, however, standard decoding simply prioritizes high-confidence tokens, which need not align with this dependency order. We identify this discrepancy as the confidence shortcut: models commit with high certainty to plausible tokens while neglecting long-range dependencies. In multi-digit addition, models predict higher-order digits without properly tracking carries through long chains. Controlled pretraining across diverse reasoning tasks confirms that confidence-guided ordering often selects suboptimal sequences, and confidence-aligned training schemes can exacerbate these failures---for example, increasing addition error rates by an order of magnitude. Our findings caution against relying solely on confidence to choose generation orders and against training objectives that reinforce this preference. The experimental code is available at https://github.com/jinha2536/mdm-arithmetic.
Dueun Kim, Albert No
Department of Artificial Intelligence, Yonsei University
Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.
Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3