Continuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation. The field has grown rapidly, and the capabilities of CDLMs have begun to rival those of discrete diffusion. Yet, unlike for continuous modalities such as images or videos, our grasp of the levers that matter when training CDLMs remains limited. In this work, we focus on two of these levers, namely self-conditioning and embedding geometry, and shed light on some of the mechanisms behind them, which in turn suggest cheaper and more effective designs. We begin by showing that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training. These gains can be recovered without an extra forward pass by a two-level interpolant, coined DoubleCat, which improves TinyGSM accuracy more than threefold. Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians\unicodex2013an issue known as token identifiability. To address it directly, we introduce discrete reveals to DoubleCat, yielding SemiCat, which achieves discrete diffusion-level performance on TinyGSM, reaching 49.94% with standard training hyperparameters and a tuned sampler. At the 1.7B scale, SemiCat outperforms discrete diffusion on four of six likelihood-based evaluations, and surpasses one-hot flows on unconditional generation. On embedded spaces, we find that cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one. We briefly explore a range of further options, such as hybrid corruption processes and noise schedules, including asynchronous ones. We believe this exploration can help the community better understand and advance the design space of CDLMs.
Figures & tables
Figure 1: (a) One-hot flows are always exposed to nearly clean samples for t≈1 , and only track the clean target, X1ℓ . (b) At inference, they track instead the dynamic predictions of their denoiser, πθ , which always lies on the interior of the simplex. (c) DoubleCat addresses this issue at train-time by simply adding an additional corruption process on the output space of its denoiser, the simplex.
Figure 2: Entropy of a one-hot flow’s denoiser, πθ , along validation set and inference trajectories.
Method
Accuracy (%, ↑ )
Best (%)
T=1
T=0.5
T=0.2
top-k=1
Runs ( ↑ )
Name err. ( ↓ )
Discrete diffusion baselines
MDM
8.79
21.68
27.14
28.28
88.63
11.22
USDM
12.43
29.34
33.06
33.74
91.96
7.81
One-hot based
Base one-hot flow
3.28
4.42
3.89
3.64
50.91
48.07
Table 1: TinyGSM results at 256 NFEs. Accuracy is reported per sampler; the last two columns report the best value attained over those samplers. T denotes the temperature applied to πθ . The models are trained with batch size 256 for 200 k steps. † refers to standard hyperparameters for the TinyGSM benchmark in the literature (including batch size of 512 , 250 k steps). See Section E.1 for more details.
Figure 3: Gen. PPL/entropy Pareto frontiers as measured by GPT-2 for one-hot flows trained on OpenWebText with sequence lengths of 1024 tokens, using the GPT-2 tokeniser.
Method
Accuracy (%, ↑ )
Best (%)
T=1
T=0.5
T=0.2
top-k=1
Runs ( ↑ )
Name err. ( ↓ )
Ambient sphere
d=16
6.97
9.40
8.95
8.87
58.83
40.71
d=128
7.88
9.25
9.63
10.24
56.48
42.76
d=768
8.64
12.89
13.12
12.28
66.72
32.75
Plaid, d=16
Table 2: TinyGSM results at 256 NFE. Plaid learns its log-SNR endpoints; the variants all pin them. “Without self-cond.” refers to checkpoints trained without self-conditioning. T denotes the temperature applied to πθ . The models are trained with batch size 256 for 200 k steps. See Section E.1 for more details.
Figure 4: DER of embedded geometries, and the quality/diversity trade-off across methods on OWT. (Left) The DER of a trained sphere model on OWT and that of Plaid both with d=768 , against the known one-hot DER. (Right) Gen. PPL/Entropy Pareto frontiers on OWT, with Plaid for d=16 .
Figure 5: Gen. PPL as measured by Qwen 2.57 B for models trained on Nemotron with sequence lengths of 2048 tokens, using the TikToken tokeniser.
Method
ARC-E
BoolQ
OBQA
PIQA
RACE
SIQA
Chance
24.7
50.4
26.6
51.6
24.2
32.2
LLaDA-8B-Base ( Nie et al., 2025 )
–
–
–
74.4
–
–
AR † (autoregressive)
72.7
71.9
40.4
78.1
36.2
41.9
AR ⋆ (autoregressive)
73.1
63.6
40.2
76.9
36.3
43.5
Duo † (uniform diffusion)
53.4
59.6
33.0
62.7
35.0
39.0
MDLM † (masked diffusion)
50.5
62.8
32.0
62.2
34.7
39.2
Table 3: Zero-shot accuracy (%) on likelihood-based benchmarks. ⋆ Models from Davis et al. (2026) . † Models from Sahoo et al. (2026) . Both with 1.7 B parameters, trained on Nemotron for 2.1 T tokens. SemiCat and one-hot flow likelihoods are estimated by the cross-entropy loss of the continuation (see equations 2.2 and 3.7 ). Best non-AR results at 1.7 B parameters per column are in bold .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
OpenWebText
TinyGSM
Nemotron ( 1.7 B)
Architecture (DiT)
Hidden size
768
768
2304
adaLN cond. dim
128
128
128
Layers
12
12
18
Attention heads
12
12
18
FFN (SwiGLU) multiplier
4
4
3
Appendix
Table 4: Hyperparameters of our models on OpenWebText, TinyGSM and Nemotron.
TinyGSM (ours)
TinyGSM (standard)
Global batch size
256
512
Total steps
200000
250000
Schedule
Cosine decay w/ linear warmup
Constant w/ linear warmup
Warmup steps
3000
2500
Peak / final LR
3×10−4 / 10−5
3×10−4 / 3×10−4
Weight decay
0.1
0
Appendix
Table 5: A comparison of TinyGSM hyperparameters between ours and the standard ones found in the literature. We run most of our experiments on our own, and report SemiCat ’s numbers on the standard configuration too ( Table 1 ).
Accuracy (%)
Best (%)
Y0ℓ
T=1
T=0.5
T=0.2
top-k=1
Runs ( ↑ )
Name err. ( ↓ )
Barycentre
4.70
6.52
6.29
7.51
55.57
43.90
Uniform
6.14
9.17
10.84
11.30
53.53
46.10
Uniform vertex
5.31
7.05
7.81
8.49
51.48
47.99
Appendix
Table 6: Ablation on the simplex prior Y0ℓ for DoubleCat , at NFE 256 . Accuracy is reported per sampler; the last two columns report the best value attained over those samplers. “Barycentre” is δ∣V∣1 , “Uniform” is uniform over the simplex, U(Δ∣V∣) , and “uniform vertex” is uniform over the vertices of the simplex ( i.e. , the one-hot encoding of U(V) ). The inner schedule is set to κt≡t2 . We retain the uniform distribution which consistently outperforms the other choices on accuracy.
Accuracy (%)
Best (%)
κt
T=1
T=0.5
T=0.2
top-k=1
Runs ( ↑ )
Name err. ( ↓ )
t
9.70
13.50
14.18
14.78
63.61
36.16
t
6.97
9.55
10.61
10.84
55.19
44.43
t2
6.14
9.17
10.84
11.30
53.53
46.10
t3
5.00
7.35
7.05
8.79
51.18
48.45
Appendix
Table 7: Ablation on the simplex schedule κt for DoubleCat , at NFE 256 , with Y0ℓ∼U(Δ∣V∣) for all runs. Accuracy is reported per sampler; the last two columns report the best value attained over those samplers.
Accuracy (%, ↑ )
Best (%)
Method
Self-cond.
T=1
T=0.5
T=0.2
top-k=1
Runs ( ↑ )
Name err. ( ↓ )
One-hot flow
Off
3.28
4.42
3.89
3.64
50.91
48.07
On
6.44
6.75
8.34
6.97
61.79
37.60
DoubleCat
Off
11.45
15.85
16.00
15.92
69.67
30.02
On
16.83
20.77
20.70
19.79
75.21
24.34
Appendix
Table 8: Self-conditioning effect on one-hot flows and DoubleCat for schedule κt≡t .
Figure 6: Gen. PPL as measured by GPT-2 for models trained on OpenWebText with sequence lengths of 1024 tokens, using the GPT-2 tokeniser. All Plaid runs for different embedding dimensionalities, d .
Figure 7: Gen. PPL as measured by GPT-2 for models trained on OpenWebText with sequence lengths of 1024 tokens, using the GPT-2 tokeniser. Plaid is for d=16 .
Gen. PPL ( ↓ ) (entropy)
Flip probability ρt
NFE =50
NFE =100
NFE =200
Baseline, 1 M steps, NFE =128
One-hot flow
85.64\penalty(5.45)
DoubleCat
66.02\penalty(5.38)
SemiCat
117.00\penalty(5.62)
no L2 weighting
Appendix
Table 9: OWT Gen. PPL (GPT-2), token entropy in parentheses, at 200 k steps, temperature 1 . The corruption flips each X1ℓ independently, with probability ρt , to a token drawn uniformly from V ; it enters the interpolant only, leaving the loss target clean. Trained with γ being the usual mixture schedule.
Accuracy (%, ↑ )
Best (%)
α
T=1
T=0.5
T=0.2
top-k=1
Runs ( ↑ )
Name err. ( ↓ )
0
7.20
9.55
9.63
8.72
69.22
29.26
0.25
13.50
18.35
17.89
17.97
79.38
19.86
0.5
19.26
23.88
25.17
25.93
84.91
14.71
0.75
15.47
25.25
29.57
29.26
87.11
12.81
1
14.56
23.86
27.74
27.37
86.73
12.81
Appendix
Table 10: TinyGSM accuracy at NFE =256 , for α-Cat at corruption levels α∈{0,0.25,0.5,0.75,1} .
Accuracy (%, ↑ )
Student
NFE 1
2
4
8
16
32
FMLM+ ( Agarwal et al., 2026 )
0.2
1.2
5.7
12.4
17.8
19.9
SemiCat teacher
–
–
–
5.5
12.1
20.1
T=1
0.2
0.5
1.6
5.4
13.0
19.0
T=0.5
0.7
1.2
2.8
9.8
16.5
24.0
T=0.3
0.5
0.8
3.9
9.2
16.3
22.7
Appendix
Table 11: TinyGSM accuracy for D-MMD distillation of SemiCat , at teacher temperature T . Students train for 20 k steps at constant learning rate 2×10−5 , with early stopping on GSM8K test set accuracy at 32 NFE.
Figure 8: Statistics across training steps, for a Plaid model trained on OWT. (Left) The evolution of cosine similarity between Wc (the part of the input matrix for self-conditioning) and Wx (the part that multiplies the input). (Middle) The validation loss. (Right) The Gen. PPL.
Figure 9: The entropic schedule against the base DER, mixture and linear schedules. (Left) The schedules themselves. (Right) The expected remaining entropy, S(γt)=EX0ℓ[H(X1ℓ∣Itℓ)] .
Figure 10: Gen. PPL as measured by GPT-2 for models trained on OpenWebText with sequence lengths of 1024 tokens, using the GPT-2 tokeniser.
Figure 11: Local per-position times tℓ as a function of global time tg at concentration k=1 and sequence length L=128 . (Left) Each curve traces one position as global time advances. (Right) Each curve is the spatial profile at one global time, monotone decreasing in ℓ : the sequence is revealed left to right. The dashed horizontals mark the per-position mean, which equals tg at every setting.
Accuracy (%, ↑ )
Best (%)
InfoNoise
Schedule
Asynchronous
T=1
T=0.5
T=0.2
top-k=1
Runs ( ↑ )
Name err. ( ↓ )
Off
Linear
Off
6.82
9.40
11.14
9.40
63.61
36.01
On
7.88
10.99
12.28
10.99
66.26
33.43
Mixture
Off
10.39
14.78
13.65
13.87
68.16
31.54
On
12.21
15.47
16.00
14.63
73.39
26.38
On
Linear
Off
9.63
17.44
18.42
15.69
67.78
31.69
Appendix
Table 12: TinyGSM accuracy at NFE =256 . For linear and mixture outer schedules γt and inner schedule κt=t , we compare a scalar time shared across all positions against an asynchronous left-to-right time per token position, each with and without dynamic allocation via InfoNoise. All settings are DoubleCat sampled as a one-hot flow. Better of each matched pair in each column is shown in bold .
Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the right discrete substitute is the exact posterior bridge, the closed-form conditional law linking any two noise levels, which is available for broad corruptions including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage training framework that does not require an already trained teacher model. Our CDLM objective recovers masked diffusion, continuous consistency models, and progressive or discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.
Hasan Amin, Yuan Gao, Yaser Souri +4
Department of Computer Science, Purdue University · Microsoft
Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces. In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead. Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel. These modalities represent tokens at different semantic granularities: in our instantiation, the tokens themselves and coarser clusters obtained by clustering pretrained token embeddings. We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities. Applied to CoBit, this yields H-CoBit, which delivers large empirical gains across benchmarks. At dataset entropy, H-CoBit improves MAUVE and reaches a generative perplexity (GenPPL) of 49.4 on LM1B and 50.4 on OWT, improving on the baseline by 24.2 and 20.7 points and surpassing even discrete DLMs of comparable size. On GSM8K, it reaches 27.4% accuracy, outperforming prior continuous diffusion and flow-based models. We further apply H-CDLM to the flow matching model FLM, obtaining consistent gains with H-FLM and demonstrating that the framework generalizes across continuous generative paradigms. Our code will be made publicly available at https://github.com/matol-16/HCDLM.git .
Mathias Ollu, Nikos Komodakis
Ecole Polytechnique · Archimedes, Athena RC · University of Crete +1
Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.