Hierarchical Continuous Diffusion Language Models

Hui Ren; Zihan Li; Chang Liu; Huidong Liu; Alexander Schwing

cs.CLOct 1, 2026

Hierarchical Continuous Diffusion Language Models

Authors: Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing

Organizations: University of Illinois Urbana-Champaign · Amazon.com, Inc.

Abstract

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.

Figures & tables

Figure 1: HC-DLM coupled diffusion process. The forward direction independently corrupts the continuous latent trajectory

x_{0}\to x_{T}

and the discrete token trajectory

k_{0}\to k_{T}

via

q(x_{t}\mid x_{t-1})

and

q(k_{t}\mid k_{t-1})

. The reverse direction couples the two channels:

p_{\phi}(x_{t-1}\mid x_{t},k_{t})

advances the continuous state conditioned on the current token state, while

p_{\theta}(k_{t}\mid x_{t})

reads token distributions from the latent state, iterating until the clean states

(x_{0},k_{0})

are reached.

Figure 2: HC-DLM training pipeline. The encoder

q_{\psi}(x_{0}\mid k_{0})

maps clean tokens

k_{0}

to a continuous latent

x_{0}

, while independent forward kernels produce the noisy pair

(x_{t},k_{t})

. The token-conditioned denoiser

d_{\phi}(x_{t},k_{t},t/T)

produces the clean-latent estimate

\hat{x}_{0}

(loss

\mathcal{J}_{\mathrm{cont}}

), the token predictor

p_{\theta}(k_{0}\mid x_{0})

decodes

x_{0}

\hat{k}_{0}

(loss

\mathcal{J}_{\mathrm{recon}}

), and the encoder is regularized by

\mathcal{J}_{\mathrm{ent}}

Method	#Params	Easy	Hard
Autoregressive
ARM (w/o ordering)	42M	9.73	–
ARM (with ordering)	42M	87.18	32.57
Discrete diffusion
MDM (vanilla)	6M	6.88	3.62
MDM (top-prob.)		18.51	9.44

Table 1: Sudoku puzzle-solving accuracy (%

\uparrow

) on easy and hard splits. Best result per column in bold .

Method	#Params	CD4	CD5
Autoregressive
GPT-2 Scratch	6M	31.9	4.3
	85M	45.8	5.1
	303M	41.3	4.5
Stream-of-Search	250M	54.2	–
LLaMA	7B	41.1	6.7

Table 2: Countdown accuracy (%

\uparrow

). Best overall result per subtask in bold ; best result at the 6M parameter scale underlined .

Method	#Params	Gen. PPL $\downarrow$
Autoregressive
Transformer	108M	66.7
Diffusion
MDM	116M	103.9
SEDD	116M	115.9
Duo	116M	97.6

Table 3: Unconditional generation results on LM1B. Baseline results are from models retrained by the LangFlow authors ( Chen et al., 2026 ) .

Method	Cont. latent	Disc. cond.	Easy	Hard
MDM (top-prob. margin)	$\times$	$\times$	89.49	49.88
Latent DM (w/o disc. cond.)	$\checkmark$	$\times$	50.46	24.74
HC-DLM (ours)	$\checkmark$	$\checkmark$	94.21	72.41

Table 4: Component ablation on Sudoku: accuracy (%

\uparrow

). Cont. latent : latent trajectory

x_{t}

present. Disc. cond. :

k_{t}

passed to denoiser

d_{\phi}

Noise type	Token ordering	Easy	Hard
Absorbing-state	Random	69.60	43.74
	Top-prob.	74.72	47.50
	Top-prob. margin	75.59	48.25
Uniform-state	Parallel update	94.21	72.41

Table 5: Ablation on Sudoku: discrete noise type and token-ordering heuristic. Accuracy (%

\uparrow

). Best result in bold .

Panel of Figure 3 — Figure 3: (a) Accuracy of the decoded intermediate prediction $\hat{x}_{0}$ at each normalized step $t/T$ ; HC-DLM improves faster. (b) Final accuracy on Hard Sudoku vs. step count; HC-DLM stays robust with fewer steps.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training time
LangFlow and retrained baselines †	$\sim$ 292 h
Plaid	$\sim$ 375 h
HC-DLM (ours)	$\sim$ 48 h

Appendix

Table 6: Reported wall-clock training time for LM1B models.

NFE	Gen. PPL	Entropy
128	75.5	4.21
64	80.1	4.21
32	93.0	4.21
16	117.2	4.18
8	168.8	4.02
Ground truth	40.4	4.32

Appendix

Table 7: Sample quality across sampling budgets. Lower Gen. PPL is better.

Figure 4: Wall-clock LM1B sampling time per batch at 128 NFEs and sequence length 128. HC-DLM is faster from batch size 16 onward, with the gap widening as the batch grows. Lower is better.

Figure 5: Effect of token-channel classifier-free guidance. We report generative perplexity (lower is better) and entropy (higher is better) as the guidance scale

w

varies. The main LM1B result uses

w=2.75

;

w=1

corresponds to sampling without guidance extrapolation.

Example	Generated text
Example 1	first 21 points of the quarter because they rallied by 21 points .<\|endoftext\|>It is , after all , Obama ’s favourite to win for president .<\|endoftext\|>Around 700,000 to 800,000 are required to pay their retails , including winter holidays , year-course , morning holidays , trips , and rigs taxes .<\|endoftext\|>Even with his lawyer , Dorothy Kinaitz , who has planned a separate appeal , he could soon appeal to the jury .<\|endoftext\|>Although Geoff Karlfeld ’s Irish blood was redbed by judges Alec Dominan and Dr. McGerson , the American men
Example 2	alcohol .<\|endoftext\|>This year , NDO has cut its profit forecast for the second quarter of 2009 .<\|endoftext\|>Ÿes , you are going to have a real team that is going to get the support and remain out of the race as to how very good they ’re going through , M̈cCain said .<\|endoftext\|>An aggressive Revolutionary Union militia , Tim Lappba , were killed in the fighting .<\|endoftext\|>The impact of the April quake was striking , with anticipated power managers warning from the city ’s summer thunderstorms .<\|endoftext\|>The Blue Millions Rural Care is likely to be offered if the county fails to comply .<\|endoftext\|>Publishing professionals
Example 3	.<\|endoftext\|>Ẅe have to work together to make sure that it is difficult to do all of the things that we don ’t do in the future , ḧe said .<\|endoftext\|>Meanwhile , a couple of elections in the northern port city of Yivazoum sent a complaint saying his support for Mr Tsvangirai and Mr Mugabe to a disastrous bid for a second term .<\|endoftext\|>Bill Greenberg , a Republican on the Finance Committee , said that if Congress was for delay , the State program linked to Bush ’s overhaul of the financial institutions was not upstate to slip out the majority .<\|endoftext\|>Undle Sarben

Appendix

Table 8: Unconditional samples generated by HC-DLM on LM1B.

$M$	Easy	Hard
1	81.73	55.81
2	91.59	67.34
4	93.75	71.11
8	71.82	45.52
16	78.80	51.89

Appendix

Table 9: Ablation on latent sequence length

M

(Sudoku, accuracy %

\uparrow

). All other hyperparameters follow the main configuration in Section B.4 ;

M=1

is the setting used throughout the Sudoku and Countdown experiments, while LM1B defaults to

M=128

, with one latent per token.. For a fair comparison of early-to-mid stage optimization efficiency, we report accuracy at the same checkpoint (560 epochs).

Figure 6: Wall-clock inference time versus denoising steps. Runtime per generated batch scales approximately linearly for both methods. Across tested step counts, HC-DLM is consistently faster, reflecting its efficient DiT implementation.

Method	Continuous variable	Update of the continuous variable	Input of the token predictor	Decoded token can change later
MDM	none	—	$k_{t}$	no
VMD	global latent	not updated (sampled once)	$k_{t},\,z$	no
CADD	noised token embeddings	no learned update	$k_{t},\,x_{t}$	no
CCDD	frozen token embeddings	learned denoiser	$k_{t},\,x_{t}$	no
HC-DLM (ours)	global latent	learned denoiser	$\hat{x}_{0}$ only	yes

Appendix

Table 10: Structural comparison with hybrid discrete–continuous diffusion models. MDM ( Sahoo et al., 2024 ) , VMD ( Zhang et al., 2025 ) , CADD ( Zheng et al., 2026 ) and CCDD ( Zhou et al., 2026 ) use the absorbing kernel, under which a token stays fixed once it is unmasked. For HC-DLM,

\hat{x}_{0}

is the denoiser’s clean-latent estimate.

Explore similar work

May 18, 2026cs.CL

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only

20\times

compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of

22.1

among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.

Zhihan Yang, Wei Guo, Shuibai Zhang +5

NVIDIA & Cornell · NVIDIA & Georgia Tech · UW-Madison +3

Diffusion Language Models Diffusion Models Reparameterization

Ask PDF

Apr 30, 2026cs.LG

Consistent Diffusion Language Models

Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the right discrete substitute is the exact posterior bridge, the closed-form conditional law linking any two noise levels, which is available for broad corruptions including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage training framework that does not require an already trained teacher model. Our CDLM objective recovers masked diffusion, continuous consistency models, and progressive or discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.

Hasan Amin, Yuan Gao, Yaser Souri +4

Department of Computer Science, Purdue University · Microsoft

Diffusion Language Models Diffusion Models

Ask PDF

May 22, 2026cs.LG

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.

Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3

NVIDIA

Diffusion Language Models Dataset Distillation Language Modeling

Ask PDF