Hierarchical Continuous Diffusion Language Models

Hui Ren; Zihan Li; Chang Liu; Huidong Liu; Alexander Schwing

cs.CLOct 1, 2026

Hierarchical Continuous Diffusion Language Models

Authors: Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing

Organizations: University of Illinois Urbana-Champaign · Amazon.com, Inc.

Abstract

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.

Figures & tables

Figure 1: HC-DLM coupled diffusion process. The forward direction independently corrupts the continuous latent trajectory

x_{0}\to x_{T}

and the discrete token trajectory

k_{0}\to k_{T}

via

q(x_{t}\mid x_{t-1})

and

q(k_{t}\mid k_{t-1})

. The reverse direction couples the two channels:

p_{\phi}(x_{t-1}\mid x_{t},k_{t})

advances the continuous state conditioned on the current token state, while

p_{\theta}(k_{t}\mid x_{t})

reads token distributions from the latent state, iterating until the clean states

(x_{0},k_{0})

are reached.

Figure 2: HC-DLM training pipeline. The encoder

q_{\psi}(x_{0}\mid k_{0})

maps clean tokens

k_{0}

to a continuous latent

x_{0}

, while independent forward kernels produce the noisy pair

(x_{t},k_{t})

. The token-conditioned denoiser

d_{\phi}(x_{t},k_{t},t/T)

produces the clean-latent estimate

\hat{x}_{0}

(loss

\mathcal{J}_{\mathrm{cont}}

), the token predictor

p_{\theta}(k_{0}\mid x_{0})

decodes

x_{0}

\hat{k}_{0}

(loss

\mathcal{J}_{\mathrm{recon}}

), and the encoder is regularized by

\mathcal{J}_{\mathrm{ent}}

Method	#Params	Easy	Hard
Autoregressive
ARM (w/o ordering)	42M	9.73	–
ARM (with ordering)	42M	87.18	32.57
Discrete diffusion
MDM (vanilla)	6M	6.88	3.62
MDM (top-prob.)		18.51	9.44

Table 1: Sudoku puzzle-solving accuracy (%

\uparrow

) on easy and hard splits. Best result per column in bold .

Method	#Params	CD4	CD5
Autoregressive
GPT-2 Scratch	6M	31.9	4.3
	85M	45.8	5.1
	303M	41.3	4.5
Stream-of-Search	250M	54.2	–
LLaMA	7B	41.1	6.7

Table 2: Countdown accuracy (%

\uparrow

). Best overall result per subtask in bold ; best result at the 6M parameter scale underlined .

Method	#Params	Gen. PPL $\downarrow$
Autoregressive
Transformer	108M	66.7
Diffusion
MDM	116M	103.9
SEDD	116M	115.9
Duo	116M	97.6

Table 3: Unconditional generation results on LM1B. Baseline results are from models retrained by the LangFlow authors ( Chen et al., 2026 ) .

Method	Cont. latent	Disc. cond.	Easy	Hard
MDM (top-prob. margin)	$\times$	$\times$	89.49	49.88
Latent DM (w/o disc. cond.)	$\checkmark$	$\times$	50.46	24.74
HC-DLM (ours)	$\checkmark$	$\checkmark$	94.21	72.41

Table 4: Component ablation on Sudoku: accuracy (%

\uparrow

). Cont. latent : latent trajectory

x_{t}

present. Disc. cond. :

k_{t}

passed to denoiser

d_{\phi}

Noise type	Token ordering	Easy	Hard
Absorbing-state	Random	69.60	43.74
	Top-prob.	74.72	47.50
	Top-prob. margin	75.59	48.25
Uniform-state	Parallel update	94.21	72.41

Table 5: Ablation on Sudoku: discrete noise type and token-ordering heuristic. Accuracy (%

\uparrow

). Best result in bold .

Panel of Figure 3 — Figure 3: (a) Accuracy of the decoded intermediate prediction $\hat{x}_{0}$ at each normalized step $t/T$ ; HC-DLM improves faster. (b) Final accuracy on Hard Sudoku vs. step count; HC-DLM stays robust with fewer steps.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training time
LangFlow and retrained baselines †	$\sim$ 292 h
Plaid	$\sim$ 375 h
HC-DLM (ours)	$\sim$ 48 h

Appendix

Table 6: Reported wall-clock training time for LM1B models.

NFE	Gen. PPL	Entropy
128	75.5	4.21
64	80.1	4.21
32	93.0	4.21
16	117.2	4.18
8	168.8	4.02
Ground truth	40.4	4.32

Appendix

Table 7: Sample quality across sampling budgets. Lower Gen. PPL is better.

Figure 4: Wall-clock LM1B sampling time per batch at 128 NFEs and sequence length 128. HC-DLM is faster from batch size 16 onward, with the gap widening as the batch grows. Lower is better.

Figure 5: Effect of token-channel classifier-free guidance. We report generative perplexity (lower is better) and entropy (higher is better) as the guidance scale

w

varies. The main LM1B result uses

w=2.75

;

w=1

corresponds to sampling without guidance extrapolation.

Example	Generated text
Example 1	first 21 points of the quarter because they rallied by 21 points .<\|endoftext\|>It is , after all , Obama ’s favourite to win for president .<\|endoftext\|>Around 700,000 to 800,000 are required to pay their retails , including winter holidays , year-course , morning holidays , trips , and rigs taxes .<\|endoftext\|>Even with his lawyer , Dorothy Kinaitz , who has planned a separate appeal , he could soon appeal to the jury .<\|endoftext\|>Although Geoff Karlfeld ’s Irish blood was redbed by judges Alec Dominan and Dr. McGerson , the American men
Example 2	alcohol .<\|endoftext\|>This year , NDO has cut its profit forecast for the second quarter of 2009 .<\|endoftext\|>Ÿes , you are going to have a real team that is going to get the support and remain out of the race as to how very good they ’re going through , M̈cCain said .<\|endoftext\|>An aggressive Revolutionary Union militia , Tim Lappba , were killed in the fighting .<\|endoftext\|>The impact of the April quake was striking , with anticipated power managers warning from the city ’s summer thunderstorms .<\|endoftext\|>The Blue Millions Rural Care is likely to be offered if the county fails to comply .<\|endoftext\|>Publishing professionals
Example 3	.<\|endoftext\|>Ẅe have to work together to make sure that it is difficult to do all of the things that we don ’t do in the future , ḧe said .<\|endoftext\|>Meanwhile , a couple of elections in the northern port city of Yivazoum sent a complaint saying his support for Mr Tsvangirai and Mr Mugabe to a disastrous bid for a second term .<\|endoftext\|>Bill Greenberg , a Republican on the Finance Committee , said that if Congress was for delay , the State program linked to Bush ’s overhaul of the financial institutions was not upstate to slip out the majority .<\|endoftext\|>Undle Sarben

Appendix

Table 8: Unconditional samples generated by HC-DLM on LM1B.

$M$	Easy	Hard
1	81.73	55.81
2	91.59	67.34
4	93.75	71.11
8	71.82	45.52
16	78.80	51.89

Appendix

Table 9: Ablation on latent sequence length

M

(Sudoku, accuracy %

\uparrow

). All other hyperparameters follow the main configuration in Section B.4 ;

M=1

is the setting used throughout the Sudoku and Countdown experiments, while LM1B defaults to

M=128

, with one latent per token.. For a fair comparison of early-to-mid stage optimization efficiency, we report accuracy at the same checkpoint (560 epochs).

Figure 6: Wall-clock inference time versus denoising steps. Runtime per generated batch scales approximately linearly for both methods. Across tested step counts, HC-DLM is consistently faster, reflecting its efficient DiT implementation.

Method	Continuous variable	Update of the continuous variable	Input of the token predictor	Decoded token can change later
MDM	none	—	$k_{t}$	no
VMD	global latent	not updated (sampled once)	$k_{t},\,z$	no
CADD	noised token embeddings	no learned update	$k_{t},\,x_{t}$	no
CCDD	frozen token embeddings	learned denoiser	$k_{t},\,x_{t}$	no
HC-DLM (ours)	global latent	learned denoiser	$\hat{x}_{0}$ only	yes

Appendix

Table 10: Structural comparison with hybrid discrete–continuous diffusion models. MDM ( Sahoo et al., 2024 ) , VMD ( Zhang et al., 2025 ) , CADD ( Zheng et al., 2026 ) and CCDD ( Zhou et al., 2026 ) use the absorbing kernel, under which a token stays fixed once it is unmasked. For HC-DLM,

\hat{x}_{0}

is the denoiser’s clean-latent estimate.

Explore similar work

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
May 18, 2026Zhihan Yang, Wei Guo, Shuibai Zhang +5Diffusion Language Models Diffusion Models
NVIDIA & Cornell · NVIDIA & Georgia Tech · UW-Madison +3
Consistent Diffusion Language Models
Apr 30, 2026Hasan Amin, Yuan Gao, Yaser Souri +4Diffusion Language Models Diffusion Models
Department of Computer Science, Purdue University · Microsoft
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language Models Dataset Distillation
NVIDIA

May 18, 2026, cs.CLJ/K move · Enter open · S save

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo+4

NVIDIA & Cornell · NVIDIA & Georgia Tech · UW-Madison+3

Diffusion Language Models Diffusion Models Reparameterization

Open paper AskPDF arXiv

While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only

20\times

compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of

22.1

among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.

Abstract

Explore similar work

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

Consistent Diffusion Language Models

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling