Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

Mathias Ollu; Nikos Komodakis

cs.CLOct 6, 2026

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

Authors: Mathias Ollu, Nikos Komodakis

Organizations: Ecole Polytechnique · Archimedes, Athena RC · University of Crete · IACM-Forth

Abstract

Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces. In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead. Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel. These modalities represent tokens at different semantic granularities: in our instantiation, the tokens themselves and coarser clusters obtained by clustering pretrained token embeddings. We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities. Applied to CoBit, this yields H-CoBit, which delivers large empirical gains across benchmarks. At dataset entropy, H-CoBit improves MAUVE and reaches a generative perplexity (GenPPL) of 49.4 on LM1B and 50.4 on OWT, improving on the baseline by 24.2 and 20.7 points and surpassing even discrete DLMs of comparable size. On GSM8K, it reaches 27.4% accuracy, outperforming prior continuous diffusion and flow-based models. We further apply H-CDLM to the flow matching model FLM, obtaining consistent gains with H-FLM and demonstrating that the framework generalizes across continuous generative paradigms. Our code will be made publicly available at https://github.com/matol-16/HCDLM.git .

Figures & tables

Panel of Figure 2 — Figure 2: Hierarchical continuous diffusion ( $k{=}2$ ). (a) A surjective map $\Gamma$ , obtained by $k$ -means ++ on pretrained embeddings, sends every token to a cluster, yielding a second symbol sequence of the same length $L$ . (b) The cluster index is encoded with the host model’s own token encoder. The two codes are concatenated at each position and corrupted at their own noise levels $\sigma_{1},\sigma_{2}$ . One network denoises both modalities. (c) Clusters act as a semantic anchor for tokens. This interplay can be optimized via per-modality scheduler/sampler parameter tuning.

Panel of Figure 3 — Figure 3: Token vs. cluster information-uniform noise schedules for H-CoBit: (a) density over log-noise levels; (b) cumulative distributions, with the median shifting from $\sigma_{50}=0.89$ to $1.13$ .

Panel of Figure 4 — Figure 4: Generation quality against diversity. GenPPL versus token unigram entropy for (a) LM1B, 1M vs 600k steps, (b) OpenWebText, matched 750k steps. Each model is swept over churn $\gamma$ and NFE budget; the solid curve is the Pareto frontier. The dashed rule is the data’s unigram entropy and the star is real data. H-CoBit ( $n{=}16$ ) uses cluster loss weight $\lambda_{2}=\frac{4}{15}$ in (a) and $\lambda_{2}=0.1$ in (b). Per-modality sampler resamples the same checkpoint with cluster churn confined to the late half of the schedule, at $\gamma_{2}\!=\!0.4$ in (a) and $\gamma_{2}\!=\!0.3$ with global $s_{\mathrm{noise}}\!=\!1.05$ in (b). $s_{\mathrm{noise}}$ is model-agnostic, so (b) also plots CoBit at $s_{\mathrm{noise}}\!=\!1.05$ for fair comparison.

	LM1B ( $L{=}128$ )				OpenWebText ( $L{=}1024$ )
Model	Steps	NFE	$H\!\uparrow$	GenPPL $\downarrow$	Steps	NFE	$H\!\uparrow$	GenPPL $\downarrow$
Real text (test split)	–	–	4.34	53.1	–	–	5.46	14.9
Autoregressive
AR baseline ( Franca and Tong, 2026 )					–	–	5.61	40.2
Discrete diffusion
SEDD ( Lou et al., 2024 )					–	–	5.65	125.7

Table 1: Generation quality, diversity at one operating point. Every row is one generation scored by GenPPL and entropy

H

. All reported continuous DLMs have

\approx

130M parameters (except ELF, 342M). H-CoBit is our hierarchical continuous DLM framework on CoBit-S. H-CoBit uses cluster-loss weights

\lambda_{2}{=}0.27

0.1

for (LM1B / OWT). We report GenPPL at matched dataset entropy when reached by the model. Per-Modality Sampler (PMS) samples with tuned churn per-modality: for H-CoBit, LM1B uses

\mathcal{W}^{2}_{\text{late}}

\gamma_{2}=0.4

and OWT uses

\mathcal{W}_{\text{late}}

\gamma_{2}=0.3

+ global

s_{noise}=1.05

(compared to Cobit-S,

s_{noise}=1.05

). The bold is best at matched H. Baselines are taken from original papers, Franca and Tong (2026) or Pynadath et al. (2026b) , except for S-FLM and CoBit which we re-evaluate.

Model	Family	NFE	Acc. (%)
Autoregressive (reference ceiling)
AR (greedy)	autoregressive LM	–	63.3
AR (sampling)	autoregressive LM	–	53.9
Discrete DLMs
MDLM (low-T)	masked discrete diffusion	1024	$\sim$ 33
Duo (low-T)	discrete diffusion	1024	$\sim$ 36

Table 2: Mathematical reasoning accuracy. GSM8K accuracy at matched training budget. CoBit/H-CoBit are reported at their best churn. H-CoBit (

\lambda_{2}=0.3

) beats CoBit by +2.1pts at half the NFE, +4pts at matched NFE. Baselines are from Deschenaux and Gulcehre (2026) ; Batzolis et al. (2026) .

Panel of Figure 5 — Figure 5: H-CoBit cluster design: semantics, weight, count. LM1B at 100k steps. (a) GenPPL/entropy frontier for $n{=}16$ clusters from $k$ -means over LangFlow ( Chen et al., 2026 ) embeddings, against a random cluster assignment control. Random clusters yield no improvement on the baseline. (b) GenPPL gain over the baseline at dataset entropy with respect to loss weight and $n$ . (c) Cluster-count ablation with each $n$ given the $\lambda_{2}$ that is best for it at three entropy levels. $n=16$ is best at every $H$ . For (b) and (c), the grey rule is what H-CoBit variants have to beat (lower is better).

Arm	$H{\approx}4.28$	$H{\approx}4.30$	$H{\approx}4.32$	$H{\approx}4.34$	MAUVE
			GenPPL at token entropy $H$
dual (per-modality schedule)	46.4	46.9	51.6	57.3	0.973
single (shared schedule)	44.6	45.1	52.8	57.4	0.965

Table 3: Per-modality vs single schedule, H-CoBit. LM1B

n{=}16

. MAUVE is the best reached.

Panel of Figure 6 — Figure 6: H-FLM vs. FLM on LM1B (1M steps each; H-FLM: $n\!=\!16$ , $\lambda_{2}\!=\!1.0$ ). Each curve is a sampler’s Pareto frontier over temperature. Stochastic sampling, each model’s best parameters: $\gamma\!=\!0.03$ (FLM) or $\gamma\!=\!0.3$ plus late window cluster churn with cluster temperature 0 (H-FLM).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Embeddings	pretrained LangFlow/S-FLM input embeddings, $D=768$
Metric	Euclidean on unit-normalized rows ( $\equiv$ cosine)
Clusters $n$	LM1B $\{8,\ldots,256\}$ , OWT / TinyGSM $\{16,32,64\}$ ; $n=16$ default
Initialization	$K$ -means++, $n_{\mathrm{init}}=10$ restarts, best inertia
Max iterations $T$ / tolerance $\varepsilon$	$300$ / $10^{-4}$ (relative centroid shift)
Seed	$42$

Appendix

Table 4: Clustering hyperparameters.

\Gamma

is computed once per (tokenizer,

n

) pair and frozen before training.

Corpus	$\|\mathcal{V}\|$	$n_{\mathrm{sem}}$	size min / mean / max	Sil. $\uparrow$	DBI $\downarrow$	CH $\uparrow$	$H_{\mathrm{type}}$
LM1B	30,522	16	848 / 1908 / 4210	0.050	3.50	506	0.969
OpenWebText	65,536	15	2279 / 4369 / 7671	0.057	3.41	1300	0.978

Appendix

Table 5: Measured clustering quality at

n=16

n_{\mathrm{sem}}

excludes the reserved structural cluster. Silhouette / DBI / CH are computed on the fitted tokens only, silhouette on a

10{,}000

-token subsample.

H_{\mathrm{type}}

is the type-occupancy entropy normalized by

\log_{2}n_{\mathrm{sem}}

Id	Usage	Most frequent members
11	16.5%	the , of , in , a , for , of the , in the , with , by , on
12	13.0%	to , that , is , ’s , are , as , it , were , has , have
9	7.9%	, , and , , the , , and , or , , a , , but , -- , ;
0	6.6%	. , : , . The , .\n\nThe , .’’ , . He , ?
2	6.2%	was , had , said , ed , has been , made , did , left , says , called
4	4.7%	two , 2 , 1 , 3 , 4 , three , 5 , $ , 10

Appendix

Table 6: OpenWebText clusters at

n=16

. The seven most-used semantic clusters, with their share of a

35.5

M-token sample and their most frequent gpt2id_bpe16 code tokens (which may span several GPT-2 tokens). Ids are the

K

-means labels, prior to the code rebalancing of Section A.4 , which permutes them without changing membership.

Model	Representation $x_{0}=\mathcal{E}(y)$	Corruption path	VE-equivalent $\sigma$	Decoder $\mathcal{D}$
Analog Bits	$m=\lceil\log_{2}\|\mathcal{V}\|\rceil$ analog bits in $\{\pm 1\}$ ; $d=m$	VP (cosine), $x_{t}=\sqrt{\bar{\alpha}_{t}}\,x_{0}+\sqrt{1-\bar{\alpha}_{t}}\,\epsilon$	$\sqrt{(1-\bar{\alpha}_{t})/\bar{\alpha}_{t}}$	threshold bits at $0$
CoBit	analog bits in $\{0,1\}$ , $m=15$ (LM1B) or $16$ (OWT codec); $d=m$ , patched to length $L$	VE, native: $x_{\sigma}=x_{0}+\sigma\epsilon$ , $\sigma_{\mathrm{data}}=1/2$	$\sigma$ (native)	threshold bit probabilities, invert codec
LangFlow	learned embeddings, $\ell_{2}$ -normalized and scaled by $\sqrt{D}$ ; $d=D=768$	VP $\gamma$ -path, $z_{l}=\alpha_{l}z+\sigma_{l}\epsilon$ , $\sigma_{l}^{2}=\mathrm{sigmoid}(l)$	$e^{l/2}$ , i.e. $l=2\log\sigma$	$\arg\max$ of predicted token probabilities
FLM, FMLM	one-hot vectors; $d=\|\mathcal{V}\|$	linear interpolant $I_{t}=(1-t)\,\epsilon+t\,x_{1}$	$(1-t)/t$	$\arg\max$ per position
ELF	frozen T5-small contextual embeddings ( $512$ ), linear bottleneck to $128$	rectified flow, $z_{t}=t\,x+(1-t)\,\epsilon$	$(1-t)/t$	network in decode mode at $t=1$ , then unembedding
S-FLM	learned unit-norm embeddings, $E=\mathbb{S}$ ; $d=768$	geodesic, $z_{t}=\mathrm{SLERP}(z_{0},z_{1},\alpha_{t})$ , $z_{0}\sim\mathcal{U}(\mathbb{S}^{d-1})$	— (not additive Gaussian)	$\arg\max$ of $p^{\theta}_{1\|t}$

Appendix

Table 7: Representation and corruption process. Instantiation of the encoder

\mathcal{E}

, of the forward process, and of the decoder

\mathcal{D}

of Section 3 . The VE-equivalent noise level follows from the rescaling equation 8 .

Model	Network output	Objective	Noise / time schedule	Sampler, stochasticity
Analog Bits ( 2023 )	$\hat{x}_{0}$ (analog bits)	MSE, $w\equiv 1$	cosine (VP)	DDIM / ancestral; SC ( $p=0.5$ ); asymmetric time intervals
CoBit ( 2026 )	bit posteriors in $(0,1)^{Lm}$ via matched-filter residual $\ell_{\theta}=r_{\theta}+\mathrm{clip}\big((x_{\sigma}-\tfrac{1}{2})/\sigma^{2}\big)$	weighted MSE, EDM $w(\sigma)=\frac{\sigma^{2}+\sigma_{\mathrm{data}}^{2}}{\sigma^{2}\sigma_{\mathrm{data}}^{2}}$	online entropy-rate, $\pi_{\alpha}(u)\propto g(\sigma)\,\hat{h}_{\log}(\sigma)^{\alpha}$ with $\alpha=\tfrac{1}{2}$ and $\hat{h}_{\log}\approx e(\sigma)/\sigma^{2}$	DDIM PF-ODE $+$ full-band EDM churn $\gamma_{i}=S_{\mathrm{churn}}/N$ , entropy-gated $\lambda_{\mathrm{ent}}\approx S_{\mathrm{churn}}\pi_{\alpha}$ ; SC ( $p=0.5$ , carry mode)
LangFlow ( 2026 )	token probabilities $\hat{x}_{\theta}\in\Delta^{\|\mathcal{V}\|-1}$ ; denoiser $\hat{z}_{\theta}=E^{\top}\hat{x}_{\theta}$	cross-entropy (Bregman flow matching) $+$ scheduler loss	learnable Gumbel density over $l$ fitted to the information gain $H^{\prime}_{\gamma}\approx\|\mathrm{d}\hat{\mathcal{L}}/\mathrm{d}t\|$	Euler on the $l$ -path, deterministic by design; SC ( $p=0.25$ )
FLM, FMLM ( 2026 )	posterior $p_{1\|t}$ through a tokenwise softmax ( $=\mathbb{E}[x_{1}\mid I_{t}]$ )	cross-entropy; FMLM: KL semigroup loss on the two-time denoiser $\delta_{s,t}$	decoding-error reparametrization $\tau(t)=1-\tfrac{\|\mathcal{V}\|}{\|\mathcal{V}\|-1}P_{e}(t)$	Euler ODE; FMLM performs one/few-step flow-map jumps; no SC. (We add stochastic sampling).
ELF ( 2026 )	clean embeddings ( $x$ -prediction), shared-weight decoding head at $t=1$	velocity MSE ( $w\propto(1-t)^{-2}$ , $80\%$ ) $+$ token cross-entropy at $t=1$ ( $20\%$ )	logit-normal over $t$ ( $P_{\mathrm{mean}}=-1.5$ , $P_{\mathrm{std}}=0.8$ ), at training and inference	Euler ODE, or noise re-injection (“SDE”, scale $\gamma$ ); SC ( $p=0.5$ ) $+$ training-time CFG
S-FLM ( 2026 )	posterior $p^{\theta}_{1\|t}$ ; tangent average $\bar{u}=\sum_{v}p^{\theta}_{1\|t}(v)\,\log_{z_{t}}(\hat{e}_{v})$	cross-entropy	truncation to the high-noise band $\alpha\leq\alpha^{\star}(\delta)$ , plus adaptive refit from $\|\mathrm{d}\hat{\mathcal{L}}/\mathrm{d}t\|$ (information gain).	geodesic Euler; exact, one-sample stochastic or top- $k$ velocity; temperature $T$ ; no SC

Appendix

Table 8: Prediction target, objective, schedule and sampler of continuous DLMs. All models estimate the same posterior, but differ in how it is parametrized, how noise levels are allocated, and how much stochasticity the sampler injects.

Model	Cluster encoder $\mathcal{E}_{2}$	Added state per position, $n=16$	Per-modality schedule and remarks
CoBit	$\lceil\log_{2}n\rceil$ analog bits appended to the token’s bit patch	$+4$ bits on $m=15/16$ (approx. $+1\%$ parameters, Section 5.1 )	Own entropy-rate density $\pi^{2}_{\alpha}$ fitted on the cluster loss; churn can be set per-modality. Encoding the cluster inside the token code instead of appending it underperforms.
LangFlow	learned cluster embedding table $E^{(2)}\in\mathbb{R}^{n\times d_{2}}$ , concatenated per position	$+d_{2}$ channels on $D=768$	A second learnable Gumbel scheduler over $l^{2}$ ; the deterministic sampler leaves no churn to tune.
FLM, FMLM	one-hot over $\mathcal{C}$	$+16$ on $\|\mathcal{V}\|=30522$	A second decoding-error curve $P_{e}^{2}(t)$ , hence a second reparametrization $\tau_{2}$ .
ELF	extra embedding channels for the cluster, plus a second unembedding head at the decoding step	$+d_{2}$ channels	No information-uniform schedule to fit; the logit-normal density would have to be shifted per-modality by hand, as in image latent forcing ( Baade et al., 2026 ) .
S-FLM	second unit-norm cluster table on $\mathbb{S}^{d_{2}-1}$ , with its own SLERP interpolation	$+d_{2}$ channels on $d=768$	Own truncation threshold $\alpha^{\star}_{2}(\delta)$ , which predicts the coarse-to-fine ordering.

Appendix

Table 9: Hierarchical instantiation of existing continuous DLMs. How the

k=2

construction of Section 4 applies to each model, with

n=|\mathcal{C}|

clusters and

d_{2}

the cluster embedding dimension. Added state is reported for

n=16

, the value we find optimal (Figure 5 ). The CoBit row corresponds to H-CoBit (Section 5 ).

	LM1B	OpenWebText	TinyGSM
Tokenizer	BERT-base-uncased	gpt2id_bpe16	SmolLM-135M
Vocabulary $\|\mathcal{V}\|$	30,522	65,536	49,153
Sequence length $L$	128	1024	512
Bits/token (CoBit / H-CoBit)	$15\to 19$	$16\to 20$	$16\to 20$
Steps (CoBit / H-CoBit)	1M / 600k	750k / 750k	250k / 250k
LR schedule	cosine (1M)	cosine (1M)	constant

Appendix

Table 10: Main-run settings for H-CoBit. All other hyperparameters follow Batzolis et al. (2026) .

LM1B (final checkpoints)
			MAUVE $\uparrow$ at fixed entropy (best NFE)
Model	steps	$\lambda_{2}$	$H_{1}$	$H_{\mathrm{data}}$	$H_{2}$
target $H$			$4.36$ ( $H_{1}$ )	$4.34$ ( $H_{\mathrm{data}}$ )	$4.29$ ( $H_{2}$ )
CoBit	1M	–	0.870 (64)	0.861 (128)	0.697 (256)
H-CoBit	600k	$\frac{4}{15}$	0.900 (64)	0.903 (128)	0.648 (512)
OpenWebText (final checkpoints)

Appendix

Table 11: mauve at fixed entropy levels, best budget per level. Entropy-matched MAUVE (

H_{1}

+0.02

nats off it,

H_{\mathrm{data}}

= the dataset’s own,

H_{2}

-0.05

nats off it); cell is the best mauve any swept NFE reaches at that level, winning NFE in parentheses. Target entropies (nats): LM1B 4.36/4.34/4.29; OWT 5.48/5.46/5.41.

Dataset	bits/tok	CoBit	H-CoBit	$\Delta$	CoBit	H-CoBit	$\Delta$
		Training (ms/step)			Inference ( $\mu$ s/call)
LM1B	15 $\to$ 19	123.5	125.8	+1.9%	292	301	+2.9%
TinyGSM / GSM8K	16 $\to$ 20	231.5	237.3	+2.5%	1174	1213	+3.3%
OpenWebText	16 $\to$ 20	244.3	250.0	+2.3%	2401	2503	+4.2%
$n{=}64$	16 $\to$ 22	244.3	252.2	+3.2%	2401	2520	+5.0%

Appendix

Table 12: Inference and training costs. Both arms built and timed in one process on one A100-SXM-64GB; training is forward, backward and optimizer step with no dataloading, inference is deterministic sampling with self-conditioning at NFE=128. Per-call time is the inference unit because it is what multiplies by NFE to give generation time. Each dataset is read at the largest batch it was profiled at, where the GPU is saturated.

Panel of Figure 7 — Figure 7: For H-CoBit, embedding the cluster index in the bit sequence is cheaper, but it does not pay off. LM1B at 100k steps, GenPPL with respect to sample entropy. Embedding clusters in token bit codes is done via a variable length prefix.

Panel of Figure 8 — Figure 8: Token/cluster agreement during sampling, H-CoBit (LM1B, $n\!=\!16$ , 600k steps; NFE 128, 256 samples). Top: share of positions whose current argmax already equals the final sample, for the token, the cluster block, and the cluster implied by the token block ( $\sum_{x^{1},\Gamma(x^{1})=c}p(x^{1})$ ). Bottom: Kullback-Leibler divergence between the token-implied cluster distribution and the cluster block.

LM1B: $n{=}16$ , 600k steps, $\lambda_{2}{=}4/15$ , $H_{\mathrm{data}}=4.34$
Gen-PPL			$\Delta$ vs. control (%) at NFE
Arm	@ $H_{\mathrm{data}}$ (NFE)	$H$ reach	64	128	256	512
control (shared churn)	57.3 (256)	4.356	0	0	0	0
Cluster-churn magnitude $\gamma_{2}$
$\gamma_{2}{=}0$ (frozen)	57.8 (512)	4.356	-1.5	+8.2	+4.7	-2.0
$\gamma_{2}{=}0.1$	59.7 (256)	4.357	-3.5	+0.9	-0.8	-1.1

Appendix

Table 13: Every multimodal-churn arm on both datasets for H-CoBit.

\Delta

Gen-PPL: the mean % change against the control at matched entropy over the interval both arms’ envelopes cover (negative is better; – where they share none). Bold is best. Magnitude alone has no effect, it must be combined with late churning to yield improvements.

Explore similar work

May 7, 2026cs.CL

CoBit: Language Modeling with Bitstream Diffusion

Diffusion language models (DLMs) promise parallel, order-agnostic generation, but on standard benchmarks they have historically lagged behind autoregressive models in sample quality and diversity. Recent continuous flow and diffusion approaches have narrowed this gap. In this work, we further close the autoregressive gap by modeling text as a continuous diffusion process over fixed-width binary bitstreams. We refer to the resulting model as CoBit (Continuous Bitstream Diffusion). Our approach represents semantic tokens as analog bit sequences and uses a matched-filter residual parameterization to isolate contextual learning from analytic independent-bit posteriors. Crucially, we adopt a stochastic sampler that applies Langevin-type corrections gated by the entropy-rate profile, concentrating stochasticity in high-information regions while remaining nearly deterministic elsewhere. On LM1B, our 130M-parameter model reaches a generative perplexity (GenPPL) of 59.76 at matched real-data entropy (4.31) using 256 neural function evaluations (NFEs), outperforming prior DLM baselines and reaching the autoregressive reference. On OpenWebText (OWT), our sampler establishes a new continuous-DLM Pareto frontier, achieving GenPPL 27.06 at entropy 5.26 using 4x fewer steps than previous 1024-NFE baselines. Scaling the same recipe to a 462M-parameter model (CoBit-M) further improves the OWT GenPPL-entropy frontier over the 130M model (CoBit-S) and over medium-scale continuous and discrete DLM baselines, reaching GenPPL 19.5 at entropy 5.40, near real-data entropy (5.44), and approaching pretrained GPT-2 Medium over the high-quality region. As an additional benefit, bitstream diffusion removes the O(V) vocabulary scaling bottleneck of standard DLMs: by predicting O(log V) bitwise logits via semantic bit-patching, it lowers memory and raises throughput, a scalable paradigm as vocabulary sizes grow.

Georgios Batzolis, Mark Girolami, Luca Ambrogioni

Department of Engineering, University of Cambridge. · Devotion AI Labs Ltd. · Donders Institute for Brain, Cognition, and Behaviour, Radboud University.

Diffusion Language Models Diffusion Models Language Modeling

Ask PDF

Oct 1, 2026cs.CL

Hierarchical Continuous Diffusion Language Models

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.

Hui Ren, Zihan Li, Chang Liu +2

University of Illinois Urbana-Champaign · Amazon.com, Inc.

Diffusion Language Models Diffusion Models

Ask PDF

Apr 30, 2026cs.LG

Consistent Diffusion Language Models

Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the right discrete substitute is the exact posterior bridge, the closed-form conditional law linking any two noise levels, which is available for broad corruptions including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage training framework that does not require an already trained teacher model. Our CDLM objective recovers masked diffusion, continuous consistency models, and progressive or discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.

Hasan Amin, Yuan Gao, Yaser Souri +4

Department of Computer Science, Purdue University · Microsoft

Diffusion Language Models Diffusion Models

Ask PDF