CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

Junzhe Shen; Jieru Zhao; Ziwei He; Zhouhan Lin

cs.CLMar 3, 2026

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

Authors: Junzhe Shen, Jieru Zhao, Ziwei He, Zhouhan Lin

Organizations: LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai AI Laboratory

Abstract

We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.

Figures & tables

Figure 1: Framework of CoDAR . Starting from a noisy latent sequence $x_{T}$ , a reverse diffusion process progressively denoises the hidden states to $x_{0}$ . $\mathbf{x}_{T},\ldots,\mathbf{x}_{0}\in\mathrm{R}^{L\times d}$ , where $L$ denotes the sequence length and $d$ denotes the size of hidden states. After that, an autoregressive Transformer decoder conditions on the denoised $\mathbf{x}_{0}$ with cross-attention to translate $\mathbf{x}_{0}$ to discrete tokens $\mathbf{y}_{1},\ldots,\mathbf{y}_{L}$ .

Figure 2: Token recovery rate of point-wise linear classifier and autoregressive Transformer decoder under different sizes of hidden states.

Model	Gen. PPL( $\downarrow$ )	Diversity( $\uparrow$ )
Training set	16.75	0.2191
Recovered training set	25.07	0.2282
Decoded noise	14.24	0.0380
MDLM ∗	123.73	0.4784
SEDD	129.57	0.4742
CoDAR ( $T=0.00$ )	47.71	0.1660

Table 1: Unconditional generation on OpenWebText. We compare CoDAR to discrete baselines (MDLM and SEDD) using 1,000 generated samples per method, reporting generative perplexity and an n-gram–based diversity score. We also include three diagnostic reference points: training set text, decoded ground-truth embeddings (“recovered training set”), and decoded Gaussian noise, to contextualize the metrics. We use 250 sampling steps for all runs while varying the decoder temperature for CoDAR .

Model	Gen.PPL( $\downarrow$ )	Diversity( $\uparrow$ )
LD4LG	167.47	0.5797
CoDAR	104.76	0.3264

Table 2: Unconditional generation on LM1b.

Solver	Sampling Steps	Gen.PPL( $\downarrow$ )	Div.( $\uparrow$ )
Ancestral	25	214.86	0.4251
	50	206.54	0.4734
	100	185.91	0.4757
	250	164.89	0.4842
DPM-Solver	25	212.32	0.4929
	50	178.82	0.4942

Table 3: Solver ablation for CoDAR . We compare ancestral sampling to DPM-Solver for unconditional generation across sampling budgets with decoder temperature set to 1.0.

Sampling Steps	Model	Gen.PPL( $\downarrow$ )	Div.( $\uparrow$ )
25	MDLM ∗	232.78	0.5287
	SEDD	221.63	0.5171
	CoDAR	212.32	0.4929
50	MDLM ∗	165.71	0.5046
	SEDD	164.24	0.492
	CoDAR	178.82	0.4942

Table 4: Few-step unconditional generation on OpenWebText. We compare our model sampled with DPM-Solver (decoder temperature

T=1

) against discrete diffusion baselines (MDLM, SEDD) under matched step budgets.

Hidden Size	Solver	Gen.PPL( $\downarrow$ )	Div.( $\uparrow$ )
768	Ancestral	523.07	0.5764
768	DPM-Solver	546.10	0.6212
256	Ancestral	294.42	0.5056
256	DPM-Solver	300.01	0.5475
64	Ancestral	164.90	0.4842
64	DPM-Solver	147.53	0.488

Table 5: Ablation on dimension of hidden states.

Decoder	Gen.PPL( $\downarrow$ )	Div.( $\uparrow$ )
Linear	153.44	0.1238
Transformer Decoder	164.90	0.4842

Table 6: Ablation on choice of decoder

Continuous Diffusion Language Models

Continuous diffusion language models generate text by gradually denoising continuous token embeddings or latent representations.

See all 43 papers

CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language
Mar 2, 2026Roy Uziel, Omer Belhasin, Itay Levy +4Diffusion Language Models Continuous Diffusion Language Models
NVIDIA
Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
Oct 6, 2026Mathias Ollu, Nikos KomodakisLanguage Modeling Diffusion Language Models
Ecole Polytechnique · Archimedes, Athena RC · University of Crete +1
CoBit: Language Modeling with Bitstream Diffusion
May 7, 2026Georgios Batzolis, Mark Girolami, Luca AmbrogioniDiffusion Language Model Decoding Diffusion Language Models
Department of Engineering, University of Cambridge. · Devotion AI Labs Ltd. · Donders Institute for Brain, Cognition, and Behaviour, Radboud University.

Mar 2, 2026J/K move · Enter open · S save

CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

Roy Uziel, Omer Belhasin, Itay Levy, Akhiad Bercovich+3

NVIDIA

Diffusion Language Models Continuous Diffusion Language Models Text Generation Masked Diffusion Models Diffusion Model Fine-Tuning

Open paper AskPDF arXiv

Masked Diffusion Models (MDMs) provide an efficient non-causal alternative to autoregressive generation but often struggle with token dependencies and semantic incoherence due to their reliance on discrete marginal distributions. We address these limitations by shifting the diffusion process into a continuous sentence-level semantic space. We propose CRoCoDiL (Continuous and Robust Conditioned Diffusion for Language), a unified fine-tuning approach that jointly trains an encoder-demasker architecture, grounding the MDM demasking in continuous latent representations. This leads to the formation of a novel autoencoder in which decoding is obtained by an MDM algorithm. Relying on the same framework, we introduce two unconditional text synthesis algorithms: Continuous-Then-Discrete (ConThenDisc), a hybrid-diffusion approach that first generates latent representations in continuous space and then decodes these to tokens via an MDM, and Continuous-Within-Discrete (ConWithinDisc), a multi-diffusion strategy that refines latent representations throughout the discrete sampling process. Experiments using LLaDA show that our methods achieve superior generation quality and more than 10x faster sampling speeds in an unconditional setting.

Abstract

Continuous Diffusion Language Models

Explore similar work

CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

CoBit: Language Modeling with Bitstream Diffusion