CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

Roy Uziel; Omer Belhasin; Itay Levy; Akhiad Bercovich; Ran El-Yaniv; Ran Zilberstein; Michael Elad

cs.CLMar 2, 2026

CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

Authors: Roy Uziel, Omer Belhasin, Itay Levy, Akhiad Bercovich, Ran El-Yaniv, Ran Zilberstein, Michael Elad

Organizations: NVIDIA

Abstract

Masked Diffusion Models (MDMs) provide an efficient non-causal alternative to autoregressive generation but often struggle with token dependencies and semantic incoherence due to their reliance on discrete marginal distributions. We address these limitations by shifting the diffusion process into a continuous sentence-level semantic space. We propose CRoCoDiL (Continuous and Robust Conditioned Diffusion for Language), a unified fine-tuning approach that jointly trains an encoder-demasker architecture, grounding the MDM demasking in continuous latent representations. This leads to the formation of a novel autoencoder in which decoding is obtained by an MDM algorithm. Relying on the same framework, we introduce two unconditional text synthesis algorithms: Continuous-Then-Discrete (ConThenDisc), a hybrid-diffusion approach that first generates latent representations in continuous space and then decodes these to tokens via an MDM, and Continuous-Within-Discrete (ConWithinDisc), a multi-diffusion strategy that refines latent representations throughout the discrete sampling process. Experiments using LLaDA show that our methods achieve superior generation quality and more than 10x faster sampling speeds in an unconditional setting.

Figures & tables

Figure 1: The CRoCoDiL framework: Building on a learned encoder of text sequences and a demasker guided by this continuous representation, we introduce (a) an autoencoder and (b,c) two text generation algorithms, ConThenDisc and ConWithinDisc . A regular MDM serves in all cases as a decoder that converts the latent ${\mathbf{z}}_{0}$ into a sequence ${\hat{\mathbf{x}}}_{0}$ . The text generation algorithms rely on learned diffusion models that operate in the representation domain.

Figure 2: The training framework for the embedding network ${\mathbf{z}}_{0}=h_{\phi}({\mathbf{x}}_{0})$ and the guided demasker $f_{\theta}({\mathbf{x}}_{t},t,{\mathbf{z}}_{0})$ . Flame indicates a trained network.

Figure 3: The ConThenDisc and ConWithinDisc text generation algorithms. In both, a continuous diffusion model generates a starting latent

{\mathbf{z}}_{0}

, which is decoded to tokens by a regular MDM. ConWithinDisc includes a refinement of the latent (the purple part) based on the partially synthesized text.

Panel of Figure 4 — Figure 4: Guided-Demasker: Cross-Entropy (top) and top-1 token prediction performance versus masking probability ( $0$ for no-mask), measured on a validation set of $1000$ sequences.

Figure 5: Autoencoder: A source program and its encoded-decoded outcome (Gen-Length= $256$ tokens as one block). The two outcomes correspond to NFE= $4$ (CER= $0.12$ ) and NFE= $16$ (CER= $0.03$ ).

Block	NFE	Gen-PPL	Bert-Score	CER
32	8	59.538	0.901	0.422
32	16	20.263	0.936	0.210
32	32	13.525	0.957	0.150
32	64	11.301	0.968	0.130
32	128	10.401	0.970	0.123
32	256	10.085	0.972	0.118

Table 1: Autoencoder: Performance measured via Generative Perplexity, and recovery error evaluated via Bert-Score and Character Error Rate (CER). The Table explores varying hyper-parameters of the MDM decoder, for a generative length of

256

tokens.

Panel of Figure 6 — Figure 6: Text Generation: MAUVE and Gen. Perplexity for base LLaDA, ConThenDisc and ConWithinDisc , vs. complexity (NFE). $n=512,K=8$ (top) and $n=1024,K=16$ (bottom). In all cases, the whole sequence is treated as one block.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Stability under latent perturbations. We add i.i.d. Gaussian noise of standard deviation $\sigma$ to the conditioning register bank at evaluation time and report masked-token Top-1 accuracy (top) and mean cross-entropy (bottom). The Guided Demasker trained with register-noise augmentation degrades gradually as $\sigma$ increases, while the variant trained without this augmentation deteriorates substantially earlier and more sharply. These results refer to $90\%$ masking.

Figure 8: Register-space interpolation aggregated over $1000$ program pairs. We linearly interpolate between two encoded register banks and decode at $\alpha\in\{0,0.25,0.5,0.75,1\}$ . We report MAUVE (higher is better); numbers denote Generative Perplexity (lower is better), averaged over the same decoded sets. Quality remains high across the interpolation path, including at the midpoint between two random programs.

Figure 9: PSNR of denoiser predictions against ground-truth registers across diffusion time. Conditioning ( ConWithinDisc ) yields consistently higher PSNR than the unconditional model ( ConThenDisc ), demonstrating improved reconstruction from noisy registers.

Figure 10: The proposed autoencoder: a token sequence

{\mathbf{x}}_{0}

is encoded to a latent representation

{\mathbf{z}}_{0}

, and its decoding is done via a regular MDM.

Block	NFE	Gen-PPL	Bert-Score	CER
32	16	35.477	0.833	1.149
32	32	15.554	0.877	0.237
32	64	9.630	0.904	0.146
32	128	8.170	0.959	0.118
32	256	7.583	0.960	0.105
64	8	33.738	0.764	1.100

Appendix

Table 2: Autoencoder: Performance measured via Generative Perplexity, and recovery error evaluated via Bert-Score and Character Error Rate (CER). The Table explores varying hyper-parameters of the MDM decoder, for a generative length of

512

tokens.

Panel of Figure 11 — Figure 11: Ablations at length 512. Top: register-count ablation under ConThenDisc (varying $K$ ). Increasing the number of registers beyond the baseline does not improve and can degrade performance, suggesting that learning and exploiting additional fine registers is challenging and not necessarily beneficial for discrete demasking. Bottom: conditioning refresh ablation, updating after a fraction $\alpha$ of LLaDA steps. The midpoint update ( @0.50 ) is most effective; updating too early resembles the no-refresh baseline ( @0 , i.e., ConThenDisc ), while updating too late leaves limited room to affect generation as more tokens are already committed and remain unchanged during demasking.

Figure 12: Text Generation: MAUVE and Generative Perplexity of generated text with base LLaDA, ConThenDisc and ConWithinDisc . The sequence generated is of length $256$ tokens, handled as one block, and the number of registers in the embedding is $K=8$ .

Panel of Figure 13 — Figure 13: Qualitative code generations (length 256), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (256 discrete denoising steps, single block).

Panel of Figure 14 — Figure 14: Qualitative code generations (length 256), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (128 discrete denoising steps, single block).

Panel of Figure 15 — Figure 15: Qualitative code generations (length 256), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (64 discrete denoising steps, single block).

Panel of Figure 16 — Figure 16: Qualitative code generations (length 256), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (32 discrete denoising steps, single block).

Panel of Figure 17 — Figure 17: Qualitative code generations (length 512), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (256 discrete denoising steps, single block).

Panel of Figure 18 — Figure 18: Qualitative code generations (length 512), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (128 discrete denoising steps, single block).

Panel of Figure 19 — Figure 19: Qualitative code generations (length 512), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (64 discrete denoising steps, single block).

Panel of Figure 20 — Figure 20: Qualitative code generations (length 512), samples 1–2. Continuous registers are sampled with ConThenDisc (128 DDIM steps) and decoded with LLaDA (32 discrete denoising steps, single block).

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 443 papers

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
Mar 3, 2026Junzhe Shen, Jieru Zhao, Ziwei He +1Continuous Diffusion Language Models Diffusion Language Model Decoding
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute +1
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language Models Diffusion Language Model Distillation
NVIDIA
Consistent Diffusion Language Models
Apr 30, 2026Hasan Amin, Yuan Gao, Yaser Souri +4Few-Step Diffusion Sampling Diffusion Model Acceleration
Department of Computer Science, Purdue University · Microsoft

Mar 3, 2026J/K move · Enter open · S save

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

Junzhe Shen, Jieru Zhao, Ziwei He, Zhouhan Lin

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute+1

Continuous Diffusion Language Models Diffusion Language Model Decoding

Open paper AskPDF arXiv

We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.

Abstract

Diffusion Language Models

Explore similar work

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Consistent Diffusion Language Models