Continuous Diffusion Language Models

Continuous diffusion language models study how to generate text by refining noisy continuous representations rather than choosing each token in a strictly left-to-right sequence. Common approaches use Gaussian diffusion or flow matching in token-embedding space, or learn compressed semantic latents with an encoder and decoder; some combine these with masked diffusion, autoregressive decoding, or distillation for faster sampling. These models are increasingly competitive with discrete diffusion and autoregressive baselines on language modeling, summarization, reasoning, and code benchmarks, with methods targeting generation in fewer denoising steps. Key challenges include reliably converting continuous states into coherent tokens, preventing repetition, and balancing sample quality, long-range coherence, and inference speed.