LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Yuxin Chen; Chumeng Liang; Hangke Sui; Ruihan Guo; Chaoran Cheng; Jiaxuan You; Ge Liu

cs.CLApr 13, 2026

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Authors: Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu

Organizations: UIUC

Abstract

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Figures & tables

Figure 1: LangFlow takes noisy token embeddings as inputs and predicts clean probabilities. In the training ( Left ) , we first map discrete token sequence ${\bm{x}}$ to token embeddings ${\bm{z}}$ via a learnable embedding matrix ${\mathbf{E}}$ . Then, we perturb ${\bm{z}}$ into noisy embeddings ${\bm{z}}_{\gamma}$ according to $\gamma$ , the sampled logarithmic noise-to-signal ratio. Finally, we use a denoiser network to predict categorical distribution $\hat{\bm{x}}_{\bm{\theta}}({\bm{z}}_{\gamma},\gamma)$ and supervise this prediction with the cross-entropy loss. Self-conditioning is randomly activated. In the sampling ( Right ) , we start from Gaussian noise ${\bm{z}}_{\gamma_{0}}$ and iteratively denoise it towards clean token embeddings ${\bm{z}}_{\gamma_{N}}$ . We define the moving target $\hat{{\bm{z}}}_{\bm{\theta}}$ for every step by embedding predicted clean probabilities $\hat{\bm{x}}_{\bm{\theta}}$ with ${\mathbf{E}}$ . The final token sequence is obtained by $\arg\max$ decoding of $\hat{\bm{x}}_{\bm{\theta}}({\bm{z}}_{\gamma_{N}},\gamma_{N})$ .

Panel of Figure 2 — Figure 2: Loss geometry. (Left) Training loss as a function of the OT flow matching time $t=\sigmoid(\gamma/2)$ . (Middle) Loss over $\gamma=\log\mathrm{NSR}$ reveals a consistent geometry across training stages. (Right) The derivative $\partial\mathcal{L}/\partial\gamma$ concentrates in a narrow region, forming a stable structure.

Model	Gen. PPL $\downarrow$	$\Delta$	PPL $\downarrow$	$\Delta$
MDLM	103.9	–	31.0	–
+Self-Conditioning	94.9	-9.0	32.7	+1.7
LangFlow	154.2	–	49.0	–
+Self-Conditioning	81.5	-72.7	30.0	-19.0

Table 1: Self-conditioning ablations on LM1B.

Autoregressive
Model	LM1B		OWT
	Gen. PPL $\downarrow$	PPL $\downarrow$	Gen. PPL $\downarrow$	PPL $\downarrow$
Transformer	66.7 ‡	22.8 ‡	35.9	17.5
Diffusion (Absorbing-state)
D3PM Absorb	–	76.9	–	–
DiffusionBert	–	63.8	–	–

Table 2: Performance comparison on LM1B and OWT. We report upper bounds on perplexity (PPL) for DLMs and generative perplexities (Gen. PPL) based on GPT-2 ( Radford et al., 2019 ) . ‡ notes results of retrained baselines. We use gold , silver , and bronze to note the 1st, 2nd, and 3rd place among DLMs, respectively.

Autoregressive
Model	PTB	Wikitext	LM1B	Lambada	AG News	Pubmed	Arxiv
Transformer †	82.05	25.75	51.25	51.28	52.09	49.01	41.73
Diffusion (Absorbing-state)
SEDD Absorb †	100.09	34.28	68.20	49.86	62.09	44.53	38.48
D3PM Absorb ¶	200.82	50.86	138.92	93.47	–	–	–
MDLM †	95.26	32.83	67.01	47.52	61.15	41.89	37.37

Table 3: Zero-shot perplexities (

\downarrow

) of models trained for 1M steps on OWT. All perplexities for DLMs are upper bounds.

\dagger

Taken from Sahoo et al. (2024) .

\lx@paragraphsign

Taken from Lou et al. (2023) , where these baselines were trained for 1.3M steps as opposed to ours and other baselines that were trained for 1M steps. ‡ Taken from Sahoo et al. (2025a) . We use gold , silver , and bronze to note the 1st, 2nd, and 3rd place among DLMs, respectively.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6

Feature	Ours & Other Baselines	Plaid
# Parameters	130M	108M
Backbone	Transformer	Transformer
Embedding Dim	768	16
Layers / Hidden Dim	16 / 768	16 / 768
Time Conditioning	AdaLN	Addition
Training Budget (GPU hours / GPU)	$\sim$ 292	$\sim$ 375

Appendix

Table 4: Model and training setup.

Figure 3: Comparison of Nearest Neighbor Distance (NND) among embeddings of four language modeling architectures: autoregressive (AR), MDLM ( Sahoo et al., 2024 ) , Plaid ( Gulrajani and Hashimoto, 2023 ) , and LangFlow. Compared to the other three architectures, the NNDs of Plaid cluster around zero, indicating the indistinguishability of different embeddings (i.e., mode collapse).

Figure 4: Self-conditioning dynamics on LM1B. The clean token is “run”. Dashed lines denote pass 1 (without SC), and solid lines denote pass 2 (with SC). We visualize the evolution of $p(x\mid z_{\gamma})$ across token groups.

Data Entropy=4.32
NFE	Gen. PPL	Entropy
128	92.24	4.31
64	104.83	4.32
32	127.32	4.33
16	179.60	4.35

Appendix

Table 5: NFE comparison on LM1B. We report the Gen. PPL and entropy for LangFlow under different NFEs.

	LangFlow		Duo		SEDD Uniform		MDLM		SEDD Absorb
NFE	Gen. PPL	Entropy	Gen. PPL	Entropy	Gen. PPL	Entropy	Gen. PPL	Entropy	Gen. PPL	Entropy
1024	36.53	5.25	77.69	5.55	99.90	5.56	104.85	5.63	105.03	5.62
512	41.52	5.31	78.14	5.55	100.44	5.56	104.43	5.63	104.45	5.62
256	49.24	5.37	78.62	5.55	103.41	5.56	112.70	5.66	109.82	5.63
128	60.09	5.43	80.02	5.55	105.82	5.57	120.77	5.67	117.28	5.65
64	80.34	5.51	85.62	5.57	113.02	5.57	143.88	5.70	138.42	5.67

Appendix

Table 6: NFE comparison on OWT. We compare LangFlow with Duo, MDLM, and SEDD under different sampling budgets. The data entropy (5.44) is quoted from Lee et al. (2026) .

	LangFlow	AR	Duo	MDLM
Word	health	body	NYCFC	Mitchell
Count	11	4	8	13
Entropy	5.3375	5.5742	5.4827	5.2111

Appendix

Table 7: Maximum frequency of a single content word in generated samples. We report the most frequent content word and its occurrence count for each method.

Self-Conditioning

See all 10 papers

Flow Map Language Models: One-step Language Modeling via Continuous Denoising
Feb 18, 2026Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal +6Continuous Diffusion Language Models One-Step Generative Modeling
KAIST · Carnegie Mellon University
ELF: Embedded Language Flows
May 11, 2026Keya Hu, Linlu Qiu, Yiyang Lu +5Flow-Based Generative Modeling Diffusion Language Models
MIT
ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings
Aug 24, 2026Na Li, Yuchen Jiao, Changxiao Cai +1Flow-Based Generative Modeling Continuous Diffusion Language Models
Department of Statistics and Data Science, Chinese University of Hong Kong, Hong Kong · Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA

Feb 18, 2026J/K move · Enter open · S save

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah+5

KAIST · Carnegie Mellon University

Continuous Diffusion Language Models One-Step Generative Modeling Flow-Based Generative Modeling Generative Flow Networks Language Model Distillation

Open paper AskPDF arXiv

Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.

Abstract

Self-Conditioning

Explore similar work

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

ELF: Embedded Language Flows

ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings