Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Chanhyuk Lee; Jaehoon Yoo; Manan Agarwal; Sheel Shah; Jerry Huang; Aditi Raghunathan; Seunghoon Hong; Nicholas M. Boffi

cs.CLFeb 18, 2026

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Authors: Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Seunghoon Hong, Nicholas M. Boffi,

Organizations: KAIST · Carnegie Mellon University

Abstract

Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.

Figures & tables

Figure 1: Flow map language models. Our FMLM outperforms discrete diffusion models ( gray ) and matches the 8-step generation performance of distilled discrete diffusion models ( light purple ) in only one step ( dark purple ).

Figure 2: Overview. (Left) We leverage a simple continuous interpolation between Gaussian noise and a one-hot encoding of language data. (Middle) Our FLM learns a denoiser that predicts the posterior over clean data, which we convert into a flow for sampling. (Right) Our distilled FMLM directly transports states between distant timepoints, enabling few-step generation.

Figure 3: Factorization error in discrete diffusion. A toy dataset with two correlated modes new - york and san - diego . (Left) In many-step sampling, both continuous flows and discrete diffusion models generate valid data. (Right) With few-step sampling, the factorized transition of discrete diffusion yields a spurious mixture of all possible combinations (including the invalid pairings new - diego and san - york ).

Figure 4: Semigroup on the simplex.

X_{s,u}({\bf x})

leaves the simplex, but

\delta_{s,u}({\bf x})

and

\delta_{u,t}(X_{s,u})

always lie on it.

\delta_{s,t}({\bf x})

is their convex combination, providing a training signal for distillation.

Figure 5

Figure 6

Figure 9: Decoding error rate. Our time reparameterization

\tau(t)

redistributes time so each step contributes uniformly to the denoising signal; time samples shown in ticks.

Model	LM1B		OWT
Model	Gen. PPL ( $\downarrow$ )	Entropy (4.31)	Gen. PPL ( $\downarrow$ )	Entropy (5.44)
RDLM	268.21	4.33	-	-
CANDI	120.99	4.35	143.13	5.71
MDLM	109.21	4.32	105.15	5.63
Duo	98.14	4.31	77.69	5.55
FLM (Ours)	96.91	4.29	62.23	5.33

Table 10: FLM Performance. Generation performance of FLM at 1024 sampling steps in comparison to discrete diffusion baselines. Ground truth dataset entropy shown in parentheses. Our approach attains state-of-the-art generative perplexity while maintaining entropy close to the data.

Figure 11: FLM generation quality. Generation performance of FLM on LM1B ( left ) and OWT ( right ) compared to diffusion baselines. FLM outperforms baselines at large step counts. Its performance degrades at low step counts, as it has not yet been distilled into an FMLM .

Figure 12: FMLM few-step generation. Few-step generation performance of FMLM on LM1B ( left ) and OWT ( right ) compared to distilled discrete diffusion. FMLM maintains strong generative perplexity across step counts and achieves state-of-the-art performance in the very few-step regime. Performance degrades slightly as the step count decreases and can be improved with further distillation.

LM1B	MDLM + SDTT		MDLM + Di4C		Duo + DCD		Duo + Di4C		FMLM (Ours)
Steps	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.
1	1429.48	4.31	1217.10	4.38	1224.52	4.33	292.94	3.79	119.34	4.16
2	602.14	4.28	621.59	4.37	520.08	4.20	247.69	3.87	110.19	4.21
4	241.01	4.28	247.32	4.00	210.88	4.23	150.67	4.00	98.76	4.21
OWT	MDLM + SDTT		MDLM + Di4C		Duo + DCD		Duo + Di4C		FMLM (Ours)
Steps	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.	Gen. PPL $(\downarrow)$	Ent.

Table 13: FMLM performance. Comparison between FMLM and baseline few-step distilled discrete diffusion models in the extreme few-step regime. Similar to the many-step regime, our model attains state of the art performance and while maintaining the entropy of the generated samples.

Panel of Figure 14 — Figure 14: Qualitative one-step generation. One-step samples from FMLM and distilled discrete diffusion baselines trained on LM1B. FMLM produces coherent, grammatical text, while discrete diffusion baselines generate incoherent token sequences ( red , Gen. PPL $>$ 1000) or repetitive tokens with collapsed entropy ( red , Entropy $<$ 4).

	Gen. PPL ( $\downarrow$ )	Ent.
MDLM + SDTT	374.70	4.09
MDLM + Di4C	376.84	4.10
Duo + DCD	703.18	4.18
Duo + Di4C	451.29	4.08
FMLM (Ours)	141.66	4.14

Table 15: FMLM one-step conditional generation.

Figure 16: Autoguidance stability. FLM maintains stable generation quality across guidance scales

\eta

up to 100, while discrete baselines fail at

\eta\geq 10

. Shaded region shows Gen. PPL

>1000

or entropy

<3.9

, indicating nonsensical or collapsed generation. Results shown on LM1B across 128–1024 sampling steps.

	Topic	Grammaticality	Sentiment	Safety
Method	Gen.PPL $(\downarrow)$ / Reward $(\uparrow)$	Gen.PPL $(\downarrow)$ / Reward $(\uparrow)$	Gen.PPL $(\downarrow)$ / Reward $(\uparrow)$	Gen.PPL $(\downarrow)$ / Reward $(\uparrow)$
1024 Steps
MDLM	115.1 / 0.703	106.3 / 0.255	118.6 / 0.614	123.9 / 0.835
Duo	87.5 / 0.840	96.0 / 0.202	94.9 / 0.867	95.2 / 0.871
FLM (Ours)	69.3 / 0.921	59.8 / 0.240	84.0 / 0.940	73.1 / 0.930
8 Steps

Table 17: Reward-guided generation. We apply FMRG [ 50 ] to steer generation toward four target properties (topic, grammaticality, sentiment, safety) across varying step counts on OWT. At 1024 steps, the undistilled teacher models (MDLM, Duo) are used; at fewer steps, their distilled counterparts (MDLM+SDTT, Duo+DCD) are used. FMLM achieves higher reward scores while maintaining lower generative perplexity than discrete baselines at all step counts.

FLM , 1024-step generation
Configuration	Gen. PPL ( $\downarrow$ )	Entropy
Parameterization and loss
Velocity, MSE	3801.36	4.85
Denoiser, MSE	129.04	3.97
Denoiser + softmax, MSE	120.16	4.28
Denoiser + softmax, CE	96.91	4.29

Table 18: Ablation results on the LM1B dataset. * denotes 300k training steps. All embedding ablation models use learned time reparameterization from Dieleman et al. [13] .

	Unconditional				Conditional, held-out puzzles				Conditional, Sudoku-Extreme
Steps	1	2	4	1024	1	2	4	1024	1	2	4	1024
MDLM+SDTT [ 7 ]	0.00	0.00	0.02	92.19	0.00	2.93	8.40	98.63	0.00	0.00	7.62	24.12
Duo+DCD [ 19 ]	0.00	0.39	17.19	91.41	0.00	30.85	73.82	96.88	0.00	8.01	16.80	27.73
FMLM (Ours)	9.38	57.03	79.69	94.92	39.84	85.15	92.19	98.83	16.01	20.70	25.00	26.56

Table 22: Sudoku generation and solving validity (% among 1024 samples).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 23: Eulerian and Lagrangian objectives on the simplex. (Left) The Eulerian teacher

\bar{\delta}_{s,t}

is constructed from

\hat{D}_{s}(I_{s})

and derivatives of

\hat{\delta}_{s,t}

. (Right) The Lagrangian teacher is constructed from

\hat{D}_{t}(\hat{X}_{s,t}(I_{s}))

, requiring an intermediate flow map evaluation off the simplex. In both cases, the teacher may transiently leave the simplex during training due to derivative correction terms, but the cross-entropy loss remains well-defined since only the student

\hat{\delta}_{s,t}(I_{s})

(parameterized with softmax) must lie on the simplex. At optimality, all quantities lie on the simplex.

Model Size	Small (179M)	Medium (424M)	Large (870M)
Gen. PPL ( $\downarrow$ )	75.57	65.94	61.59
Entropy	5.40	5.42	5.39

Appendix

Table 27: Performance of FLM trained on OWT for 150k steps, across different model sizes.

Prior	Uniform	Gaussian
Gen. PPL ( $\downarrow$ )	660.33	105.48
Entropy	4.12	4.31

Appendix

Table 28: Performance of FMLM with uniform and Gaussian priors trained on LM1B dataset for 200k steps.

Figure 31: Ablation on flow map characterization. Training loss and generation quality (Gen. PPL) of different flow map distillation objectives.

	$\|V\|=30k$ , $L=128$	$\|V\|=30k$ , $L=1024$	$\|V\|=50k$ , $L=128$	$\|V\|=50k$ , $\|L\|=1024$
MDLM [ 29 ]	8.9GB / 0.0693s	40.8GB / 0.3086s	11.3GB / 0.0703s	54.5GB / 0.3922s
Duo [ 19 ]	10.2GB / 0.0707s	54.7GB / 0.3559s	13.5GB / 0.0861s	78.0GB / 0.4717s
FLM	10.3GB / 0.0661s	48.3GB / 0.2959s	14.0GB / 0.0674s	63.4GB / 0.3759s

Appendix

Table 32: Memory and time cost per training step. Peak GPU memory (GB) and wall-clock time (seconds) per training step for FLM and discrete diffusion baselines, measured across vocabulary sizes

|V|\in\{30k,50k\}

and sequence lengths

L\in\{128,1024\}

Panel of Figure 33 — Figure 33: Qualitative one-step conditional generation. One-step conditional samples from FMLM and distilled discrete diffusion baselines trained on OWT. Generated text are colored . While FMLM generates grammarly correct sentence highly correlated with the given prefix, other baselines fail to generate proper sentence or be aligned with the given prefix.

Panel of Figure 34 — Figure 34: Samples generated by FLM trained on LM1B with different sampling steps.

Panel of Figure 35 — Figure 35: Samples generated by FLM trained on OWT with different sampling steps.

Panel of Figure 36 — Figure 36: One-step samples generated by FMLM trained on LM1B.

Panel of Figure 37 — Figure 37: One-step conditional samples generated by FMLM trained on OWT. Generated text are colored .

Panel of Figure 38 — Figure 38: One-step samples generated by FMLM trained on OWT.

Panel of Figure 39 — Figure 39: One-step samples generated by few-step masked discrete diffusion baselines trained on OWT.

Panel of Figure 40 — Figure 40: One-step samples generated by few-step uniform discrete diffusion baselines trained on OWT.

Panel of Figure 41 — Figure 41: Samples from FMLM trained on LM1B from fixed starting noise and varying the number of steps.

Panel of Figure 42 — Figure 42: Samples from FMLM trained on OWT from fixed starting noise and varying the number of steps.

Panel of Figure 43 — Figure 43: Samples generated by MDLM + SDTT [ 10 ] trained on LM1B from fixed initial random seed and varying the number of sampling steps.

Panel of Figure 44 — Figure 44: Samples generated by Duo + DCD [ 19 ] trained on LM1B from fixed initial random seed and varying the number of sampling steps.

Panel of Figure 45 — Figure 45: A sample from FMLM +FMRG ( Section 5.3 ), rewarded by safety (TweetVal-Offensive [ 64 ] , Label=Non-offensive).

Panel of Figure 46 — Figure 46: A sample from FMLM +FMRG ( Section 5.3 ), rewarded by topic (AG News [ 61 ] , Label=Sports).

Continuous Diffusion Language Models

Continuous diffusion language models generate text by gradually denoising continuous token embeddings or latent representations.

See all 43 papers

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
Apr 13, 2026Yuxin Chen, Chumeng Liang, Hangke Sui +4Self-Conditioning Diffusion Language Models
UIUC
ELF: Embedded Language Flows
May 11, 2026Keya Hu, Linlu Qiu, Yiyang Lu +5Flow-Based Generative Modeling Diffusion Language Models
MIT
FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation
Apr 6, 2026Runzhe Zhang, Letian Chen, Wenpeng Zhang +2Language Modeling Flow-Based Generative Modeling
Shanghai Jiao Tong University · ByteDance

Apr 13, 2026J/K move · Enter open · S save

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo+3

UIUC

Self-Conditioning Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training Flow Matching

Open paper AskPDF arXiv

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Abstract

Continuous Diffusion Language Models

Explore similar work

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

ELF: Embedded Language Flows

FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation