Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.
Figures & tables
Figure 1: Flow map language models. Our FMLM outperforms discrete diffusion models ( gray ) and matches the 8-step generation performance of distilled discrete diffusion models ( light purple ) in only one step ( dark purple ).
Figure 2: Overview. (Left) We leverage a simple continuous interpolation between Gaussian noise and a one-hot encoding of language data. (Middle) Our FLM learns a denoiser that predicts the posterior over clean data, which we convert into a flow for sampling. (Right) Our distilled FMLM directly transports states between distant timepoints, enabling few-step generation.
Figure 3: Factorization error in discrete diffusion. A toy dataset with two correlated modes new - york and san - diego . (Left) In many-step sampling, both continuous flows and discrete diffusion models generate valid data. (Right) With few-step sampling, the factorized transition of discrete diffusion yields a spurious mixture of all possible combinations (including the invalid pairings new - diego and san - york ).
Figure 4: Semigroup on the simplex. Xs,u(x) leaves the simplex, but δs,u(x) and δu,t(Xs,u) always lie on it. δs,t(x) is their convex combination, providing a training signal for distillation.
Figure 5Figure 6
Figure 9: Decoding error rate. Our time reparameterization τ(t) redistributes time so each step contributes uniformly to the denoising signal; time samples shown in ticks.
Model
LM1B
OWT
Gen. PPL ( ↓ )
Entropy (4.31)
Gen. PPL ( ↓ )
Entropy (5.44)
RDLM
268.21
4.33
-
-
CANDI
120.99
4.35
143.13
5.71
MDLM
109.21
4.32
105.15
5.63
Duo
98.14
4.31
77.69
5.55
FLM (Ours)
96.91
4.29
62.23
5.33
Table 10: FLM Performance. Generation performance of FLM at 1024 sampling steps in comparison to discrete diffusion baselines. Ground truth dataset entropy shown in parentheses. Our approach attains state-of-the-art generative perplexity while maintaining entropy close to the data.
Figure 11: FLM generation quality. Generation performance of FLM on LM1B ( left ) and OWT ( right ) compared to diffusion baselines. FLM outperforms baselines at large step counts. Its performance degrades at low step counts, as it has not yet been distilled into an FMLM .
Figure 12: FMLM few-step generation. Few-step generation performance of FMLM on LM1B ( left ) and OWT ( right ) compared to distilled discrete diffusion. FMLM maintains strong generative perplexity across step counts and achieves state-of-the-art performance in the very few-step regime. Performance degrades slightly as the step count decreases and can be improved with further distillation.
LM1B
MDLM + SDTT
MDLM + Di4C
Duo + DCD
Duo + Di4C
FMLM (Ours)
Steps
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
1
1429.48
4.31
1217.10
4.38
1224.52
4.33
292.94
3.79
119.34
4.16
2
602.14
4.28
621.59
4.37
520.08
4.20
247.69
3.87
110.19
4.21
4
241.01
4.28
247.32
4.00
210.88
4.23
150.67
4.00
98.76
4.21
OWT
MDLM + SDTT
MDLM + Di4C
Duo + DCD
Duo + Di4C
FMLM (Ours)
Steps
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
Gen. PPL (↓)
Ent.
Table 13: FMLM performance. Comparison between FMLM and baseline few-step distilled discrete diffusion models in the extreme few-step regime. Similar to the many-step regime, our model attains state of the art performance and while maintaining the entropy of the generated samples.
Figure 14: Qualitative one-step generation. One-step samples from FMLM and distilled discrete diffusion baselines trained on LM1B. FMLM produces coherent, grammatical text, while discrete diffusion baselines generate incoherent token sequences ( red , Gen. PPL > 1000) or repetitive tokens with collapsed entropy ( red , Entropy < 4).
Gen. PPL ( ↓ )
Ent.
MDLM + SDTT
374.70
4.09
MDLM + Di4C
376.84
4.10
Duo + DCD
703.18
4.18
Duo + Di4C
451.29
4.08
FMLM (Ours)
141.66
4.14
Table 15: FMLM one-step conditional generation.
Figure 16: Autoguidance stability. FLM maintains stable generation quality across guidance scales η up to 100, while discrete baselines fail at η≥10 . Shaded region shows Gen. PPL >1000 or entropy <3.9 , indicating nonsensical or collapsed generation. Results shown on LM1B across 128–1024 sampling steps.
Topic
Grammaticality
Sentiment
Safety
Method
Gen.PPL (↓) / Reward (↑)
Gen.PPL (↓) / Reward (↑)
Gen.PPL (↓) / Reward (↑)
Gen.PPL (↓) / Reward (↑)
1024 Steps
MDLM
115.1 / 0.703
106.3 / 0.255
118.6 / 0.614
123.9 / 0.835
Duo
87.5 / 0.840
96.0 / 0.202
94.9 / 0.867
95.2 / 0.871
FLM (Ours)
69.3 / 0.921
59.8 / 0.240
84.0 / 0.940
73.1 / 0.930
8 Steps
Table 17: Reward-guided generation. We apply FMRG [ 50 ] to steer generation toward four target properties (topic, grammaticality, sentiment, safety) across varying step counts on OWT. At 1024 steps, the undistilled teacher models (MDLM, Duo) are used; at fewer steps, their distilled counterparts (MDLM+SDTT, Duo+DCD) are used. FMLM achieves higher reward scores while maintaining lower generative perplexity than discrete baselines at all step counts.
FLM , 1024-step generation
Configuration
Gen. PPL ( ↓ )
Entropy
Parameterization and loss
Velocity, MSE
3801.36
4.85
Denoiser, MSE
129.04
3.97
Denoiser + softmax, MSE
120.16
4.28
Denoiser + softmax, CE
96.91
4.29
Table 18: Ablation results on the LM1B dataset. * denotes 300k training steps. All embedding ablation models use learned time reparameterization from Dieleman et al. [13] .
Figure 17
Unconditional
Conditional, held-out puzzles
Conditional, Sudoku-Extreme
Steps
1
2
4
1024
1
2
4
1024
1
2
4
1024
MDLM+SDTT [ 7 ]
0.00
0.00
0.02
92.19
0.00
2.93
8.40
98.63
0.00
0.00
7.62
24.12
Duo+DCD [ 19 ]
0.00
0.39
17.19
91.41
0.00
30.85
73.82
96.88
0.00
8.01
16.80
27.73
FMLM (Ours)
9.38
57.03
79.69
94.92
39.84
85.15
92.19
98.83
16.01
20.70
25.00
26.56
Table 22: Sudoku generation and solving validity (% among 1024 samples).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 23: Eulerian and Lagrangian objectives on the simplex. (Left) The Eulerian teacher δˉs,t is constructed from D^s(Is) and derivatives of δ^s,t . (Right) The Lagrangian teacher is constructed from D^t(X^s,t(Is)) , requiring an intermediate flow map evaluation off the simplex. In both cases, the teacher may transiently leave the simplex during training due to derivative correction terms, but the cross-entropy loss remains well-defined since only the student δ^s,t(Is) (parameterized with softmax) must lie on the simplex. At optimality, all quantities lie on the simplex.
Model Size
Small (179M)
Medium (424M)
Large (870M)
Gen. PPL ( ↓ )
75.57
65.94
61.59
Entropy
5.40
5.42
5.39
Appendix
Table 27: Performance of FLM trained on OWT for 150k steps, across different model sizes.
Prior
Uniform
Gaussian
Gen. PPL ( ↓ )
660.33
105.48
Entropy
4.12
4.31
Appendix
Table 28: Performance of FMLM with uniform and Gaussian priors trained on LM1B dataset for 200k steps.
Figure 31: Ablation on flow map characterization. Training loss and generation quality (Gen. PPL) of different flow map distillation objectives.
∣V∣=30k , L=128
∣V∣=30k , L=1024
∣V∣=50k , L=128
∣V∣=50k , ∣L∣=1024
MDLM [ 29 ]
8.9GB / 0.0693s
40.8GB / 0.3086s
11.3GB / 0.0703s
54.5GB / 0.3922s
Duo [ 19 ]
10.2GB / 0.0707s
54.7GB / 0.3559s
13.5GB / 0.0861s
78.0GB / 0.4717s
FLM
10.3GB / 0.0661s
48.3GB / 0.2959s
14.0GB / 0.0674s
63.4GB / 0.3759s
Appendix
Table 32: Memory and time cost per training step. Peak GPU memory (GB) and wall-clock time (seconds) per training step for FLM and discrete diffusion baselines, measured across vocabulary sizes ∣V∣∈{30k,50k} and sequence lengths L∈{128,1024} .
Figure 33: Qualitative one-step conditional generation. One-step conditional samples from FMLM and distilled discrete diffusion baselines trained on OWT. Generated text are colored . While FMLM generates grammarly correct sentence highly correlated with the given prefix, other baselines fail to generate proper sentence or be aligned with the given prefix.
Figure 34: Samples generated by FLM trained on LM1B with different sampling steps.
Figure 35: Samples generated by FLM trained on OWT with different sampling steps.
Figure 36: One-step samples generated by FMLM trained on LM1B.
Figure 37: One-step conditional samples generated by FMLM trained on OWT. Generated text are colored .
Figure 38: One-step samples generated by FMLM trained on OWT.
Figure 39: One-step samples generated by few-step masked discrete diffusion baselines trained on OWT.
Figure 40: One-step samples generated by few-step uniform discrete diffusion baselines trained on OWT.
Figure 41: Samples from FMLM trained on LM1B from fixed starting noise and varying the number of steps.
Figure 42: Samples from FMLM trained on OWT from fixed starting noise and varying the number of steps.
Figure 43: Samples generated by MDLM + SDTT [ 10 ] trained on LM1B from fixed initial random seed and varying the number of sampling steps.
Figure 44: Samples generated by Duo + DCD [ 19 ] trained on LM1B from fixed initial random seed and varying the number of sampling steps.
Figure 45: A sample from FMLM +FMRG ( Section 5.3 ), rewarded by safety (TweetVal-Offensive [ 64 ] , Label=Non-offensive).
Figure 46: A sample from FMLM +FMRG ( Section 5.3 ), rewarded by topic (AG News [ 61 ] , Label=Sports).
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.
We present FlowLM, a flow matching language model transformed from pre-trained diffusion language models via efficient fine-tuning. By re-aligning the curved sampling trajectories of diffusion models into straight-line flows, FlowLM enables high quality few-step generation that rivals or even outperforms the quality of 2,000-step diffusion sampling with very few training epochs. Remarkably, finetuned FlowLM reaches performance saturation with only half as many training epochs as training from scratch, both approaches greatly outperforming the original diffusion model, thereby validating our method. Furthermore, we validate a more effective training objective for flow matching: predicting clean data to consistently guide the sampling process towards the true data distribution. Empirical results demonstrate that our approach is highly effective for high-quality, few-step text generation.