The sequential nature of autoregressive next-token prediction imposes a fundamental speed limit on large language models. While continuous flow models offer a path to parallel generation, they traditionally demand expensive iterative integration. Flow Maps bypass this bottleneck by compressing generative trajectories into single-step mappings, theoretically enabling the generation of full text sequences from noise in a single forward pass. However, standard formulations rely on Euclidean regression losses that are geometrically ill-suited for discrete data. In this work, we resolve this conflict with Discrete Flow Maps, a framework that reconciles trajectory compression with the geometry of the probability simplex. We recast standard flow map training for the discrete domain, aligning the training dynamics with the discrete nature of language. Empirically, this strict geometric alignment allows our method to surpass previous state-of-the-art results in discrete flow modeling.
Figures & tables
Figure 1: Generative perplexity as a function of the number of sampling steps on the LM1B dataset, comparing Discrete Flow Maps (DFM) with other accelerated methods. This highlights the ability of DFMs to generate higher-quality text with fewer sampling steps.
Figure 2: Overview of the geometry of the discrete flow map. Left: The instantaneous denoiser ψs,s(xs) at any time s is always on the simplex by weighted sum. Right: By derivation of ( 19 ), the mean denoiser ψs,t(xs) is also always a projection onto the simplex, making possible a direct parameterization of the flow map Xs,t that enables our cross-entropy loss functions.
Figure 3: Interpolants with initial point I0∈RK from ρ0 randomly connecting to simplex vertices.
Figure 4: Illustration of block generation with mixed attention masking. A fixed context (green) is processed with causal attention, while a block of future tokens (red) is generated in parallel using bidirectional attention within the block and cross-attention to the context. The attention mask (left) visualizes the masking structure, and the right panel shows how cached key–value (KV) states enable efficient reuse of context while generating successive blocks.
LM1B
OWT
Method
Metric
1
2
4
Method
Metric
1
2
4
Duo + DCD
gen. PPL ↓
1224.52
520.08
210.88
Duo + DCD
gen. PPL ↓
5743.29
891.16
250.86
entropy ↑
4.33
4.20
4.23
entropy ↑
6.02
5.41
5.37
Duo + Di4C
gen. PPL
292.94
247.69
150.67
Duo + Di4C
gen. PPL
370.51
210.22
154.67
entropy
3.79
3.87
4.00
entropy
3.92
4.63
4.85
MDLM + SDTT
gen. PPL
1429.48
602.14
241.01
MDLM + SDTT
gen. PPL
1260.86
877.22
339.73
Table 1: Generative perplexity ( ↓ ) and entropy ( ↑ ) across number of function evaluations (NFEs) for LM1B and OWT.
ω
gen. PPL ↓
entropy ↑
0.0
56.31
5.20
0.5
46.78
5.06
1.0
36.44
4.94
1.5
33.22
4.87
2.0
30.98
4.81
Table 2: gen. PPL ( ↓ ) and entropy ( ↑ ) across CFG scales, ω . Samples are generated in four blocks of 256 tokens, each with 1024 steps.
Dataset
Stage
Metric
1
2
4
8
16
128
256
1024
LM1B
Diagonal
gen. PPL ↓
2.05
69.59
204.13
124.64
101.89
75.82
72.75
68.33
entropy ↑
0.76
2.87
4.37
4.32
4.27
4.19
4.17
4.16
+ Distillation (PSD)
gen. PPL
94.08
87.42
78.89
69.90
64.90
58.38
56.59
56.31
entropy
4.06
4.08
4.10
4.10
4.11
4.10
4.10
4.11
+ Distillation (ESD)
gen. PPL
68.11
77.60
71.53
65.61
59.92
55.70
56.88
58.27
entropy
3.79
4.11
4.13
4.13
4.13
4.11
4.12
4.12
Table 3: Generative perplexity ( ↓ ) and entropy ( ↑ ) across number of function evaluations (NFEs).
Figure 5: Example LM1B generations using 2 and 1024 function evaluations.
Figure 6: Example OWT generation using CFG ( ω=0.0 ) and block sampling to generate 4 blocks of size 256. A change in text colour denotes a new block.
Figure 7: Example OWT generation using CFG ( ω=1.0 ) and block sampling to generate 4 blocks of size 256. A change in text colour denotes a new block.
Figure 8: Example OWT generation using CFG ( ω=2.0 ) and block sampling to generate 4 blocks of size 256. A change in text colour denotes a new block.
Figure 9: Example OWT generation using CFG ( ω=3.0 ) and block sampling to generate 4 blocks of size 256. A change in text colour denotes a new block.
Flow map learning trains neural models to predict finite-time state or distribution transitions directly, enabling fast simulation, control, and generation.
Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.
Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established. We present Lafa (Large flow language model), a fast, high-performing flow language model scaled to 8B parameters in a data-efficient manner. Lafa is a blockwise flow model with causal attention on the prefix and bidirectional, asynchronous flow within each block. This design enables efficient training by adapting pretrained autoregressive (AR) models and fast inference via prefix caching and blockwise parallel decoding. We train Lafa at 0.5B, 7B, and 8B scales with fewer than 30B tokens, more efficient than previous diffusion models, and evaluate it on instruction following, reasoning, and coding, where the learned flow matches or surpasses the quality of AR and can be distilled into flow maps for one- and few-step generation. Notably, flow training preserves or even improves pretrained AR capabilities via a multi-token prediction effect, which we leverage in a self-speculative decoding scheme that verifies flow-based drafts autoregressively within a single model. This yields 2.54x the throughput of an AR model at comparable quality, outperforming multi-token prediction and discrete diffusion methods. Our findings show that continuous flows can serve as a scalable foundation for language modeling.
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow