Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established. We present Lafa (Large flow language model), a fast, high-performing flow language model scaled to 8B parameters in a data-efficient manner. Lafa is a blockwise flow model with causal attention on the prefix and bidirectional, asynchronous flow within each block. This design enables efficient training by adapting pretrained autoregressive (AR) models and fast inference via prefix caching and blockwise parallel decoding. We train Lafa at 0.5B, 7B, and 8B scales with fewer than 30B tokens, more efficient than previous diffusion models, and evaluate it on instruction following, reasoning, and coding, where the learned flow matches or surpasses the quality of AR and can be distilled into flow maps for one- and few-step generation. Notably, flow training preserves or even improves pretrained AR capabilities via a multi-token prediction effect, which we leverage in a self-speculative decoding scheme that verifies flow-based drafts autoregressively within a single model. This yields 2.54x the throughput of an AR model at comparable quality, outperforming multi-token prediction and discrete diffusion methods. Our findings show that continuous flows can serve as a scalable foundation for language modeling.
Figures & tables
Figure 1: Large flow language models. Lafa 7B attains 2.02–2.54 × the throughput of an AR model at a matched or better quality, outperforming existing MTP and diffusion-based approaches.
Figure 2: Overview. (Top) Lafa learns a block denoiser that also performs AR prediction on the prefix. (Bottom) Lafa supports flow decoding, which can be further distilled into a flow map for one-step generation. Together, AR and flow support self-speculative decoding, which accepts flow-based drafts from left to right via AR verification.
Figure 3: Attention mask for block-bidirectional attention.
Figure 4: Parallelized training over a sequence composed of two blocks of size three. The uncorrupted sequence uses causal attention and the AR loss, while each noisy block attends to the preceding clean tokens and bidirectionally within itself, and uses the denoising losses.
Figure 5: Tokenwise time for each block position learned by Lafa 7B.
Figure 6
Figure 7: Self-speculative decoding. Accuracy ↑ vs. TPF ↑ averaged over all six benchmarks. In the right panel, Efficient-DLM and LLaDA are 8B and the other baselines are 7B. “Canvas” denotes iterative completion over the full sequence, while “32/8” denotes decoding 32-token blocks in four 8-token sub-blocks. Further details in Section C.1 .
Figure 8: Accuracy ↑ vs. end-to-end throughput measured in tokens per second (TPS) ↑ at 7–8B scales, averaged over IFEval, GSM8K, and HumanEval.
Configuration
Average (%)
Training
Fully causal attention
32.7
Block-bidirectional attention
38.0
Synchronous time
36.2
Asynchronous tokenwise time
38.0
Frozen embeddings
34.4
Table 3: Ablations. We use flow decoding and report mean accuracy over IFEval, GSM8K, and HumanEval.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11Figure 12
Figure 9: Fused self-speculative decoding with block size three. A single forward pass verifies previous draft with AR prediction and performs next-block drafting. This is done by decoding next drafts corresponding to every possible acceptance of the previous draft in parallel and selecting one based on verification.
Setting
Lafa 0.5B
Lafa 7B
Lafa 8B
Model and data
Initialization
Qwen2.5 0.5B Instruct
Qwen2.5 7B Instruct
Qwen3 8B Base
Training data / epochs / tokens
Dolci-Instruct-SFT / 5 epochs / approximately 9B
Block sizes B
4, 8, 16
4, 8, 16
4
Maximum sequence length
512–1024
512–1024
512
adaLN
Per-layer
Shared across layers
Shared across layers
Appendix
Table 4: Main hyperparameters. For block size four, we use batches of 384 sequences for training, whereas for larger blocks, we use memory-matched microbatches and, when applicable, continue training from smaller-block checkpoint.
0.5B scale
7B scale
Method
B
Euler steps
GSM8K
HumanEval
IFEval
Average
GSM8K
HumanEval
IFEval
Average
AR SFT
–
–
32.0
35.4
48.4
38.6
69.1
78.7
73.8
73.9
Flow decoding
Lafa (Ours)
16
1
0.8
3.0
20.3
8.1
2.4
10.4
29.8
14.2
2
1.7
6.1
23.5
10.4
18.7
21.3
36.2
25.4
4
11.7
12.2
27.0
17.0
47.5
40.2
44.7
44.1
Appendix
Table 5: Flow and flow map decoding accuracies.
Method
IFEval
BBH
MMLU-Pro
GPQA
GSM8K
HumanEval
Average
0.5B scale
Qwen2.5 SFT †
48.4
26.3
11.9
17.9
32.0
35.4
28.7
MTP † [ 6 ]
50.6
17.3
9.2
18.8
27.1
32.3
25.9
DeepSeek-MTP † [ 8 ]
53.2
24.1
14.0
18.5
35.5
36.0
30.2
MARS † [ 33 ]
51.3
26.6
12.4
19.4
32.8
40.2
30.4
SBD † [ 46 ]
55.3
20.6
9.2
22.5
27.2
30.8
27.6
Appendix
Table 6: Single-token decoding. For diffusion models, we decode exactly one highest-probability token per forward. Models marked † are trained by us from the matched AR model on Dolci-Instruct-SFT for five epochs.
IFEval
BBH
MMLU-Pro
GPQA
GSM8K
HumanEval
Average
Method
B
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
0.5B scale
AR SFT
–
48.4
1.0
26.3
1.0
11.9
1.0
17.9
1.0
32.0
1.0
35.4
1.0
28.7
1.0
MARS
4
51.0
1.8
31.7
2.3
13.9
1.7
18.3
2.0
31.7
2.1
32.3
1.9
29.8
2.0
MTP
4
51.2
2.5
17.6
3.3
9.4
3.1
17.9
2.8
26.8
2.9
33.5
2.9
26.1
2.9
DeepSeek-MTP
2
51.9
1.6
23.7
1.8
12.1
1.9
18.8
1.9
35.5
1.8
35.4
1.7
29.6
1.8
Appendix
Table 7: Self-speculative and parallel decoding accuracy (%) and TPF across model scales.
IFEval
GSM8K
HumanEval
Average
Method
Setting
Threshold
Accuracy
TPS
Accuracy
TPS
Accuracy
TPS
Accuracy
TPS
AR SFT
–
–
73.8
40.9
69.1
40.6
78.7
37.0
73.9
39.5
DeepSeek-MTP
B=2
–
78.0
45.1
70.3
57.2
80.5
58.4
76.3
53.6
MARS
B=4
0.95
77.6
42.0
70.7
59.8
79.9
43.3
76.1
48.4
B=8
0.95
77.3
40.9
69.1
62.7
78.0
43.1
74.8
48.9
B=16
0.95
75.0
41.1
72.9
61.3
75.0
44.6
74.3
49.0
Appendix
Table 8: Per-task accuracy (%) and throughput in tokens per second (TPS) at 7–8B scales, measured using one H100 GPU with batch size one. Efficient-DLM is 8B and other methods are 7B.
Figure 10: Confidence decoding, averaged over GSM8K, HumanEval, and IFEval. In the right panel, Lafa uses 8B for B=4 and 7B for B∈{8,16} . Efficient-DLM ( B=32 ) and LLaDA are 8B, and other baselines are 7B.
Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of autoregressive models remains challenging. We present Rectified Language Flow (ReLF), a language model that generates text by transporting noise to token embeddings using rectified flow. ReLF combines several complementary design choices: fixed token embeddings constructed with Hellinger-SVD, self-conditioning guidance trained with cross-entropy, and regularized reflow for few-step generation. We further introduce singular logits, which combine an analytically computed term that drives commitment to a valid token embedding as noise vanishes with a bounded neural residual that learns contextual token preferences. With 64 function evaluations on OpenWebText, ReLF achieves lower generative perplexity than autoregressive baselines at matched diversity across all three model sizes tested, while also outperforming prior continuous flow methods in this setting. After two reflow rounds, ReLF still outperforms autoregressive baselines with 16 function evaluations while generating 1,024-token sequences 15-19 times faster at batch size 1.