Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established. We present Lafa (Large flow language model), a fast, high-performing flow language model scaled to 8B parameters in a data-efficient manner. Lafa is a blockwise flow model with causal attention on the prefix and bidirectional, asynchronous flow within each block. This design enables efficient training by adapting pretrained autoregressive (AR) models and fast inference via prefix caching and blockwise parallel decoding. We train Lafa at 0.5B, 7B, and 8B scales with fewer than 30B tokens, more efficient than previous diffusion models, and evaluate it on instruction following, reasoning, and coding, where the learned flow matches or surpasses the quality of AR and can be distilled into flow maps for one- and few-step generation. Notably, flow training preserves or even improves pretrained AR capabilities via a multi-token prediction effect, which we leverage in a self-speculative decoding scheme that verifies flow-based drafts autoregressively within a single model. This yields 2.54x the throughput of an AR model at comparable quality, outperforming multi-token prediction and discrete diffusion methods. Our findings show that continuous flows can serve as a scalable foundation for language modeling.
Figures & tables
Figure 1: Large flow language models. Lafa 7B attains 2.02–2.54 × the throughput of an AR model at a matched or better quality, outperforming existing MTP and diffusion-based approaches.
Figure 2: Overview. (Top) Lafa learns a block denoiser that also performs AR prediction on the prefix. (Bottom) Lafa supports flow decoding, which can be further distilled into a flow map for one-step generation. Together, AR and flow support self-speculative decoding, which accepts flow-based drafts from left to right via AR verification.
Figure 3: Attention mask for block-bidirectional attention.
Figure 4: Parallelized training over a sequence composed of two blocks of size three. The uncorrupted sequence uses causal attention and the AR loss, while each noisy block attends to the preceding clean tokens and bidirectionally within itself, and uses the denoising losses.
Figure 5: Tokenwise time for each block position learned by Lafa 7B.
Figure 6
Figure 7: Self-speculative decoding. Accuracy ↑ vs. TPF ↑ averaged over all six benchmarks. In the right panel, Efficient-DLM and LLaDA are 8B and the other baselines are 7B. “Canvas” denotes iterative completion over the full sequence, while “32/8” denotes decoding 32-token blocks in four 8-token sub-blocks. Further details in Section C.1 .
Figure 8: Accuracy ↑ vs. end-to-end throughput measured in tokens per second (TPS) ↑ at 7–8B scales, averaged over IFEval, GSM8K, and HumanEval.
Configuration
Average (%)
Training
Fully causal attention
32.7
Block-bidirectional attention
38.0
Synchronous time
36.2
Asynchronous tokenwise time
38.0
Frozen embeddings
34.4
Table 3: Ablations. We use flow decoding and report mean accuracy over IFEval, GSM8K, and HumanEval.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11Figure 12
Figure 9: Fused self-speculative decoding with block size three. A single forward pass verifies previous draft with AR prediction and performs next-block drafting. This is done by decoding next drafts corresponding to every possible acceptance of the previous draft in parallel and selecting one based on verification.
Setting
Lafa 0.5B
Lafa 7B
Lafa 8B
Model and data
Initialization
Qwen2.5 0.5B Instruct
Qwen2.5 7B Instruct
Qwen3 8B Base
Training data / epochs / tokens
Dolci-Instruct-SFT / 5 epochs / approximately 9B
Block sizes B
4, 8, 16
4, 8, 16
4
Maximum sequence length
512–1024
512–1024
512
adaLN
Per-layer
Shared across layers
Shared across layers
Appendix
Table 4: Main hyperparameters. For block size four, we use batches of 384 sequences for training, whereas for larger blocks, we use memory-matched microbatches and, when applicable, continue training from smaller-block checkpoint.
0.5B scale
7B scale
Method
B
Euler steps
GSM8K
HumanEval
IFEval
Average
GSM8K
HumanEval
IFEval
Average
AR SFT
–
–
32.0
35.4
48.4
38.6
69.1
78.7
73.8
73.9
Flow decoding
Lafa (Ours)
16
1
0.8
3.0
20.3
8.1
2.4
10.4
29.8
14.2
2
1.7
6.1
23.5
10.4
18.7
21.3
36.2
25.4
4
11.7
12.2
27.0
17.0
47.5
40.2
44.7
44.1
Appendix
Table 5: Flow and flow map decoding accuracies.
Method
IFEval
BBH
MMLU-Pro
GPQA
GSM8K
HumanEval
Average
0.5B scale
Qwen2.5 SFT †
48.4
26.3
11.9
17.9
32.0
35.4
28.7
MTP † [ 6 ]
50.6
17.3
9.2
18.8
27.1
32.3
25.9
DeepSeek-MTP † [ 8 ]
53.2
24.1
14.0
18.5
35.5
36.0
30.2
MARS † [ 33 ]
51.3
26.6
12.4
19.4
32.8
40.2
30.4
SBD † [ 46 ]
55.3
20.6
9.2
22.5
27.2
30.8
27.6
Appendix
Table 6: Single-token decoding. For diffusion models, we decode exactly one highest-probability token per forward. Models marked † are trained by us from the matched AR model on Dolci-Instruct-SFT for five epochs.
IFEval
BBH
MMLU-Pro
GPQA
GSM8K
HumanEval
Average
Method
B
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
Acc.
TPF
0.5B scale
AR SFT
–
48.4
1.0
26.3
1.0
11.9
1.0
17.9
1.0
32.0
1.0
35.4
1.0
28.7
1.0
MARS
4
51.0
1.8
31.7
2.3
13.9
1.7
18.3
2.0
31.7
2.1
32.3
1.9
29.8
2.0
MTP
4
51.2
2.5
17.6
3.3
9.4
3.1
17.9
2.8
26.8
2.9
33.5
2.9
26.1
2.9
DeepSeek-MTP
2
51.9
1.6
23.7
1.8
12.1
1.9
18.8
1.9
35.5
1.8
35.4
1.7
29.6
1.8
Appendix
Table 7: Self-speculative and parallel decoding accuracy (%) and TPF across model scales.
IFEval
GSM8K
HumanEval
Average
Method
Setting
Threshold
Accuracy
TPS
Accuracy
TPS
Accuracy
TPS
Accuracy
TPS
AR SFT
–
–
73.8
40.9
69.1
40.6
78.7
37.0
73.9
39.5
DeepSeek-MTP
B=2
–
78.0
45.1
70.3
57.2
80.5
58.4
76.3
53.6
MARS
B=4
0.95
77.6
42.0
70.7
59.8
79.9
43.3
76.1
48.4
B=8
0.95
77.3
40.9
69.1
62.7
78.0
43.1
74.8
48.9
B=16
0.95
75.0
41.1
72.9
61.3
75.0
44.6
74.3
49.0
Appendix
Table 8: Per-task accuracy (%) and throughput in tokens per second (TPS) at 7–8B scales, measured using one H100 GPU with batch size one. Efficient-DLM is 8B and other methods are 7B.
Figure 10: Confidence decoding, averaged over GSM8K, HumanEval, and IFEval. In the right panel, Lafa uses 8B for B=4 and 7B for B∈{8,16} . Efficient-DLM ( B=32 ) and LLaDA are 8B, and other baselines are 7B.