Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of autoregressive models remains challenging. We present Rectified Language Flow (ReLF), a language model that generates text by transporting noise to token embeddings using rectified flow. ReLF combines several complementary design choices: fixed token embeddings constructed with Hellinger-SVD, self-conditioning guidance trained with cross-entropy, and regularized reflow for few-step generation. We further introduce singular logits, which combine an analytically computed term that drives commitment to a valid token embedding as noise vanishes with a bounded neural residual that learns contextual token preferences. With 64 function evaluations on OpenWebText, ReLF achieves lower generative perplexity than autoregressive baselines at matched diversity across all three model sizes tested, while also outperforming prior continuous flow methods in this setting. After two reflow rounds, ReLF still outperforms autoregressive baselines with 16 function evaluations while generating 1,024-token sequences 15-19 times faster at batch size 1.
Figures & tables
Figure 1: (a) Quality–diversity Pareto frontiers for ReLF and ELF [ 27 ] at 64 NFE, with AR baselines; curves are traced by varying guidance scale for flows and temperature for AR. ReLF improves both metrics over same-size ELF models. ReLF-Large reaches Generative PPL 17.3 at entropy 5.51 , close to real text ( 17.3 , 5.49 ). (b) Wall-clock speedup over same-size AR models for 1,024-token generation on one GPU. With 16 NFE after two reflow rounds, ReLF achieves lower Generative PPL than these AR models at similar entropy.
Figure 2: ReLF architecture. From the noisy state Zt and self-conditioning input C , the Transformer outputs residual logits fθ , added to singular logits logrt . X1 is a probability-weighted mean of token embeddings, fed back via stop-gradient sg as the next C (dashed).
Figure 3: Singular logits connect commitment to embedding convergence. A four-token example in which both models share the bounded residual fθ and the initial state Z0 , with pt=pθ(⋅∣Zt,t) along each trajectory. (a) With bounded logits, pt retains mass on competing tokens. (b) Adding logrt drives pt to a one-hot distribution. (c) Correspondingly, the flow ends at an embedding mixture (dashed) or at E(w1) (solid).
Figure 4: Guidance on logits. Training (left): two no-gradient passes give cold logits lu ( C=0 ) and warm logits lc (conditioned on X1u ); their difference, scaled by k(1−1/ω) , is the shift Δ subtracted from the network’s logits before CE at the clean token w . Sampling (right): one evaluation at the chosen ω uses the unshifted logits, whose clean-state estimate X1 gives the velocity and the next C . Shading separates flow space (top) from logit space (bottom).
Figure 5: Few-step generation across reflow rounds. Gen. PPL (top) and entropy (bottom) versus NFE for Small, Medium, and Large ReLF models after zero to three reflow rounds. At 8 NFE, two reflow rounds reduce Gen. PPL by 40% , 41% , and 35% for Small, Medium, and Large, respectively, relative to the model before reflow. After two rounds, ReLF-Small outperforms ELF+PD [ 27 ] from 8 NFE, and all three sizes match or outperform their AR references from 16 NFE. Dotted lines denote same-size AR models; long-dashed lines denote real OpenWebText. Arrows at 4 NFE mark entropies of the model before reflow that fall below the plotted range.
Figure 6: Ablations of ReLF. Results use ReLF-Small at 64 NFE unless otherwise indicated. (a) Singular logits under momentum guidance α . (b) Embedding matrix. (c) CE and MSE training objectives over guidance scale ω . (d) Flow dimension d . (e) NFE at ω=1.7 . (f) Guidance response for Small, Medium, and Large models. Panels (a)–(d) show Gen. PPL versus entropy; (e)–(f) show Gen. PPL above and entropy below. Black reference lines in (e)–(f) denote real text. At entropy 5.45 , CE reaches Gen. PPL 24.1 versus 25.6 for MSE. Larger models lose less entropy as guidance increases. Each point uses 1,024 sequences unless noted.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Embedding
Reported configuration
Dimension
tend
Hellinger–SVD
Default
32
0.943
Gaussian
Small embedding ablation
32
0.620
Scaled one-hot
Small embedding ablation
50,257
0.557
Appendix
Table 1: Fixed embedding configurations. Each representation has unigram-weighted mean squared norm 32 and its own precomputed training-time distribution. tend is the endpoint of the training-time distribution.
Model
Parameters (M)
Layers
Hidden width
Attention heads
FFN width
Small
138
12
768
12
2,048
Medium
421
24
1,056
11
2,816
Large
784
32
1,280
20
3,456
Appendix
Table 2: Architecture configurations of ReLF. Parameter counts include all trainable projections, attention gates, and time-conditioning layers, but exclude the fixed token embeddings. FFN width denotes the intermediate dimension of the SwiGLU network.
d
Gen. PPL / entropy
16
ω
1.7
1.8
1.9
2.0
2.1
46.5 / 5.32
43.0 / 5.31
39.7 / 5.30
36.7 / 5.30
33.9 / 5.29
32
ω
1.7
1.8
1.9
2.0
2.1
24.1 / 5.45
23.6 / 5.45
19.1 / 5.37
14.9 / 5.22
11.6 / 5.07
64
ω
2.2
2.4
2.6
2.8
3.0
25.4 / 5.51
23.7 / 5.48
22.0 / 5.43
20.4 / 5.38
19.3 / 5.32
Appendix
Table 3: Flow embedding dimension sweep for ReLF-Small. Selected settings from the sweep in Figure 6 (d); SDE sampling at 64 NFE. For each d , the first row gives the guidance scale ω and the second the Gen. PPL / entropy.
Figure 7: Two reflow rounds at half the NFE. Each panel compares the base model at 32 NFE (dashed) with the model after two reflow rounds at 16 NFE (solid), with vertices Gen. PPL, entropy, and NFE; outward is better on every axis. Gen. PPL is normalized over the six plotted runs, with the highest value at the center, and entropy and NFE over the reflow-round sweep ( k=0 – 3 at 8, 16, and 32 NFE for all sizes); Gen. PPL and NFE use a log scale.
Model
Guidance ω
Entropy (nats) ↑
Gen. PPL ↓
ReLF-Small
1.7
5.45
24.1
1.8
5.45
23.6
1.9
5.37
19.1
2.0
5.22
14.9
2.1
5.07
11.6
2.3
4.94
9.8
Appendix
Table 4: OpenWebText Pareto results and autoregressive baselines. ReLF uses 64 NFE; its frontiers are computed separately within each model size over ω∈{1.5,1.6,1.7,1.8,1.9,2.0,2.1,2.3} . AR baselines use ancestral sampling over the temperatures shown in Figure 1 (a). Entropy is mean per-sequence unigram entropy; Gen. PPL is evaluated by GPT-2 Large.
Guidance ω
Entropy (nats) ↑
Gen. PPL ↓
1.5
5.49
18.0
1.6
5.50
17.7
1.7
5.50
17.4
1.8
5.50
17.4
1.9 †
5.51
17.3
2.0 †
5.51
17.6
Appendix
Table 5: Full guidance-scale sweep for ReLF-Large. All eight settings used for the ReLF-Large frontier in Table 4 , with the same evaluation protocol. † Non-dominated settings listed in Table 4 .
Model
NFE
Entropy (nats) ↑
Gen. PPL ↓
ReLF-Small
4
5.41
116.4
8
5.42
51.2
16
5.41
26.4
32
5.43
22.9
ReLF-Medium
4
5.42
113.7
8
5.42
42.7
Appendix
Table 6: Few-step generation at matched entropy. ReLF rows use two reflow rounds. Except for FMLM, each entry is the lowest Gen. PPL found with entropy at least 5.40 . FMLM values are as reported by Lee et al. [34] .
Figure 8: Quality–diversity trade-offs by model size. ReLF over guidance scale ω∈[1.5,2.5] and ELF [ 27 ] at 64 NFE, compared with AR temperature sweeps. Higher guidance or lower temperature generally reduces entropy. ReLF uses the SDE sampler; ELF includes both SDE and ODE results as reported in the original study. Stars mark real OpenWebText. Results use 1,024 sequences per point for ReLF and AR, and 1,000 for ELF.
Figure 9: Guidance and temperature across diversity statistics. Top: ReLF at 64 NFE with the SDE sampler over guidance scale ω ; bottom: AR over temperature T . Solid polygons are the settings compared in Section 4.2 ; dashed and dotted polygons are a weaker and a stronger setting of the same control. On every spoke, real OpenWebText lies on the rim and outward is closer to real text; a value beyond the rim or the center is drawn on it. Vendi score and recall are computed on Qwen3-Embedding-4B embeddings of 1,024 generated documents. Distinct-4 is the fraction of distinct word 4-grams, and rep-8 is the share of a sequence’s token 8-grams that recur within it. Diamonds mark the rep-8 guardrail ( 3.1% , 1.2× real text).
Figure 10: Reflow rounds at fixed NFE. Gen. PPL (top) and entropy (bottom) for ReLF-Small, ReLF-Medium, and ReLF-Large at 8, 16, and 32 NFE. Round 0 denotes the model before reflow; indigo markers show round 3. Annotations give Gen. PPL before and after two rounds and the percentage reduction. ReLF and AR results use 1,024 sequences per point. AR reference temperatures are 0.97 for Small and 0.98 for Medium and Large, matched to the 64-NFE ReLF entropy. Reference lines show same-size AR models, ELF+PD [ 27 ] at the same NFE, and real OpenWebText.
Figure 11: Latency and throughput at matched quality. ReLF at 16 NFE and AR on one GPU, using KV caching for AR and CUDA graphs for both methods. After two reflow rounds, ReLF achieves lower Gen. PPL than same-size AR models. Top: time per batch of 1,024-token sequences. Bottom: throughput in tokens per second. Labels above ReLF bars indicate speedup over AR. Batch sizes are 1, 4, 16, and 64, showing how the relative speed changes as more sequences are generated together.
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established. We present Lafa (Large flow language model), a fast, high-performing flow language model scaled to 8B parameters in a data-efficient manner. Lafa is a blockwise flow model with causal attention on the prefix and bidirectional, asynchronous flow within each block. This design enables efficient training by adapting pretrained autoregressive (AR) models and fast inference via prefix caching and blockwise parallel decoding. We train Lafa at 0.5B, 7B, and 8B scales with fewer than 30B tokens, more efficient than previous diffusion models, and evaluate it on instruction following, reasoning, and coding, where the learned flow matches or surpasses the quality of AR and can be distilled into flow maps for one- and few-step generation. Notably, flow training preserves or even improves pretrained AR capabilities via a multi-token prediction effect, which we leverage in a self-speculative decoding scheme that verifies flow-based drafts autoregressively within a single model. This yields 2.54x the throughput of an AR model at comparable quality, outperforming multi-token prediction and discrete diffusion methods. Our findings show that continuous flows can serve as a scalable foundation for language modeling.
Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.