Rectified Language Flow

Baiyu Su; Runlong Liao; Lizhang Chen; Qiang Liu

cs.AIOct 24, 2026

Rectified Language Flow

Authors: Baiyu Su, Runlong Liao, Lizhang Chen, Qiang Liu

Organizations: The University of Texas at Austin

Abstract

Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of autoregressive models remains challenging. We present Rectified Language Flow (ReLF), a language model that generates text by transporting noise to token embeddings using rectified flow. ReLF combines several complementary design choices: fixed token embeddings constructed with Hellinger-SVD, self-conditioning guidance trained with cross-entropy, and regularized reflow for few-step generation. We further introduce singular logits, which combine an analytically computed term that drives commitment to a valid token embedding as noise vanishes with a bounded neural residual that learns contextual token preferences. With 64 function evaluations on OpenWebText, ReLF achieves lower generative perplexity than autoregressive baselines at matched diversity across all three model sizes tested, while also outperforming prior continuous flow methods in this setting. After two reflow rounds, ReLF still outperforms autoregressive baselines with 16 function evaluations while generating 1,024-token sequences 15-19 times faster at batch size 1.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Quality–diversity Pareto frontiers for ReLF and ELF [ 27 ] at 64 NFE, with AR baselines; curves are traced by varying guidance scale for flows and temperature for AR. ReLF improves both metrics over same-size ELF models. ReLF-Large reaches Generative PPL $17.3$ at entropy $5.51$ , close to real text ( $17.3$ , $5.49$ ). (b) Wall-clock speedup over same-size AR models for 1,024-token generation on one GPU. With 16 NFE after two reflow rounds, ReLF achieves lower Generative PPL than these AR models at similar entropy.

Figure 2: ReLF architecture. From the noisy state

Z_{t}

and self-conditioning input

{\bm{C}}

, the Transformer outputs residual logits

f_{\theta}

, added to singular logits

\log r_{t}

\widehat{X}_{1}

is a probability-weighted mean of token embeddings, fed back via stop-gradient

\operatorname{sg}

as the next

{\bm{C}}

(dashed).

Figure 3: Singular logits connect commitment to embedding convergence. A four-token example in which both models share the bounded residual

f_{\theta}

and the initial state

Z_{0}

, with

p_{t}=p_{\theta}(\cdot\mid Z_{t},t)

along each trajectory. (a) With bounded logits,

p_{t}

retains mass on competing tokens. (b) Adding

\log r_{t}

drives

p_{t}

to a one-hot distribution. (c) Correspondingly, the flow ends at an embedding mixture (dashed) or at

E(w_{1})

(solid).

Figure 4: Guidance on logits. Training (left): two no-gradient passes give cold logits $l^{\mathrm{u}}$ ( ${\bm{C}}=\mathbf{0}$ ) and warm logits $l^{\mathrm{c}}$ (conditioned on $\widehat{X}_{1}^{\mathrm{u}}$ ); their difference, scaled by $k(1-1/\omega)$ , is the shift $\Delta$ subtracted from the network’s logits before CE at the clean token $w$ . Sampling (right): one evaluation at the chosen $\omega$ uses the unshifted logits, whose clean-state estimate $\widehat{X}_{1}$ gives the velocity and the next ${\bm{C}}$ . Shading separates flow space (top) from logit space (bottom).

Figure 5: Few-step generation across reflow rounds. Gen. PPL (top) and entropy (bottom) versus NFE for Small, Medium, and Large ReLF models after zero to three reflow rounds. At 8 NFE, two reflow rounds reduce Gen. PPL by

40\%

41\%

, and

35\%

for Small, Medium, and Large, respectively, relative to the model before reflow. After two rounds, ReLF-Small outperforms ELF+PD [ 27 ] from 8 NFE, and all three sizes match or outperform their AR references from 16 NFE. Dotted lines denote same-size AR models; long-dashed lines denote real OpenWebText. Arrows at 4 NFE mark entropies of the model before reflow that fall below the plotted range.

Panel of Figure 6 — Figure 6: Ablations of ReLF. Results use ReLF-Small at 64 NFE unless otherwise indicated. (a) Singular logits under momentum guidance $\alpha$ . (b) Embedding matrix. (c) CE and MSE training objectives over guidance scale $\omega$ . (d) Flow dimension $d$ . (e) NFE at $\omega=1.7$ . (f) Guidance response for Small, Medium, and Large models. Panels (a)–(d) show Gen. PPL versus entropy; (e)–(f) show Gen. PPL above and entropy below. Black reference lines in (e)–(f) denote real text. At entropy $5.45$ , CE reaches Gen. PPL $24.1$ versus $25.6$ for MSE. Larger models lose less entropy as guidance increases. Each point uses 1,024 sequences unless noted.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Embedding	Reported configuration	Dimension	$t_{\mathrm{end}}$
Hellinger–SVD	Default	32	0.943
Gaussian	Small embedding ablation	32	0.620
Scaled one-hot	Small embedding ablation	50,257	0.557

Appendix

Table 1: Fixed embedding configurations. Each representation has unigram-weighted mean squared norm 32 and its own precomputed training-time distribution.

t_{\mathrm{end}}

is the endpoint of the training-time distribution.

Model	Parameters (M)	Layers	Hidden width	Attention heads	FFN width
Small	138	12	768	12	2,048
Medium	421	24	1,056	11	2,816
Large	784	32	1,280	20	3,456

Appendix

Table 2: Architecture configurations of ReLF. Parameter counts include all trainable projections, attention gates, and time-conditioning layers, but exclude the fixed token embeddings. FFN width denotes the intermediate dimension of the SwiGLU network.

$d$		Gen. PPL / entropy
16	$\omega$	1.7	1.8	1.9	2.0	2.1
		46.5 / 5.32	43.0 / 5.31	39.7 / 5.30	36.7 / 5.30	33.9 / 5.29
32	$\omega$	1.7	1.8	1.9	2.0	2.1
		24.1 / 5.45	23.6 / 5.45	19.1 / 5.37	14.9 / 5.22	11.6 / 5.07
64	$\omega$	2.2	2.4	2.6	2.8	3.0
		25.4 / 5.51	23.7 / 5.48	22.0 / 5.43	20.4 / 5.38	19.3 / 5.32

Appendix

Table 3: Flow embedding dimension sweep for ReLF-Small. Selected settings from the sweep in Figure 6 (d); SDE sampling at 64 NFE. For each

d

, the first row gives the guidance scale

\omega

and the second the Gen. PPL / entropy.

Figure 7: Two reflow rounds at half the NFE. Each panel compares the base model at 32 NFE (dashed) with the model after two reflow rounds at 16 NFE (solid), with vertices Gen. PPL, entropy, and NFE; outward is better on every axis. Gen. PPL is normalized over the six plotted runs, with the highest value at the center, and entropy and NFE over the reflow-round sweep (

k=0

–

3

at 8, 16, and 32 NFE for all sizes); Gen. PPL and NFE use a log scale.

Model	Guidance $\omega$	Entropy (nats) $\uparrow$	Gen. PPL $\downarrow$
ReLF-Small	1.7	5.45	24.1
	1.8	5.45	23.6
	1.9	5.37	19.1
	2.0	5.22	14.9
	2.1	5.07	11.6
	2.3	4.94	9.8

Appendix

Table 4: OpenWebText Pareto results and autoregressive baselines. ReLF uses 64 NFE; its frontiers are computed separately within each model size over

\omega\in\{1.5,1.6,1.7,1.8,1.9,2.0,2.1,2.3\}

. AR baselines use ancestral sampling over the temperatures shown in Figure 1 (a). Entropy is mean per-sequence unigram entropy; Gen. PPL is evaluated by GPT-2 Large.

Guidance $\omega$	Entropy (nats) $\uparrow$	Gen. PPL $\downarrow$
1.5	5.49	18.0
1.6	5.50	17.7
1.7	5.50	17.4
1.8	5.50	17.4
1.9 †	5.51	17.3
2.0 †	5.51	17.6

Appendix

Table 5: Full guidance-scale sweep for ReLF-Large. All eight settings used for the ReLF-Large frontier in Table 4 , with the same evaluation protocol. † Non-dominated settings listed in Table 4 .

Model	NFE	Entropy (nats) $\uparrow$	Gen. PPL $\downarrow$
ReLF-Small	4	5.41	116.4
	8	5.42	51.2
	16	5.41	26.4
	32	5.43	22.9
ReLF-Medium	4	5.42	113.7
	8	5.42	42.7

Appendix

Table 6: Few-step generation at matched entropy. ReLF rows use two reflow rounds. Except for FMLM, each entry is the lowest Gen. PPL found with entropy at least

5.40

. FMLM values are as reported by Lee et al. [34] .

Figure 8: Quality–diversity trade-offs by model size. ReLF over guidance scale

\omega\in[1.5,2.5]

and ELF [ 27 ] at 64 NFE, compared with AR temperature sweeps. Higher guidance or lower temperature generally reduces entropy. ReLF uses the SDE sampler; ELF includes both SDE and ODE results as reported in the original study. Stars mark real OpenWebText. Results use 1,024 sequences per point for ReLF and AR, and 1,000 for ELF.

Figure 9: Guidance and temperature across diversity statistics. Top: ReLF at 64 NFE with the SDE sampler over guidance scale

\omega

; bottom: AR over temperature

T

. Solid polygons are the settings compared in Section 4.2 ; dashed and dotted polygons are a weaker and a stronger setting of the same control. On every spoke, real OpenWebText lies on the rim and outward is closer to real text; a value beyond the rim or the center is drawn on it. Vendi score and recall are computed on Qwen3-Embedding-4B embeddings of 1,024 generated documents. Distinct-4 is the fraction of distinct word 4-grams, and rep-8 is the share of a sequence’s token 8-grams that recur within it. Diamonds mark the rep-8 guardrail (

3.1\%

1.2\times

real text).

Figure 10: Reflow rounds at fixed NFE. Gen. PPL (top) and entropy (bottom) for ReLF-Small, ReLF-Medium, and ReLF-Large at 8, 16, and 32 NFE. Round 0 denotes the model before reflow; indigo markers show round 3. Annotations give Gen. PPL before and after two rounds and the percentage reduction. ReLF and AR results use 1,024 sequences per point. AR reference temperatures are

0.97

for Small and

0.98

for Medium and Large, matched to the 64-NFE ReLF entropy. Reference lines show same-size AR models, ELF+PD [ 27 ] at the same NFE, and real OpenWebText.

Figure 11: Latency and throughput at matched quality. ReLF at 16 NFE and AR on one GPU, using KV caching for AR and CUDA graphs for both methods. After two reflow rounds, ReLF achieves lower Gen. PPL than same-size AR models. Top: time per batch of 1,024-token sequences. Bottom: throughput in tokens per second. Labels above ReLF bars indicate speedup over AR. Batch sizes are 1, 4, 16, and 64, showing how the relative speed changes as more sequences are generated together.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
Apr 13, 2026Yuxin Chen, Chumeng Liang, Hangke Sui +4Self-Conditioning Diffusion Language Models
UIUC
Large Flow Language Models
Oct 20, 2026Arnav Chavan, Chanhyuk Lee, Udbhav Bamba +4Language Modeling Flow-Based Generative Modeling
UC Berkeley · Amazon · KAIST +1
ELF: Embedded Language Flows
May 11, 2026Keya Hu, Linlu Qiu, Yiyang Lu +5Flow-Based Generative Modeling Diffusion Language Models
MIT

Apr 13, 2026J/K move · Enter open · S save

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo+3

UIUC

Self-Conditioning Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training Flow Matching

Open paper AskPDF arXiv

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Abstract

Language Modeling

Explore similar work

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Large Flow Language Models

ELF: Embedded Language Flows