ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings

Na Li; Yuchen Jiao; Changxiao Cai; Gen Li

cs.CLAug 24, 2026

ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings

Authors: Na Li, Yuchen Jiao, Changxiao Cai, Gen Li

Organizations: Department of Statistics and Data Science, Chinese University of Hong Kong, Hong Kong · Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA

Abstract

Recent advances in continuous diffusion and flow-based language models (LMs) have achieved performance competitive with discrete LMs. However, existing continuous frameworks still rely on decoders supervised with cross entropy (CE) because the flow trajectories are not guaranteed to terminate at valid token embeddings. Motivated by this limitation, we introduce ConvergeFlow, an embedding-space flow-based LM, which constrains the data predictor to the convex hull of token embeddings and trains it solely with the mean squared error objective induced by flow matching. Under suitable regularity conditions, we prove that the resulting flow converges to valid token embeddings despite errors in the data predictor, enabling direct token prediction without a CE-supervised decoder. We further develop three sampling mechanisms for controlling the trade-off between the generative perplexity and entropy. Experiments on OpenWebText demonstrate that ConvergeFlow achieves performance competitive with existing continuous and discrete diffusion LMs. These findings demonstrate the potential of the flow-based paradigm for language modeling. Our code is available at https://github.com/Na-Li66/ConvergeFlow.

Figures & tables

Figure 1: Gen. PPL-entropy trade-off across different models on the OWT dataset ( 61 ) . Curves show the trade-offs achieved by ConvergeFlow, LangFlow, ELF-SDE, and ELF-ODE, while individual markers indicate the reported results of the remaining baselines. Schedules A and B denote two time-adaptive guidance schedules for ConvergeFlow, defined in Section 3.2 . The red dashed line marks the entropy of the OWT dataset,

5.44

Model	Gen. PPL ( $\downarrow$ )	Entropy ( $\uparrow$ )	Model Size
Autoregressive
Transformer ‡	35.90	5.58	170M
Discrete DLMs
MDLM ‡ ( 64 )	104.85	5.63	170M
Duo ‡ ( 65 )	77.69	5.55	170M
SEDD Uniform ‡ ( 47 )	99.90	5.56	170M

Table 1: Comparison of Gen. PPL and entropy across different models. For ELF and LangFlow, we report the Gen. PPL at the point whose entropy is closest to that of the dataset; their complete Gen. PPL-entropy curves are shown in Figure 1 . For ConvergeFlow, we report the Gen. PPL at the dataset entropy using the complete results in Figure 1 and Table 10 . Results and model sizes marked with ‡ are taken from Duo. At the dataset entropy of

5.44

, our method achieves a Gen. PPL of

33.17

, whereas the lowest Gen. PPL among the continuous flow-based LMs is approximately

60

, even though these models are evaluated at entropies below the dataset entropy.

Technique	Control parameter	Intended effect
Self-conditioning guidance	Coefficient $w_{\mathsf{scg}}$	Amplify refinement from self-conditioning
Iterative self-conditioning refinement	Iteration count $K$	Refine data prediction
Unconditional guidance	Coefficient $w_{\mathsf{ug}}$	Strengthen movement toward gradient of PPL

Table 2: Summary of sampling techniques for quality-diversity trade-offs.

Panel of Figure 2 — Figure 2: Comparison of flow convergence between the embedding-weighted and direct data predictors. The two models use the same fixed LangFlow embeddings, architecture, and training configuration, differing only in their output parameterization. We plot the smallest and second-smallest normalized distances $\|x_{t}^{(i)}-\alpha_{t}e_{j}\|_{2}/(\sigma_{t}\sqrt{d})$ from the flow state associated with each token position $x_{t}^{(i)}$ to the scaled token embeddings as functions of log-SNR.

Panel of Figure 3 — Figure 3: Effect of the training objective on Gen. PPL. (a) Starting from the same pre-trained LangFlow checkpoint, MSE-based training yields an immediate and persistent improvement over CE-based training. (b) In the crossover experiment, the CE-trained checkpoint improves rapidly after switching to MSE, whereas the MSE-trained checkpoint degrades after switching to CE. The horizontal axis denotes additional training steps after the objective switch.

Panel of Figure 4 — Figure 4: Gen. PPL-entropy trade-offs of the three sampling techniques at NFE= $64$ . Each curve is obtained by varying the corresponding nominal control parameter. (a)–(c) compare the standard and time-adaptive variants of each sampling technique, where the adaptive variants increase the guidance strength or self-conditioning refinement count toward the data endpoint. (d) compares the time-adaptive variants of the three sampling techniques.

Figure 5: Joint effect of iterative self-conditioning refinement and self-conditioning guidance at NFE=

64

. Each curve fixes the refinement parameter

K_{\mathsf{iscr}}

and varies the guidance strength

w_{\mathsf{scg}}

Panel of Figure 6 — Figure 6: Effect of adding unconditional guidance to a sampler combining iterative self-conditioning refinement and self-conditioning guidance at $\mathrm{NFE}=64$ .

Panel of Figure 7 — Figure 7: Comparison of two token prediction rules for NFE budgets of $64$ and $128$ .

Panel of Figure 8 — Figure 8: Comparison of the three configurations for NFE budgets of $64$ and $128$ . Each curve is obtained by varying the self-conditioning guidance strength $w_{\mathsf{scg}}$ .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Effect of the sampling schedule on Gen. PPL. We compare the original schedule by LangFlow with the proposed adaptive schedule

t_{i}=(i+0.5)/N

i=0,\ldots,N-1

, across different steps

N

Sampling technique	Standard			Time-adaptive
Sampling technique	Value	Gen. PPL	Entropy	Nominal value	Gen. PPL	Entropy
Iterative self-conditioning refinement	1	105.0006	5.5373	10	98.3887	5.5373
	2	85.4298	5.5139	15	90.3650	5.5303
	3	77.6263	5.5053	20	85.2866	5.5228
	5	73.3118	5.5027	40	78.6051	5.5176
Self-conditioning guidance	1	102.9869	5.5387	20	93.5983	5.5349

Appendix

Table 3: Gen. PPL-entropy trade-offs produced by the three sampling techniques under a fixed NFE budget of

64

. We compare each standard variant with its time-adaptive counterpart. For the time-adaptive variants, “Nominal value” denotes the control parameter before time-dependent scaling.

Sampling technique	Standard			Time-adaptive
Sampling technique	Value	Gen. PPL	Entropy	Nominal value	Gen. PPL	Entropy
Iterative self-conditioning refinement	1	93.3518	5.5078	10	89.2618	5.5052
	2	73.9209	5.4694	15	80.2898	5.4941
	3	64.3410	5.4497	20	74.6253	5.4828
	5	56.5077	5.4300	40	64.9262	5.4636
Self-conditioning guidance	1	93.3518	5.5078	20	84.4239	5.5039

Appendix

Table 4: Gen. PPL-entropy trade-offs produced by the three sampling techniques under a fixed NFE budget of

128

. We compare each standard variant with its time-adaptive counterpart. For the time-adaptive variants, “Nominal value” denotes the control parameter before time-dependent scaling.

		Self-conditioning guidance strength $w_{\mathsf{scg}}$
Refinement parameter $K_{\mathsf{iscr}}$	Metric	10	15	20	25	30	40
25	Gen. PPL	96.5033	76.8682	64.2796	55.8742	51.1444	45.0416
25	Entropy	5.5532	5.5299	5.5109	5.4922	5.4794	5.4549
50	Gen. PPL	97.3557	69.4861	53.9569	45.1607	40.1334	35.691
50	Entropy	5.5654	5.5341	5.505	5.4796	5.4592	5.4306
100	Gen. PPL	118.1948	74.3155	52.0762	41.6154	36.5026	32.5312

Appendix

Table 5: Joint effect of iterative self-conditioning refinement and self-conditioning guidance under

\mathrm{NFE}=64

		Self-conditioning guidance strength $w_{\mathsf{scg}}$
Refinement parameter $K_{\mathsf{iscr}}$	Metric	10	15	20	25	30	40
50	Gen. PPL	79.8379	57.2238	44.7869	37.8605	33.7153	29.8325
50	Entropy	5.5191	5.4771	5.443	5.4145	5.3898	5.3587
100	Gen. PPL	85.1575	54.8722	39.9296	32.6275	28.7678	25.5304
100	Entropy	5.5417	5.4906	5.4453	5.4049	5.3741	5.3379
200	Gen. PPL	103.3736	59.3922	40.3115	32.0478	27.9034	24.925

Appendix

Table 6: Joint effect of iterative self-conditioning refinement and self-conditioning guidance under

\mathrm{NFE}=128

			Self-conditioning guidance strength $w_{\mathsf{scg}}$
$w_{\mathsf{ug}}$	$K_{\mathsf{iscr}}$	Metric	10	15	20	25	30	40
0.0	50	Gen. PPL	97.3557	69.4861	53.9569	45.1607	40.1334	35.6910
	50	Entropy	5.5654	5.5341	5.5050	5.4796	5.4592	5.4306
	100	Gen. PPL	118.1948	74.3155	52.0762	41.6154	36.5026	32.5312
	100	Entropy	5.5909	5.5579	5.5237	5.4914	5.4634	5.4274
0.25	50	Gen. PPL	88.4084	64.1003	50.0962	42.6205	37.9129	33.7868

Appendix

Table 7: Joint effect of unconditional guidance, iterative self-conditioning refinement, and self-conditioning guidance under NFE

=64

			Self-conditioning guidance strength $w_{\mathsf{scg}}$
NFE	$K$	Metric	1.5	2	2.5	3	3.5
64	4	Gen. PPL	45.7076	32.4898	26.1870	22.6047	20.7017
	4	Entropy	5.4124	5.3412	5.2835	5.2272	5.1831
	6	Gen. PPL	42.8333	29.9203	24.1211	20.8427	19.2514
	6	Entropy	5.4107	5.3332	5.2610	5.1853	5.1118
	8	Gen. PPL	39.4945	26.9691	21.1209	18.1815	16.7697

Appendix

Table 8: Results for Configuration A that uses a constant refinement count

K

and self-conditioning guidance strength

w_{\mathsf{scg}}

			Self-conditioning guidance strength $w_{\mathsf{scg}}$
NFE	$K_{\mathsf{iscr}}$	Metric	6	8	10	15
64	10	Gen. PPL	54.4313	43.7583	38.1201	32.4360
	10	Entropy	5.4732	5.4399	5.4151	5.3710
	15	Gen. PPL	48.0552	37.5346	32.1775	27.5590
	15	Entropy	5.4593	5.4161	5.3818	5.3216
	20	Gen. PPL	44.6766	34.0579	29.1159	25.1861

Appendix

Table 9: Results for Configuration B, in which both the refinement count and self-conditioning guidance strength are scaled by

1/(1+\sqrt{\sigma_{t_{i}}/\alpha_{t_{i}}})

Configuration	Metric	$w_{\mathsf{scg}}$
		1.5 1.75 2 2.25 2.5 2.75 3 3.25 3.5
A	Gen. PPL	39.49 31.67 26.97 23.30 21.12 19.58 18.18 17.45 16.77
A	Entropy	5.393 5.348 5.301 5.243 5.191 5.144 5.078 5.020 4.971
		6 7 8 9 10 12 15
B	Gen. PPL	42.75 36.23 32.16 29.36 27.54 25.37 24.44
B	Entropy	5.452 5.425 5.399 5.379 5.357 5.328 5.290

Appendix

Table 10: Detailed Gen. PPL and entropy results for Configurations A–C under a fixed NFE budget of

64

. Within each configuration, the columns vary the nominal self-conditioning guidance strength

w_{\mathsf{scg}}

Configuration	Metric	$w_{\mathsf{scg}}$
		1.5 1.75 2 2.25 2.5 2.75 3 3.25 3.5
A	Gen. PPL	28.68 23.10 19.49 16.99 15.22 14.02 12.99 12.23 11.54
A	Entropy	5.288 5.222 5.160 5.092 5.012 4.948 4.862 4.789 4.696
		6 7 8 9 10 12 15
B	Gen. PPL	30.72 25.81 22.68 20.87 19.33 17.64 17.03
B	Entropy	5.364 5.325 5.288 5.261 5.226 5.168 5.102

Appendix

Table 11: Detailed Gen. PPL and entropy results for Configurations A–C under a fixed NFE budget of

128

. Within each configuration, the columns vary the nominal self-conditioning guidance strength

w_{\mathsf{scg}}

Flow-Based Generative Modeling

Flow-based generative modeling learns probability transports from simple noise to data for fast sampling, density estimation, and controllable generation.

See all 538 papers

Apr 13, 2026

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Yuxin Chen, Chumeng Liang, Hangke Sui +4

UIUC

Self-Conditioning Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training Flow Matching

Ask PDF

Feb 18, 2026

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.

Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal +6

KAIST · Carnegie Mellon University

Continuous Diffusion Language Models One-Step Generative Modeling Flow-Based Generative Modeling Generative Flow Networks Language Model Distillation

Ask PDF

May 11, 2026

ELF: Embedded Language Flows

Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.

Keya Hu, Linlu Qiu, Yiyang Lu +5

MIT

Flow-Based Generative Modeling Diffusion Language Models Continuous Diffusion Language Models

Ask PDF

Abstract

Flow-Based Generative Modeling

Explore similar work

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

ELF: Embedded Language Flows