FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation

Runzhe Zhang; Letian Chen; Wenpeng Zhang; Zhouhan Lin; Peilin Zhao

cs.CLApr 6, 2026

FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation

Authors: Runzhe Zhang, Letian Chen, Wenpeng Zhang, Zhouhan Lin, Peilin Zhao

Organizations: Shanghai Jiao Tong University · ByteDance

Abstract

We present FlowLM, a flow matching language model transformed from pre-trained diffusion language models via efficient fine-tuning. By re-aligning the curved sampling trajectories of diffusion models into straight-line flows, FlowLM enables high quality few-step generation that rivals or even outperforms the quality of 2,000-step diffusion sampling with very few training epochs. Remarkably, finetuned FlowLM reaches performance saturation with only half as many training epochs as training from scratch, both approaches greatly outperforming the original diffusion model, thereby validating our method. Furthermore, we validate a more effective training objective for flow matching: predicting clean data to consistently guide the sampling process towards the true data distribution. Empirical results demonstrate that our approach is highly effective for high-quality, few-step text generation.

Figures & tables

Panel of Figure 1 — Figure 1: Schematic of FlowLM

Method	$\mathcal{L}_{q_{0}}$	$\mathcal{L}_{q_{1}}$	$\mathcal{L}_{q_{2}}$	$\mathcal{L}_{q_{3}}$
Diffusion	$1.48\times 10^{-2}$	$1.46\times 10^{-2}$	$1.49\times 10^{-2}$	$1.47\times 10^{-2}$
Flow Matching	$9.04\times 10^{-4}$	$1.66\times 10^{-3}$	$2.70\times 10^{-3}$	$6.39\times 10^{-3}$

Table 1: Comparison of loss distribution in different time step intervals between Diffusion and Flow Matching models.

Tasks	Type	Methods	BLEU $\uparrow$	R-L $\uparrow$	BERTScore $\uparrow$	dist-1 $\uparrow$	Training epoch
Question Generation	Multi-step	DiffuSeq(step=2000)	0.1527	0.3474	0.5864	0.9113	34000
	Few-step	DiffuSeq(DPM-solver,step=10)	0.1425	0.3509	0.5730	0.8606	34000
		FlowLM(FT, step=5)	0.1596	0.3484	0.5898	0.9206	6000
		FlowLM(FT, step=3)	0.1595	0.3489	0.5878	0.9169	6000
		FlowLM(FT, step=1)	0.1524	0.3550	0.5713	0.8411	6000
Paraphrase	Multi-step	DiffuSeq(step=2000)	0.1880	0.5306	0.7918	0.9736	34000

Table 2: Comparison between FlowLM and Diffusion LM (MBR=1) . The best results of few-step generation model are bold .

Panel of Figure 2 — Figure 2: Evaluation metrics on Question Generation using MBR decoding across 1 to 10 candidates. Comparison between FlowLM (6000 epochs, step=5, 3,1) and DiffuSeq (34000 epochs, step=2000, DPM step=10).

Panel of Figure 3 — Figure 3: Comparison of gradient during the training process of different methods (Note that maximum values are different). Scr: Trained from scratch. FT: Finetuned

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Visualization of generation trajectories in 2D PCA space.

Figure 5: Schematic diagram of sampling process

Model	DiffuSeq (step=2000)	FlowLM (step=1)	FlowLM (step=3)	FlowLM (step=5)
Time (s)	0.7264	0.00014	0.00038	0.000572
Model	DiffuSeq (DPM-step=10)	FMSeq (step=1)	FMSeq (step=3)	FMSeq (step=5)
Time (s)	0.0012	0.00054	0.00159	0.002101

Appendix

Table 6: Average time needed for sampling per sample (seconds), exclude data loading.

Model	Time sample	Pred objective	Num timesteps	Regularization loss	Time difference	Self conditioning	Speed
DiffuSeq	loss-aware	x-pred	2000	No	No	No	Slow
FlowLM	uniform	x-pred	20	Yes	No	No	Faster
FMSeq	loss-aware	x-pred	2000	No	Yes	Yes	Fast

Appendix

Table 7: Detailed comparison of DiffuSeq, FlowLM, and FMSeq (Transposed)

Tasks	Type	Methods	BLEU $\uparrow$	R-L $\uparrow$	BERTScore $\uparrow$	dist-1 $\uparrow$	Training epoch
Question Generation	Few-step	FlowLM(x-pred,v-loss, step=5)	0.1557	0.3468	0.5845	0.9168	6000
		FlowLM(x-pred,v-loss, step=3)	0.1559	0.3480	0.5822	0.9149	6000
		FlowLM(x-pred,v-loss, step=1)	0.1473	0.3530	0.5616	0.8189	6000
		FlowLM(x-pred,x-loss, step=5)	0.1596	0.3484	0.5898	0.9206	6000
		FlowLM(x-pred,x-loss, step=3)	0.1595	0.3489	0.5878	0.9169	6000
		FlowLM(x-pred,x-loss, step=1)	0.1524	0.3550	0.5713	0.8411	6000

Appendix

Table 8: Comparison between x-pred x-loss and x-pred, v-loss in FlowLM (mbr=1) using uniform time sampling . The best results of few-step generation model are bold .

MBR ( $n$ )	Category	Model	BLEU	ROUGE-L	BERTScore	Dist-1
1	Multi-step	Diffuseq(2000)	0.1514	0.3477	0.5874	0.9126
	Few-step	Diffuseq(DPM, 10)	0.1434	0.3513	0.5740	0.8636
		FlowLM(step=5)	0.1596	0.3484	0.5898	0.9206
		FlowLM(step=3)	0.1600	0.3499	0.5880	0.9166
		FlowLM(step=1)	0.1527	0.3549	0.5711	0.8430
2	Multi-step	Diffuseq(2000)	0.1532	0.3488	0.5880	0.9044

Appendix

Table 11: Detailed performance metrics for Question Generation across all 10 MBR candidate levels.

MBR ( $n$ )	Category	Model	BLEU	ROUGE-L	BERTScore	Dist-1
1	Multi-step	Diffuseq(2000)	0.1868	0.5316	0.7920	0.9737
	Few-step	Diffuseq(DPM, 10)	0.1952	0.5583	0.7932	0.9566
		FlowLM(step=5)	0.1916	0.5289	0.7827	0.9785
		FlowLM(step=3)	0.1987	0.5357	0.7784	0.9757
		FlowLM(step=1)	0.1910	0.5394	0.7560	0.9446
2	Multi-step	Diffuseq(2000)	0.1856	0.5268	0.7877	0.9700

Appendix

Table 13: Detailed performance metrics for Paraphrase across all 10 MBR candidate levels.(Best in Fast is bold ).

MBR ( $n$ )	Category	Model	BLEU	ROUGE-L	BERTScore	Dist-1
1	Multi-step	Diffuseq(2000)	0.2971	0.5330	0.7787	0.9263
	Few-step	Diffuseq(DPM, 10)	0.2318	0.4674	0.6896	0.8795
		FlowLM(step=5)	0.2527	0.4850	0.7293	0.9022
		FlowLM(step=3)	0.2484	0.4798	0.7122	0.8766
		FlowLM(step=1)	0.2274	0.4440	0.6332	0.7493
2	Multi-step	Diffuseq(2000)	0.3078	0.5431	0.7833	0.9135

Appendix

Table 15: Detailed performance metrics for Text Simplification across all 10 MBR candidate levels. (Best in Fast is bold ).

Panel of Figure 9 — Figure 9: Ablation analysis on Training epochs (mapped to 1k–6k) for the Question Generation task. We compare FlowLM performance under 1, 3, and 5 sampling steps.

Panel of Figure 10 — Figure 10: Ablation analysis on Training epochs (mapped to 1k–10k) for the Paraphrase task. Results demonstrate consistent quality gains as the relative training budget increases.

MBR ( $n$ )	Category	Model	BLEU $\uparrow$	R-L $\uparrow$	BERTScore $\uparrow$	dist-1 $\uparrow$
1	Multi-step	Diffuseq(2000)	0.1868	0.5316	0.7920	0.9737
	Few-step	Diffuseq(DPM, 10)	0.1952	0.5583	0.7932	0.9566
		FlowLM (Ours, S5)	0.1916	0.5289	0.7827	0.9785
		FlowLM (Ours, S3)	0.1987	0.5357	0.7784	0.9757
		FlowLM (Ours, S1)	0.1910	0.5394	0.7560	0.9446
		FlowLM (fm2k, S5)	0.1826	0.5162	0.7744	0.9734

Appendix

Table 16: Comprehensive comparison of Paraphrase results for all versions across multiple MBR candidate sizes (

n\in\{1,3,5,10\}

Tasks	Type	Methods	BLEU $\uparrow$	R-L $\uparrow$	BERTScore $\uparrow$	dist-1 $\uparrow$	Training epoch
Paraphrase	Few-step	FlowLM(T=2000, step=5)	0.1826	0.5162	0.7744	0.9734	10000
		FlowLM(T=2000, step=3)	0.1891	0.5227	0.7702	0.9720	10000
		FlowLM(T=2000, step=1)	0.1909	0.5431	0.7612	0.9359	10000
		FlowLM(T=100, step=5)	0.1839	0.5251	0.7771	0.9744	10000
		FlowLM(T=100, step=3)	0.1892	0.5238	0.7161	0.9752	10000
		FlowLM(T=100, step=1)	0.1879	0.5345	0.7701	0.9725	10000

Appendix

Table 17: Comprehensive comparison of Paraphrase results for different T values(5,20,100)

Tasks	Type	Methods	BLEU $\uparrow$	R-L $\uparrow$	BERTScore $\uparrow$	dist-1 $\uparrow$	Training epoch
Paraphrase	Few-step	FlowLM( $Reg_{rate}$ =0, step=5)	0.1596	0.3484	0.5898	0.9206	6000
		FlowLM( $Reg_{rate}$ =0, step=3)	0.1595	0.3489	0.5878	0.9169	6000
		FlowLM( $Reg_{rate}$ =0, step=1)	0.1524	0.3550	0.5713	0.8411	6000
		FlowLM( $Reg_{rate}$ =0.01, step=5)	0.1607	0.3492	0.5906	0.9217	6000
		FlowLM( $Reg_{rate}$ =0.01, step=3)	0.1605	0.3496	0.5876	0.9174	6000
		FlowLM( $Reg_{rate}$ =0.01, step=1)	0.1532	0.3562	0.5736	0.8573	6000

Appendix

Table 19: Ablation result of the regularization loss term on Question generation

Model (Steps)	Generated Question
Reference: Karl Landsteiner won the Nobel Prize for medicine in 1930 for his discovery of what?
FlowLM (Ours, $N=1$ )	karl landsteiner won a nobel prize in villains for which which discovery
	karl landsteiner won a nobel prize in harley for which in discovery
	karl landsteiner won a nobel prize in 1930 for which blanca
	the starvation theer won a nobel prize in 1930 for which medical discovery
FlowLM (Ours, $N=3$ )	karl landsteiner won a nobel prize in intimidation for which medical discovery

Appendix

Table 20: Case study on the Question Generation task. Comparison of generated samples between FlowLM (Ours) at few-step inference and DiffuSeq baselines. Semantic inconsistencies and lexical errors are highlighted in bold.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 504 papers

Flow Map Language Models: One-step Language Modeling via Continuous Denoising
Feb 18, 2026Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal +6Continuous Diffusion Language Models One-Step Generative Modeling
KAIST · Carnegie Mellon University
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
Apr 13, 2026Yuxin Chen, Chumeng Liang, Hangke Sui +4Self-Conditioning Diffusion Language Models
UIUC
Latent-Kernel Discrete Flow Maps for Few-Step Generation
Jul 29, 2026Mansoor Ahmed, Yue-Tsz Fan, Hemanth Venkateswara +1Latent Variable Models Few-Step Diffusion Sampling
Georgia State University, Atlanta, GA, USA · Georgia Institute of Technology, Atlanta, GA, USA

Feb 18, 2026J/K move · Enter open · S save

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah+5

KAIST · Carnegie Mellon University

Continuous Diffusion Language Models One-Step Generative Modeling Flow-Based Generative Modeling Generative Flow Networks Language Model Distillation

Open paper AskPDF arXiv

Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.

Abstract

Language Modeling

Explore similar work

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Latent-Kernel Discrete Flow Maps for Few-Step Generation