We introduce SERF (Self-Regularized Flow Language Models), a simple self-supervised framework for improving continuous diffusion language models by exploiting the denoising trajectory itself. While continuous flow-based language models offer a promising alternative to autoregressive generation, their training objectives typically rely on supervision from clean target tokens, leaving the denoising trajectory underexploited as a learning signal. SERF addresses this limitation through noisy-to-clean self-distillation: an exponential moving average (EMA) teacher receives a cleaner version of the student's noisy input along the same interpolation path and provides additional supervision. We investigate two variants: SERF-L, which aligns the student's intermediate representations with deeper teacher representations, and SERF-P, which aligns their output token distributions, either at the student's output or at an early student layer. All variants augment the standard training objective without requiring external pretrained models or modifying the inference procedure. Experiments across three continuous language-model paradigms (one-hot FLM, latent ELF, and hyper-spherical S-FLM) demonstrate substantial improvements in training efficiency, generation quality, and few-step generation. In particular, SERF achieves up to 7.7x wall-clock training acceleration on OpenWebText, improves MAUVE by 24.3% and GSM8K accuracy by up to 28.8%, and matches baseline performance with up to 64x fewer sampling steps. These results establish noisy-to-clean self-distillation as an effective approach to improving continuous diffusion language models.
Figures & tables
Figure 1: Performance and efficiency gains of SERF . Improvements over FLM ( Lee et al., 2026 ) on OpenWebText ( Gokaslan and Cohen, 2019 ) and over S-FLM ( Deschenaux and Gulcehre, 2026 ) on GSM8K . Left: SERF reaches the baseline’s quality with substantially fewer training steps (see Fig. 4 ), achieving wall-clock acceleration despite its small additional computational cost (see Sec. A.4 ). Middle: Improves generation quality and reasoning accuracy at the same sampling budget (see Tab. 1 and Fig. 5 ). Right: Matches the baseline’s 1024 -step performance with only a fraction of the sampling steps (see Tab. 1 , Fig. 4 and Fig. 8 ). Sources for each value are listed in Tab. 2 .
Figure 2: Overview of SERF . In flow language models, the clean data lies at x1 (one-hot token embeddings) and pure noise at x0 . SERF corrupts x1 at two noise levels along the same interpolation path using shared noise: the noisier input xs is fed to the student fθ , while the cleaner input xt ( t>s ) is fed to the EMA teacher fθˉ , whose outputs are detached via stop-gradient. The student is supervised by the standard cross-entropy loss against the ground-truth one-hot target x1l , plus a distillation loss weighted by λ , which takes one of two forms. Left: In Latent Distillation, ( SERF-L ) the hidden states of an early student block ks are mapped through a lightweight projection head gθ and aligned with those of a deeper teacher block kt>ks using a per-token cosine distance. Right: In Probability Distribution Distillation ( SERF-P ), features from student block k pass through a small MLP and the teacher’s frozen head oθˉ , giving token-wise distributions p~θl that are trained to match the teacher’s qθˉl via a cross-entropy loss; the student’s own output pθl is trained with LFLM .
Figure 3: Generated samples on OWT . Samples from FLM , and SERF-L , SERF-P k , trained for 1 M and 300 K steps and generated with 1024 sampling steps from the same initial noise. Errors are highlighted in bold . Both SERF variants produce notably more fluent text, whereas FLM exhibits grammatical errors and repetitions. We provide more examples of generated text in Appendix C .
Sampling steps
Model
# Iters
64
128
256
512
1024
Best MAUVE ( ↑ )
FLM
1 M
0.260±0.011
0.329±0.022
0.349±0.014
0.345±0.005
0.313±0.011
SERF-P k
300 K
0.341±0.019
0.414±0.026
0.399±0.010
0.409±0.026
0.388±0.004
SERF-L
300 K
0.337±0.009
0.435±0.015
0.418±0.017
0.423±0.019
0.389±0.005
Gen. PPL at H=5.44 ( ↓ )
Table 1: Best MAUVE ( ↑ , top) on OWT over τ∈{0.95,1.0,1.05} , where ± is the std over 5 MAUVE seeds, and Gen. PPL ( ↓ , bottom) at the unigram entropy of OWT text ( H=5.44 ), interpolated over each model’s temperature sweep. SERF-L and SERF-P k , trained for 300K steps, are compared against FLM trained for 1M steps. Best in bold , second best underlined . We further present the full MAUVE and Gen. PPL fronts in Fig. 8 .
Figure 4: Training convergence on OWT . SERF-L and SERF-P compared against FLM . At each iteration, we report the best MAUVE over τ∈{0.95,1.0,1.05} and the Gen. PPL interpolated to the dataset entropy (see Sec. A.5 ), using 1024 sampling steps.
Figure 5: Left: GSM8K accuracy at 250 K training steps. Diffusion models at 1024 steps with τ=1 . Best overall underlined , best continuous diffusion bolded . ∗ Denotes our evaluation. Middle: GSM8K accuracy across sampling steps. SERF-L , SERF-P , and SERF-P k compared against S-FLM , all trained on TinyGSM ( Liu et al., 2023 ) for 250 K steps. We report the mean over 5 sample seeds. Right: Comparison with ELF on OWT . SERF-L compared against ELF-B , both trained for 5 epochs ( ∼95 K steps) and sampled with 64 steps, sweeping the self-conditioning guidance scale w∈{0.5,1.0,1.5,2.0} .
Figure 6: Left: fixed ( δ=0.8 ) vs. random offset strategy for SERF-L and SERF-P , respectively. Right: The teacher’s output distribution supervises the student’s final layer ( SERF-P ) or an earlier block ( SERF-P k , k=2 ). Both trained for 100 K steps on OWT .
Figure 7: Layer ablations for SERF-L and SERF-P k . Models trained for 50 K.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Mauve and Gen.PPL fronts for SERF-L and SERF-P k at 300K steps and for the baseline FLM at 1M steps. Curves obtained by sweeping the sampling temperature τ .
Figure 9: Left: Regularization weight ablation for SERF-L and SERF-P . Right: Training convergence of SERF-L , SERF-P and SERF-P k against the S-FLM baseline, trained on TinyGSM and evaluated by GSM8K accuracy with 32 sampling steps.
Metric
Value
Comparison
Source
Left: wall-clock training acceleration
MAUVE
∼7.7×
SERF-L vs. FLM
Fig. 4 (left)
Gen. PPL
∼7.7×
SERF-L vs. FLM
Fig. 4 (right)
GSM8K Acc.
∼1.4×
SERF-L vs. S-FLM
Fig. 9 (right),
Middle: relative improvement at the same sampling budget
MAUVE
+24.3%
SERF-L vs. FLM
Tab. 1 (top)
Appendix
Table 2: Sources of the numbers in Fig. 1 . For each value, we list the compared models and where the underlying numbers are reported.
Figure 10: Generated samples on OWT . Samples from FLM (left) and SERF-L (right), all trained for 1 M and 300K steps respectively. Text generated with 1024 sampling steps from the same initial noise. Errors are highlighted in bold .
Figure 11: Generated samples on OWT . Samples from FLM (left) and SERF-L (right), all trained for 1 M and 300K steps respectively. Text generated with 1024 sampling steps from the same initial noise. Errors are highlighted in bold .
Figure 12: Generated samples on OWT . Samples from FLM (left) and SERF-P k (right), all trained for 1 M and 300K steps respectively. Text generated with 1024 sampling steps from the same initial noise. Errors are highlighted in bold .
Figure 13: Generated samples on OWT . Samples from FLM (left) and SERF-P k (right), all trained for 1 M and 300K steps respectively. Text generated with 1024 sampling steps from the same initial noise. Errors are highlighted in bold .
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.
Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of autoregressive models remains challenging. We present Rectified Language Flow (ReLF), a language model that generates text by transporting noise to token embeddings using rectified flow. ReLF combines several complementary design choices: fixed token embeddings constructed with Hellinger-SVD, self-conditioning guidance trained with cross-entropy, and regularized reflow for few-step generation. We further introduce singular logits, which combine an analytically computed term that drives commitment to a valid token embedding as noise vanishes with a bounded neural residual that learns contextual token preferences. With 64 function evaluations on OpenWebText, ReLF achieves lower generative perplexity than autoregressive baselines at matched diversity across all three model sizes tested, while also outperforming prior continuous flow methods in this setting. After two reflow rounds, ReLF still outperforms autoregressive baselines with 16 function evaluations while generating 1,024-token sequences 15-19 times faster at batch size 1.