Improving Few-Step Language Flows with Untied Self-Conditioning

Bocheng Li; Linli Xu

cs.CLAug 23, 2026

Improving Few-Step Language Flows with Untied Self-Conditioning

Authors: Bocheng Li, Linli Xu

Organizations: University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence

Abstract

Flow-matching language models refine all token positions in parallel and can trade sampling steps for latency, yet generation quality still degrades sharply with few sampling steps. We trace a source of this degradation to a train--inference mismatch in previous-prediction self-conditioning: during training, the self-conditioning input is computed from the current noisy state with no intervening solver step; during sampling, the solver folds the previous prediction into the latent before that same prediction reappears as the explicit self-conditioning input. This coupling, absent during training, creates redundancy that grows with step width. We show that the mismatch degrades both the self-conditioning input and the solver update, and derive a correction for each from the model's own structure. From the frozen projection weights we identify directions along which the self-conditioning input is redundant with the latent and dampen them; from the solver's integration structure we derive that a step-average prediction is needed and approximate it from prediction history, with scale set by offline trajectory statistics. The resulting sampler, Untied Self-Conditioning, requires no retraining and uses one evaluation per step. At 8 sampling steps on LangFlow, it reduces OpenWebText generative perplexity from $531$ to~ $62$ ( $8.6\times$ ); under an adapted Arena-Hard-Auto~~v2 protocol, its outputs are preferred in $96\%$ of pairwise comparisons. On ELF-B it reduces generative perplexity from $71$ to~~ $43$ . Improvements hold from 8 to 256 sampling steps.

Figures & tables

Figure 1: Train–inference mismatch and its correction. (a) Standard self-conditioning: the solver folds

p_{k-1}

into the latent

z_{k}

, so

z_{k}

already encodes

p_{k-1}

. Then

m_{k}=p_{k-1}

re-introduces the same information as an explicit input. During training, no solver step intervenes, so this redundancy does not arise. (b) Untied Self-Conditioning corrects both consequences: an overlap transform

F_{k}

dampens self-conditioning components the latent already represents, while a solver correction

A_{k}

shifts the prediction toward the step average. The raw prediction is stored for the next self-conditioning input.

Figure 2: GenPPL versus NFE; every curve point annotates entropy. Panels (a)–(e) compare the official sampler (dashed) with Untied Self-Conditioning (solid) for each model–dataset pair. Dotted lines mark data references taken from 8 (OWT) and 27 (LM1B).

Figure 3: System context and pairwise LLM-judge preference. (a) System-level OpenWebText context; external curves taken from 8 . (b) Pair outcomes on LangFlow OpenWebText from 1,024 comparisons per NFE. Each bar partitions Ours wins, ties, and Official wins; labels at the right endpoint give the Arena score.

Figure 4: Correlation across sampling steps changes the value of a fixed self-conditioning update. (a) The experiment varies only the dependence

\rho

between the carried prediction

m_{k}=p_{k-1}

and the next latent

z_{k}

, while preserving both marginals. The MSE-optimal interpolation coefficient along one frozen self-conditioning-induced prediction direction decreases with

\rho

(b); consequently, a fixed coefficient of one crosses from helpful to harmful (c).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Shape	Rank	$\rho_{\min}$	$\rho_{\rm med}$	$\rho_{\max}$	Mean $\rho_{i}^{2}$	$\|\rho_{i}{\geq}0.9\|$	$\|\rho_{i}{\leq}0.1\|$
ELF-B	$128{\times}512$	128	0.001	0.594	0.959	0.391	18	14
ELF-M	$128{\times}512$	128	0.001	0.581	0.950	0.381	17	13
ELF-L	$128{\times}512$	128	0.004	0.577	0.952	0.375	13	13
LangFlow	$768{\times}768$	768	full rank; all $\rho_{i}{=}1$

Appendix

Table 2: Effective input-projection geometry. ELF projections pass through a

128

-wide bottleneck, producing a well-spread principal-angle spectrum; LangFlow projections are full rank and the transform applies isotropic dampening. Canonical correlations

\rho_{i}

are the cosines of the principal angles between the two row spaces.

		GenPPL $\downarrow$		$H$
Model	NFE	Official	Ours	Official	Ours
ELF-B	8	$70.6{\pm}1.7$	$42.7{\pm}2.3$	$5.23{\pm}.02$	$5.10{\pm}.01$
	16	$32.2{\pm}0.8$	$28.7{\pm}1.0$	$5.18{\pm}.01$	$5.16{\pm}.01$
	32	$23.7{\pm}0.3$	$21.0{\pm}0.2$	$5.15{\pm}.01$	$5.14{\pm}.01$
	64	$19.0{\pm}0.2$	$18.0{\pm}0.3$	$5.08{\pm}.01$	$5.10{\pm}.01$
ELF-M	8	$108.5{\pm}7.4$	$52.3{\pm}3.5$	$5.32{\pm}.03$	$5.23{\pm}.00$

Appendix

Table 3: Untied Self-Conditioning reduces GenPPL at every ELF NFE; the gap is largest at 8 NFE. Entries are mean

\pm

standard error over six seeds, each with 1,024 samples.

		GenPPL $\downarrow$		$H$
Data	NFE	Official	Ours	Official	Ours
OWT	8	$531.2{\pm}3.4$	$61.6{\pm}0.2$	$5.81{\pm}.00$	$5.33{\pm}.00$
	16	$234.5{\pm}0.8$	$48.8{\pm}0.1$	$5.69{\pm}.00$	$5.30{\pm}.00$
	32	$123.0{\pm}0.5$	$43.8{\pm}0.1$	$5.58{\pm}.00$	$5.31{\pm}.00$
	64	$79.3{\pm}0.3$	$40.0{\pm}0.1$	$5.50{\pm}.00$	$5.31{\pm}.00$
	128	$59.4{\pm}0.1$	$38.3{\pm}0.1$	$5.43{\pm}.00$	$5.31{\pm}.00$

Appendix

Table 4: Untied Self-Conditioning improves LangFlow quality at every NFE on both datasets, reducing OpenWebText GenPPL from

531

62

at 8 NFE. Entries are mean

\pm

standard error over six seeds, each with 1,024 samples (OpenWebText: 1,024 tokens; LM1B: 128 tokens).

		Word Rep-4 ( $\times 10^{-3}$ )
Data	NFE	Official	Ours
OWT	8	$0.11{\pm}.01$	$1.82{\pm}.04$
	16	$0.51{\pm}.02$	$3.59{\pm}.10$
	32	$1.54{\pm}.06$	$5.33{\pm}.16$
	64	$3.95{\pm}.08$	$7.78{\pm}.19$
	128	$8.38{\pm}.19$	$11.92{\pm}.31$

Appendix

Table 5: Word-level 4-gram repetition rate (word Rep-4) for LangFlow endpoints. Entries are mean

\pm

standard error over six seeds.

System	NFE	$\eta_{R}$	$\eta_{T}$	$\eta_{O}$	$\alpha$
ELF-B	8	3.3	1.7	—	0.7
	16	2.3	0.5	—	0.8
	32	2.3	1.1	—	0.8
	64	1.5	—	1.5	0.8
ELF-M	8	2.3	1.5	6.0	0.8
	16	1.3	0.9	—	0.8

Appendix

Table 6: Inference-time coefficients (Algorithm 1 ). “—” marks the base value

1

(unchanged).

$\rho$	$\alpha^{*}$	risk ratio	$\alpha^{*}$	risk ratio
	8 NFE		16 NFE
0.00	0.88	0.92	0.78	0.92
0.25	0.79	0.94	0.69	0.94
0.50	0.62	0.97	0.55	0.98
0.75	0.45	1.01	0.40	1.05
1.00	0.30	1.07	0.35	1.11

Appendix

Table 7: Changing temporal correlation while preserving the marginal distribution at each noise level.

\alpha^{*}

is the risk-optimal interpolation coefficient along one fixed self-conditioning-induced prediction direction; the ratio compares prediction risk at coefficient one with the base model.

Configuration	Alignment $\uparrow$	$\Delta\mathcal{E}/\operatorname{RMS}$ $\downarrow$	GenPPL $\downarrow$
Official	–	–	64.27
Overlap-aligned	0.94	$-40.35$ k	55.64
Reversed	0.51	$-22.94$ k	73.95
Random	0.74	$-32.15$ k	59.59

Appendix

Table 8: The overlap-aligned transform outperforms matched direction assignments. Every solver correction is disabled. Overlap-aligned, reverse, and random use the same retention eigenvalues and differ only in their assignment to principal directions. Alignment is the cosine with the steepest decrease in the overlap objective

\mathcal{E}

Configuration	Alignment $\uparrow$	$\Delta\mathcal{E}/\operatorname{RMS}$ $\downarrow$	GenPPL $\downarrow$
Official SC, no solver correction	–	–	65.23
Official SC, solver correction	–	–	38.17
Overlap-aligned	0.94	$-53.62$ k	34.93
Reversed	0.54	$-32.52$ k	40.21
Random	0.75	$-43.69$ k	36.92

Appendix

Table 9: Self-conditioning transforms with the solver correction active. The three transformed configurations use the same eigenvalues, assigned to principal directions in different orders.

\mathcal{E}

is defined in equation 4 .

Configuration	GenPPL $\downarrow$	GPT-2 NLL $\downarrow$	$H$
Official	29.44	3.39	5.13
Overlap-aligned dampening, no global scaling	26.56	3.28	5.06
Overlap-aligned dampening, method scaling	27.42	3.32	5.08
Uniform scaling	29.90	3.40	5.13

Appendix

Table 10: Self-conditioning controls on 256 ELF-B samples at 16 NFE with the solver correction disabled. The overlap-aligned transform dampens principal direction

i

\max(1-\widehat{a}_{k}\rho_{i}^{2},\epsilon_{\rm fp})

before any global scaling.

Model	NFE	GPT-2 $\Delta_{\rm solver}$	OWT $\Delta_{\rm solver}$	OWT $\Delta_{\rm next\,SC}$
ELF-B	16	$-0.07$	$-0.16$	$+3.00$
ELF-M	8	$-0.38$	$-0.31$	$+2.51$
ELF-M	16	$-0.17$	$-0.18$	$+3.45$
ELF-L	8	$-0.27$	$-0.25$	$+1.63$
ELF-L	16	$-0.14$	$-0.14$	$+2.75$

Appendix

Table 11: Where to apply the solver correction across checkpoints. The first two differences apply it only in the solver update and compare with the official sampler. The last also adds it to the next self-conditioning input and compares with applying it only in the solver.

Solver correction	GenPPL $\downarrow$	$H$
History-based direction with mean/RMS scaling	45.63	5.14
Common component only	57.21	4.98
History-based direction permuted across samples	49.27	4.99
Random direction with equal RMS	71.45	5.02
Official (no correction)	74.18	5.27

Appendix

Table 12: Controls for the solver correction on ELF-B with eight model evaluations.

Sampler	GenPPL $\downarrow$	$H$
Official	74.18	5.27
Momentum	74.72	5.28
ACE	115.43	5.43
HiGS	56.90	5.18
HiGS + ACE	65.84	5.24
Untied Self-Conditioning	45.63	5.14

Appendix

Table 13: Matched sampler comparison on frozen ELF-B at 8 NFE with 1,024 matched generations. All methods use the same model, time grid, and initial latent. Momentum ( 25 ) , HiGS ( 20 ) , and ACE ( 30 ) are described in Section 6 .

Model	NFE	Official (samples/s)	Untied Self-Conditioning (samples/s)	Retained
ELF-B	8	72.98	67.59	92.6%
	16	39.39	35.93	91.2%
ELF-M	8	22.20	21.47	96.7%
	16	11.78	11.38	96.6%
ELF-L	8	10.42	10.21	98.0%
	16	5.53	5.42	98.1%

Appendix

Table 14: Sampling throughput under matched computation. Official and Untied Self-Conditioning use the same compiled model, batch size, NFE, and generated outputs. Retained is method throughput divided by official throughput.

Figure 5: One-at-a-time sensitivity of GenPPL to each coefficient on ELF-B at 8 NFE. Each panel varies one coefficient while holding the other three at the values in Table 6 . The dashed blue line marks the official sampler; the dotted red line marks the Table 6 value. All 24 evaluated points reduce GenPPL relative to the official sampler. The curves for

\eta_{R}

and

\eta_{O}

are nearly flat across the full

[\times\!\frac{1}{2},\,\times\!2]

range; doubling

\eta_{T}

gives the highest GenPPL among the tested variants.

NFE	Paired samples	Ours higher NLL	Fraction	Mean $\Delta$ NLL
8	6,144	0	$0.00\%$	$-2.16$
16	6,144	0	$0.00\%$	$-1.57$
32	6,144	2	$0.03\%$	$-1.03$
64	6,144	15	$0.24\%$	$-0.69$
128	6,144	127	$2.07\%$	$-0.44$
256	6,144	693	$11.28\%$	$-0.20$

Appendix

Table 15: Paired-sample NLL comparison on LangFlow OpenWebText (six seeds, 1,024 samples per seed per NFE). “Higher NLL” counts samples where Untied Self-Conditioning has higher GPT-2 NLL than the matched official sample.

Self-Conditioning

See all 10 papers

Jul 1, 2026

Self-conditioned Flow Map Language Models via Fixed-point Flows

Self-conditioning is a core technique that enhances continuous flow-based language models, where the model learns to denoise generated text by conditioning on its own denoising estimate. While empirically successful, its performance improvements are poorly understood. Moreover, there is growing interest in the use of few-step generators based on flow maps, for which how to leverage self-conditioning is unclear. Here, we show that flow language models with self-conditioning perform a fixed-point iteration that improves generation through iterative refinement. We use this viewpoint to formulate fixed-point flows, a two-dimensional class of self-conditioned flows, where the first dimension represents the flow process and the second represents the fixed-point iteration. We show that fixed-point flows define valid flow maps, and show that they can be distilled from self-conditioned flow models by compressing both fixed-point iterations and the flow process, the former with fixed-point distillation and the latter with flow map distillation. Our resulting flow map language model, FMLM

^\star

, outperforms state-of-the-art self-conditioned models and few-step models in one- and few-step generation on OpenWebText. Code is available at https://github.com/Ugness/self-conditioned-fmlm.

Jaehoon Yoo, Wonjung Kim, Floor Eijkelboom +4

KAIST · University of Amsterdam · Carnegie Mellon University

Normalizing Flows One-Step Generative Modeling Generative Flow Networks Flow Map Learning Continuous Normalizing Flows Language Model Distillation+2

Ask PDF

Apr 6, 2026

FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation

We present FlowLM, a flow matching language model transformed from pre-trained diffusion language models via efficient fine-tuning. By re-aligning the curved sampling trajectories of diffusion models into straight-line flows, FlowLM enables high quality few-step generation that rivals or even outperforms the quality of 2,000-step diffusion sampling with very few training epochs. Remarkably, finetuned FlowLM reaches performance saturation with only half as many training epochs as training from scratch, both approaches greatly outperforming the original diffusion model, thereby validating our method. Furthermore, we validate a more effective training objective for flow matching: predicting clean data to consistently guide the sampling process towards the true data distribution. Empirical results demonstrate that our approach is highly effective for high-quality, few-step text generation.

Runzhe Zhang, Letian Chen, Wenpeng Zhang +2

Shanghai Jiao Tong University · ByteDance

Language Modeling Flow-Based Generative Modeling Diffusion Language Models Few-Step Diffusion Sampling

Ask PDF

Apr 7, 2026

FastDiSS: Few-step Match Many-step Diffusion Language Model on Sequence-to-Sequence Generation--Full Version

Self-conditioning has been central to the success of continuous diffusion language models, as it allows models to correct previous errors. Yet its ability degrades precisely in the regime where diffusion is most attractive for deployment: few-step sampling for fast inference. In this study, we show that when models only have a few denoising steps, inaccurate self-conditioning induces a substantial approximation gap; this mistake compounds across denoising steps and ultimately dominate the sample quality. To address this, we propose a novel training framework that handles these errors during learning by perturbing the self-conditioning signal to match inference noise, improving robustness to prior estimation errors. In addition, we introduce a token-level noise-awareness mechanism that prevents training from saturation, hence improving optimization. Extensive experiments across conditional generation benchmarks demonstrate that our framework surpasses standard continuous diffusion models while providing up to 400x faster inference speed, and remains competitive against other one-step diffusion frameworks.

Dat Nguyen-Cong, Tung Kieu, Hoang Thanh-Tung

FPT Software AI Center, FPT Corporation · Department of Computer Science, Aalborg University, Denmark · Quantum AI and Cyber Security Institute, FPT Corporation

Diffusion Language Model Inference Continuous Diffusion Language Models Text Generation Conditional Diffusion Models Few-Step Diffusion Sampling

Ask PDF

Abstract

Self-Conditioning

Explore similar work

Self-conditioned Flow Map Language Models via Fixed-point Flows

FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation

FastDiSS: Few-step Match Many-step Diffusion Language Model on Sequence-to-Sequence Generation--Full Version