One-Step Text Generation by Seq-Drifting

Jiayu Liu; Ruizhi Zhao; Tianyun Ji; Yee Whye Teh; Zhenya Huang; Ning Miao

cs.AIOct 9, 2026

One-Step Text Generation by Seq-Drifting

Authors: Jiayu Liu, Ruizhi Zhao, Tianyun Ji, Yee Whye Teh, Zhenya Huang, Ning Miao

Organizations: Hong Kong Institute of AI for Science, City University of Hong Kong · Shenzhen University · State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Department of Statistics, University of Oxford ·

Abstract

Diffusion language models generate tokens in parallel, but still require multiple sampling steps to produce high-quality text. In this paper, we investigate whether a model can learn to generate an entire sequence in a single forward pass. We propose Seq-Drifting, a framework that applies drifting to sequences of continuous token representations and directly learns a map from Gaussian noise to text. Unlike existing DLMs that derive the supervision at each position directly from the observed token in a training sequence, Seq-Drifting builds a drift field whose position-wise targets adapt to the current generated context. At each position, attraction guides each output toward a set of contextually plausible token representations, while repulsion discourages collapse across samples and repetition within sequences. The generator learns from this drift field during training and produces all tokens jointly in one pass at inference. Empirically, Seq-Drifting achieves a generative perplexity of 36.01 on LM1B in one step and performs competitively in both unconditional and conditional generation. Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework beyond open-ended generation. These results suggest that drifting over token sequences offers a promising path toward effective one-step text generation. Our code is available at https://github.com/jyliu-ai/Seq-Drifting.

Figures & tables

Figure 1: Seq-Drifting achieves lower generative perplexity than prior few-step distilled DLMs, and is on par with or better than multi-step DLMs, with a

4

–

11\times

speedup over the few-step baselines and a

900

–

2500\times

speedup over the multi-step ones in wall-clock time. (Gen-PPL under GPT-2 Large; dataset: LM1B,

T{=}128

. Detailed comparison in Figure 3 and Table 6 .)

Figure 2: Seq-Drifting. In one forward pass the generator $G_{\theta}$ maps a noise vector $\epsilon\!\sim\!\mathcal{N}(0,I_{C})$ (optionally with a prefix $p_{1},\dots,p_{M}$ ) to token embeddings $\boldsymbol{h}_{1},\dots,\boldsymbol{h}_{T}$ , each projected onto the unit sphere as $\boldsymbol{e}_{t}=\boldsymbol{h}_{t}/\|\boldsymbol{h}_{t}\|\in S^{D-1}$ . Supervision is applied per position . A positive set $\mathcal{S}_{t}$ of plausible tokens attracts $\boldsymbol{e}_{t}$ , while the negative sets $N_{t}^{\mathrm{out}}$ (other samples) and $N_{t}^{\mathrm{in}}$ (other positions of the same sample) repel it, weighted by a similarity kernel $\varphi$ and strengths $\lambda_{\mathrm{in}/\mathrm{out}}$ . The two combine into the drift force $\boldsymbol{F}_{t}$ ; a single geodesic step along $\boldsymbol{F}_{t}$ gives the stop-gradient target $goal_{t}$ , onto which the generator is regressed, $\mathcal{L}=\mathbb{E}\|\boldsymbol{e}_{t}-\mathrm{sg}(goal_{t})\|^{2}$ .

Figure 3: Unconditional generation. Left: Gen-PPL (under GPT-2 Large) and unigram entropy for the few-step distilled models at $1$ / $2$ / $4$ steps and Seq-Drifting at a single pass, on LM1B ( $T{=}128$ ) and OpenWebText ( $T{=}1024$ ); multi-step baselines are in Table 6 . Right: Gen-PPL vs. sampling steps and wall-clock running time per sample, showing Seq-Drifting achieves lower Gen-PPL in a single forward pass at a fraction of the cost.

Method	Step	$64{\to}64$		$64{\to}512$
Method	Step	Gen-PPL $\downarrow$	Ent.	Gen-PPL $\downarrow$	Ent.
FMLM	1	162.20	3.78	418.37	3.32
	2	143.62	3.82	441.09	3.55
	4	124.74	3.82	320.22	3.52
Duo+DCD	1	972.09	3.77	7639.94	3.85
	2	384.35	3.72	846.20	3.72

Table 1: Conditional generation. A

64

-token prefix followed by a continuation.

		Unconditional ( $128$ )		Conditional ( $64{\to}64$ )
Decision	Variant	Gen-PPL $\downarrow$	Entropy	Gen-PPL $\downarrow$	Entropy
Seq-Drifting P		36.01	4.42	76.16	4.40
Geometry	Euclidean	diverges (loss blow-up)
Positives	off-policy	$>$ 1000	$>$ 6	$>$ 10000	$>$ 4
Anti-repetition	w/o intra	10.01	1.82	12.18	1.31
Repulsion	per-context (block)	–	–	81.27	4.27

Table 2: Ablation study on unconditional (

128

) and conditional (

64{\to}64

) generation.

Figure 4: Hyperparameter sweeps of the repulsion strength

\lambda_{\mathrm{out}}

, the anti-repetition weight

\lambda_{\mathrm{in}}

, the attraction temperature

\tau

, and the support size

k

. Setting: unconditional

128

-token generation (solid), conditional

64{\to}64

generation (dashed).

Table 7

Model	SVAMP		ASDiv		ProofWriter
Model	ACC $\uparrow$	Time/sample (ms) $\downarrow$	ACC $\uparrow$	Time/sample (ms) $\downarrow$	ACC $\uparrow$	Time/sample (ms) $\downarrow$
AR	27.92 $\pm 0.36$	49.42 $\pm 4.36$	44.97 $\pm 0.50$	31.07 $\pm 1.18$	85.90 $\pm 0.38$	41.90 $\pm 0.10$
NAR	4.72 $\pm 0.76$	2.70 $\pm 0.04$	1.73 $\pm 0.57$	1.30 $\pm 0.07$	0.00 $\pm 0.00$	1.42 $\pm 0.06$
Coconut	17.76 $\pm 0.01$	25.80 $\pm 0.52$	22.59 $\pm 0.01$	26.63 $\pm 0.13$	–	–
Seq-Drifting	13.60 $\pm 0.50$	5.88 $\pm 0.17$	19.11 $\pm 0.79$	1.05 $\pm 0.03$	79.31 $\pm 0.47$	3.68 $\pm 0.02$

Table 4: Performance on SVAMP, ASDiv, and ProofWriter.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Step	Self-BLEU
Method	Step	LM1B	OpenWebText
FMLM	1	0.201	0.249
	2	0.198	0.251
	4	0.204	0.260
Duo+DCD	1	0.042	0.016
	2	0.065	0.064

Appendix

Table 5: Self-BLEU across the step budget. Per-dataset Self-BLEU for the few-step distilled models and Seq-Drifting; a fully mode-collapsed generator scores

1.0

(bottom row).

Figure 7: Sampling time across step budgets. Seconds per sample on LM1B (

T=128

) and OpenWebText (

T=1024

). The x-axis is logarithmic in the number of sampling steps and the y-axis is logarithmic in seconds per sample. The dashed line marks Seq-Drifting.

Method	Step	LM1B ( $T{=}128$ )		OpenWebText ( $T{=}1024$ )
Method	Step	Gen-PPL $\downarrow$	Entropy	Gen-PPL $\downarrow$	Entropy
FLM	1	6.27	1.15	27.54	1.53
	8	215.84	4.24	453.49	5.20
	1024	85.24	4.29	63.97	5.35
Duo	1	1027.77	4.34	4007.98	5.80
	8	121.37	4.27	196.45	5.57

Appendix

Table 6: Unconditional generation: multi-step models across the step budget, with Seq-Drifting at a single pass. LM1B (

T{=}128

) and OpenWebText (

T{=}1024

). A dash indicates that no corresponding released checkpoint or published reference result was available for that dataset–model setting.

Method	Step	$64{\to}64$		$64{\to}512$
Method	Step	Gen-PPL $\downarrow$	Entropy	Gen-PPL $\downarrow$	Entropy
FLM	1	1.95	0.83	201.21	2.18
	8	6.00	1.81	522.85	3.63
	1024	9.47	2.19	304.01	3.44
Duo	1	1103.17	3.83	7616.55	3.86
	8	99.98	3.78	181.91	3.81

Appendix

Table 7: Conditional generation: multi-step models across the step budget, with Seq-Drifting at a single pass. A

64

-token prefix, then a continuation (

64{\to}64

on LM1B,

64{\to}512

on OWT). A dash indicates that no corresponding released checkpoint or published reference result was available for that dataset–model setting.

Method	Step	LM1B		OpenWebText
Method	Step	Pythia-1B	OPT-1.3B	Pythia-1B	OPT-1.3B
FMLM	1	94.75	97.29	170.10	152.67
	2	87.55	88.58	134.89	122.45
	4	79.86	80.37	109.44	100.41
Duo+DCD	1	829.59	849.31	4654.30	4401.95
	2	435.40	453.47	905.38	792.78

Appendix

Table 8: Unconditional Gen-PPL under alternative evaluators. Gen-PPL computed with Pythia-1B and OPT-1.3B for LM1B (

T{=}128

) and OpenWebText (

T{=}1024

Panel of Figure 8 — Figure 8: Training dynamics of the conditional runs. Gen-PPL (log scale, left axis) and unigram entropy (right axis) versus training step, for Seq-Drifting P $64{\to}64$ (top left), Seq-Drifting S $64{\to}64$ (top right), Seq-Drifting P $64{\to}512$ (bottom left), and Seq-Drifting S $64{\to}512$ (bottom right). All panels show training through $300$ k steps for both output lengths. The dashed line marks the warm-up $\to$ conditional switch at $200$ k steps.

Figure 9: Off-policy training dynamics (unconditional, length

128

). Gen-PPL (log scale, left axis) and unigram entropy (right axis) versus training step for the off-policy positives ablation. The dashed line marks the end of the temperature warm-up at

10

k steps, where

\tau

reaches

0.1

and the run begins to oscillate and degrade.

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 410 papers

Drifting Objectives for Refining Discrete Diffusion Language Models
May 19, 2026Daisuke Oba, Hiroki Furuta, Naoaki OkazakiDiffusion Model Fine-Tuning Diffusion Language Models
Institute of Science Tokyo · AIST · NII LLMC
ReLaDrift: Recurrent Language Generation via Drifting for Test-Time Scaling
Oct 16, 2026Yuki Otsuka, Daisuke Oba, Naoaki OkazakiTest-Time Scaling Recurrent Language Models
Institute of Science Tokyo · AIST · NII LLMC
DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Models
Sep 14, 2026Pengyu Ji, Zichen Zhang, Xiang Hu +1Non-Autoregressive Text Generation Diffusion Language Model Decoding
‡School of Information Science and Technology, ShanghaiTech University · §Shanghai Engineering Research Center of Intelligent Vision and Imaging · ¶Tencent

May 19, 2026J/K move · Enter open · S save

Drifting Objectives for Refining Discrete Diffusion Language Models

Daisuke Oba, Hiroki Furuta, Naoaki Okazaki

Institute of Science Tokyo · AIST · NII LLMC

Diffusion Model Fine-Tuning Diffusion Language Models Drifting Models Diffusion Models Diffusion Language Model Training

Open paper AskPDF arXiv

Discrete diffusion language models (DDLMs) generate text by iteratively denoising categorical token sequences, while recent drifting methods for continuous generators suggest that part of this sampling-time correction can instead be absorbed into training through an anti-symmetric fixed-point objective. We study how to transfer this principle to DDLMs, where the main challenge is the interface with discrete text: hard token samples are non-differentiable, and categorical predictions do not directly provide continuous samples to drift. We formulate TokenDrift, a drifting objective that lifts categorical predictions to soft-token features, applies anti-symmetric drifting in a frozen semantic space, and backpropagates the resulting stop-gradient feature target to DDLM logits. In controlled continual-training experiments with masked and uniform-state diffusion backbones, TokenDrift improves fixed-NFE generation quality over matched continuation baselines, reducing Gen.-PPL at 4 NFEs by 89% on MDLM and 86% on DUO. These results suggest that drifting can provide a practical refinement objective for DDLMs.

Abstract

Text Generation

Explore similar work

Drifting Objectives for Refining Discrete Diffusion Language Models

ReLaDrift: Recurrent Language Generation via Drifting for Test-Time Scaling

DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Models