A Step in the Right Place: Understanding and Optimizing Timestep Placement in Continuous Diffusion Language Models

Shutian Zheng; Zhihong Yu; Linye Wei; Zeyang Zhang; Meng Li

cs.CLOct 11, 2026

A Step in the Right Place: Understanding and Optimizing Timestep Placement in Continuous Diffusion Language Models

Authors: Shutian Zheng, Zhihong Yu, Linye Wei, Zeyang Zhang, Meng Li

Organizations: College of Engineering, Peking University · Weiyang College, Tsinghua University · Institute for Artificial Intelligence, Peking University · School of Integrated Circuits, Peking University ·

Abstract

Continuous diffusion language models represent an emerging paradigm for language modeling. They generate text in parallel by iteratively refining continuous representations and mapping them back to tokens through a decoder. However, realizing the efficiency advantage of parallel generation requires maintaining generation quality with only a few denoising steps. Even when the number of steps is fixed, their placement along the denoising trajectory remains flexible. In this paper, we demonstrate that changing timestep placement alone can substantially alter the final latent representations and thereby affect the decoded token distributions. Building on these findings, we introduce StepTune, an algorithm that searches for timestep placements that improve generation quality. The search is conducted on a small calibration set and guided by a language-aware objective, with the model and sampling budget fixed. Once calibrated, the timestep placement is fixed and reused across inputs. Experiments demonstrate consistent improvements in unconditional generation perplexity and conditional task metrics across all evaluated sampling budgets. In particular, StepTune enables 16-step sampling with a frozen model to outperform its native 64-step baseline on both XSum summarization and WMT14 translation and reduces end-to-end latency by approximately $3.8\times$ . These results establish timestep placement as a practical means of improving the quality-efficiency trade-off in continuous diffusion language models.

Figures & tables

Figure 1: Timestep placement shapes generation. Different timestep placements alter the sampling trajectory, eventually leading to different decoded outputs.

Figure 2: StepTune improves generation quality with fewer denoising steps. (a) On XSum summarization and WMT14 De

\rightarrow

En translation, 16-step sampling outperforms the native 64-step baseline. (b) Unconditional generation with the official ELF SDE sampler shows consistently lower perplexity at matched NFE. The inset reports unigram entropy relative to LogitNormal (

100\%

Figure 3: Timestep placement changes terminal representations. Relative displacement from uniform sampling.

Figure 4: Correlations with KL across six perturbation scales.

Figure 5: Joint discrepancy reductions persist beyond calibration. (a–c) Frozen schedules reduce normalized

F_{\alpha}

on calibration and unseen validation trajectories (LogitNormal

=1

). (d) Validation scores with 95% confidence intervals.

Figure 6: Overview of StepTune. Compress the schedule space, search using $F_{\alpha}$ under quality constraints, and select and freeze one schedule for reuse.

NFE	Method	XSum					WMT14 De $\rightarrow$ En
		R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	Lat. $\downarrow$	Thr. $\uparrow$	BLEU $\uparrow$	chrF $\uparrow$	Lat. $\downarrow$	Thr. $\uparrow$
16	Native LogitNormal	34.636	11.010	26.351	142.11	25.83	23.0295	51.0300	141.09	190.97
	LD3 adaptation	35.112	11.600	26.940	–	–	24.7199	52.2235	–	–
	Ours	36.304	12.589	28.015	142.20	25.83	26.8390	53.6881	141.07	191.00
32	Native LogitNormal	35.340	11.611	26.996	275.67	13.51	24.6668	52.1351	274.66	100.96
	LD3 adaptation	35.326	11.788	27.131	–	–	24.3254	51.8025	–	–

Table 1: Conditional generation quality and efficiency on ELF-B. Results use the full XSum and WMT14 test sets (11,334 and 3,003 examples). Lat. denotes latency in ms at batch size 1, and Thr. denotes throughput in sequences/s at batch size 64; both are measured on an NVIDIA B200. All baselines are reproduced. LD3 refers to our adaptation of the original method (Appendix B.5 ); LD3 and ours use a 64-NFE teacher for NFE–16/32 and a 128-NFE teacher for NFE–64.

Figure 7: Calibration-query efficiency. Validation

F_{\alpha}

of the best calibration schedule so far. Both axes use logarithmic compression.

Method	BLEU $\uparrow$	chrF $\uparrow$
$A$ -only	$24.361\pm 0.153$	$52.653\pm 0.108$
KL-only	$24.357\pm 0.209$	$52.648\pm 0.142$
Pure- $F$	$24.515\pm 0.121$	$52.771\pm 0.067$
Pure- $Q$	$23.814\pm 1.975$	$52.227\pm 1.543$
Full	$\mathbf{24.721}\pm 0.383$	$\mathbf{52.918}\pm 0.247$

Table 3: Objective ablation. WMT14, NFE–16; mean

\pm

standard deviation across ten calibration panels.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	NFE	Panel size	Grid evaluations	Teacher NFE
Schedule control	16/32	16	160	64
Unconditional	8/16/32	64	80 (bank)	64
XSum	16/32	64	160	64
XSum	64	64	160	128
WMT14	16/32	16	160	64
WMT14	64	16	160	128

Appendix

Table 4: ELF calibration configurations. Budgets are per NFE. Each grid evaluation covers the full calibration panel; ELF-B unconditional calibration evaluates a fixed bank.

Quantity	Unconditional	Conditional
Unigram entropy $H$	$\geq 0.98H_{0}$	–
Relative length $L/L_{0}$	$[0.90,1.10]$	$[0.80,1.20]$
Repeated-trigram rate $R_{3}$	$\leq\max(1.10R_{3,0},R_{3,0}+0.01)$	$\leq\max(1.50R_{3,0},R_{3,0}+0.01)$
EOS rate	$\geq\mathrm{EOS}_{0}-0.05$	$\geq\mathrm{EOS}_{0}-0.05$
Empty-output rate	$\leq\mathrm{Empty}_{0}+0.01$	$\leq\mathrm{Empty}_{0}+0.01$

Appendix

Table 5: ELF calibration quality requirements. Subscript

0

denotes the bank reference for unconditional generation and the uniform student schedule for conditional generation.

Variant	BLEU	chrF
A-only	$25.358\pm 0.154$	$52.653\pm 0.108$
KL-only	$25.354\pm 0.222$	$52.648\pm 0.142$
Pure- $F$	$25.519\pm 0.122$	$52.771\pm 0.067$
Pure- $Q$	$24.796\pm 2.042$	$52.227\pm 1.543$
Full	$\mathbf{25.731}\pm 0.400$	$\mathbf{52.918}\pm 0.247$

Appendix

Table 6: Component ablation under the main WMT14 scoring convention. Results are computed from the saved outputs on the complete 3,003-example test set. Entries are means

\pm

sample standard deviations across ten calibration panels; BLEU is rescored using the main-results configuration, while chrF is unchanged.

Model	Task	NFE	$t<0.10$	$t<0.25$	$t<0.50$
ELF-B	XSum	16	93.8	100.0	100.0
ELF-B	XSum	32	90.6	90.6	90.6
ELF-B	XSum	64	92.2	92.2	92.2
ELF-B	WMT14	16	93.8	93.8	93.8
ELF-B	WMT14	32	78.1	81.3	87.5
ELF-B	WMT14	64	90.6	93.8	96.9

Appendix

Table 7: Temporal allocation of the deployed schedules. Entries are percentages of the

N

model-evaluation times strictly below each threshold, computed from the saved float32 grids.

Continuous Diffusion Language Models

Continuous diffusion language models generate text by gradually denoising continuous token embeddings or latent representations.

See all 49 papers

Distribution Matching Distillation for Continuous Diffusion Language Models
Sep 30, 2026Paul Le Van Kiem, Dario Shariatian, Umut Simsekli +1Continuous Diffusion Language Models Diffusion Language Model Distillation
Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique
Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models
Oct 1, 2026Yair Schiff, Omer Belhasin, Roy Uziel +7Diffusion Language Model Inference Language Modeling
NVIDIA · Cornell University
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language Models Diffusion Language Model Distillation
NVIDIA

Sep 30, 2026J/K move · Enter open · S save

Distribution Matching Distillation for Continuous Diffusion Language Models

Paul Le Van Kiem, Dario Shariatian, Umut Simsekli, Alain Durmus

Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique

Continuous Diffusion Language Models Diffusion Language Model Distillation Diffusion Language Models Diffusion Model Distillation Distribution Matching Distillation Diffusion Model Inference Acceleration+1

Open paper AskPDF arXiv

Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.

Abstract

Continuous Diffusion Language Models

Explore similar work

Distribution Matching Distillation for Continuous Diffusion Language Models

Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling