Three Creates All: You Only Sample 3 Steps

Yuren Cai; Guangyi Wang; Zongqing Li; Li Li; Zhihui Liu; Songzhi Su

cs.LGMar 23, 2026

Three Creates All: You Only Sample 3 Steps

Authors: Yuren Cai, Guangyi Wang, Zongqing Li, Li Li, Zhihui Liu, Songzhi Su

Organizations: School of Informatics, Xiamen University · School of Artificial Intelligence, Shenzhen Polytechnic University · Algorithm Research Department, Truesight

Abstract

Diffusion models deliver high-fidelity generation but remain slow at inference time due to many sequential network evaluations. We find that standard timestep conditioning becomes a key bottleneck for few-step sampling. Motivated by layer-dependent denoising dynamics, we propose Multi-layer Time Embedding Optimization (MTEO), which freeze the pretrained diffusion backbone and distill a small set of step-wise, layer-wise time embeddings from reference trajectories. MTEO is plug-and-play with existing ODE solvers, adds no inference-time overhead, and trains only a tiny fraction of parameters. Extensive experiments across diverse datasets and backbones show state-of-the-art performance in the few-step sampling and substantially narrow the gap between distillation-based and lightweight methods. Code will be available.

Figures & tables

Figure 1: Overview. (left) The visualization of single sampling step. (mid top) Procedure from $t_{cur}$ to time embedding. (mid bottom) Two type of diffusion inner blocks. (right top) The visualization of multi-layer and vanilla embedding. Different colors represent independence across the layers, dashed box denote the embedding are produced by broadcasting. (right bottom) The inner display of FiLM exert on feature maps.

Panel of Figure 2 — Figure 2: L2 distance between different sampling step of DDIM and ground-truth trajectories on CIFAR-10. We generate the ground-truth trajectory with iPNDM, 61 steps, schedule_type= polynomial , schedule_rho= 7 . The $\tau$ is generated by 121 steps.

Panel of Figure 4 — Figure 4: L2 distance between different feature trajectories. The setting is same as Fig. 2 .

Panel of Figure 5 — Figure 5: PCA analysis on conventional embedding design and MTE. The x-axis denote first n principal components. Green : time embedding. Blue : FiLM parameters.

Method	CIFAR-10				FFHQ
	NFE
	3	4	5	6	3	4	5	6
DDIM	93.36	67.40	49.66	36.08	78.16	57.37	43.85	35.15
iPNDM	47.98	24.81	13.59	7.05	45.90	28.21	17.14	10.00
UniPC	109.60	45.54	23.98	11.51	86.34	44.73	21.36	12.82
DEIS	56.00	25.64	14.37	9.39	54.45	28.30	17.39	12.29

Table 2: FID evaluation on CIFAR-10 and FFHQ Datasets. We compare MTEO against lightweight and training-free methods. Details are presented in the appendix.

Method	FID $\downarrow$				CLIP $\uparrow$
Step (1 Step = 2 NFE)	3	4	5	6	3	4	5	6
DDIM	36.04	24.02	19.73	17.69	25.87	28.35	29.32	29.79
DPM-Solver++(2M)	35.03	21.57	17.45	15.85	25.95	28.50	29.42	29.83
AMED-Plugin	$-$	18.92	$-$	14.84	$-$	$-$	$-$	$-$
EPD-Solver	$-$	16.46	$-$	13.14	$-$	$-$	$-$	$-$
+DDIM	14.93	12.92	13.65	13.53	28.86	29.65	29.85	30.01

Table 3: FID and CLIP evaluation on MS-COCO Dataset. We omitted some of the results due to the lack of corresponding pretrained checkpoints and settings.

Table 8

Method	Model Size	Parameter	Percent	NFE	FID	Hours
DMD	212.83MB	1464.32MB	688%	1	2.66	-
CTM	212.83MB	460.80MB	216%	1	1.98	153.70
ECM	212.83MB	212.83MB	100%	1	3.60	709.52
ECM	212.83MB	212.83MB	100%	2	2.11	709.52
SFD	212.83MB	212.83MB	100%	2	4.53	1.19
SFD	212.83MB	212.83MB	100%	3	3.58	1.69

Table 6: Comparison MTEO against Distillation-Based acceleration on CIFAR-10 Dataset. We report +DDIM as main results. Hours denote solo RTX3090 GPU hour. The Model Size denote the size of original teacher model and The Parameter indicates the parameter count used for distillation in the corresponding method.

Panel of Figure 6 — Figure 6: visualization of both Gain $g^{+}_{\mathcal{T}}$ and Drop $g^{-}_{\mathcal{T}}$ view. The x axis denote subset $\mathcal{T}$ .

Method	NFE	FID $\downarrow$
Method	NFE	vanilla	+single	+MTEO
DDIM	3	93.36	15.60	5.29
	4	67.40	10.43	3.91
	5	49.66	6.81	3.02
	6	36.08	5.18	2.87
iPNDM	3	47.98	18.75	6.63

Table 7: Ablation of single-layer

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Method	ImageNet-64				LSUN Bedroom
	NFE
	3	4	5	6	3	4	5	6
DDIM	75.91	52.53	43.86	24.33	86.13	54.53	34.34	25.26
iPNDM	52.17	28.95	15.61	10.60	80.99	43.89	26.65	20.71
UniPC	85.31	49.69	20.87	12.02	112.3	49.00	23.20	11.64
DEIS	40.31	20.26	12.45	8.94	52.46	23.65	14.42	10.74

Appendix

Table 8: FID evaluation on ImageNet-64 and LSUN Bedroom Datasets

Method	Model Size	Parameter	Percent	NFE	FID	Hours
DMD-cond	–	–	–	1	2.62	–
CTM	1126.4MB	1228.8MB	109%	1	2.06	1660.86
ECM-S	1126.4MB	1126.4MB	100%	1	5.51	76.50
ECM-S	1126.4MB	1126.4MB	100%	2	3.18	76.50
SFD	1126.4MB	1126.4MB	100%	2	10.25	6.15
SFD	1126.4MB	1126.4MB	100%	3	6.35	8.53

Appendix

Table 9: Comparison MTEO against Distillation-Based acceleration on ImageNet-64 Dataset. We report +DDIM as main results. Hours denote solo RTX3090 GPU hour.

Method	Model Size	Parameter	Percent	NFE	FID
SFD	2007.04MB	2007.04MB	100%	2	10.39
SFD	2007.04MB	2007.04MB	100%	3	6.42
SFD	2007.04MB	2007.04MB	100%	4	5.26
SFD	2007.04MB	2007.04MB	100%	5	4.73
SFD-v	2007.04MB	2007.04MB	100%	2	9.25
SFD-v	2007.04MB	2007.04MB	100%	3	5.36

Appendix

Table 10: Comparison MTEO against Distillation-Based acceleration on LSUN Bedroom Dataset. We report +DDIM as main results.

Method	Schedule	$\rho$	NFE
			3	4	5	6
DDIM	polynomial	7	86.13	54.53	34.34	25.26
iPNDM	polynomial	7	80.99	43.89	26.65	20.71
DPM-Solver++(3M)	logsnr	–	111.90	49.46	23.15	12.28
MTEO (Ours)
+DDIM	polynomial	7	12.99	7.01	6.03	4.44

Appendix

Table 11: Detailed FID results on LSUN Bedroom

Method	Schedule	$\rho$	NFE
			3	4	5	6
DDIM	polynomial	7	93.36	67.40	49.66	36.08
iPNDM	polynomial	7	47.98	24.81	13.59	7.05
DPM-Solver++(3M)	logsnr	–	110.00	46.52	24.97	11.99
MTEO (Ours)
+DDIM	polynomial	7	5.29	3.91	3.02	2.87

Appendix

Table 12: Detailed FID results on CIFAR-10. We report different types of schedule on +DDIM, +iPNDM and +DPM++(3M). The time_uniform2 schedule perform best which also report as main result.

Method	Schedule	$\rho$	NFE
			3	4	5	6
DDIM	polynomial	7	78.16	57.37	43.85	35.15
iPNDM	polynomial	7	45.90	28.21	17.14	10.00
DPM-Solver++(3M)	logsnr	–	86.45	45.94	22.51	13.74
MTEO (Ours)
+DDIM	polynomial	7	9.84	6.05	4.45	3.76

Appendix

Table 13: Detailed FID results on FFHQ. We report different types of schedule on +DDIM, +iPNDM and +DPM++(3M). The time_uniform schedule perform best which also report as main result.

Method	schedule	$\rho$	Metric	NFE=3	4	5	6
DDIM	polynomial	7	IS	11.68	16.25	20.49	24.40
			FID	75.91	52.53	43.86	24.33
			sFID	70.94	47.43	33.80	26.53
			Precision	29.25%	38.20%	45.81%	51.15%
			Recall	30.24%	38.81%	44.21%	48.50%
iPNDM	polynomial	7	IS	15.37	23.54	31.34	36.06

Appendix

Table 14: IS, FID, sFID, Precision and Recall evaluation on ImageNet-64 Dataset.

Dataset

Sampler

Steps

Schedule

\rho

lr

lr_{min}

\varepsilon

\varepsilon_{min}

Patience

E_{max}

Train Seeds

Batch Size

Tea Sampler

Tea Steps

CIFAR10/FFHQ

DDIM

poly/uni

7/2

2e-2

1e-3

1e-2

1e-3

300

50000-50255

iPNDM

poly/uni

7/2

2e-2

1e-3

1e-2

1e-3

300

50000-50255

iPNDM

poly/uni

7/2

2e-2

1e-3

1e-2

1e-3

300

50000-50255

iPNDM

poly/uni

7/2

2e-2

1e-3

1e-2

1e-3

300

50000-50255

iPNDM

poly/uni

7/2

2e-2

1e-3

1e-2

1e-3

300

50000-50255

iPNDM

poly/uni

7/2

2e-2

1e-3

1e-2

1e-3

300

50000-50255

iPNDM

Appendix

Table 15: Experimental settings for each dataset.

Table 20

Table 21

Table 22

Method	NFE	FID $\downarrow$
Method	NFE	+MTEO	+MTEO-deep
DDIM	3	5.29	5.18
	4	3.91	3.93
	5	3.02	3.48
	6	2.87	3.05
iPNDM	3	6.63	6.57

Appendix

Table 22: Comparison on CIFAR10 between MTEO and MTEO-deep. We train both methods with same configuration

Panel of Figure 7 — Figure 7: Visualization of feature trajectory. See more example in appendix.

Panel of Figure 8 — Figure 8: Additional Experiments on FiLM’s Channel Correction Capability. We conducted experiments on two batches of data, 0-63 and 64-127, and reported the mean values before and after correction, along with the methods used. First, we generated the true trajectory using iPNDM with 60 NFE. Based on this, we simulated a 4-steps sampling process, corresponding to the steps from Idx0 $\to$ Idx19 $\to$ Idx39 $\to$ Idx59.

Table 26

Panel of Figure 9 — Figure 9: Additional visualization of feature trajectory with trajectory 1

Panel of Figure 10 — Figure 10: Trainging loss curve. We display the trainging procedure of MTEO+DDIM on CIFAR10 with step 4, 5, 6, 7.

Figure 11: The intermediate visualization results of training procedures with 4-step MTEO+DDIM.

Panel of Figure 12 — Figure 12: The phenomenon of mode non-convergence in Stable Diffusion sampling. As shown in the figure, the sampling result at 4 steps exhibits a pattern that is completely different from that at 11 steps, while the result at 7 steps demonstrates an intermediate transition.

Panel of Figure 13 — Figure 13: Samples on CIFAR10 32×32 with 3 NFE.

Panel of Figure 14 — Figure 14: Samples on FFHQ 64×64 with 3 NFE.

Panel of Figure 15 — Figure 15: Samples on ImageNet 64×64 with 3 NFE.

Panel of Figure 16 — Figure 16: Samples on ImageNet 64×64 with 3 NFE.

Panel of Figure 17 — Figure 17: Samples on ImageNet-256 256×256 with 3 NFE.

Panel of Figure 18 — Figure 18: Samples on ImageNet-256 256×256 with 6 NFE.

Panel of Figure 19 — Figure 19: Samples on LSUN Bedroom 256×256 with 3 NFE.

Panel of Figure 20 — Figure 20: Samples on MS COCO 512×512 with 3 NFE.

Few-Step Diffusion Sampling

Few-step diffusion sampling develops ways to generate data with far fewer denoising evaluations while retaining quality and diversity.

See all 174 papers

Optimize Your Sampling: Tuned Diffusion Sampling with Bayesian Optimization
Aug 18, 2026Travis Zhang, Christian Belardi, Justin Lovelace +4Diffusion Timestep Scheduling Few-Step Diffusion Sampling
Cornell University
Unlocking Few-Step Diffusion for Faithful Previews
Sep 28, 2026Jing Jia, Sifan Liu, Guanyang WangDiffusion Sampling Few-Step Diffusion Sampling
Department of Computer Science, Rutgers University · Department of Statistical Science, Duke University · Department of Statistics, Rutgers University
Temporal Difference Learning for Diffusion Models
Jun 13, 2026Qizhen Ying, Yangchen Pan, Victor Adrian Prisacariu +1Few-Step Diffusion Sampling RL for Diffusion Models
Department of Engineering Science, University of Oxford, Oxford, United Kingdom · School of Computer Science, Carleton University, Ottawa, Canada.

Aug 18, 2026J/K move · Enter open · S save

Optimize Your Sampling: Tuned Diffusion Sampling with Bayesian Optimization

Travis Zhang, Christian Belardi, Justin Lovelace, Jin Peng Zhou+3

Cornell University

Diffusion Timestep Scheduling Few-Step Diffusion Sampling Diffusion Model Inference Acceleration T2I Diffusion Models Diffusion-Based Denoising

Open paper AskPDF arXiv

Sampling from a diffusion model typically requires many forward passes through a large neural network, making generation computationally expensive. While much work has focused on efficient solvers and samplers, comparatively little attention has been paid to selecting the sampling timesteps themselves. A recent line of work optimizes theoretically derived surrogates for sample quality rather than the quality metric itself. We propose Optimizing Your Sampling (OYS), which instead treats timestep selection as a black-box optimization problem, optimizing the target metric directly with Bayesian optimization. OYS outperforms both the default schedules and those of Align Your Steps on text-to-image generation, and improves over the default schedules on inpainting and other image tasks, in both quantitative and human evaluations. OYS requires no additional training, is applicable even to distilled models, and improves both simple and sophisticated samplers such as Euler and DPM-Solver++. A 5-step OYS schedule retains 89%-94% of the quality of a 50-step schedule while reducing inference cost by 10x.

Abstract

Few-Step Diffusion Sampling

Explore similar work

Optimize Your Sampling: Tuned Diffusion Sampling with Bayesian Optimization

Unlocking Few-Step Diffusion for Faithful Previews

Temporal Difference Learning for Diffusion Models