Spectrally-Guided Diffusion Noise Schedules

Carlos Esteves; Ameesh Makadia

cs.CVMar 19, 2026

Spectrally-Guided Diffusion Noise Schedules

Authors: Carlos Esteves, Ameesh Makadia

Organizations: Google Research.

Abstract

Denoising diffusion models are widely used for high-quality image and video generation. Their performance depends on noise schedules, which define the distribution of noise levels applied during training and the sequence of noise levels traversed during sampling. Noise schedules are typically handcrafted and require manual tuning across different resolutions. In this work, we propose a principled way to design per-instance noise schedules for pixel diffusion, based on the image's spectral properties. By deriving theoretical bounds on the efficacy of minimum and maximum noise levels, we design ``tight'' noise schedules that eliminate redundant steps. During inference, we propose to conditionally sample such noise schedules. Experiments show that our noise schedules improve generative quality of single-stage pixel diffusion models, particularly in the low-step regime.

Figures & tables

Figure 1: Our “tight” schedules adapt to each instance’s spectrum, ensuring effective noise levels at all steps. Top: An image with low energy on low frequencies. The standard cosine noise schedule destroys the signal at $t=0.5$ , which means that at least half of the training steps would apply excessive noise for this input. Our adaptive schedule preserves the low frequency content – notice that the object outline is still visible. Bottom: An image with high energy on high frequencies. The cosine schedule barely changes the input at $t=0.1$ – notice that the RAPSD curves between the cosine schedule and the input are close and correlated. This means that at least $10\%$ of the training steps would apply insufficient noise. Our schedule is effective at destroying a part of the high-frequency content at this level.

Figure 2: Our noise schedules vary per instance based on its spectral properties. Left: Median power per frequency for ImageNet at multiple resolutions (increasing from light to dark). The power spectrum of natural images follows a power law whose trends explain current noise schedule tuning heuristics. We eschew such heuristics and use each instance’s spectrum to determine its schedule. Middle: Cosine schedule and ours for 1000 images from ImageNet

256\times 256

. Right: Median noise schedules for the same set of images, at

128\times 128

256\times 256

, and

512\times 512

(light to dark color). Our schedules avoid excessively high and low noise values, while following similar trends to the baseline across resolutions without any hyperparameter change.

ImageNet $\mathbf{512\times 512}$
Model	Params	NFE	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
JiT-G ( Li & He, 2025 )	2B	-	1.78	-	306.8	-	-
PixNerd-XL/16 ( Wang et al., 2025 )	700M	100	2.84	5.95	245.6	0.80	0.59
SiD2, small	397M	512	2.19 (2.19)	4.30	295.3	0.72	0.63
Ours, small	399M	256	1.79	4.39	306.1	0.73	0.64
SiD2, Flop Heavy	397M	512	1.53 (1.48)	3.98	306.2	0.74	0.63

Table 1: Class-conditional generation on ImageNet. We compare our spectral noise scheduling against recent single-stage pixel diffusion baselines. The fairest comparison is against SiD2 ( Hoogeboom et al., 2025 ) ; we use exactly the same architecture and training protocol except for our contributions described in Section 4 . We outperform the baselines in most metrics, while needing fewer denoising steps than SiD2. We reproduce SiD2 it and compute additional metrics besides the reported FID; originally reported results are quoted next to reproduced. Section D.3 describes each metric. Ours and SiD2 values are averaged over 5 sets of generations with different seeds. NFE : number of function evaluations (denoising steps). Adapt. : PixelFlow uses a (slower) solver with adaptive number of steps.

Figure 3: Comparison against the SiD2 ( Hoogeboom et al., 2025 ) baseline on ImageNet, at different number of function evaluations (NFE), or denoising steps. Our model outperforms the baseline at the optimal number of steps, and the gap widens as the number of steps reduces. Interestingly, our “tight” schedules exhibit a slight FID worsening at high number of steps, we investigate this further on Section C.1 .

Figure 4: Samples from ImageNet $256\times 256$ . Each $2\times 4$ block shows the SiD2 baseline on top and ours on bottom, while the number of denoising steps is, from left to right, 32, 64, 128, and 256. Our generations are noticeably of higher quality at low step counts.

Figure 5: Manipulating the sampled spectrum to modify generated image properties. Here we modify the sampled spectrum such that the energy at the highest frequency is multiplied by factors 0.1, 0.2, 0.4, 1.0, 2.5, 5.0, 10.0, respectively. This affects the noise schedule and the model conditioning, so it is a way to guide the model towards different spectral properties. In this example, the energy on high frequencies correlate to the amount of texture and details. Notice how the amount of details increase from left to right. Images are generated by the same model trained on ImageNet $256\times 256$ and same initial noise.

Baselines
Method	NFE	FID $\downarrow$	IS $\uparrow$
SiD2 Baseline	512	1.68	288.2
Ours (Mixed Schedule)	256	1.42	297.0
Schedule Design
Fixed Schedule (Median)	256	1.61	299.3
Cosine with MinMax	256	1.98	271.4

Table 2: Ablation studies on ImageNet

256\times 256

using the small model architecture. We analyze the impact of our main contributions, including scheduling, conditioning mechanisms, and guidance interval parametrization.

Originally reported
Method	Params	NFE	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$
SiD	2B	256	9.6	-	-
SiD2	-	256	8.1	-	-
Our runs, heavy models
SiD2	647M	128	7.75	17.3	33.9
SiD2	647M	160	7.56	16.2	33.6

Table 3: Zero-shot text-to-image generation on MS-COCO

512\times 512

. Our runs are based on the flop heavy models described in Section D.1 , modified to inject text conditioning. This outperforms the SiD2 reported results even before applying our contributions, which widen the gap while needing fewer denoising steps. The Oracle model is the same as Ours but using the ground truth RAPSDs instead of the trained sampler.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Baseline with increasing NFE
Model	Params	NFE	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
SiD2, flop heavy	393M	32	13.3	14.2	151.2	0.542	0.519
SiD2, flop heavy	393M	64	6.27	6.21	193.4	0.662	0.575
SiD2, flop heavy	393M	128	2.96	4.22	209.9	0.715	0.619
SiD2, flop heavy	393M	256	1.61	3.71	215.7	0.727	0.640
SiD2, flop heavy	393M	512	1.30	3.64	210.9	0.728	0.646

Appendix

Table 4: Study of degradation at high NFE on ImageNet

128\times 128

. Results show that 1) the baselines show similar but milder degradation, 2) the degradation is predominantly due to loss of diversity, not quality, and 3) increasing the noise added at each inference step by setting

\gamma=0.5

mitigates the degradation and brings the best results at higher step counts for our models.

Figure 6: Manipulating the sampled spectrum to modify generated image properties. Here we show a different manipulation than the one in Fig. 5 . We modify the sampled power spectrum by multiplying it by 1/3, 2/3, 1.0, 2.0, 3.0, respectively. The effect controls the variance of the generated pixel values (contrast). When conditioned on the modified spectra, the generated images may vary in lighting, focus, and texture. Images are generated by the same model trained on ImageNet $256\times 256$ and same initial noise.

Model	Params	NFE	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
ADM ( Dhariwal & Nichol, 2021 )	554M	250	26.2	6.35	39.7	0.61	0.63
SiD2, small	397M	512	17.2	4.95	60.8	0.55	0.67
Ours, small	399M	256	16.8	4.37	61.4	0.56	0.67

Appendix

Table 5: Unconditional generation on ImageNet

256\times 256

. Results show the same trends as observed in the conditional models – our models achieve superior performance with fewer denoising steps.

Table 12

Model	Resolution	Guidance Interval	NFE	$\kappa_{\text{min}}$	$\kappa_{\text{max}}$
Small	$128\times 128$	$(0.15,0.40)$	160	0.2	200
Heavy	$128\times 128$	$(0.15,0.35)$	160	0.2	200
Small	$256\times 256$	$(0.10,0.45)$	256	0.2	200
Heavy	$256\times 256$	$(0.10,0.45)$	256	0.2	200
Small	$512\times 512$	$(0.05,0.50)$	256	0.2	200
Heavy	$512\times 512$	$(0.05,0.50)$	320	0.2	200

Appendix

Table 8: Hyperparameters for different model configurations and resolutions. We list the optimal classifier-free guidance intervals (in terms of

t

), number of sampling steps (NFE), and the scaling factors

\kappa_{\text{min}}

and

\kappa_{\text{max}}

Figure 7: Samples generated by our class-conditional flop heavy model trained on ImageNet $512\times 512$ .

Figure 8: Samples generated by our text-to-image model at $512\times 512$ , zero-shot conditioned on MS-COCO prompts.

Diffusion-Based Image Generation

Diffusion-based image generation creates images by progressively denoising noise, with controls for content, style, and structure.

See all 312 papers

Noise Schedule Design for Diffusion Models: An Optimal Control Perspective
May 21, 2026Seo Taek Kong, Weina Wang, R. SrikantDiffusion Sampling Diffusion-Based Image Generation
ECE & CSL, University of Illinois Urbana-Champaign · Computer Science Department, Carnegie Mellon University · ECE, CSL & NCSA, University of Illinois Urbana-Champaign
Spectral Progressive Diffusion for Efficient Image and Video Generation
May 18, 2026Howard Xiao, Brian Chao, Lior Yariv +1Video Diffusion Models Diffusion Sampling
Stanford University
Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusion
Jun 13, 2026Weichen Fan, Haiwen Diao, Penghao Wu +1Pixel-Space Diffusion Models Diffusion-Based Image Generation
1S-Lab, Nanyang Technological University

May 21, 2026J/K move · Enter open · S save

Noise Schedule Design for Diffusion Models: An Optimal Control Perspective

Seo Taek Kong, Weina Wang, R. Srikant

ECE & CSL, University of Illinois Urbana-Champaign · Computer Science Department, Carnegie Mellon University · ECE, CSL & NCSA, University of Illinois Urbana-Champaign

Diffusion Sampling Diffusion-Based Image Generation Diffusion Timestep Scheduling Optimal Control Diffusion Models

Open paper AskPDF arXiv

We develop a principled framework for analyzing and designing noise schedules in diffusion models. We show that one can recast this design problem as an optimal control problem, whose state is the Fisher information of the diffusion process, which evolves according to an ODE, and whose control input is the noise schedule. The objective of the optimal control problem is a functional involving the Fisher information, which is shown to be an upper bound on the Kullback-Leibler sampling error. By solving this optimal control problem, we obtain sufficient conditions on noise schedules under which

\tilde{\mathcal{O}}(d/n)

sampling error is achievable, where

d

is the data dimension and

n

is the number of discretization steps. While existing theoretical works also prove that

\tilde{\mathcal{O}}(d/n)

sampling error bounds are achievable, these results hold for specific noise schedules that differ from those used in practice. Under additional assumptions on the data distribution, we derive a family of noise schedules that recovers the functional forms of schedules widely used in practice. We show that these schedules achieve the

\tilde{\mathcal{O}}(d/n)

error bound when their parameters are chosen according to our analysis. Our theory offers a principled approach to noise schedule design, yielding explicit functional forms and guidance for selecting their parameters.

Abstract

Diffusion-Based Image Generation

Explore similar work

Noise Schedule Design for Diffusion Models: An Optimal Control Perspective

Spectral Progressive Diffusion for Efficient Image and Video Generation

Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusion