Spectral Regularization for Diffusion Models

Satish Chandran; Nicolas Roque dos Santos; Yunshu Wu; Greg Ver Steeg; Evangelos Papalexakis

cs.LGMar 2, 2026

Spectral Regularization for Diffusion Models

Authors: Satish Chandran, Nicolas Roque dos Santos, Yunshu Wu, Greg Ver Steeg, Evangelos Papalexakis

Organizations: Department of Mathematics, University of California Riverside, Riverside, California, USA · Department of Computer Science, University of California Riverside, Riverside, California, USA

Abstract

Diffusion models are typically trained using pointwise reconstruction objectives that are agnostic to the spectral and multi-scale structure of natural signals. We propose a loss-level spectral regularization framework that augments standard diffusion training with differentiable Fourier- and wavelet-domain losses, without modifying the diffusion process, model architecture, or sampling procedure. The proposed regularizers act as soft inductive biases that encourage appropriate frequency balance and coherent multi-scale structure in generated samples. Our approach is compatible with DDPM, DDIM, and EDM formulations and introduces negligible computational overhead. Experiments on image and audio generation demonstrate consistent improvements in sample quality, with the largest gains observed on higher-resolution, unconditional datasets where fine-scale structure is most challenging to model.

Figures & tables

Panel of Figure 1 — Figure 1: Checkerboard toy experiment. Figures (a) to (c) show the ground-truth pattern, a sample from a model trained without spectral regularization, and a sample from a model trained with the proposed amplitude-and-phase loss.

Figure 2: Radially averaged power spectra (log scale) for the ground truth, baseline DDPM with MSE loss, and DDPM with our amplitude+phase spectral loss.

Dataset	EDM variant	EDM Sampling Steps	Conditioning	Regularizer	Weighted FID	Unweighted FID	EDM FID
CIFAR	VE	18	Cond	Amp	1.82 $\pm$ 0.01	1.82 $\pm$ 0.01	1.81 $\pm$ 0.01
				Amp Phase	1.82 $\pm$ 0.01	1.82 $\pm$ 0.01
				Haar	1.81 $\pm$ 0.01	1.81 $\pm$ 0.02
				Bior13	1.82 $\pm$ 0.01	1.81 $\pm$ 0.02
CIFAR	VP	18	Cond	Amp	1.84 $\pm$ 0.02	1.84 $\pm$ 0.02	1.84 $\pm$ 0.02
				Amp Phase	1.84 $\pm$ 0.02	1.84 $\pm$ 0.02

Table 1: Fréchet Inception Distance (FID) scores for different spectral regularizers and EDM variants. All models are fine-tuned from pretrained EDM baselines. Lower values indicate better generative performance.

Method	$\lambda$	FAD $\downarrow$	UTMOS $\uparrow$	PESQ $\uparrow$	MR-STFT $\downarrow$	NDB $\downarrow$
Diffwave	–	1.994 $\pm$ 0.008	3.941 $\pm$ 0.005	3.440 $\pm$ 0.002	1.217 $\pm$ 0.001	0.63 $\pm$ 0.02
Amp	$10^{-4}$	1.462 $\pm$ 0.006	3.953 $\pm$ 0.009	3.477 $\pm$ 0.003	1.1802 $\pm$ 0.0005	0.65 $\pm$ 0.03
	$10^{-5}$	1.609 $\pm$ 0.007	3.953 $\pm$ 0.009	3.476 $\pm$ 0.001	1.1930 $\pm$ 0.0004	0.63 $\pm$ 0.02
	$10^{-6}$	1.775 $\pm$ 0.008	3.952 $\pm$ 0.006	3.491 $\pm$ 0.003	1.1958 $\pm$ 0.0005	0.64 $\pm$ 0.03
Amp+Phase	$10^{-4}$	1.694 $\pm$ 0.009	3.969 $\pm$ 0.008	3.516 $\pm$ 0.002	1.1896 $\pm$ 0.0003	0.66 $\pm$ 0.02
	$10^{-5}$	1.543 $\pm$ 0.012	3.988 $\pm$ 0.003	3.495 $\pm$ 0.002	1.1773 $\pm$ 0.0004	0.59 $\pm$ 0.01

Table 2: Audio generation quality metrics for DiffWave fine-tuning with spectral regularization. FAD measures distributional similarity in audio embedding space (lower is better), UTMOS estimates perceptual naturalness (higher is better), PESQ measures perceptual speech quality (higher is better), MR-STFT measures multi-resolution spectral error (lower is better), and NDB evaluates distributional coverage and mode balance (lower is better). We report the average and standard deviation of five runs with different seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Duration	Batch	LR	CRes	Dropout	Augment
CIFAR-10 (cond/uncond)	0.5	16	$2{\times}10^{-4}$	–	–	–
AFHQ-64 (uncond)	0.5	16	$5{\times}10^{-5}$	1,2,2,2	0.25	0.15
FFHQ-64 (uncond)	–	16	$5{\times}10^{-5}$	1,2,2,2	0.05	0.15

Appendix

Table 3: EDM fine-tuning hyperparameters used for all experiments.

Panel of Figure 3 — Figure 3: Generated AFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude loss under different EDM formulations.

Panel of Figure 4 — Figure 4: Generated AFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude+phase loss under different EDM formulations.

Panel of Figure 5 — Figure 5: Generated AFHQ samples obtained by fine-tuning with the unweighted Haar wavelet loss under different EDM formulations.

Panel of Figure 6 — Figure 6: Generated AFHQ samples obtained by fine-tuning with the unweighted bi-orthogonal 1.3 wavelet loss under different EDM formulations.

Panel of Figure 7 — Figure 7: Generated FFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude loss under different EDM formulations.

Panel of Figure 8 — Figure 8: Generated FFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude+phase loss under different EDM formulations.

Panel of Figure 9 — Figure 9: Generated FFHQ samples obtained by fine-tuning with the unweighted Haar wavelet loss under different EDM formulations.

Panel of Figure 10 — Figure 10: Generated FFHQ samples obtained by fine-tuning with the unweighted bi-orthogonal 1.3 wavelet loss under different EDM formulations.

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,163 papers

Revisiting Spectral Representations in Generative Diffusion Models
Sep 8, 2026Yuehao Wang, Peihao Wang, Hanwen Jiang +3Diffusion Model Alignment Self-Supervised Learning
University of Texas at Austin, USA · 2Adobe Research, USA
Spectral Progressive Diffusion for Efficient Image and Video Generation
May 18, 2026Howard Xiao, Brian Chao, Lior Yariv +1Video Diffusion Models Diffusion Sampling
Stanford University
An Analysis of Regularization and Fokker-Planck Residuals in Diffusion Models for Image Generation
Apr 16, 2026Onno Niemann, Gonzalo Martínez Muñoz, Alberto Suárez GonzalezDiffusion-Based Image Generation Score-Based Generative Modeling
Universidad Autónoma de Madrid

Sep 8, 2026J/K move · Enter open · S save

Revisiting Spectral Representations in Generative Diffusion Models

Yuehao Wang, Peihao Wang, Hanwen Jiang, Ziyi Yang+2

University of Texas at Austin, USA · 2Adobe Research, USA

Diffusion Model Alignment Self-Supervised Learning Diffusion Models

Open paper AskPDF arXiv

Diffusion models have shown remarkable performance on diverse generation tasks. Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality, yet the mechanism driving this synergy remains insufficiently understood. In this paper, we investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels. On the diffusion side, samples (e.g., images, videos) are produced by reversing a stochastic noise-injection process specified by Gaussian kernels; on the spectral representation side, spectral embeddings emerge from contrasting positive and negative relations induced by random perturbation kernels. Motivated by this, we propose a self-supervised spectral representation alignment method to facilitate diffusion model training. In addition, we clarify how joint spectral learning can benefit diffusion training from a geometric perspective. Furthermore, we find that the optimization of the spectral alignment objective is in an equivalent form of diffusion score distillation in the representation space. Building on these findings, we integrate a spectral regularizer into diffusion training objectives to improve the performance of diffusion models on multiple datasets. Experiments across images and 3D point clouds show consistent gains in generation quality. Code is released at https://github.com/yuehaowang/spectral-reg-diffusion.

Abstract

Diffusion Models

Explore similar work

Revisiting Spectral Representations in Generative Diffusion Models

Spectral Progressive Diffusion for Efficient Image and Video Generation

An Analysis of Regularization and Fokker-Planck Residuals in Diffusion Models for Image Generation