Authors: Satish Chandran, Nicolas Roque dos Santos, Yunshu Wu, Greg Ver Steeg, Evangelos Papalexakis
Organizations: Department of Mathematics, University of California Riverside, Riverside, California, USA · Department of Computer Science, University of California Riverside, Riverside, California, USA
Diffusion models are typically trained using pointwise reconstruction objectives that are agnostic to the spectral and multi-scale structure of natural signals. We propose a loss-level spectral regularization framework that augments standard diffusion training with differentiable Fourier- and wavelet-domain losses, without modifying the diffusion process, model architecture, or sampling procedure. The proposed regularizers act as soft inductive biases that encourage appropriate frequency balance and coherent multi-scale structure in generated samples. Our approach is compatible with DDPM, DDIM, and EDM formulations and introduces negligible computational overhead. Experiments on image and audio generation demonstrate consistent improvements in sample quality, with the largest gains observed on higher-resolution, unconditional datasets where fine-scale structure is most challenging to model.
Figures & tables
Figure 1: Checkerboard toy experiment. Figures (a) to (c) show the ground-truth pattern, a sample from a model trained without spectral regularization, and a sample from a model trained with the proposed amplitude-and-phase loss.
Figure 2: Radially averaged power spectra (log scale) for the ground truth, baseline DDPM with MSE loss, and DDPM with our amplitude+phase spectral loss.
Dataset
EDM variant
EDM Sampling Steps
Conditioning
Regularizer
Weighted FID
Unweighted FID
EDM FID
CIFAR
VE
18
Cond
Amp
1.82 ± 0.01
1.82 ± 0.01
1.81 ± 0.01
Amp Phase
1.82 ± 0.01
1.82 ± 0.01
Haar
1.81 ± 0.01
1.81 ± 0.02
Bior13
1.82 ± 0.01
1.81 ± 0.02
CIFAR
VP
18
Cond
Amp
1.84 ± 0.02
1.84 ± 0.02
1.84 ± 0.02
Amp Phase
1.84 ± 0.02
1.84 ± 0.02
Table 1: Fréchet Inception Distance (FID) scores for different spectral regularizers and EDM variants. All models are fine-tuned from pretrained EDM baselines. Lower values indicate better generative performance.
Method
λ
FAD ↓
UTMOS ↑
PESQ ↑
MR-STFT ↓
NDB ↓
Diffwave
–
1.994 ± 0.008
3.941 ± 0.005
3.440 ± 0.002
1.217 ± 0.001
0.63 ± 0.02
Amp
10−4
1.462 ± 0.006
3.953 ± 0.009
3.477 ± 0.003
1.1802 ± 0.0005
0.65 ± 0.03
10−5
1.609 ± 0.007
3.953 ± 0.009
3.476 ± 0.001
1.1930 ± 0.0004
0.63 ± 0.02
10−6
1.775 ± 0.008
3.952 ± 0.006
3.491 ± 0.003
1.1958 ± 0.0005
0.64 ± 0.03
Amp+Phase
10−4
1.694 ± 0.009
3.969 ± 0.008
3.516 ± 0.002
1.1896 ± 0.0003
0.66 ± 0.02
10−5
1.543 ± 0.012
3.988 ± 0.003
3.495 ± 0.002
1.1773 ± 0.0004
0.59 ± 0.01
Table 2: Audio generation quality metrics for DiffWave fine-tuning with spectral regularization. FAD measures distributional similarity in audio embedding space (lower is better), UTMOS estimates perceptual naturalness (higher is better), PESQ measures perceptual speech quality (higher is better), MR-STFT measures multi-resolution spectral error (lower is better), and NDB evaluates distributional coverage and mode balance (lower is better). We report the average and standard deviation of five runs with different seeds.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Duration
Batch
LR
CRes
Dropout
Augment
CIFAR-10 (cond/uncond)
0.5
16
2×10−4
–
–
–
AFHQ-64 (uncond)
0.5
16
5×10−5
1,2,2,2
0.25
0.15
FFHQ-64 (uncond)
–
16
5×10−5
1,2,2,2
0.05
0.15
Appendix
Table 3: EDM fine-tuning hyperparameters used for all experiments.
Figure 3: Generated AFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude loss under different EDM formulations.
Figure 4: Generated AFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude+phase loss under different EDM formulations.
Figure 5: Generated AFHQ samples obtained by fine-tuning with the unweighted Haar wavelet loss under different EDM formulations.
Figure 6: Generated AFHQ samples obtained by fine-tuning with the unweighted bi-orthogonal 1.3 wavelet loss under different EDM formulations.
Figure 7: Generated FFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude loss under different EDM formulations.
Figure 8: Generated FFHQ samples obtained by fine-tuning with the unweighted Fourier amplitude+phase loss under different EDM formulations.
Figure 9: Generated FFHQ samples obtained by fine-tuning with the unweighted Haar wavelet loss under different EDM formulations.
Figure 10: Generated FFHQ samples obtained by fine-tuning with the unweighted bi-orthogonal 1.3 wavelet loss under different EDM formulations.
Diffusion models have shown remarkable performance on diverse generation tasks. Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality, yet the mechanism driving this synergy remains insufficiently understood. In this paper, we investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels. On the diffusion side, samples (e.g., images, videos) are produced by reversing a stochastic noise-injection process specified by Gaussian kernels; on the spectral representation side, spectral embeddings emerge from contrasting positive and negative relations induced by random perturbation kernels. Motivated by this, we propose a self-supervised spectral representation alignment method to facilitate diffusion model training. In addition, we clarify how joint spectral learning can benefit diffusion training from a geometric perspective. Furthermore, we find that the optimization of the spectral alignment objective is in an equivalent form of diffusion score distillation in the representation space. Building on these findings, we integrate a spectral regularizer into diffusion training objectives to improve the performance of diffusion models on multiple datasets. Experiments across images and 3D point clouds show consistent gains in generation quality. Code is released at https://github.com/yuehaowang/spectral-reg-diffusion.
Yuehao Wang, Peihao Wang, Hanwen Jiang +3
University of Texas at Austin, USA · 2Adobe Research, USA
Diffusion models have been shown to implicitly generate visual content autoregressively in the frequency domain, where low-frequency components are generated earlier in the denoising process while high-frequency details emerge only in later timesteps. This structure offers a natural opportunity for efficient generation, as high-resolution computation on noise-dominated frequencies is largely redundant. We propose Spectral Progressive Diffusion, a general framework that progressively grows resolution along the denoising trajectory of pretrained diffusion models. To this end, we develop a spectral noise expansion mechanism and derive an optimal resolution schedule from the model's power spectrum. Our framework supports training-free acceleration and a novel fine-tuning recipe that further improves efficiency and quality. We demonstrate significant speedups on state-of-the-art pretrained image and video generation models while preserving visual quality.
Recent work has shown that diffusion models trained with the denoising score matching (DSM) objective often violate the Fokker--Planck (FP) equation that governs the evolution of the true data density. Directly penalizing these deviations in the objective function reduces their magnitude but introduces a significant computational overhead. It is also observed that enforcing strict adherence to the FP equation does not necessarily lead to improvements in the quality of the generated samples, as often the best results are obtained with weaker FP regularization. In this paper, we investigate whether simpler penalty terms can provide similar benefits. We empirically analyze several lightweight regularizers, study their effect on FP residuals and generation quality, and show that the benefits of FP regularization are available at substantially lower computational cost. Our code is available at https://github.com/OnnoNiemann/fp_diffusion_analysis.
Onno Niemann, Gonzalo Martínez Muñoz, Alberto Suárez Gonzalez