Sparsely Supervised Diffusion

Wenshuai Zhao; Zhiyuan Li; Yi Zhao; Mohammad Hassan Vali; Martin Trapp; Joni Pajarinen; Juho Kannala; Arno Solin

cs.LGFeb 2, 2026

Sparsely Supervised Diffusion

Authors: Wenshuai Zhao, Zhiyuan Li, Yi Zhao, Mohammad Hassan Vali, Martin Trapp, Joni Pajarinen, Juho Kannala, Arno Solin

Organizations: Department of Computer Science, Aalto University, Finland · Department of Electrical Engineering and Automation, Aalto University, Finland.

Abstract

Diffusion models have shown remarkable success across a wide range of generative tasks. However, they often suffer from spatially inconsistent generation, arguably due to excessive correlations learned by the model. This can produce samples that are locally plausible but globally inconsistent. We propose a principled method to mitigate this issue. Our method, sparsely supervised diffusion (SSD), is a simple yet effective masking framework that can be implemented with only a few lines of code. We analytically show that SSD fundamentally alters diffusion model training by modifying the spectrum of the data covariance and that it suppresses correlations in the data covariance matrix. Experiments show that our method more accurately approximates the underlying population score function, reduces memorization on small datasets, and promotes the use of essential contextual information during generation. Moreover, even when up to 98% of pixels are masked, it achieves competitive FID scores across a range of datasets and, importantly, avoids the training instability commonly observed on small datasets.

Figures & tables

Panel of Figure 1 — Figure 1: Masking up to 98% of pixels improves training . SSD learns exclusively from unmasked pixels while being encouraged to generalize over the masked regions.

Panel of Figure 2 — Figure 2: Overview of the proposed method in comparison with standard flow matching (FM).

Panel of Figure 3 — Figure 3: Effect of masking during training. When $\eta=0$ , the model reduces to the baseline flow matching [ 49 ] . Across three datasets, SSD with up to 80% masked pixels can still achieve comparable performance as the baseline. Notably, the baseline in Fig. 3(b) eventually explodes while SSD remains stable. The shaded regions indicate the 95% confidence interval over four runs.

Panel of Figure 4 — Figure 4: Masking leads to more realistic generations. Non-curated samples from models trained on CelebA. Sample #1 shows finer neck wrinkles at $\eta=0.8$ . In sample #6, an unrealistic bulge on the head at $\eta=0$ gradually disappears as $\eta$ increases. A similar artifact in sample #9 of the baseline is also removed by masking.

Dataset	Baseline (FM/DiT)	SSD (ours)
CIFAR10	2.28 (0.04)	2.24 (0.05)
LSUN	1.40 (0.04)	1.39 (0.01)
CelebA-50K	2.23 (0.23)	1.85 (0.14)
CelebA-10K	13.72 (–)	5.10 (–)
ImageNet	4.98 (–)	4.83 (–)

Table 1: FID (

\sigma

) (

\downarrow

) on different datasets.

Method	Baseline ( $\eta=0.0$ )	Ours ( $\eta=0.98$ )
FID ( $\downarrow$ )	13.72	5.10
$\ell_{2}$ distance ( $\uparrow$ )	42.32	46.02

Table 2: Masking mitigates memorization. FID (

\downarrow

) and average

\ell_{2}

distance (

\uparrow

) to the nearest training samples over 10k generated images on CelebA-10K. Larger

\ell_{2}

indicates less memorization.

Figure 5: Average

\ell_{2}

distance to the nearest training samples over training iterations on CelebA-10K. Larger values indicate less memorization.

Panel of Figure 6 — Figure 6: Masking helps capturing essential contextual information. (a) Distribution of $\ell_{1}$ norm of gradients computed per image and, (b) distribution of gradient magnitude per pixel over 10k images for CIFAR10 dataset.

Panel of Figure 7 — Figure 7: Demonstration of improved spatial consistency. After training for 5k epochs on a binary image dataset of triangles ( ) and squares ( ) ( 1st row ), our model generates images with less scattered dots ( 3rd row ), i.e. enhanced structural integrity, compared to the standard diffusion model without masking ( 2nd row ). Quantitative results are reported in Table 3 .

Method	No. of scatters ( $\downarrow$ )	No. of inconsistent images ( $\downarrow$ )
Baseline	11,208	3,899
Ours	1,014	792

Table 3: Masking improves spatial consistency . The table shows the number of scattered pixel clusters and number of images containing such clusters over 5k generated images for the illustrative dataset shown in Fig. 7 .

Error type ( $\downarrow$ )	Baseline ( $\eta=0$ )	SSD ( $\eta=0.8$ )
Six fingers	26	8
Four fingers	9	14
Total	35	22

Table 4: Comparison of finger-count errors among 1,000 generated 11kHands samples.

Panel of Figure 8 — Figure 8: Masking improves estimating the population score function . Visualization of population scores (ground truth) and empirical scores estimated with $\eta=0$ (baseline) and $\eta=0.5$ (ours), along with a comparison of score estimation errors (right two figures) such that the error is color coded by 0 10.63. The baseline model only accurately estimates scores near observed data points (shown as ), whereas the proposed masking enables more accurate estimation over broader regions (squared areas) and yields lower overall errors.

Metric	Baseline	W/ Masking (ours)
Mean error ( $\downarrow$ )	3.65	2.78
Max error ( $\downarrow$ )	10.63	6.18

Table 5: Comparison of empirical score estimation errors on a 2D Gaussian data distribution. Lower (

\downarrow

) is better.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	CelebA-50K	CIFAR10	LSUN Bedroom
Training Configuration
Epochs	2000	3000	2000
Effective Batch Size	128	256	1024
Learning Rate	0.0001	0.0001	0.0001
Optimizer Betas	[0.9, 0.95]	[0.9, 0.95]	[0.9, 0.95]
Use EMA	True	True	True

Appendix

Table 6: Key hyperparameters for experiments on three datasets

Method	CelebA-10K FID ( $\downarrow$ )	CIFAR10 FID ( $\downarrow$ )
MDT	24.72	–
Dropout ( $p=0.30$ )	13.72	2.28
Dropout ( $p=0.98$ )	13.38	9.37
SSD (ours, $\eta=0.98$ )	5.10	2.20

Appendix

Table 7: Comparison with dropout and MDT baselines. Both dropout and MDT fail to prevent divergence during training. All methods on CelebA-10K are trained for 2000 epochs.

Training data	Training steps	Baseline	SSD ( $\eta=0.8$ )	SSD ( $\eta=0.98$ )
ImageNet-100K	200K	25.99	18.45	15.58
Full ImageNet	180K	4.98	–	4.83

Appendix

Table 8: ImageNet FID at CFG=1.5 using DiT-XL/2 at 256

\times

256.

Masking strategy	Mask ratio $\eta$	Final FID ( $\downarrow$ )
None (baseline)	0	13.72
Element-wise	0.5	13.29
Element-wise	0.95	6.36
Element-wise	0.98	5.10
Patch-wise (8 $\times$ 8)	0.6	7.61
Patch-wise (8 $\times$ 8)	0.9	5.14

Appendix

Table 9: Masking-granularity ablation on CelebA-10K. Results are reported at 3,500 epochs.

Panel of Figure 9 — Figure 9: Comparison of generated samples on the 11kHands dataset. Each panel presents 200 samples generated using the same random seed. Thin red outlines highlight samples exhibiting spatial inconsistencies, specifically an incorrect number of fingers. (a) The baseline model frequently generates hands with six fingers, (b) whereas SSD mitigates this issue but occasionally produces hands with only four fingers. The full evaluation results are reported in Table 4 .

Panel of Figure 10 — Figure 10: Sample paths from same initial noises with models trained with different $\eta$ across three datasets.

Figure 11: More samples generated by models trained on CelebA-50K with different mask ratios.

Figure 12: More samples generated by models trained on CIFAR10 with different mask ratios.

Figure 13: More samples generated by models trained on LSUN Bedroom with different mask ratios.

Spatially Grounded Image Generation

See all 8 papers

Skipping the Zeros in Diffusion Models for Sparse Data Generation
May 3, 2026Phil Sidney Ostheimer, Mayank Nagda, Andriy Balinskyy +6Synthetic Data Generation Diffusion Model Acceleration
1RPTU University Kaiserslautern-Landau · 2Heidelberg Univer · University of California, Irvine.
SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
Feb 13, 2026Jintao Zhang, Kai Jiang, Chendong Xiang +5Sparse Attention Diffusion Model Acceleration
Tsinghua University · UC Berkeley
Guided Star-Shaped Masked Diffusion
Oct 9, 2025Viacheslav Meshchaninov, Egor Shibaev, Artem Makoian +5Diffusion Sampling Diffusion Language Models
Constructor University · Higher School of Economics · T-Tech +1

May 3, 2026J/K move · Enter open · S save

Skipping the Zeros in Diffusion Models for Sparse Data Generation

Phil Sidney Ostheimer, Mayank Nagda, Andriy Balinskyy, Gabriel Vicente Rodrigues+5

1RPTU University Kaiserslautern-Landau · 2Heidelberg Univer · University of California, Irvine.

Synthetic Data Generation Diffusion Model Acceleration Diffusion Models

Open paper AskPDF arXiv

Diffusion models (DMs) excel on dense continuous data, but are not designed for sparse continuous data. They do not model exact zeros that represent the deliberate absence of a signal. As a result, they erase sparsity patterns and perform unnecessary computation on mostly zero entries. With Sparsity-Exploiting Diffusion (SED), we model only non-zero values, preserving sparsity. SED delivers computational savings while maintaining or improving generation quality by skipping zeros during training and inference. Across physics and biology benchmarks, SED matches or surpasses conventional DMs and domain-specific baselines, while vision experiments provide intuitive insights into the limitations of dense DMs and the benefits of SED.

Abstract

Spatially Grounded Image Generation

Explore similar work

Skipping the Zeros in Diffusion Models for Sparse Data Generation

SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning

Guided Star-Shaped Masked Diffusion