EigDiff: Eigensample Diffusion - or how to significantly reduce training dataset size with marginal performance variation?

Andrei Jarca; Vlad Hondru; Florinel-Alin Croitoru; Radu Tudor Ionescu

cs.CVOct 14, 2026

EigDiff: Eigensample Diffusion - or how to significantly reduce training dataset size with marginal performance variation?

Authors: Andrei Jarca, Vlad Hondru, Florinel-Alin Croitoru, Radu Tudor Ionescu

Organizations: University of Bucharest, Romania

Abstract

Diffusion models are notoriously expensive to train, requiring large-scale datasets to learn how to gradually reverse the noise from images over multiple small denoising steps. Current approaches focus on distilling the number of diffusion steps, neglecting potential redundancies of the training data. In this paper, we explore the potential of harnessing eigensamples to train diffusion models on significantly smaller datasets, while preserving model performance. To mitigate the sensitivity to affine transformations of eigenimages, we harness the latent space of a pre-trained CLIP model. We next compute latent eigenvectors by performing principal component analysis (PCA) in CLIP space. Further, we employ eigenvectors as reference points to guide the selection of a representative training set comprising comprising both exemplars (images whose embeddings are closest to eigenvectors) and diverse images. Our comprehensive experiments with different diffusion models applied on multiple datasets indicate that our approach (EigDiff) can reduce training data by up to 14x and accelerate subset selection by up to 1607x, with marginal performance differences.

Figures & tables

Figure 1: Overview of EigDiff (eigen-guided data selection). Preference image pairs are CLIP-encoded into embeddings, on which PCA yields the principal (eigen) components. Each pair is scored by its projection onto these directions ( $\mathbf{w}\cdot\mathbf{x}^{\top}$ for SFT, $\mathbf{w}\cdot(\mathbf{x}-\mathbf{y})^{\top}$ for DPO). Per component, Stage 1 deterministically selects the exemplars (highest score). The figure illustrates the near to far spread. Stage 2 stratifies the full remaining score range via diversity quantiles. Best viewed in color.

Dataset	Backbone	Full-scale	Eigen (ours)	$n_{1}$	$\delta$	$B$	$(\tau_{1},\tau_{2})$
Dataset	Backbone	Steps	Steps	$n_{1}$	$\delta$	$B$	$(\tau_{1},\tau_{2})$
Pick-a-Pic	SD1.5	$50{,}000$	$3{,}584$	$1{,}024$	$6$	$4$	$(0.6,0.8)$
Pick-a-Pic	PixArt- $\alpha$	$25{,}000$	$1{,}792$	$1{,}024$	$6$	$4$	$(0.6,0.8)$
HPDv2	SD1.5	$50{,}000$	$7{,}168$	$2{,}048$	$6$	$4$	$(0.6,0.8)$
HPDv2	PixArt- $\alpha$	$25{,}000$	$3{,}584$	$2{,}048$	$6$	$4$	$(0.6,0.8)$
Rapidata	SD1.5	$22{,}515$	$3{,}584$	$1{,}024$	$6$	$4$	$(0.4,0.6)$

Table 1: Per-dataset training steps for the full-scale baseline and our eigen-based selection method. Additionally, we include the hyperparameters of our method.

Objective	Dataset	SD1.5			PixArt- $\alpha$
Objective	Dataset	Full-scale	EigDiff (ours)	Speedup	Full-scale	EigDiff (ours)	Speedup
DPO	Pick-a-Pic	407	29.175	13.95 $\times$	184.26	13.208	13.95 $\times$
	HPDv2	407	58.35	6.98 $\times$	184.26	26.416	6.98 $\times$
	Rapidata	183.27	29.175	6.28 $\times$	82.97	13.207	6.28 $\times$
SFT	Pick-a-Pic	183.13	13.122	13.95 $\times$	97.20	6.97	13.95 $\times$
	HPDv2	183.13	26.244	6.98 $\times$	97.20	13.94	6.98 $\times$

Table 2: Total fine-tuning time (in minutes) for full-scale vs. distilled datasets. Experiments are performed with an equivalent number of epochs, on one Intel i9-14900F CPU with 64GB RAM and an NVIDIA RTX 5090 GPU. Less time is better.

Objective	Dataset	K-Center	Facility Loc.	K-Means $++$	SemDeDup	EigDiff (ours)	Speedup
DPO	Pick-a-Pic	$358.1_{\pm 0.3}$	$1525.5_{\pm 18.0}$	$360.6_{\pm 0.7}$	$5.27_{\pm 0.24}$	$1.36_{\pm 0.02}$	3.9–1123 $\times$
	HPDv2	$708.2_{\pm 6.0}$	$2079.8_{\pm 43.0}$	$715.9_{\pm 5.7}$	$4.48_{\pm 0.21}$	$1.46_{\pm 0.03}$	3.1–1424 $\times$
	Rapidata	$106.6_{\pm 1.6}$	$468.5_{\pm 0.1}$	$110.8_{\pm 0.8}$	$2.06_{\pm 0.29}$	$0.46_{\pm 0.01}$	4.4–1011 $\times$
SFT	Pick-a-Pic	$358.1_{\pm 0.3}$	$1525.5_{\pm 18.0}$	$360.6_{\pm 0.7}$	$5.27_{\pm 0.24}$	$1.23_{\pm 0.04}$	4.3–1244 $\times$
	HPDv2	$708.2_{\pm 6.0}$	$2079.8_{\pm 43.0}$	$715.9_{\pm 5.7}$	$4.48_{\pm 0.21}$	$1.30_{\pm 0.02}$	3.4–1598 $\times$
	Rapidata	$106.6_{\pm 1.6}$	$468.5_{\pm 0.1}$	$110.8_{\pm 0.8}$	$2.06_{\pm 0.29}$	$0.38_{\pm 0.02}$	5.4–1230 $\times$

Table 3: Average selection time comparison (in seconds) over three seeds with standard deviation. Selections run on their respective pre-computed features, thus the timing is identical regardless of the model. We showcase the speedup interval between EigDiff, and the slowest/fastest method (SemDeDup to Facility Locator). We used an Intel i9-14900F CPU with 64GB of RAM for all selections, each approach being measured with 4 threads. Less time is better.

Figure 2: Comparing the experiments with Stable Diffusion 1.5 on Pick-a-Pic, fine-tuned on 50000 steps, on the best performing full-scale seed.

Model	Strategy	Pick-a-Pic	HPDv2	Rapidata
SD1.5	Pre-trained	$20.536$	$20.779$	$20.373$
	K-Means $++$ ( Arthur and Vassilvitskii, 2007 )	$20.691_{\pm 0.004}$	$\underline{20.989}_{\pm 0.014}$	$20.390_{\pm 0.013}$
	K-Center ( Gonzalez, 1985 )	$20.645_{\pm 0.008}$	$20.571_{\pm 0.030}$	$20.394_{\pm 0.009}$
	Facility Location ( Mirzasoleiman et al., 2015 )	$\underline{20.702}_{\pm 0.010}$	$20.983_{\pm 0.065}$	$20.414_{\pm 0.028}$
	SemDeDup ( Abbas et al., 2023 )	$20.614_{\pm 0.013}$	$20.898_{\pm 0.029}$	$20.398_{\pm 0.005}$
	Random selection	$20.669_{\pm 0.013}$	$20.977_{\pm 0.016}$	$20.432_{\pm 0.022}$

Table 4: PickScore results (higher is better) using SD1.5 and PixArt-

\alpha

under the DPO regime, on the Pick-a-Pic, HPDv2, and Rapidata datasets, comparing our approach with the baselines. We report the average PickScore over three runs. For model and dataset, we highlight the best and second-best scores among coreset selection / dataset distillation strategies in bold and underline, respectively. The pre-trained and full-scale experiments are reference points, thus not ranked.

Model	Strategy	Pick-a-Pic	HPDv2	Rapidata	ROCO
SD1.5	Pre-trained	$20.536$	$20.779$	$20.373$	$15.4887$
	K-Means $++$ ( Arthur and Vassilvitskii, 2007 )	$20.808_{\pm 0.005}$	$20.777_{\pm 0.009}$	$20.561_{\pm 0.013}$	$21.031_{\pm 0.124}$
	K-Center ( Gonzalez, 1985 )	$20.778_{\pm 0.026}$	$20.721_{\pm 0.019}$	$20.529_{\pm 0.006}$	$21.098_{\pm 0.225}$
	Fac. Loc. ( Mirzasoleiman et al., 2015 )	$\underline{20.820}_{\pm 0.014}$	$\underline{20.791}_{\pm 0.023}$	$\underline{20.579}_{\pm 0.019}$	$20.814_{\pm 0.368}$
	SemDeDup ( Abbas et al., 2023 )	$20.718_{\pm 0.005}$	$20.786_{\pm 0.020}$	$20.521_{\pm 0.042}$	$20.375_{\pm 0.061}$
	Random selection	$20.811_{\pm 0.011}$	$20.773_{\pm 0.002}$	$20.566_{\pm 0.016}$	$\underline{21.184}_{\pm 0.186}$

Table 5: PickScore/BiomedCLIP results (higher is better) using SD1.5 and PixArt-

\alpha

under the SFT regime, on the Pick-a-Pic, HPDv2, Rapidata and ROCO datasets, comparing our approach with the baselines. We report the average PickScore over three runs. For model and dataset, we highlight the best and second-best scores among coreset selection / dataset distillation strategies in bold and underline, respectively. The pre-trained and full-scale experiments are reference points, thus not ranked.

Stage 1	Stage 2	Variance-based	PickScore
Stage 1	Stage 2	selection	PickScore
✓	✓	✓	$\textbf{20.705}_{\pm 0.023}$
✓	✓	✗	$20.678_{\pm 0.023}$
✓	✗	✓	$20.703_{\pm 0.022}$
✗	✓	✓	$20.685_{\pm 0.013}$
✗	✗	✗	$20.669_{\pm 0.013}$

Table 6: Ablation of our sample selection strategy on the Pick-a-Pic dataset. This analysis demonstrates the relative contribution of each stage, as well as the effectiveness of the variance-based selection method introduced in Eq. ( 3 ).

$B$	PickScore
1	$20.675{\pm 0.003}$
2	$20.699\pm 0.013$
3	$20.672\pm 0.009$
4	$\textbf{20.705}\pm 0.023$
5	$20.689\pm 0.003$
10	$20.671\pm 0.023$

Table 7: Ablation on the number of bins

B

employed in Eq. ( 6 ). Results are reported on the Pick-a-Pic dataset using SD1.5.

Ratio	PickScore
2	$20.611\pm 0.01$
3	$20.638\pm 0.018$
4	$20.639\pm 0.018$
5	$20.664\pm 0.012$
6	$\textbf{20.705}\pm 0.023$

Table 8: Ablation on the sample ratio

\delta=\frac{n_{2}}{n_{1}}

. Results are reported on the Pick-a-Pic dataset using SD1.5.

Thresholds $(\tau_{1},\tau_{2})$	PickScore
(0.2, 0.4)	$20.695\pm 0.014$
(0.4, 0.6)	$20.677\pm 0.014$
(0.6, 0.8)	$\textbf{20.705}\pm 0.023$
(0.8, 0.95)	$20.685\pm 0.03$

Table 9: Ablation on PCA thresholds for the two stages of selection. Results are reported on the Pick-a-Pic dataset using SD1.5.

$n_{1}$	PickScore
512	$20.627\pm 0.013$
1024	$20.705\pm 0.023$
2048	$20.777\pm 0.026$
4096	$\textbf{20.864}\pm 0.013$

Table 10: Ablation on Stage 1 budget. Results are reported on the Pick-a-Pic dataset using SD1.5.

Figure 3: Qualitative results with the DPO regime across all datasets with SD1.5 and PixArt- $\alpha$ . Best seen in color.

Figure 4: Qualitative results with the SFT regime across all datasets with SD1.5 and PixArt- $\alpha$ . Best seen in color.

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,176 papers

Geometry-Aware Dataset Condensation for Diffusion Model Training
Jun 4, 2026Xiao Cui, Yulei Qin, Mo Zhu +3Dataset Distillation Dataset Condensation
University of Science and Technology of China · Independent Researcher · Zhejiang University +1
Learnability-Guided Diffusion for Dataset Distillation
Apr 1, 2026Jeffrey A. Chan-Santiago, Mubarak ShahDataset Distillation Diffusion-Based Dataset Distillation
Institute of Artificial Intelligence, University of Central Florida
DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models
May 5, 2026Qichao Wang, Yunhong Lu, Hengyuan Cao +2Diffusion-Based Dataset Distillation Diffusion Model Guidance
Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences

Jun 4, 2026J/K move · Enter open · S save

Geometry-Aware Dataset Condensation for Diffusion Model Training

Xiao Cui, Yulei Qin, Mo Zhu, Wengang Zhou+2

University of Science and Technology of China · Independent Researcher · Zhejiang University+1

Dataset Distillation Dataset Condensation Diffusion Models

Open paper AskPDF arXiv

Dataset condensation aims to construct compact datasets from real data via synthesis or selection. However, existing approaches are ill-suited for diffusion model training: synthetic data generation often yields low-fidelity samples unsuitable for authentic modeling, while real subset selection typically fails to preserve the distributional geometry required by diffusion likelihood objectives. To address this, we propose to reformulate real subset selection as a geometry-aware distribution alignment problem. By incorporating one-sided partial optimal transport, our method selectively aligns a compact subset with the full data distribution while allowing unmatched mass in low-density regions, ensuring the preserved geometric structure necessary for effective diffusion model training. To further ensure distributional fidelity, we complement geometric alignment with lightweight feature-statistics and semantic consistency regularization. An efficient two-stage discrete optimization strategy is proposed to achieve this alignment objective. Extensive experiments across diffusion variants, subset sizes, image resolutions, and training rounds show that our method achieves superior fidelity and distributional coverage in diffusion model training. Codes are available at https://github.com/2018cx/GADC.

Abstract

Diffusion Models

Explore similar work

Geometry-Aware Dataset Condensation for Diffusion Model Training

Learnability-Guided Diffusion for Dataset Distillation

DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models