Data-Efficient Classifier Pretraining Through Diffusion Models

Qinyu Li; Yee Whye Teh; Chris Holmes; Joel Jaskari

cs.LGOct 16, 2026

Data-Efficient Classifier Pretraining Through Diffusion Models

Authors: Qinyu Li, Yee Whye Teh, Chris Holmes, Joel Jaskari

Organizations: Department of Statistics, University of Oxford · Faculty of Medicine and Health Technology, Tampere University · Tampere Institute for Advanced Study, Tampere University

Abstract

Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks. Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time. Motivated by these traditional approaches, we propose DiffPT, a simple diffusion modeling approach to pretrain a given classifier backbone for limited domain-specific training data. We experiment on time-series data, standard natural-image benchmarks, and medical imaging datasets, and contrast the diffusion approach to a multitude of modern and traditional un/self-supervised methods. While contrastive pretraining performs better on CIFAR-10 and Tiny ImageNet in low-data regimes, DiffPT is particularly effective on medical imaging and time-series tasks, where it outperforms self-supervised baselines across a range of low-data regimes. Our results suggest that DiffPT is a simple and effective approach for data-efficient classifier pretraining on different modalities.

Figures & tables

Figure 1: Examples from the datasets used in our experiments. CIFAR-10 and Tiny ImageNet show images from four randomly chosen classes. PTB-XL shows examples from four diagnostic superclasses (truncated to only ECG leads I–III for visualisation purposes). WISDM shows tri-axial phone accelerometer signals from four activities. For EyePACS the top row images are randomly selected, while the bottom row images are selected to present biomarkers suitable for visualisation. The red boxes in the bottom row are zoomed-in regions containing discriminative biomarkers, highlighting their extreme localised nature.

	CIFAR-10			Tiny ImageNet
Method	1%	10%	100%	1%	10%	100%
AE	$44.2$	$75.5$	$92.4$	$5.3$	$27.7$	$57.8$
DAE	$44.4$	$75.9$	$92.3$	$5.8$	$27.5$	$57.5$
DINO	$42.9$	$76.0$	$\boldsymbol{93.1}$	$5.2$	$27.0$	$\boldsymbol{58.2}$
SimCLR	$\boldsymbol{54.3}$	$\boldsymbol{79.1}$	$91.2$	$\boldsymbol{8.8}$	$\boldsymbol{30.8}$	$54.9$
DiffPT (ours)	$49.7$	$78.7$	$93.0$	$7.6$	$29.5$	$\boldsymbol{58.2}$

Table 1: Accuracy (%) on CIFAR-10 and Tiny ImageNet across subset sizes. Bold indicates best, underline indicates second-best per column among pretraining methods. AE: auto-encoder, DAE: denoising auto-encoder.

	WISDM (Acc %)		PTB-XL (AUROC)
Method	50%	100%	5%	10%	100%
AE	$58.16$	$70.69$	$0.869$	$0.880$	$0.927$
DAE	$57.45$	$72.34$	$0.869$	$0.880$	$0.927$
DINO	$56.50$	$70.69$	$0.859$	$0.870$	$\boldsymbol{0.928}$
SimCLR	$55.08$	$69.98$	$0.862$	$0.871$	$0.926$
DiffPT (ours)	$\boldsymbol{59.81}$	$\boldsymbol{74.00}$	$\boldsymbol{0.872}$	$\boldsymbol{0.881}$	$0.927$

Table 2: WISDM accuracy (%) and PTB-XL macro-averaged AUROC across subset sizes. Bold indicates best, underline indicates second-best per column among pretraining methods. AE: auto-encoder, DAE: denoising auto-encoder.

	HAM10000		EyePACS
Method	10%	100%	1%	10%	100%
AE	$0.801$	$0.891$	$0.623$	$0.746$	$0.890$
DAE	$0.804$	$0.889$	$0.660$	$0.840$	$0.915$
DINO	$0.797$	$0.893$	$0.654$	$0.659$	$0.827$
SimCLR	$\boldsymbol{0.855}$	$\boldsymbol{0.925}$	$0.643$	$0.825$	$0.941$
DiffPT (ours)	$0.828$	$0.905$	$\boldsymbol{0.744}$	$\boldsymbol{0.854}$	$\boldsymbol{0.946}$

Table 3: HAM10000 (macro-averaged) and EyePACS AUROC across subset sizes. Bold indicates best, underline indicates second-best per column among pretraining methods. AE: auto-encoder, DAE: denoising auto-encoder.

Method	Extra params (M)	Peak mem. (GB)	Rel. mem.	Rel. throughput
From scratch	–	62.0	1.00 $\times$	1.00 $\times$
AE	2.8	69.0	1.11 $\times$	0.91 $\times$
DAE	2.8	69.1	1.12 $\times$	0.91 $\times$
DINO	7.9 $\dagger$	128.5	2.07 $\times$	0.41 $\times$
SimCLR	1.3	123.7	2.00 $\times$	0.51 $\times$
DiffPT (ours)	3.0	76.5	1.24 $\times$	0.89 $\times$

Table 4: Computational cost of a training step for each pretraining method on the HAM10000 dataset. The batch size is 128 and inputs are

256{\times}256

RGB images. Using 32-bit floating-point precision and a single NVIDIA GH200 GPU. All methods share the same ResNet-34 backbone (21.3M parameters), which is the only part transferred to the downstream classifier. Extra params are trainable parameters used only during pretraining (decoder, projector, or DINO heads). Memory is peak allocated GPU memory; throughput counts source images per second (excluding CPU data loading but including method specific augmentation). Relative values are with respect to the classifier baseline trained from scratch. AE: auto-encoder, DAE: denoising auto-encoder.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Input	Training augmentation	Evaluation
CIFAR-10	$32\times 32$	Random crop (4 px padding); horizontal flip ( $p{=}0.5$ )	–
Tiny ImageNet	$64\times 64$	Random crop (4 px padding); horizontal flip ( $p{=}0.5$ )	–
EyePACS	$256\times 256$	Random flip (none / horizontal / vertical / both, uniformly); rotation $\in[-180^{\circ},180^{\circ}]$ ; symmetric padding of up to 5% per side then random crop back to $256\times 256$ ; scaling $\in[0.9,1.1]$	–
HAM10000	$256\times 256$	Resize from $600\times 450$ , then as EyePACS	Resize
PTB-XL	$12\times 250$ (2.5 s @ 100 Hz)	Random window from the 10 s record	Sliding windows, 50% overlap; max over windows
WISDM	$12\times 200$ (10 s @ 20 Hz)	Random window within the record’s valid length	Sliding windows, 50% overlap; mean over windows

Appendix

Table 5: Base augmentation.

Dataset	View size	Crop scale	$s$ / $p_{g}$	Global size	Global scale	Local size	Local scale	$N_{\ell}$ , $s$ / $p_{g}$
	SimCLR			DINO
CIFAR-10	32	$[0.2,1]$	1.0 / 0.2	32	$[0.5,1]$	24	$[0.2,0.6]$	6, 1.0 / 0.2
Tiny ImageNet	64	$[0.2,1]$	1.0 / 0.2	64	$[0.35,1]$	32	$[0.1,0.4]$	6, 1.0 / 0.2
EyePACS	256	$[0.5,1]$	0.3 / 0	224	$[0.6,1]$	96	$[0.3,0.6]$	4, 0.3 / 0
HAM10000 †	256	$[0.5,1]$	0.5 / 0	224	$[0.6,1]$	96	$[0.3,0.6]$	4, 0.5 / 0

Appendix

Table 6: SimCLR and DINO view-generation settings for the image datasets. Crop scales are fractions of the image area passed to RandomResizedCrop;

s

is the colour-jitter strength and

p_{g}

the grayscale probability. (see text).

	PTB-XL	WISDM
Sampling rate	100 Hz	20 Hz
SimCLR view length	250 (2.5 s)	200 (10 s)
DINO global crops	$2\times 250$ (2.5 s)	$2\times 200$ (10 s)
DINO local crops	$6\times 125$ (1.25 s)	$6\times 100$ (5 s)
Gaussian noise $\sigma$	0.15	0.10
Time-mask width (max), global / local	25 / 12	20 / 10

Appendix

Table 7: SimCLR and DINO view generation for the time-series datasets. Window lengths are in samples, with durations in parentheses.

Data-Efficient Learning

Data-efficient learning develops methods that achieve accurate, robust models with fewer labeled examples, training samples, or interactions.

See all 233 papers

Self-Improving Diffusion Classifiers with Minority Preference Optimization
Jul 4, 2026Hyunsoo Kim, Jungmyung Wi, Soobin Um +2Diffusion Model Alignment Long-Tailed Classification
1Korea University · 2Kook Min University · 3Kyung Hee University
Diffusion-Based Data Augmentation for Image Recognition: A Systematic Analysis and Evaluation
Mar 9, 2026Zekun Li, Yinghuan Shi, Yang Gao +1Diffusion-Based Data Augmentation Data Augmentation
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China. · School of Computing and Data Science, The University of Hong Kong, Hong Kong SAR, China.
MiniDog: Dog-Breed Text-to-image Flow Matching from Pretraining to Fine-Tuning on 4 Consumer GPUs
Oct 13, 2026Reyhaneh Esmailizadeh, Xingjian Leng, Zhanhao Liang +1T2I Generation T2I Diffusion Models

Jul 4, 2026J/K move · Enter open · S save

Self-Improving Diffusion Classifiers with Minority Preference Optimization

Hyunsoo Kim, Jungmyung Wi, Soobin Um, Donghyun Kim+1

1Korea University · 2Kook Min University · 3Kyung Hee University

Diffusion Model Alignment Long-Tailed Classification Diffusion Model Fine-Tuning RL for Diffusion Models Zero-Shot Image Classification

Open paper AskPDF arXiv

Prior studies have demonstrated that diffusion classifiers achieve robust zero-shot classification performance. However, their effectiveness is strongly tied to the pretraining data distribution: they perform well in majority, high-density regions of the data manifold, but are significantly less accurate in minority, low-density regions. Although prior works on minority sampling have focused on generating more minority-like images, what minority sampling fundamentally enables beyond generation remains underexplored. In this paper, we reveal a direct relationship between minority sampling in generation and the perception capability of diffusion classifiers. Specifically, we show that enhancing minority sampling broadens the coverage of underrepresented regions on the data manifold, thereby improving diffusion-based recognition. To exploit this connection, we propose Self-Improving Diffusion Classifiers with Minority Preference Optimization (MiPO), which fine-tunes a pretrained diffusion model using minority preference rewards. Using only arbitrary caption data, MiPO generates candidate samples, rewards those that better cover minority regions, and optimizes the model with LoRA and Group Relative Policy Optimization, without additional image data, external foundation models, or external reward models. This enables stable, prompt-adaptive minority sampling and translates low-density generative coverage into improved zero-shot diffusion classification. To sum up, we show that diffusion classifier perception is biased toward majority regions, demonstrate that this bias can be alleviated through minority preference optimization, and evaluate MiPO on five standard datasets.

Abstract

Data-Efficient Learning

Explore similar work

Self-Improving Diffusion Classifiers with Minority Preference Optimization

Diffusion-Based Data Augmentation for Image Recognition: A Systematic Analysis and Evaluation

MiniDog: Dog-Breed Text-to-image Flow Matching from Pretraining to Fine-Tuning on 4 Consumer GPUs