Data Warmup: Complexity-Aware Curricula for Efficient Diffusion Training

Jinhong Lin; Pan Wang; Zitong Zhan; Lin Zhang; Pedro Morgado

cs.LGApr 8, 2026

Data Warmup: Complexity-Aware Curricula for Efficient Diffusion Training

Authors: Jinhong Lin, Pan Wang, Zitong Zhan, Lin Zhang, Pedro Morgado

Organizations: University of Wisconsin–Madison · University of Pittsburgh · University at Buffalo, SUNY

Abstract

A key inefficiency in diffusion training occurs when a randomly initialized network, lacking visual priors, encounters gradients from the full complexity spectrum--most of which it lacks the capacity to resolve. We propose Data Warmup, a curriculum strategy that schedules training images from simple to complex without modifying the model or loss. Each image is scored offline by a semantic-aware complexity metric combining foreground dominance (how much of the image salient objects occupy) and foreground typicality (how closely the salient content matches learned visual prototypes). A temperature-controlled sampler then prioritizes low-complexity images early and anneals toward uniform sampling. On ImageNet 256x256 with SiT backbones (S/2 to XL/2), Data Warmup improves IS by up to 6.11 and FID by up to 3.41, reaching baseline quality tens of thousands of iterations earlier. Reversing the curriculum (exposing hard images first) degrades performance below the uniform baseline, confirming that the simple-to-complex ordering itself drives the gains. The method combines with orthogonal accelerators such as REPA and requires only ~10 minutes of one-time preprocessing with zero per-iteration overhead.

Figures & tables

Figure 1: Image complexity. Data warmup is based on image complexity. We quantify image complexity via two factors: foreground dominance and foreground typicality. Foreground separation leverages features from DINO-v2 to isolate salient regions. Foreground dominance ( $\Omega_{dom}$ ) measures the prominence of foreground elements, while foreground typicality ( $\Omega_{prot}$ ) assesses how representative a foreground is relative to learned prototypes. The overall complexity score ( $\Omega_{i}$ ) is derived by combining these two factors.

Figure 2: Data warmup. During diffusion training, images are sampled according to their complexity scores through a temperature-based scheduler. Initially, simpler images (low complexity scores) are prioritized, with the sampling distribution gradually shifting to uniformly cover the full dataset complexity.

Figure 3: Illustration of the complexity correction function $\Omega_{dom}(r_{bg};\kappa,v_{\text{min}})$ (Eq. 3 ). Left: Varying steepness $\kappa$ with fixed $v_{\text{min}}=0.002$ . Higher $\kappa$ leads to a sharper transition. Right: Varying minimum value $v_{\text{min}}$ with fixed steepness $\kappa=12.0$ .

Figure 4: ImageNet samples organized into low and high complexity according to $\Omega$ scores. Each row displays images from the same category.

Figure 5: Schedule. Evolution of the effective dataset size

|{\cal D}_{\tau(t)}|

(normalized by

|{\cal D}|

) using the power schedule. Warmup iterations set to

T_{w}=400

	IS ↑	FID ↓	sFID ↓	Pre. ↑	Rec. ↑
SiT-B/2 [ 19 ]	41.40	36.16	6.80	0.52	0.63
+ Data Warmup	45.70 $\uparrow$ 4.30	32.75 $\downarrow$ 3.41	6.56 $\downarrow$ 0.24	0.54 $\uparrow$ 0.02	0.63
+ Inverse Data Warmup	36.60 $\downarrow$ 4.80	41.05 $\uparrow$ 4.89	7.19 $\uparrow$ 0.39	0.49 $\downarrow$ 0.03	0.62 $\downarrow$ 0.01

Table 1: Performance of SiT-B/2 on ImageNet-1K (256

\times

256) with and without data warmup. The baseline employs uniform data sampling. “Inverse Data Warmup” prioritizes high-complexity samples during early training phases. ↑/↓ denote that higher/lower is better. Superscripts denote performance gains/losses in green/red.

	IS ↑	FID ↓	sFID ↓	Pre. ↑	Rec. ↑
REPA [ 37 ]	55.36	27.54	6.91	0.56	0.65
+ Data Warmup	58.08 $\uparrow$ 2.72	25.84 $\downarrow$ 1.70	6.89 $\downarrow$ 0.02	0.57 $\uparrow$ 0.01	0.64 $\downarrow$ 0.01

Table 2: Performance of REPA on ImageNet-1K (256×256) with and without data warmup using SiT-B/2 backbone. ↑/↓ denotes that higher/lower is better. Superscripts denote performance gains/losses in green/red.

Dataset	Model	IS ↑	FID ↓	sFID ↓
	SiT-B/2	43.05	79.38	220.69
IN-100	+ Warmup	28.75 $\downarrow$ 14.30	100.97 $\uparrow$ 21.59	225.38 $\uparrow$ 4.69
	SiT-B/2	44.79	35.32	10.08
IN-500	+ Warmup	50.90 $\uparrow$ 6.11	31.95 $\downarrow$ 3.37	10.03 $\downarrow$ 0.05
	SiT-B/2	41.40	36.16	6.80
IN-1K	+ Warmup	45.70 $\uparrow$ 4.30	32.75 $\downarrow$ 3.41	6.56 $\downarrow$ 0.24

Table 3: SiT-B/2 evaluated on datasets of increasing size. ↑/↓ denotes that higher/lower is better. Superscripts denote performance gains/losses in green/red.

Figure 6: Training curve. Inception scores of an SiT model trained on ImageNet-500 with and without data warmup.

T_{w}=200k

Model	IS ↑	FID ↓	sFID ↓
SiT-S	24.39	58.34	9.30
+ Warmup	25.55 $\uparrow$ 1.16	55.87 $\downarrow$ 2.47	9.16 $\downarrow$ 0.14
SiT-B	41.40	36.16	6.80
+ Warmup	45.70 $\uparrow$ 4.30	32.75 $\downarrow$ 3.41	6.56 $\downarrow$ 0.24
SiT-L	71.48	18.79	5.14
+ Warmup	77.44 $\uparrow$ 5.96	17.17 $\downarrow$ 1.62	5.14 $\downarrow$ 0.00

Table 4: SiT backbones of varying size evaluated on ImageNet 1000 (256×256). ↑/↓ denotes that higher/lower is better. Superscripts denote performance gains/losses in green/red.

	IS ↑	FID ↓	sFID ↓
Baseline	41.40	36.16	6.80
+ $\Omega_{prot}$	42.91 $\uparrow$ 1.51	35.01 $\downarrow$ 1.15	6.72 $\downarrow$ 0.08
+ $\Omega_{dom}$	44.42 $\uparrow$ 3.02	33.94 $\downarrow$ 2.22	6.53 $\downarrow$ 0.27
+ $\Omega_{prot}$ and $\Omega_{dom}$	45.70 $\uparrow$ 4.30	32.75 $\downarrow$ 3.41	6.56 $\downarrow$ 0.24

Table 5: Performance comparison of SiT training strategies using the SiT-B/2 model on ImageNet-1K (256

\times

256). ↑/↓ denotes that higher/lower is better. Superscripts denote performance gains/losses in green/red.

Param	Value	IS ↑	FID ↓	sFID ↓
	10	45.01	33.14	6.63
	12	45.69	32.75	6.56
$\kappa$	16	44.94	33.21	6.63
	0.02	44.19	34.07	6.68
$v_{\min}$	0.002	45.69	32.75	6.56

Table 6: Study of

\Omega_{dom}

hyper-parameters

\kappa

and

v_{min}

, evaluated on SiT-B/2 (256×256). The impact of

\kappa

and

v_{min}

is evaluated around the optimal setting

\kappa=12,v_{min}=0.002

Figure 7: Generated samples for the class “Loggerhead Sea Turtle” from a SiT-XL/2 model trained with REPA and data warmup for 2M iterations (classifier-free guidance $w=4.0$ ).

Diffusion-Based Image Generation

Diffusion-based image generation creates images by progressively denoising noise, with controls for content, style, and structure.

See all 327 papers

Data-free Multistep Diffusion Distillation
Oct 13, 2026Ngoc Duy Tran, Kien Do, Duc Thanh Nguyen +1Diffusion Model Distillation Diffusion Distillation
Deakin Applied Artificial Intelligence Initiative, Deakin University, Australia · VinSmart Future, Vietnam · School of Information Technology, Deakin University, Australia
EMPURPLE: A Free Lunch for Diffusion Distillation based on the Information Bottleneck
Jul 5, 2026Zilai Li, Lujia BaiDistribution Shift Diffusion Model Distillation
Independent researcher, University of Nottingham, Nottingham, UK
Improved Distributional Diffusion Models
Sep 29, 2026Tommaso Martorella, Alexandre Galashov, Felix Krause +4Few-Step Diffusion Sampling Diffusion-Based Image Generation
Munich Center for Machine Learning (MCML) · Google DeepMind

Oct 13, 2026J/K move · Enter open · S save

Data-free Multistep Diffusion Distillation

Ngoc Duy Tran, Kien Do, Duc Thanh Nguyen, Truyen Tran

Deakin Applied Artificial Intelligence Initiative, Deakin University, Australia · VinSmart Future, Vietnam · School of Information Technology, Deakin University, Australia

Diffusion Model Distillation Diffusion Distillation Data-Free Knowledge Distillation Few-Step Diffusion Sampling T2I Diffusion Models

Open paper AskPDF arXiv

One-step diffusion distillation achieves strong performance on small datasets but remains insufficient for high-resolution or complex text-to-image settings. Extending to multistep distillation often requires training data, suffers from early-step degradation, inconsistent outputs across sampling budgets, and incurs high computational cost. We identify that these limitations stem from simultaneous training, where a single adapted network approximates multiple score functions of different student distributions at once, causing biased distribution matching and unstable optimization. Therefore, we propose Progressive Multistep Data-free Distillation (PMDD), which assigns a dedicated adapted network per student distribution while avoiding the memory overhead of maintaining multiple adapted networks. Combined with a stage-wise unforget loss to preserve earlier-step generations, PMDD reduces optimization bias in the student, provides faster convergence, and a single unified model that supports multiple inference budgets. Extensive experiments show that PMDD matches or outperforms data-free few-step baselines at every multistep budget, reaches FID 1.99 on ImageNet 64x64 in two steps, below its 512-NFE teacher (2.32), and achieves FID 7.76 on zero-shot COCO 512x512, while requiring 4-5x less training compute than data-free DMDv2 to support three inference budgets.

Abstract

Diffusion-Based Image Generation

Explore similar work

Data-free Multistep Diffusion Distillation

EMPURPLE: A Free Lunch for Diffusion Distillation based on the Information Bottleneck

Improved Distributional Diffusion Models