A First-Step Bias Restores Lost Diversity in Diffusion Model Distillation

Wenwu Tang; Dong Wang; Yun Cheng; Olga Saukh

cs.CVOct 27, 2026

A First-Step Bias Restores Lost Diversity in Diffusion Model Distillation

Authors: Wenwu Tang, Dong Wang, Yun Cheng, Olga Saukh

Organizations: Graz University of Technology · Swiss Data Science Center, Switzerland · Complexity Science Hub, Austria

Abstract

Distribution-matching distillation enables high-quality image generation in only a few denoising steps. However, distilled student models often map different initial noise samples to similar outputs under the same prompt. To locate where diversity is lost, we compare the teacher and student along the sampling trajectory. The largest measured reduction in the response to different initial noises occurs during the first, high-noise step. When we replace the first step with the teacher's corresponding trajectory and leave the final three student steps unchanged, most of the output diversity returns. This suggests that the first step is a key point for restoring diversity. Motivated by this finding, we introduce ReSeed, a post-distillation method that adds a seed-response bias adapter only to the first student step and aligns its velocity with the teacher's first-interval secant velocity. All student weights and later denoising steps remain unchanged. We evaluate ReSeed on three text-to-image models (SD3-Medium, SD3.5-Medium, and FLUX.1-dev) and one text-to-video model (Wan2.1-T2V-1.3B). For a distilled 4-step FLUX.1-dev model, ReSeed requires only $0.00196\%$ additional trainable parameters relative to the base model and is trained on 8,000 prompt--noise pairs, while keeping the distilled student fully frozen and requiring no target images, perceptual losses, or adversarial losses. It improves DINO diversity by $48.3\%$ while maintaining competitive visual quality. These results show that substantial diversity can be recovered through a lightweight, data-efficient tuning of only the first denoising step, without finetuning the full student model.

Figures & tables

Figure 1: ReSeed recovers visible same-prompt diversity. All panels use the prompt “A smiling woman with long dark brown hair, hazel eyes, and a warm expression” and the same 25 fixed initial noises, matched cell by cell. (a) The 40-step SD3.5-Medium teacher at CFG $4.5$ . (b) The frozen 4-step DMD student, and (c) The same frozen 4-step DMD student with ReSeed.

Figure 2: First-step replacement on SD3-Medium. (a) With student steps 2–4 fixed, replacing step 1 by the teacher raises diversity for

98/100

prompts; ReSeed moves them toward the teacher-prefix result. (b,c) Replacing the first

k

steps shows that

k=1

gives

91\%

of the diversity gain. CLIPScore changes little until later steps are replaced. Bands show the standard deviation over three seeds.

Figure 3: Overview of ReSeed . (a) First-step contraction can merge seed trajectories; the teacher and ReSeed retain alternative paths. (b) Training fits first-step block-output biases to the teacher’s interval-average velocity with frozen base weights. (c) The bias is off in steps 2–4.

Figure 4: Seed-response error versus DINOv3 diversity on COCO5K ( $5{,}000$ prompts $\times$ 10 seeds). The error axis is reversed; black curve is checkpoint means.

SD3-Medium (teacher CFG $=7$ )
Method	NFEs	Diversity			Quality
Method	NFEs	DINO $\uparrow$	CLIP dist. $\uparrow$	LPIPS $\uparrow$	CLIP $\uparrow$	Pick $\uparrow$	HPS $\uparrow$	AeS $\uparrow$	FID $\downarrow$
Base Model	56	0.269	0.148	0.696	0.3534	22.516	0.2895	5.33	32.15
DMD	4	$0.214_{\pm 0.001}$	$0.134_{\pm 0.0003}$	$0.655_{\pm 0.006}$	$\mathbf{0.3442}_{\pm 0.0011}$	$\mathbf{22.654}_{\pm 0.015}$	$\mathbf{0.3094}_{\pm 0.0007}$	$\underline{5.61}_{\pm 0.01}$	$37.38_{\pm 0.40}$
DP-DMD	4	$\underline{0.282}_{\pm 0.007}$	$\underline{0.167}_{\pm 0.002}$	$0.666_{\pm 0.016}$	$0.3276_{\pm 0.0004}$	$21.983_{\pm 0.096}$	$0.2805_{\pm 0.0078}$	$5.20_{\pm 0.09}$	$43.29_{\pm 2.12}$
GAD	4	$0.234_{\pm 0.002}$	$0.132_{\pm 0.002}$	$\underline{0.668}_{\pm 0.017}$	$0.3398_{\pm 0.0020}$	$\underline{22.587}_{\pm 0.057}$	$0.3046_{\pm 0.0003}$	$5.53_{\pm 0.01}$	$\underline{35.82}_{\pm 0.39}$

Table 1: Results on SD3-Medium and SD3.5-Medium under a common evaluation protocol.

Model	NFEs	Diversity			Quality
Model	NFEs	DINO $\uparrow$	CLIP dist. $\uparrow$	LPIPS $\uparrow$	CLIP score $\uparrow$	Pick $\uparrow$	HPS $\uparrow$	AeS $\uparrow$	FID $\downarrow$
FLUX.1-dev	28	0.242	0.140	0.649	0.3472	23.050	0.3033	5.70	33.49
SenseFlow	4	$0.158$	$0.090$	$0.533$	$0.3431$	$\mathbf{23.152}$	$\mathbf{0.3128}$	$\mathbf{5.87}$	$40.22$
SenseFlow + ReSeed (ours)	4	$\mathbf{0.234}$ $\mathbf{\pm 0.001}$	$\mathbf{0.126}$ $\mathbf{\pm 0.002}$	$\mathbf{0.618}$ $\mathbf{\pm 0.003}$	$\mathbf{0.3450}$ $\mathbf{\pm 0.0003}$	$23.121$ $\pm 0.011$	$0.3118$ $\pm 0.0003$	$5.85$ $\pm 0.01$	$\mathbf{34.65}$ $\mathbf{\pm 0.15}$

Table 2: FLUX quality–diversity comparison.

Panel of Figure 6 — Figure 6: 4-step FLUX.1-dev student during bias adapter training.

Method	Steps	Video quality (VBench)							Text align. $\uparrow$	Video diversity
Method	Steps	Subject $\uparrow$	Backg. $\uparrow$	Flicker $\uparrow$	Smooth. $\uparrow$	Dynamic $\uparrow$	AeS $\uparrow$	Imaging $\uparrow$	Text align. $\uparrow$	CLIP mean $\uparrow$	CLIP frame $\uparrow$	DINO mean $\uparrow$	DINO frame $\uparrow$
Teacher	50	0.970	0.961	0.983	0.991	0.371	0.590	0.703	0.2425	0.1634	0.1899	0.5005	0.5352
DMD2	4	0.970	0.962	0.983	0.991	0.531	0.615	0.698	0.2419	0.1267	0.1530	0.3374	0.3748
DFD	4	0.967	0.955	0.984	0.991	0.529	0.611	0.711	0.2398	0.1351	0.1659	0.3358	0.3775
DMD2 + ReSeed (ours)	4	0.967	0.961	0.986	0.992	0.329	0.631	0.675	0.2365	0.1437	0.1733	0.4290	0.4716

Table 3: Wan2.1-T2V-1.3B on 320 VBench prompts. Bold marks the best 4-step result

Student model	Params	of denoiser	Before $\to$ after	Gain	change	Step 1	Per sample	Overhead
	Adapter		DINO diversity		Peak act.	Latency (ms)
SD3-Medium	72,192	$0.00356\%$	$0.2142\to 0.2936$	$+37.1\%$	$+0.00$ MiB	$37.10\!\to\!37.68$	$148.4\!\to\!149.0$	$+0.410\%$
SD3.5-Medium	72,192	$0.00322\%$	$0.2356\to 0.3282$	$+39.3\%$	$+0.00$ MiB	$55.69\!\to\!56.25$	$222.8\!\to\!223.2$	$+0.194\%$
FLUX.1-dev	233,472	$0.00196\%$	$0.1575\to 0.2336$	$+48.3\%$	$+1.00$ MiB	$253.4\!\to\!255.9$	$1013.6\!\to\!1016.2$	$+0.250\%$
Wan2.1-T2V-1.3B	46,080	$0.00325\%$	$0.3374\to 0.4290$	$+27.1\%$	$+0.00$ MiB	$992.7\!\to\!996.1$	$3978.9\!\to\!3984.3$	$+0.137\%$

Table 4: ReSeed training and inference overhead.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Source type	Repository or checkpoint
SD3-Medium teacher	Public	stabilityai/stable-diffusion-3- medium-diffusers
SD3.5-Medium teacher	Public	stabilityai/ stable-diffusion-3.5-medium
FLUX.1-dev teacher	Public	black-forest-labs/FLUX.1-dev
SenseFlow-FLUX student	Public	domiso/SenseFlow , SenseFlow-FLUX/ transformer [ Ge et al., 2026 ]
Wan2.1-T2V-1.3B teacher	Public	Wan-AI/ Wan2.1-T2V-1.3B-Diffusers
Wan2.1 DMD2 student	Public	csusupergear/ wan_t2v_checkpoints_release , dmd2_t2v_vipe/0001000 [ Chen et al., 2026 ]

Appendix

Table 5: Model and checkpoint sources. Repository identifiers link to the public weight releases; locally trained students are listed separately.

Category	Teacher	DMD2	DFD	+ ReSeed	Change [95% CI]	Improved
Animal	0.4235	0.2555	0.2471	0.3313	$+0.0758\;[+0.053,+0.100]$	34/40
Architecture	0.5948	0.4391	0.4374	0.5278	$+0.0888\;[+0.056,+0.125]$	31/40
Food	0.4371	0.2797	0.2719	0.3904	$+0.1107\;[+0.087,+0.136]$	40/40
Human	0.4518	0.2564	0.2635	0.3800	$+0.1236\;[+0.097,+0.152]$	37/40
Lifestyle	0.4792	0.2837	0.3117	0.4093	$+0.1255\;[+0.096,+0.155]$	38/40
Plant	0.5813	0.4125	0.4146	0.4818	$+0.0693\;[+0.034,+0.104]$	31/40

Appendix

Table 6: Video-level DINOv2 diversity by VBench category (40 prompts each). The paired change and the number of improved prompts compare ReSeed with DMD2.

Figure 7: Training trajectories for five teacher guidance scales.

Teacher $w$	Diversity			Quality
Teacher $w$	DINO $\uparrow$	CLIP dist. $\uparrow$	LPIPS $\uparrow$	CLIP score $\uparrow$	Pick $\uparrow$	HPS $\uparrow$	AeS $\uparrow$	FID-1K $\downarrow$
$\mathbf{1}$	$\mathbf{0.3132}$	$\mathbf{0.1813}$	$\mathbf{0.7441}$	$0.33832$	$22.318$	$0.2998$	$5.617$	$\mathbf{71.35}$
$3$	$0.3060$	$0.1764$	$0.7391$	$0.33952$	$22.384$	$0.3012$	$5.616$	$71.54$
$5$	$0.2990$	$0.1737$	$0.7332$	$0.34072$	$22.433$	$0.3019$	$5.619$	$72.08$
$7$	$0.2951$	$0.1697$	$0.7279$	$0.34049$	$22.454$	$0.3026$	$5.617$	$72.47$
$10$	$0.2873$	$0.1676$	$0.7230$	$\mathbf{0.34094}$	$\mathbf{22.471}$	$\mathbf{0.3027}$	$\mathbf{5.621}$	$72.63$

Appendix

Table 7: Teacher guidance for the first-interval regression target, SD3-Medium, read at

2{,}000

updates. Only the guidance used to build the target changes; the student, its sampler at CFG

1

, the COCO1K prompts, the

100\times 10

diversity bank, the seed and the block range are shared. Bold marks the best column entry. Diversity falls and alignment generally increases with

w

, and FID is best at the lowest guidance.

Variant	CLIP score	Pick	HPS	AeS	FID $\downarrow$	DINO $\uparrow$	CLIPd $\uparrow$	LPIPS $\uparrow$
4-step student (frozen)	0.34290	22.637	0.3100	5.618	37.471	0.2142	0.1334	0.6609
28-step teacher	0.35343	22.516	0.2895	5.325	32.148	0.2686	0.1478	0.6961
$\alpha=-1$	0.34346	22.597	0.3118	5.623	38.468	0.1815	0.1230	0.5855
$\alpha=0.5$	0.34222	22.605	0.3081	5.628	35.598	0.2545	0.1509	0.6979
$\alpha=1$	0.34051	22.471	0.3041	5.623	33.376	0.2914	0.1698	0.7283
$\alpha=1.5$	0.33918	22.304	0.2980	5.600	31.476	0.3190	0.1862	0.7535

Appendix

Table 8: COCO5K quality and diversity.

Panel of Figure 8 — Figure 8: ReSeed training dynamics. Left: training loss for the three image models. Middle and right: bias magnitude and cosine with the final bias during training.

Figure 9: SD3-Medium and SD3.5-Medium over ReSeed training.

Figure 10: Prompt-level trajectories for one FLUX.1-dev adapter on 100 prompts $\times$ 10 initial noises. Each point compares one prompt at a training checkpoint (vertical axis) with the original student (horizontal axis); colour denotes the checkpoint and the black line shows the prompt mean. Left: DINOv3 diversity improves for 95 of 100 prompts, from $0.158$ to $0.242$ on average. Right: CLIP returns to its initial mean ( $0.33748\!\rightarrow\!0.33753$ ).

Step	LHS	RHS	LHS/RHS	$s$	DINOv3 div. $\uparrow$	CLIP div. $\uparrow$
0	5.70	8.53	0.68	0.39	0.213	0.122
100	3.09	3.66	0.85	0.23	0.253	0.139
200	2.74	3.08	0.90	0.19	0.267	0.145
300	2.60	2.93	0.89	0.20	0.277	0.149
400	2.46	2.73	0.91	0.18	0.279	0.150
700	2.28	2.49	0.92	0.17	0.281	0.152

Appendix

Table 9: Pairwise response error, its bound, and final-image diversity on COCO5K (

5{,}000

prompts

\times

10 seeds) for one SD3-Medium ReSeed run. Step 0 is the DMD 4-step student. LHS and RHS are the empirical pairwise error

\widehat{P}

and uncorrected bound

\widehat{B}

, averaged over prompts and reported in units of

10^{-3}

. LHS/RHS and the estimated mean-error share

s

are medians over prompts.

Figure 11: Empirical analysis of Proposition 1 on COCO5K. Each point is one prompt at one checkpoint, coloured by training step; step 0 is the frozen student. The black curve shows per-step means over prompts. The left-hand side of Equation 7 is plotted against its right-hand side $2(t_{1}-t_{0})^{2}\mathcal{L}_{\mathrm{ReSeed}}$ . The diagonal marks $\widehat{P}=\widehat{B}$ ; the exact finite-sample bound is given in Equation 13 . The median LHS/RHS rises from $0.68$ to $0.92$ during training (Table 9 ).

Figure 12: Matched-noise SD3.5-Medium comparisons. Each method uses the same six seeds. The figure continues on the next page.

Panel of Figure 13 — Figure 13: Matched-noise SD3-Medium comparisons. Each method uses the same six noises.

Panel of Figure 14 — Figure 14: Matched-seed FLUX.1-dev comparisons. Each method uses the same nine noises for each prompt.

Panel of Figure 15 — Figure 15: Matched-seed FLUX.1-dev comparisons (continued).

Panel of Figure 16 — Figure 16: Matched-noise frame strips for Wan2.1-T2V-1.3B. In each panel, the top, middle, and bottom blocks show the 50-step teacher, the 4-step DMD2 student, and the same student with ReSeed (300 updates); within a block, the eight rows are seeds 0–7 with the same initial noise across blocks, and the twelve columns are twelve evenly spaced frames from 0 to 80. Prompts and video-level DINO diversity (teacher / DMD2 / ReSeed, eight seeds): (a) “A barista pours latte art into a cappuccino, viewed from directly above as a delicate rosetta forms” , $0.122/0.058/0.245$ ; (b) “an elegant ceramic plant pot and hanging plant on indoor” , $0.607/0.247/0.588$ ; (c) “A young boy lights sparklers in his backyard as twilight settles, his face illuminated in flickering gold” , $0.113/0.082/0.132$ ; (d) “scenic view from wooden pathway” , $0.599/0.339/0.475$ .

Figure 17: SD3-Medium: a gallery of 4-step ReSeed samples at $1024\times 1024$ .

Figure 18: SD3.5-Medium: a gallery of 4-step ReSeed samples at $1024\times 1024$ .

Figure 19: FLUX.1-dev: a gallery of 4-step ReSeed samples at $1024\times 1024$ .

Diffusion Model Distillation

Diffusion model distillation transfers a large model’s generative behavior to faster or smaller models that need fewer sampling steps.

See all 191 papers

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation
Apr 5, 2026Haoyu Li, Tingyan Wen, Lin Qi +6Diffusion Model Distillation Diffusion Sampling
Tsinghua University · Central Media Technology Institute, Huawei
CrossDistill: Balancing Quality and Diversity via Trajectory-Level Hybrid Few-Step Distillation
Sep 13, 2026Yuxi Liu, Haoyu Li, Yixiang Cai +8Video Diffusion Models Diffusion Model Distillation
Peking University, Melon Group · Alibaba Group · Tsinghua University
Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations
May 8, 2026Yuan Zhang, Chenyi Li, Guoqing Ma +7Diffusion Model Distillation Diffusion Sampling
JD Explore Academy, China · The Hong Kong University of Science and Technology · Tsinghua University +1

Apr 5, 2026J/K move · Enter open · S save

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation

Haoyu Li, Tingyan Wen, Lin Qi, Zhe Wu+5

Tsinghua University · Central Media Technology Institute, Huawei

Diffusion Model Distillation Diffusion Sampling Few-Step Diffusion Sampling Distribution Matching Distillation T2I Diffusion Models

Open paper AskPDF arXiv

Diffusion models produce high-quality text-to-image results, but their iterative denoising is computationally expensive.Distribution Matching Distillation (DMD) emerges as a promising path to few-step distillation, but suffers from diversity collapse and fidelity degradation when reduced to two steps or fewer. We present 1.x-Distill, the first fractional-step distillation framework that breaks the integer-step constraint of prior few-step methods and establishes 1.x-step generation as a practical regime for distilled diffusion models.Specifically, we first analyze the overlooked role of teacher CFG in DMD and introduce a simple yet effective modification to suppress mode collapse. Then, to improve performance under extreme steps, we introduce Stagewise Focused Distillation, a two-stage strategy that learns coarse structure through diversity-preserving distribution matching and refines details with inference-consistent adversarial distillation. Furthermore, we design a lightweight compensation module for Distill--Cache co-Training, which naturally incorporates block-level caching into our distillation pipeline.Experiments on SD3-Medium and SD3.5-Large show that 1.x-Distill surpasses prior few-step methods, achieving better quality and diversity at 1.67 and 1.74 effective NFEs, respectively, with up to 33x speedup over original 28x2 NFE sampling.

Abstract

Diffusion Model Distillation

Explore similar work

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation

CrossDistill: Balancing Quality and Diversity via Trajectory-Level Hybrid Few-Step Distillation

Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations