Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

Liangyu Yuan; Yufei Huang; Mingkun Lei; Tong Zhao; Ruoyu Wang; Changxi Chi; Yiwei Wang; Chi Zhang

cs.CVMar 21, 2026

Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

Authors: Liangyu Yuan, Yufei Huang, Mingkun Lei, Tong Zhao, Ruoyu Wang, Changxi Chi, Yiwei Wang, Chi Zhang

Organizations: Westlake University · Tongji University · Zhejiang University · University of California at Merced

Abstract

Diffusion models generate synthetic images through an iterative refinement process. However, the misalignment between the simulation-free objective and the iterative process often causes accumulated gradient error along the sampling trajectory, which leads to unsatisfactory results and a failure to generalize. Guidance techniques like Classifier Free Guidance (CFG) and AutoGuidance (AG) alleviate this by extrapolating between the main and inferior signal for stronger generalization. Despite empirical success, the effective operational regimes of prevalent guidance methods are still under-explored, leading to ambiguity when selecting the appropriate guidance method given a precondition. In this work, we first conduct synthetic comparisons to isolate and demonstrate the effective regime of guidance methods represented by CFG and AG from the perspective of weak-to-strong principle. Based on this, we propose a hybrid instantiation called SGG under the principle, taking the benefits of both. Furthermore, we demonstrate that the W2S principle along with SGG can be migrated into the training objective, improving the generalization ability of unguided diffusion models. We validate our approach with comprehensive experiments. At inference time, evaluations on SD3 and SD3.5 confirm that SGG outperforms existing training-free guidance variants. Training-time experiments on transformer architectures demonstrate the effective migration and performance gains in both conditional and unconditional settings. Code is available at https://github.com/851695e35/SGG.

Figures & tables

Figure 1: $\mathrm{I}$ : Weak-to-strong guidance principle: Guidance methods serve as tools for improving generalization capacity, we propose SGG to combine the benefits of condition-dependent (CDG) and condition-agnostic guidance (CAG). $\mathrm{II}$ : Integration to the training framework, improving the generalization ability of unguided diffusion models.

Figure 2: Recursive toy example with varying class complexity and in-class distribution (granular of the condition). 1st row: In a well fitted model and the conditional information is blurry, CFG [ 18 ] exhibits mode-seeking capacity while lack diversity. 2nd row: In a less fitted model and the conditional information is sharp, AG [ 24 ] improves diversity while leads to outliers. 3rd row: In practice, SGG incorporates the mode-seeking capacity of CFG in high noise levels while applying AG in low noise levels to preserve the in-class distribution.

Figure 3: Applying guidance reduces the gap to optimal velocity $\dot{\mathbf{v}}$ . The error-correction of CFG is prominent at high noise levels, while the effect of AG is prominent at low noise levels.

Figure 4: $\mathrm{I}$ : Two groups of construction of the weak models, condition-dependent and condition-agnostic. $\mathrm{II}$ : Segmented Guidance applied in training and sampling

Models		SD3-medium				SD3.5-medium
Dataset		MS-COCO-1K		LAION-5B-1K		MS-COCO-1K		LAION-5B-1K
Metrics	NFE/s	HPSv2.1	Aes.	HPSv2.1	Aes.	HPSv2.1	Aes.	HPSv2.1	Aes.
Conditional (w/o Guidance)	1	21.118	4.864	20.215	4.938	21.204	4.978	20.801	4.978
CFG [ 18 ]	2	29.199	5.267	28.174	5.193	29.199	5.279	28.333	5.203
SLG [ 20 ]	2	26.685	5.581	25.000	5.415	27.295	5.714	26.050	5.512
CFG+SLG [ 20 ]	3	27.759	5.468	26.147	5.285	28.331	5.678	27.246	5.421

Table 1: Quantitative comparison of guidance methods on MS-COCO-1K and LAION-5B-1K, evaluating both HPSv2.1 and aesthetic scores for SD3 and SD3.5 models. Best results are in bold , second-best are underlined .

Conditional Generation
Model	NFE/s	time/it	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$
SiT-B/2	1	1.00	31.22	6.41	49.59
+ CFG	2	1.00	6.02	5.47	183.83
AG	1	1.27	13.96	88.36	4.68
BR	1	1.02	16.02	5.13	76.21
MG	1	1.23	5.88	6.19	253.74

Table 2: Training-time integration results on ImageNet 256

\times

256 with SiT-B/2, in conditional and unconditional settings.

Figure 5: Qualitative comparison between Conditional (w/o guidance), CFG [ 18 ] , SLG [ 20 ] , SGG (Ours).

Figure 6: Ablation study on the segmentation timestep

\tau

. We vary

\tau

from 4 (lightest) to 24 (darkest) in increments of 4, out of 28 total sampling steps. The results indicate that a mid-range segmentation point yields the best performance.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	$T_{\mathrm{main}}$	$T_{\mathrm{weak}}$	$\tau$	$w_{\mathrm{cfg}}$	$w_{\mathrm{ag}}$
Config A	$2^{15}$	$2^{11}$	0.5	2.0	2.0
Config B	$2^{12}$	$2^{10}$	0.1	2.0	2.0
Config C	$2^{15}$	$2^{11}$	0.3	2.0	2.0

Appendix

Table 4: Hyperparameter settings for toy experimentation.

Panel of Figure 7 — Figure 7: Quantitative comparison of guidance scale ( $w$ ) for CFG and SLG with SD3.5 on LAION-5B-1K and MSCOCO-1K datasets.

Method	Avg	Aesthetic	Overall Cons.	Imaging Qual.	Subject Cons.	Dynamic Deg.	Motion Smooth.
CFG	0.6877	0.6003	0.2317	0.6553	0.9391	0.7222	0.9776
SLG	0.6809	0.5822	0.2026	0.6597	0.9504	0.7083	0.9820
SGG	0.7001	0.6107	0.2314	0.6624	0.9402	0.7778	0.9785

Appendix

Table 5: Comparison on WAN-1.3B [ 53 ] with CFG, SLG and SGG (Ours). Best results are bolded , and second-best results are underlined .

	Conditional					Unconditional
Parameter	Baseline	AG	BR	MG	SGG	Baseline	AG	BR
Training configuration
Generation Type	Cond	Cond	Cond	Cond	Cond	Uncond	Uncond	Uncond
Batch-size	256	256	256	256	256	256	256	256
Num. GPUs (A100)	4	4	4	4	4	4	4	4
Noise Schedules	OT	OT	OT	OT	OT	OT	OT	OT

Appendix

Table 6: Training information for W2S guidance experiments. All models are trained on SiT-B/2. For AG, we train a separate weak model with T/4 iterations, where T is the strong model’s iteration.

Method	MS-COCO-1K				LAION-5B-1K
	HPSv2.1	Aesthetic	CLIP	GenEval	HPSv2.1	Aesthetic	CLIP	GenEval
CFG	29.219	5.279	26.316	0.628	28.234	5.203	27.521	0.674
SLG	27.295	5.714	25.145	0.507	26.050	5.512	25.440	0.523
CFG+SLG	28.931	5.678	26.485	0.598	27.246	5.421	27.154	0.638
CFG (linear schedule)	28.833	5.261	26.351	0.633	27.954	5.220	27.777	0.665
SGG (swap orders)	27.393	5.366	25.523	0.534	26.270	5.311	26.283	0.579

Appendix

Table 7: SD3.5-medium on MS-COCO-1K and LAION-5B-1K

		Conditional			Unconditional
Method	Guidance	FID $\downarrow$	sFID $\downarrow$	Inception Score $\uparrow$	FID $\downarrow$	sFID $\downarrow$	Inception Score $\uparrow$
Baseline	$w=0.0$	31.22	6.41	49.59	61.27	7.00	17.33
BR	$w=0.2$	18.33	4.71	70.45	46.37	4.93	20.23
	$w=0.3$	16.02	5.13	76.21	43.25	5.11	20.66
	$w=0.4$	15.09	7.64	80.29	42.13	7.59	20.43
AG	$w=0.2$	18.71	4.97	73.30	50.97	5.52	19.40

Appendix

Table 8: Ablation study on guidance scale (

w

) for W2S training methods (BR and AG) in both conditional and unconditional settings. All models are SiT-B/2 trained on ImageNet 256x256.

Figure 8: SLG in training, applied in different iterations.

Figure 9: Qualitative comparison of CFG and SGG (Ours) on video generation

Figure 10: Qualitative Comparison between Unguided Conditional, CFG, SLG and SGG (Ours) (1/2)

Figure 11: Qualitative Comparison between Unguided Conditional, CFG, SLG and SGG (Ours) (2/2)

Figure 12: Qualitative Comparison between SiT-B/2 (Baseline), SGG (Ours), REPA, REPA+SGG

Explore similar work

Guiding a Diffusion Model by Swapping Its Tokens
Apr 9, 2026Weijia Zhang, Yuehao Liu, Shanyan Guan +4Diffusion Model Guidance Diffusion Guidance
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.
Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models
Apr 29, 2026Haosen Li, Wenshuo Chen, Lei Wang +4Diffusion Model Guidance Classifier-Free Guidance
1The Hong Kong University of Science and Technology (Guangzhou) · 2Griffith University & Data61/CSIRO
Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models
Aug 17, 2026Artem Sergievskii, Artyom Turevich, Sergey KastryulinDiffusion Model Guidance Classifier-Free Guidance
HSE University · Yandex, HSE University

Apr 9, 2026J/K move · Enter open · S save

Guiding a Diffusion Model by Swapping Its Tokens

Weijia Zhang, Yuehao Liu, Shanyan Guan, Wu Ran+3

MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.

Diffusion Model Guidance Diffusion Guidance Image Generation T2I Diffusion Models

Open paper AskPDF arXiv

Classifier-Free Guidance (CFG) is a widely used inference-time technique to boost the image quality of diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular diffusion models, outperforms previous condition-free methods in image fidelity and prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any diffusion model as a plug-in to gain immediate improvements.

Abstract

Explore similar work

Guiding a Diffusion Model by Swapping Its Tokens

Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models

Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models