Guiding a Diffusion Model by Swapping Its Tokens

Weijia Zhang; Yuehao Liu; Shanyan Guan; Wu Ran; Yanhao Ge; Wei Li; Chao Ma

cs.CVApr 9, 2026

Guiding a Diffusion Model by Swapping Its Tokens

Authors: Weijia Zhang, Yuehao Liu, Shanyan Guan, Wu Ran, Yanhao Ge, Wei Li, Chao Ma

Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.

Abstract

Classifier-Free Guidance (CFG) is a widely used inference-time technique to boost the image quality of diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular diffusion models, outperforms previous condition-free methods in image fidelity and prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any diffusion model as a plug-in to gain immediate improvements.

Figures & tables

Figure 1: Self-Swap Guidance (SSG) generates higher-fidelity images over a wider range of guidance scale. In contrast, existing methods [ 18 , 1 , 19 ] suffer from poor details at lower guidance scale, or noise, oversaturation, and oversimplified details at higher guidance scale.

Figure 2: Visualisations of guidance patterns and the iteratively denoised images across different timesteps. The text prompt used is “ A loft bed with a dresser underneath it ”.

Figure 3: Qualitative comparison of unconditional image generation by SDXL.

Table 4

Figure 4: Qualitative comparison of conditional image generation by SDXL.

Table 6

Figure 5: Effect of varying guidance scale and swap ratio on image quality and prompt alignment.

Figure 6: Visualising the effect of using different swap ratio and guidance scale values on generated images.

Method	FID $\downarrow$	CLIP $\uparrow$	IS $\uparrow$	AES $\uparrow$	PickScore $\uparrow$	IR $\uparrow$
SAG [ 18 ]	43.97	0.295	22.12	5.756	20.65	-0.483
SEG [ 19 ]	38.15	0.303	26.15	5.888	21.38	-0.0239
PAG [ 1 ]	36.79	0.306	29.42	5.827	21.57	0.00171
Random	32.28	0.312	34.67	5.928	22.14	0.283
Similar	28.74	0.309	32.81	5.826	21.80	0.110
Dissimilar	31.41	0.313	34.44	5.901	22.18	0.297

Table 5: Importance of adversarial token swap. Swapping dissimilar tokens achieves the best generation quality overall. Random swap yields slightly worse results and swapping similar tokens perform worst, but they still substantially outperform.

Figure 7: Visualising the effect of different token swap policies. Swapping dissimilar tokens further refines local details and global coherence compared to random swap. In contrast, swapping similar tokens leads to poor generation that resembles the vanilla diffusion model’s output.

Spatial	Channel	FID $\downarrow$	CLIP $\uparrow$	IS $\uparrow$	AES $\uparrow$	PickScore $\uparrow$	IR $\uparrow$
✓	✗	31.96	0.313	33.33	5.931	22.11	0.272
✗	✓	31.30	0.313	34.95	5.892	22.17	0.286
✓	✓	31.41	0.313	34.44	5.901	22.18	0.297

Table 6: Ablation on two types of token swap.

SSG	CFG	FID $\downarrow$	CLIP $\uparrow$	IS $\uparrow$	AES $\uparrow$	PickScore $\uparrow$	IR $\uparrow$
✗	✗	54.96	0.281	21.20	5.676	20.22	-0.834
✓	✗	31.41	0.313	34.44	5.901	22.18	0.297
✓	✓	30.82	0.319	36.37	5.858	22.44	0.492

Table 7: Compatibility of SSG with CFG.

Figure 8: Compatibility between SSG and CFG. SSG can be applied on top of CFG to further refine image quality.

Explore similar work

Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance
Mar 21, 2026Liangyu Yuan, Yufei Huang, Mingkun Lei +5Diffusion Model Guidance Diffusion Models
Westlake University · Tongji University · Zhejiang University +1
ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts
Nov 26, 2024Jinho Chang, Changsun Lee, Hyungjin Chung +1Diffusion Model Guidance Classifier-Free Guidance
Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology, Seoul, South Korea · EverEx, Seoul, South Korea
Guiding Diffusion Models with Semantically Degraded Conditions
Mar 11, 2026Shilong Han, Yuming Zhang, Hongxia WangDiffusion Model Guidance T2I Diffusion Models
College of Science, National University of Defense Technology

Mar 21, 2026J/K move · Enter open · S save

Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

Liangyu Yuan, Yufei Huang, Mingkun Lei, Tong Zhao+4

Westlake University · Tongji University · Zhejiang University+1

Diffusion Model Guidance Diffusion Models Diffusion Guidance Classifier-Free Guidance

Open paper AskPDF arXiv

Diffusion models generate synthetic images through an iterative refinement process. However, the misalignment between the simulation-free objective and the iterative process often causes accumulated gradient error along the sampling trajectory, which leads to unsatisfactory results and a failure to generalize. Guidance techniques like Classifier Free Guidance (CFG) and AutoGuidance (AG) alleviate this by extrapolating between the main and inferior signal for stronger generalization. Despite empirical success, the effective operational regimes of prevalent guidance methods are still under-explored, leading to ambiguity when selecting the appropriate guidance method given a precondition. In this work, we first conduct synthetic comparisons to isolate and demonstrate the effective regime of guidance methods represented by CFG and AG from the perspective of weak-to-strong principle. Based on this, we propose a hybrid instantiation called SGG under the principle, taking the benefits of both. Furthermore, we demonstrate that the W2S principle along with SGG can be migrated into the training objective, improving the generalization ability of unguided diffusion models. We validate our approach with comprehensive experiments. At inference time, evaluations on SD3 and SD3.5 confirm that SGG outperforms existing training-free guidance variants. Training-time experiments on transformer architectures demonstrate the effective migration and performance gains in both conditional and unconditional settings. Code is available at https://github.com/851695e35/SGG.

Abstract

Explore similar work

Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts

Guiding Diffusion Models with Semantically Degraded Conditions