SWIFT: Self-guided Weak-to-strong dIFfusion models via Target extrapolation

Yanjin He; Cuifeng Shen; Wanzhou Lei; Maohua Li; Tao Lan; Kan Liu; Hanlin Tang; Qing Qu

cs.LGOct 19, 2026

SWIFT: Self-guided Weak-to-strong dIFfusion models via Target extrapolation

Authors: Yanjin He, Cuifeng Shen, Wanzhou Lei, Maohua Li, Tao Lan, Kan Liu, Hanlin Tang, Qing Qu

Organizations: University of Michigan, Ann Arbor · Alibaba Group · University of California, Berkeley · Nanjing University

Abstract

Training high-quality diffusion models is computationally expensive, and inference-time guidance often adds further sampling cost. In this work, we introduce Swift, a simple yet effective self-guided training method that uses a frozen weak denoiser to improve a stronger model without adding inference overhead. For each noisy input, Swift extrapolates the clean regression target away from the weak denoiser's prediction, amplifying components that remain underfit. The weak guide can be an early training checkpoint, requires no backpropagation during main-model training, and is discarded afterward. The resulting model retains the original sampler and supports both conditional and unconditional generation without label dropout or an unconditional branch. We explain this mechanism through a spectral analysis of finite-time population gradient flow in an RKHS. With an early-stopped weak guide, we prove that a nonempty range of extrapolation strengths yields strictly lower posterior-mean error than matched standard training, despite shifting the regression target away from the posterior mean. Experiments on CIFAR-10 and ImageNet-256 demonstrate improvements across model scales and representations. In one ImageNet-256 configuration, Swift surpasses the baseline's best FID using roughly half the total training compute, including guide preparation and training overhead. On RAEv2, it achieves FID 1.15 without inference-time guidance and 0.99 with guidance. These results establish target extrapolation as an effective approach to weak-to-strong diffusion training with efficient single-model inference.

Figures & tables

Figure 2: Swift on a two-class, tree-shaped 2D toy distribution. Ground truth and samples from CFG, AG, MG, SGG, IG, GFT, and Swift . Each compared method uses independently tuned settings. Blue boxes mark corresponding branching regions. Lower KL and outlier rates and higher $\log p$ are better.

615M parameters, 80 epochs; guidance = CFG
	Without guidance						With guidance
Family: representation	FID $\downarrow$	sFID $\downarrow$	$\overline{\mathrm{FD}}_{r}\downarrow$	$\overline{\mathrm{MIND}}\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	$\overline{\mathrm{FD}}_{r}\downarrow$	$\overline{\mathrm{MIND}}\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$
RAE: DINOv2-B	2.09	4.48	4.13	21.34	0.83	0.57	1.98	4.50	4.17	20.71	0.81	0.59
$+$ Swift	1.75	4.26	3.69	21.54	0.81	0.61	1.59	4.30	3.57	19.40	0.80	0.62
VAE: SD-VAE (EMA) $+$ reg	5.49	5.52	11.08	87.84	0.71	0.66	2.32	4.70	7.39	32.52	0.80	0.60
$+$ Swift	3.77	5.06	8.44	57.83	0.73	0.66	1.81	4.42	5.93	26.15	0.80	0.61

Table 1: Representative ImageNet-256 results. Each pair uses the same model configuration and sampler. Guidance is tuned separately for re-evaluated pairs; source-reported guided baselines retain their published settings ( Section 10.2 ).

\overline{\mathrm{FD}}_{r}

and

\overline{\mathrm{MIND}}

are arithmetic means across Inception, ConvNeXt, DINOv2, MAE, and SigLIP features.

Figure 3: ImageNet-256 samples from Swift on RAEv2/DINOv3-L (gFID-50K $0.99$ ). Samples are generated with the Swift -trained model using retuned internal guidance and a 50-step ODE sampler ( Section 10.2 ).

Panel of Figure 4 — Figure 4: Swift remains beneficial with inference-time guidance. (a) FID with autoguidance on the DiffusionBench models and native internal REPA-head guidance on RAEv2. (b) On RAEv2, Swift shifts the preferred internal-guidance scale from $1.78$ to $1.20$ .

Panel of Figure 5 — Figure 5: Training-compute comparison on DINOv2-B. Swift surpasses the best baseline FID observed through 320 epochs using approximately $0.45\times$ the FLOPs needed to reach that baseline checkpoint.

Panel of Figure 6 — Figure 6: Choosing $\alpha$ and the weak guide on CIFAR-10. Moderate positive $\alpha$ , a half-width guide, and a sufficiently trained weak guide give the strongest results. The CIFAR-10 training and evaluation settings are in Sections 10.1 and 10.2 .

Method	Guidance reference	No unconditional prediction	No extra denoising head	One forward pass per prediction
CFG [ 9 ]	unconditional prediction (same model)	$\times$	$\checkmark$	$\times$
AG [ 11 ]	frozen weak model (same conditioning)	$\checkmark$	$\checkmark$	$\times$
MG [ 28 ]	conditional and unconditional EMA predictions	$\times$	$\checkmark$	$\checkmark$
SGG (train) [ 36 ]	unconditional or intermediate prediction, depending on noise level	$\times$	$\times$	$\checkmark$
IG (main) [ 39 ]	supervised intermediate prediction	$\checkmark$	$\times$	$\checkmark$
IG (training) [ 39 ]	final and intermediate EMA predictions	$\checkmark$	$\times$	$\checkmark$

Table 2: Training components and sampling. Checks indicate the stated property. Unconditional predictions omit the class label; extra heads are intermediate outputs with their own denoising loss. One pass means one backbone evaluation per sampling prediction, including both IG outputs. SGG is its conditional training variant. Optional backbone components and additional guidance are excluded.

Arm	Pointwise loss	Estimate of $\mathcal{D}^{\star}$
baseline	$\\|\mathcal{D}_{\theta}-\bm{x}_{0}\\|_{2}^{2}$	$\mathcal{D}_{\theta}$
Swift	$\\|\mathcal{D}_{\theta}-(1+\alpha)\bm{x}_{0}+\alpha\mathcal{D}_{\mathrm{w}}\\|_{2}^{2}$	$(\mathcal{D}_{\theta}+\alpha\mathcal{D}_{\mathrm{w}})/(1+\alpha)$
MG [ 28 ]	$\\|\mathcal{D}_{\theta}-\bm{x}_{0}-w(\mathcal{D}_{\mathrm{c}}-\mathcal{D}_{\mathrm{u}})\\|_{2}^{2}$	$\mathcal{D}_{\theta}-w(\mathcal{D}_{\mathrm{c}}-\mathcal{D}_{\mathrm{u}})$
GFT [ 2 ]	$\\|\beta\mathcal{D}_{\theta}+(1-\beta)\operatorname{sg}[\mathcal{D}_{\mathrm{u}}]-\bm{x}_{0}\\|_{2}^{2}$	$\beta\mathcal{D}_{\theta}+(1-\beta)\mathcal{D}_{\mathrm{u}}$
IG (aux. loss) [ 39 ]	$\\|\mathcal{D}_{\mathrm{f}}-\bm{x}_{0}\\|_{2}^{2}+\gamma\\|\mathcal{D}_{\mathrm{i}}-\bm{x}_{0}\\|_{2}^{2}$	$\mathcal{D}_{\mathrm{f}}$

Table 3: Pointwise training losses and evaluated estimates. The expectation over training examples and noise is implicit. Here

\mathcal{D}_{\mathrm{w}}

is the frozen class-A weak guide,

\mathcal{D}_{\mathrm{c}}

the frozen class-A baseline, and

\mathcal{D}_{\mathrm{u}}

the frozen A

+

B denoiser;

\operatorname{sg}

denotes stop-gradient.

$d=16$ (40 seeds)
Method	$\sigma=0.05$	$0.1$	$0.2$	$0.5$	$1.0$
baseline	0.3100	0.2600	0.2311	0.2125	0.2272
Swift	0.2566	0.2175	0.1952	0.1865	0.1988
MG	0.3179	0.2685	0.2388	0.2216	0.2358
GFT	0.2617	0.2203	0.1972	0.1904	0.2075
IG	0.3418	0.2885	0.2522	0.2258	0.2312

Table 4: Distance to the class-A posterior mean. For each method and dimension, its best setting is selected by the lowest arithmetic mean distance across the five noise levels; the whole row uses that one setting. Means are over independent seeds; all arms share the main backbone. Lower is better; bold marks the best displayed entry at each noise level.

	CIFAR-10 (EDM)	ImageNet-256 (DiffusionBench)	ImageNet-256 (RAEv2)
resolution / space	$32\times 32$ , pixels	$256\times 256$ , latents or pixels	$256\times 256$ , DINOv3-L latents
sampler	EDM deterministic (Heun) [ 10 ]	Euler [ 16 ]	Euler [ 26 , App. A, Table 11]
solver steps	18	50	50
guidance	none	CFG, tuned independently	internal guidance, tuned independently
generated samples	50k	50k	50k
reference statistics	CIFAR-10 training set	ImageNet-256 training set	ImageNet-256 training set

Table 5: Sampling and evaluation settings used for our runs. Within a re-evaluated pair, the baseline and Swift use the same sampler, generated-sample count, and metric implementation. CIFAR-10’s 18 Heun steps use 35 denoiser evaluations; inference guidance may add network passes to the other models’ stated solver steps.

Representation autoencoders (RAE); 615M, 80 epochs
						FD r $\downarrow$					MIND $\downarrow$
Configuration (baseline)	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	Inception	ConvNeXt	DINOv2	MAE	SigLIP	Inception	ConvNeXt	DINOv2	MAE	SigLIP
DINOv2-B	2.09	4.48	210.6	0.83	0.57	1.32	2.34	3.19	6.25	7.57	2.45	70.70	26.72	0.44	6.41
$+$ Swift	1.75	4.26	202.8	0.81	0.61	1.09	2.43	2.87	5.18	6.89	1.55	76.44	23.60	0.35	5.76
DINOv2-B $+$ reg	2.07	4.55	208.2	0.82	0.58	1.29	2.42	3.23	6.33	7.67	2.27	77.41	27.21	0.44	6.21
$+$ Swift	1.71	4.18	205.2	0.81	0.61	1.09	2.40	2.90	5.29	6.75	1.56	74.47	23.36	0.35	5.79

Table 6: All 19 ImageNet-256 configurations without guidance. Swift uses the same default coefficient (

\alpha{=}0.4

) with a smaller pretrained guide frozen before main-model training, without per-configuration coefficient tuning. FD r and MIND are reported in five feature spaces; the lower FID of each pair is bold.

Representation autoencoders (RAE); 615M, 80 epochs
						FD r $\downarrow$					MIND $\downarrow$
Configuration (baseline)	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	Inception	ConvNeXt	DINOv2	MAE	SigLIP	Inception	ConvNeXt	DINOv2	MAE	SigLIP
DINOv2-B	1.98	4.50	223.1	0.81	0.59	1.26	2.21	3.40	6.12	7.84	1.91	66.75	27.95	0.43	6.52
$+$ Swift	1.59	4.30	221.6	0.80	0.62	1.00	2.27	2.78	5.14	6.65	1.15	66.61	23.23	0.33	5.70
DINOv2-B $+$ reg	1.84	4.55	236.2	0.82	0.58	1.16	2.16	3.24	6.37	7.96	1.91	64.08	27.03	0.43	6.01
$+$ Swift	1.52	4.18	232.9	0.81	0.61	0.98	2.10	2.73	5.31	6.57	1.31	62.96	22.34	0.35	5.92

Table 7: All 19 ImageNet-256 configurations with guidance. Inference-time guidance is tuned separately for re-evaluated pairs; source-reported guided baselines retain their published settings. The training recipe is otherwise the same as in Table 6 . FD r and MIND are reported in five feature spaces; the lower FID of each pair is bold.

Panel of Figure 7 — Figure 7: Guided FID at each model’s best available CFG scale for six DiffusionBench configurations from Table 7 . Open circles denote matched baselines whose guidance scale is swept independently; open diamonds denote source-reported guided baselines. Filled dots show Swift at its own best scale, and annotations give the FID change. The preferred scale can shift substantially (e.g., $2.2\!\to\!1.8$ for SD-VAE $+$ reg and $2.4\!\to\!1.8$ for PixNerd), motivating separate tuning on the two sides.

Model	Type	Base FID	Guided FID	Guidance
DiT-XL/2 [ 24 ]	latent diffusion	9.62	2.27	CFG
SiT-XL/2 [ 20 ]	latent diffusion	8.30	2.06	CFG
MDT-XL/2 [ 5 ]	latent diffusion	6.23	1.79	CFG
DDT-XL/2 [ 30 ]	latent diffusion	6.27	1.26	CFG, int.
REPA [ 35 ]	latent diffusion	5.90	1.42	CFG, int.
MAR-H [ 17 ]	autoregressive	2.35	1.55	CFG

Table 8: Published ImageNet-256 results shown in Figure 1 (left). FID-50K values are taken directly from the corresponding papers. Evaluation procedures differ between papers: some sample 50 images per class, others sample class labels at random, which can change FID by about

0.1

[ 38 , Table 14] . Guidance: CFG, classifier-free guidance; int., with a guidance interval; AG, autoguidance; IG, internal guidance [ 39 ] ; REPA guid., the REPA guidance of Singh et al. [26] . ∗ Without CFG, but still with internal guidance. Our two RAEv2 rows use the settings of Section 11.3 .

Panel of Figure 8 — Figure 8: RAEv2 unguided training trajectories. The dense Swift traces are used to study training dynamics; the canonical RAEv2 results are reported in Tables 6 and 7 . The diamond marks the released epoch-80 checkpoint.

Panel of Figure 9 — Figure 9: The same fixed Swift recipe improves four different ImageNet-256 training trajectories. Each panel shows a different DiffusionBench stage-2 model and representation. Grey is the matched $\alpha{=}0$ baseline and blue is Swift with the same fixed recipe ( $\alpha{=}0.4$ , half-width guide pretrained for half the standard main-model duration and then frozen), with no per-model tuning. Swift is ahead by epoch 40 in all four cases and remains ahead through the end of training, across baselines spanning roughly an order of magnitude in FID. The $y$ -axis is logarithmic; faint markers denote individual FID evaluations. The trajectory baselines reproduce the corresponding DiffusionBench endpoints within $0.6$ – $6\%$ ; the comparisons in Table 6 instead re-evaluate both sides with 50k samples.

Epoch	Baseline	Swift	$\Delta$	Epoch	Baseline	Swift	$\Delta$
20	3.63	3.52	$-0.12$	140	1.81	1.51	$-0.30$
40	2.67	2.32	$-0.36$	160	1.80	1.46	$-0.34$
60	2.36	2.00	$-0.36$	180	1.75	1.49	$-0.26$
80	2.13	1.77	$-0.37$	200	1.72	1.44	$-0.28$
100	1.99	1.60	$-0.39$	220	1.67	1.41	$-0.26$
120	1.91	1.50	$-0.41$	240	1.63	1.38	$-0.26$

Table 9: DINOv2-B over 240 matched training epochs. Unguided FID from the training-time evaluator, sampled every 20 epochs. The two blocks list epochs in order;

\Delta

is Swift minus baseline, computed before rounding the FIDs displayed here. Figure 10 plots the same 20-epoch checkpoints.

Panel of Figure 10 — Figure 10: DINOv2-B training trajectories through epoch 240. The matched unguided FID curves are plotted every 20 epochs, using the rounded values in Table 9 . Swift remains below the baseline throughout the displayed schedule.

Method	Best FID
Single-stage guide-capacity switch	1.501
Time profile: late peak	1.507
Combined time/noise coefficients	1.549
Recorded guide ahead of student training age	1.557
Time profile: late hold	1.559
Multi-stage guide-capacity schedule	1.561

Table 10: Additional methods and their best FIDs. Each entry is the lowest FID at the common 200k endpoint across all completed settings and runs of that method, rather than a run average. Online adaptive rules are preliminary.

Figure 11: Samples from Swift on RAEv2/DINOv3-L: birds and mammals. Internal guidance, 50-step Euler ODE sampler; 28 samples per class.

Figure 12: Samples from Swift on RAEv2/DINOv3-L: marine life. Internal guidance, 50-step Euler ODE sampler; 28 samples per class.

Figure 13: Samples from Swift on RAEv2/DINOv3-L: vehicles, objects, and bread. Internal guidance, 50-step Euler ODE sampler; 28 samples per class.

Figure 14: Samples from Swift on RAEv2/DINOv3-L: fruit, food, and plants. Internal guidance, 50-step Euler ODE sampler; 28 samples per class.

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,169 papers

Sep 6, 2026

SwiftExplorer: Training-free Diffusion Model Alignment with Swift Diversity Exploration

Diffusion models have general generative abilities but struggle to align with specific objectives. Fine-tuning can improve alignment, yet its training cost is often prohibitive. This led to training-free methods that apply objective-guided terms in sampling to bias the generation distribution toward designated regions, e.g., high-reward areas. However, these methods face two issues: (1) the strong directional bias narrows the pretrained distribution and generation diversity, and (2) indiscriminate constant guidance fails to prune redundant signals, hurting both quality and efficiency. To address the above challenges, we propose SwiftExplorer, a plugin that mitigates distribution collapse caused by excessive diversity loss and reduces compute costs. First, we adopt an Inheritance-Restart exploration mechanism to avoid early convergence, while exploration also increases the likelihood of high-reward trajectories. Additionally, it balances diversity and fidelity, adding diversity without causing a distribution over-shift. Second, our Quality-Efficiency arbitration mechanism improves guidance by removing incorrect signals, and it reduces computation by dynamically stopping generation when completeness and marginal reward gain are optimal. In an extensive number of experiments and different types of evaluation metrics, the proposed SwiftExplorer achieves excellent performance on all metrics, including preference, fidelity, diversity, and richness.

Renye Yan, Jikang Cheng, You Wu +5

Peking University Beijing, China · Nanjing University Nanjing, China · Huazhong Agricultural University Wuhan, China +1

Diffusion Model Alignment Diffusion Model Guidance Diffusion Guidance Diffusion Sampling Diffusion Model Inference Acceleration

Ask PDF

Mar 21, 2026

Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

Diffusion models generate synthetic images through an iterative refinement process. However, the misalignment between the simulation-free objective and the iterative process often causes accumulated gradient error along the sampling trajectory, which leads to unsatisfactory results and a failure to generalize. Guidance techniques like Classifier Free Guidance (CFG) and AutoGuidance (AG) alleviate this by extrapolating between the main and inferior signal for stronger generalization. Despite empirical success, the effective operational regimes of prevalent guidance methods are still under-explored, leading to ambiguity when selecting the appropriate guidance method given a precondition. In this work, we first conduct synthetic comparisons to isolate and demonstrate the effective regime of guidance methods represented by CFG and AG from the perspective of weak-to-strong principle. Based on this, we propose a hybrid instantiation called SGG under the principle, taking the benefits of both. Furthermore, we demonstrate that the W2S principle along with SGG can be migrated into the training objective, improving the generalization ability of unguided diffusion models. We validate our approach with comprehensive experiments. At inference time, evaluations on SD3 and SD3.5 confirm that SGG outperforms existing training-free guidance variants. Training-time experiments on transformer architectures demonstrate the effective migration and performance gains in both conditional and unconditional settings. Code is available at https://github.com/851695e35/SGG.

Liangyu Yuan, Yufei Huang, Mingkun Lei +5

Westlake University · Tongji University · Zhejiang University +1

Diffusion Model Guidance Diffusion Models Diffusion Guidance Classifier-Free Guidance

Ask PDF

Feb 11, 2026

General and Efficient Steering of Diffusion Models

Steering diffusion models toward conditions unseen during training typically requires either retraining with conditional inputs or per-step gradient computations, both of which incur substantial computational overhead. We present Noise-Aligned RFM Steering (NA-RFM), a general recipe for efficiently steering diffusion models without gradient guidance during inference, enabling fast controllable generation. The method combines two offline-computed signals: noise alignment, a high-noise correction from PCA statistics of the target examples and the full data, and Recursive Feature Machine (RFM) activation steering, which learns a target-discriminative direction from labeled forward-process activations. During sampling, noise alignment provides coarse control at high noise, while the RFM direction is reused over intermediate/late timesteps through lightweight activation edits. Experiments on CIFAR-10, ImageNet, CelebA, and fine-grained bird species show improved target accuracy over gradient-based post-hoc guidance baselines, improved FID on the class-guidance benchmarks, and substantial inference speedups. Code: https://github.com/isotrivial/na-rfm.

Qingsong Wang, Mikhail Belkin, Yusu Wang

Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA

Diffusion Model Guidance Activation Steering Controllable Image Generation Diffusion Model Inference Acceleration Diffusion-Based Denoising

Ask PDF

Abstract

Diffusion Models

Explore similar work

SwiftExplorer: Training-free Diffusion Model Alignment with Swift Diversity Exploration

Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

General and Efficient Steering of Diffusion Models