Specificity-Aware Diffusion Steering via Variance-Reduced Sequential Monte Carlo

Luran Wang; Linrui Ma; Hannes Stärk; Regina Barzilay

cs.LGSep 30, 2026

Specificity-Aware Diffusion Steering via Variance-Reduced Sequential Monte Carlo

Authors: Luran Wang, Linrui Ma, Hannes Stärk, Regina Barzilay

Organizations: CSAIL, Massachusetts Institute of Technology

Abstract

Inference-time steering enables pretrained diffusion models to satisfy new constraints without full retraining. However, specificity-aware generation is difficult: repelling samples from a negative reference distribution can also erode the positive distribution where the two overlap. The key challenge is to suppress negative mass while minimally distorting the positive distribution. We address this problem by formulating specificity-aware steering as a target-design problem and deriving a target distribution from an overlap-based objective. The resulting target keeps the desired reference distribution only in regions where it is sufficiently preferred over the undesired reference distribution, giving a likelihood-ratio interpretation of specificity. To sample from the corresponding time-dependent target path, we develop a Sequential Monte Carlo sampler with a variance-minimized local proposal. We further introduce a practical fixed-noise optimization procedure with the Jacobian--vector products with the desired and undesired score fields. Experiments on synthetic task, class-contrastive generation, text-to-image tasks and peptide-MHC (p-MHC) binder show that the proposed method suppresses undesired regions more effectively, reduces mode shift, and improves sampling stability by decreasing the SMC weight collapse compared with negative-guidance baselines. Code is available at: https://github.com/WangLuran/Specificity-Aware-Diffusion-Steering

Figures & tables

Panel of Figure 1 — Figure 1 : Comparison of target distributions in two one-dimensional examples. Blue and orange denote the desired density $p^{A}$ and undesired density $p^{B}$ , respectively; green denotes the induced target. The proposed target preserves regions where $p^{A}$ dominates $p^{B}$ and removes regions with high undesired density. By contrast, DNG can shift mass away from desired modes, while the negative-prompt (NP) construction does not consistently suppress the undesired region.

Figure 2 : Visualization on the 40-MoG synthetic task. The task setting contains $A$ -only target modes, $B$ -only negative modes, and $A\cap B$ suppressed modes. The target distribution retains only the $A$ -only modes.

Panel of Figure 3 — Figure 3 : CIFAR-10 specificity Pareto frontiers with $N=10{,}240$ samples. Cols. 1–2 show class removal (§ 5.2.1 ); Cols. 3–4 show multiclass selection (§ 5.2.2 ). Bottom row: FID versus Wrong%. Top row: KL versus Wrong% for class removal and Recall versus Wrong% for multiclass selection. † Safe Denoiser is omitted because its Wrong% does not reach the plotted region across multiple kernel bandwidths; see App. D .

	Related Negative			Unrelated Negative			Overall
Method	CLIP +	CLIP -	Margin	CLIP +	CLIP -	Margin	Margin
Ours (Proposition 2)	0.2722	0.1413	0.1309	0.2679	0.0655	0.2024	0.1666
CFG	0.2475	0.1473	0.1002	0.2561	0.0632	0.1928	0.1465
DNG	0.2758	0.1766	0.0992	0.2748	0.0812	0.1937	0.1465
FKC	0.2788	0.1652	0.1136	0.2796	0.0822	0.1974	0.1555
FK steering	0.2695	0.1758	0.0937	0.2727	0.0843	0.1884	0.1411

Table 1 : Text-to-image specificity with SD v1.4 and

100

reverse steps, averaged over the 10 DNG prompt pairs. For each prompt and method, a single hyperparameter setting is selected and used for both related and unrelated negatives. CLIP + is similarity to the positive prompt (

\uparrow

), CLIP - is similarity to the active negative prompt (

\downarrow

), and margin is CLIP

{}_{+}-

CLIP - (

\uparrow

Figure 4 : Samples for “medieval feast.” The first row uses proposal optimization and the second uses direct SMC. Proposal optimization substantially reduces visual collapse.

Method	$\boldsymbol{N}$	Mean $\uparrow$	Median $\uparrow$	Q75 $\uparrow$	$\boldsymbol{\Delta>0}$ $\uparrow$
BoltzGen (target A only)	64	0.1786	0.1587	0.2862	87.5%
Fixed guidance	64	0.1654	0.1453	0.2833	93.8%
DNG	64	0.1483	0.1581	0.2623	89.1%
Ours (Proposition 2)	64	0.1856	0.1822	0.3337	84.4%

Table 2 : Specificity on the pMHC binder-design task. Here,

N

is the number of evaluated binders, Q75 is the

75

th percentile, and

\Delta>0

is the fraction of binders with a positive specificity margin. Higher values are better.

Panel of Figure 5 — Figure 5 : Representative predicted complexes for a specificity-selective binder. The binder is shown in green, and the peptide is shown in red. The remaining chains form the shared HLA-A*02:01– $\beta_{2}$ m scaffold. The desired target supports a peptide-adjacent binder pose, whereas the displayed prediction for the undesired P4F target places the binder away from the peptide.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6 : Wrong %–KL Pareto for the forward multiclass-selection task (keep $\{0,1,2\}$ ). The method ordering is consistent with the Recall plot in Fig. 3 . The reverse direction exhibits the same method ordering on FID and Recall but not on KL: with only three keep classes the histogram is too coarse to discriminate adaptive methods (DNG, SMC) from uniform-strength baselines (Composition, SLD) on class balance alone.

No.	Scene	Positive prompt	Related negative	Unrelated negative
1	Medieval feast	A grand medieval feast set in a great hall, filled with long tables covered in bountiful platters of food, goblets, and flickering light, with knights and nobles enjoying the lavish spread.	Chalices, candles	Kids playing football
2	English breakfast	A classic British breakfast, featuring a diverse selection of cooked delights, perfect for a filling and flavorful start to the day.	Egg, sausage	The view of a skyline
3	Dinner table	A beautifully set dinner table, with elegant arrangements, a variety of enticing dishes, and delicate decor that speaks to the sophistication of the meal.	Flowers, wine glasses	A person wearing sunglasses
4	Art workshop	An inspiring art workshop filled with creativity, featuring a wide range of tools and materials used by participants working on their individual projects.	Paintbrush, canvases	A bicycle in the rain
5	Antique store	A charming antique store with shelves and tables filled with unique and historical treasures, offering a glimpse into the past.	Lamps, old books	An electric toothbrush

Appendix

Table 3 : Text-to-image prompts.

Proposition 2 surrogate	Pearson $\uparrow$	Centered MAE $\downarrow$	Centered rel. RMSE $\downarrow$	Normalized-ESS MAE $\downarrow$
Without JVP	$0.689$	$2.094$	$0.783$	$0.115$
With JVP	$\mathbf{0.804}$	$\mathbf{1.633}$	$\mathbf{0.681}$	$\mathbf{0.107}$

Appendix

Table 4 : Agreement of the raw Proposition 2 surrogate with the exact finite one-step Gaussian log-weight on seed

10{,}000

. Metrics are computed after particle-wise centering within each non-degenerate step.

Proposal	$\boldsymbol{\mathrm{Margin}}\uparrow$	$\boldsymbol{R_{\mathrm{eff}}}\uparrow$	Unique roots $\uparrow$	$\boldsymbol{F_{\max}}\downarrow$	Resamples $\downarrow$
Fixed $\rho_{2}=0$	$0.1294\pm 0.0130$	$3.49\pm 1.41$	$4.20\pm 1.64$	$0.425\pm 0.143$	$1.80\pm 0.84$
Full-ESS optimized $\rho_{2}$	$\mathbf{0.1363\pm 0.0080}$	$\mathbf{5.07\pm 1.74}$	$\mathbf{5.80\pm 1.30}$	$\mathbf{0.300\pm 0.112}$	$\mathbf{0.80\pm 0.45}$

Appendix

Table 5 : Effect of full-ESS optimization of

\rho_{2}

on CLIP specificity and particle genealogy. Values are mean

\pm

standard deviation over five paired seeds when 8 images generated.

Figure 7 : Qualitative samples from our method for the medieval-feast prompt with the related negative prompt.

Figure 8 : Qualitative samples from our method for the English-breakfast prompt with the related negative prompt.

Figure 9 : Qualitative samples from our method for the dinner-table prompt with the related negative prompt.

Figure 10 : Qualitative samples from our method for the art-workshop prompt with the related negative prompt.

Figure 11 : Qualitative samples from our method for the antique-store prompt with the related negative prompt.

Method	$\boldsymbol{N}$	Mean	Median	Q75	$\boldsymbol{\Delta>0}$
BoltzGen (target A only)	64	0.1786	0.1587	0.2862	87.5%
Fixed guidance	64	0.1654	0.1453	0.2833	93.8%
DNG	64	0.1483	0.1581	0.2623	89.1%
Ours (Proposition 2)	64	0.1856	0.1822	0.3337	84.4%

Appendix

Table 6: Specificity on the HLA-A02:01– NLVPMVATV versus HLA-A02:01– NLVFMVATV binder-design task. Here,

N

is the number of evaluated binders and

K

is the number of Boltz-2 predictions per binder and target condition. All methods use

K=3

; higher values are better.

Explore similar work

General and Efficient Steering of Diffusion Models
Feb 11, 2026Qingsong Wang, Mikhail Belkin, Yusu WangDiffusion Models
Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
Inference-Time Alignment of Diffusion Models via Trust-Region Iterative Twisted Sequential Monte Carlo
May 24, 2026Weixin Wang, Yu Yang, Wei Deng +1Diffusion Alignment Diffusion Models
Duke University · Morgan Stanley
Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion
May 22, 2026Jaihoon Kim, Taehoon Yoon, Prin Phunyaphibarn +3Diffusion Models Monte Carlo
1KAIST · University of Michigan · 3NVIDIA

Feb 11, 2026, cs.LGJ/K move · Enter open · S save

General and Efficient Steering of Diffusion Models

Qingsong Wang, Mikhail Belkin, Yusu Wang

Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA

Diffusion Models

Open paper AskPDF arXiv

Steering diffusion models toward conditions unseen during training typically requires either retraining with conditional inputs or per-step gradient computations, both of which incur substantial computational overhead. We present Noise-Aligned RFM Steering (NA-RFM), a general recipe for efficiently steering diffusion models without gradient guidance during inference, enabling fast controllable generation. The method combines two offline-computed signals: noise alignment, a high-noise correction from PCA statistics of the target examples and the full data, and Recursive Feature Machine (RFM) activation steering, which learns a target-discriminative direction from labeled forward-process activations. During sampling, noise alignment provides coarse control at high noise, while the RFM direction is reused over intermediate/late timesteps through lightweight activation edits. Experiments on CIFAR-10, ImageNet, CelebA, and fine-grained bird species show improved target accuracy over gradient-based post-hoc guidance baselines, improved FID on the class-guidance benchmarks, and substantial inference speedups. Code: https://github.com/isotrivial/na-rfm.

Abstract

Explore similar work

General and Efficient Steering of Diffusion Models

Inference-Time Alignment of Diffusion Models via Trust-Region Iterative Twisted Sequential Monte Carlo

Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion