Negative guidance of diffusion models generates samples that avoid an undesired condition, specified either by reference samples or by a conditional score alongside an unconditional model. Existing methods inject repulsive terms derived from local information at each sample, so the resulting sampler does not target an explicit density, and its bias cannot be systematically reduced by increasing computation. We instead present a negative-guidance method that samples from an explicit target density, which uniformly handles both reference-sample and score-based specifications and suppresses probability mass near the undesired condition with an avoidance strength that parametrizes the target density itself. Realizing this target requires estimating, at each diffusion timestep, how likely each sample is to belong to the undesired condition. We cast this estimation as a Positive-Unlabeled learning problem that admits lightweight online training along the diffusion trajectory, using condition samples as positives and diffusion particles as unlabeled data. Because the guided particles drift from the unconditional marginal, we further correct the residual mismatch via Sequential Monte Carlo, yielding a sampler that is asymptotically exact in the particle limit and that turns any existing negative-guidance method into its proposal kernel. Empirically, our method approaches the ground-truth target on Gaussian mixtures and Pareto-dominates existing methods on trade-offs between avoidance and generation quality, distributional bias, and diversity, including in an image-generation setting with a pretrained score network.
Figures & tables
Figure 1: Generated samples on one trial. Each point is a sample in its first two coordinates, colored by the true posterior p(c∣x)∈[0,1] (red: p(c∣x)=1 , blue: 0 ). Oracle and Ours avoid the condition c and cover all remaining modes. NegCFG suppresses leak by collapsing onto a few remaining modes, and DNG keeps coverage but retains residual samples on the avoided modes.
Figure 2: Avoidance rate vs. generation quality on CIFAR-10. Top: plane. Bottom: four-class vehicles. Columns plot target-class rate against FID, KL, and average pairwise LPIPS (left to right); lower is better except for LPIPS. The Safe Denoiser sweeps η . Ours uses the Safe Denoiser as its proposal kernel, fixes η∈{0.2,0.3,0.5} , and sweeps λ , tracing the Pareto frontier in all panels.
Figure 3: Results of Van Gogh style avoidance on Stable Diffusion v1.4. Left: generated examples of (a) the avoided style and (b) ours at M=128 , λ=16 , each showing 16 randomly selected images. These are the top two rows of the random 8×8 grids in Figure 6 of Appendix D.3 , which also shows the other methods. Right: avoided-style rate against FID; lower left is better. The baselines sweep their strengths, and ours uses SLD as its proposal kernel, reaching the oracle-floor rate with the lowest FID while keeping the prompted scene. Full results are in Appendix D.3 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Four GMM layouts drawn at random. Each panel projects 3000 unconditional samples onto the xy -plane. Red points are samples from the avoided component density p(⋅∣c) and blue points are samples from p(⋅∣¬c) .
η
Leak (mean) ↓
W2↓
Mode coverage ↑
0.0 (uncond)
0.253±0.016
3.6±0.6
1.00
0.1
0.193±0.014
2.7±0.4
1.00
0.2
0.150±0.013
3.5±0.7
1.00
0.5
0.068±0.008
9.7±2.4
0.99
0.7
0.041±0.007
14.1±3.8
0.97
1.0
0.018±0.005
20.5±6.3
0.94
Appendix
Table 2: NegCFG η sweep, 50 trials, mean ± std. Mode coverage =P[mink∈¬c∥x−μk∥<5] .
Table 4: Timestep-count ablation for DNG on the Gaussian mixture, mean ± std over 15 trials, at (λ0,τ)=(1,1) and M=1024 . The Oracle is recomputed at each T , and Gap is the difference between the two leak columns.
M
Oracle leak
Ours leak ↓
Oracle W2
Ours W2↓
ESS /M
Resamples
64
0.011
0.009±0.014
3.6
3.5±0.7
0.96
3.1
256
0.014
0.021±0.028
2.5
2.1±0.4
0.96
2.7
1024
0.012
0.008±0.006
1.5
1.9±0.6
0.97
2.2
Appendix
Table 5: Particle-count ablation on the Gaussian mixture with true scores, mean ± std over 15 trials. The Oracle is recomputed at each M . The row at M=1024 differs from Table 1 , which averages over 50 trials.
Class
Method
η
Rate (%) ↓
FID ↓
KL ↓
LPIPS ↑
Plane
Safe Denoiser
0.3
4.78
20.75
0.127
0.200
Safe Denoiser
0.5
2.95
45.96
0.279
0.175
Ours, λreg=10−3
0.3
3.50
24.65
0.148
0.197
Ours, λreg=1.0
0.3
4.39
23.66
0.136
0.198
Cat
Safe Denoiser
0.3
7.16
20.63
0.150
0.207
Safe Denoiser
0.5
6.10
44.53
0.282
0.183
Appendix
Table 6: PU regularization ablation at λ=1.5 , 80 seeds per cell. Our method is run at η=0.3 , while the Safe Denoiser baseline is shown at η∈{0.3,0.5} , the two values that bracket the knee of its rate–FID curve in Figure 2 and Figure 5 .
Figure 5: Avoidance rate vs. generation quality on CIFAR-10 for the cat (top), frog (middle), and ship (bottom) single-class conditions. Columns plot target-class rate against FID, KL, and average pairwise LPIPS (left to right); lower is better except for LPIPS.
Class
λ
ESS (time avg.)
ESS (min)
Ancestors /128
Resamples
Plane
0.5
126.5±0.6
59.6±22.7
105.4±19.0
1.3±1.4
Plane
1.0
125.9±0.8
43.9±25.5
85.2±27.0
2.9±2.2
Plane
2.0
125.1±0.9
37.5±24.1
66.6±28.2
4.7±3.0
Vehicles
0.5
121.9±1.8
72.5±18.0
121.1±9.2
0.4±0.5
Vehicles
1.0
116.4±2.8
33.2±18.6
94.0±14.1
2.3±1.3
Vehicles
2.0
110.4±3.1
10.1±7.1
56.2±18.4
6.4±2.3
Appendix
Table 7: ESS diagnostics on CIFAR-10 at η=0.3 , M=128 , T=250 , mean ± std over 80 runs. Ancestors counts the distinct ancestors among the final 128 particles.
Method
Plane
Vehicles
Safe Denoiser
4.78±1.99
27.60±3.98
Ours ( λ=0.5 )
3.88±2.31
25.86±5.27
Ours ( λ=1.0 )
3.28±2.66
19.87±6.09
Ours ( λ=2.0 )
2.54±3.01
20.93±18.06
Appendix
Table 8: Target-class rate [%] per 128 -sample batch on CIFAR-10 at η=0.3 , mean ± std over 80 seeds.
Figure 6: Randomly selected 8×8 image grids, from Table 9 : PU Steering at M=128 , λ=16 read as the equal-weight output after resampling, SLD at xxmax , and NegCFG at η=3 , its lowest-FID strength.
Method
Config
Rate ↓
FID ↓
KID ↓
Style-FD ↓
ESS /M
Oracle floor
2048 vs. 2048
0.238
18.52
0.0002
0.009
—
Content (no avoidance)
0.409
60.98
0.0552
0.213
—
Prompted Van Gogh
0.645
164.32
0.1982
0.661
—
SLD
max
0.334
43.08
0.0294
0.114
—
SLD
xxmax
0.317
41.82
0.0263
0.118
—
SLD
x3max
0.296
43.70
0.0284
0.180
—
Appendix
Table 9: Stable Diffusion v1.4 avoiding Van Gogh style, 2048 images per configuration, lower is better throughout. Gray rows are anchors rather than competing methods. For PU Steering, “u.” evaluates the plain images and “w.” the SMC-weighted estimate, ESS /M is the mean per-run terminal effective sample size, and λ is the terminal tempering strength. Bold marks the best FID, KID, and Style-FD among the methods outside the no-resampling ablation. Representative baseline strengths are listed; Figure 7 plots every configuration of the sweeps.
Figure 7: Stable Diffusion, avoided-style rate against FID (left) and Style-FD (right); lower left is better in both panels. Every configuration of Table 9 is plotted, including the two no-resampling points, which are the rightmost M=64 markers.
Inference-time steering enables pretrained diffusion models to satisfy new constraints without full retraining. However, specificity-aware generation is difficult: repelling samples from a negative reference distribution can also erode the positive distribution where the two overlap. The key challenge is to suppress negative mass while minimally distorting the positive distribution. We address this problem by formulating specificity-aware steering as a target-design problem and deriving a target distribution from an overlap-based objective. The resulting target keeps the desired reference distribution only in regions where it is sufficiently preferred over the undesired reference distribution, giving a likelihood-ratio interpretation of specificity. To sample from the corresponding time-dependent target path, we develop a Sequential Monte Carlo sampler with a variance-minimized local proposal. We further introduce a practical fixed-noise optimization procedure with the Jacobian--vector products with the desired and undesired score fields. Experiments on synthetic task, class-contrastive generation, text-to-image tasks and peptide-MHC (p-MHC) binder show that the proposed method suppresses undesired regions more effectively, reduces mode shift, and improves sampling stability by decreasing the SMC weight collapse compared with negative-guidance baselines. Code is available at: https://github.com/WangLuran/Specificity-Aware-Diffusion-Steering
Diffusion models have emerged as state-of-the-art generative models for high-fidelity image synthesis, particularly in their classifier-free guided and classifier-guided forms. However, standard classifier guidance concentrates probability mass around high-density class mean, leading to poor coverage of rare samples in the tails of the class-conditional distributions. Recent work on diffusion-based tail sampling mitigates this by training an additional low-density-seeking classifier with a synthetic-vs-real discriminator, at the cost of additional networks and training. In parallel, a number of samplers and distillation techniques accelerate or refine diffusion sampling, but do not explicitly address long-tail coverage. We propose a purely sampling-time, density-aware extension of classifier-guided conditional diffusion model that targets low-density regions without any additional training. We have applied guidance at noisy images not on predicted noise like most diffusion models. Starting from a pretrained conditional diffusion model and classifier on ImageNet, we modify the guided reverse dynamics by steering trajectories toward low-confidence regions via the modified classifier gradient, and at each time step, we also guide the sampling process toward the predicted real image. 1st guidance helps explore low-probability samples, and 2nd guidance helps to generate samples to be close to the real data manifold. The proposed sampler consistently improves ADM model recall at 64x64 resolution while maintaining a comparable FID, and with a 256x256 ADM model, we showed the results visually with different combinations of both guidance. We also showed that standard ADM classifier guidance, combined with predicted real image guidance, helps generate high perceptual quality samples with a 256x256 ADM model on ImageNet.
Stochastic differential equation (SDE)-based generative models have achieved substantial progress in conditional generation via training-free differentiable loss-guided approaches. However, existing methodologies utilizing posterior sam- pling typically confront a substantial estimation error, which results in inaccu- rate gradients for guidance and leading to inconsistent generation results. To mitigate this issue, we propose that performing an additional backward denois- ing step and Monte-Carlo sampling (ABMS) can achieve better guided diffu- sion, which is a plug-and-play adjustment strategy. To verify the effectiveness of our method, we provide theoretical analysis and propose the adoption of a dual-focus evaluation framework, which further serves to highlight the critical problem of cross-condition interference prevalent in existing approaches. We conduct experiments across various task settings and data types, mainly includ- ing conditional online handwritten trajectory generation, image inverse problems (inpainting, super resolution and gaussian deblurring) molecular inverse design and so on. Experimental results demonstrate that our approach can be effec- tively used with higher order samplers and consistently improves the quality of generation samples across all the different scenarios.
Minsi Ren, Wenhao Deng, Ruiqi Feng +1
AI for Scientific Simulation and Discovery Lab, Westlake University