Negative guidance of diffusion models generates samples that avoid an undesired condition, specified either by reference samples or by a conditional score alongside an unconditional model. Existing methods inject repulsive terms derived from local information at each sample, so the resulting sampler does not target an explicit density, and its bias cannot be systematically reduced by increasing computation. We instead present a negative-guidance method that samples from an explicit target density, which uniformly handles both reference-sample and score-based specifications and suppresses probability mass near the undesired condition with an avoidance strength that parametrizes the target density itself. Realizing this target requires estimating, at each diffusion timestep, how likely each sample is to belong to the undesired condition. We cast this estimation as a Positive-Unlabeled learning problem that admits lightweight online training along the diffusion trajectory, using condition samples as positives and diffusion particles as unlabeled data. Because the guided particles drift from the unconditional marginal, we further correct the residual mismatch via Sequential Monte Carlo, yielding a sampler that is asymptotically exact in the particle limit and that turns any existing negative-guidance method into its proposal kernel. Empirically, our method approaches the ground-truth target on Gaussian mixtures and Pareto-dominates existing methods on trade-offs between avoidance and generation quality, distributional bias, and diversity, including in an image-generation setting with a pretrained score network.
Figures & tables
Figure 1: Generated samples on one trial. Each point is a sample in its first two coordinates, colored by the true posterior p(c∣x)∈[0,1] (red: p(c∣x)=1 , blue: 0 ). Oracle and Ours avoid the condition c and cover all remaining modes. NegCFG suppresses leak by collapsing onto a few remaining modes, and DNG keeps coverage but retains residual samples on the avoided modes.
Figure 2: Avoidance rate vs. generation quality on CIFAR-10. Top: plane. Bottom: four-class vehicles. Columns plot target-class rate against FID, KL, and average pairwise LPIPS (left to right); lower is better except for LPIPS. The Safe Denoiser sweeps η . Ours uses the Safe Denoiser as its proposal kernel, fixes η∈{0.2,0.3,0.5} , and sweeps λ , tracing the Pareto frontier in all panels.
Figure 3: Results of Van Gogh style avoidance on Stable Diffusion v1.4. Left: generated examples of (a) the avoided style and (b) ours at M=128 , λ=16 , each showing 16 randomly selected images. These are the top two rows of the random 8×8 grids in Figure 6 of Appendix D.3 , which also shows the other methods. Right: avoided-style rate against FID; lower left is better. The baselines sweep their strengths, and ours uses SLD as its proposal kernel, reaching the oracle-floor rate with the lowest FID while keeping the prompted scene. Full results are in Appendix D.3 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Four GMM layouts drawn at random. Each panel projects 3000 unconditional samples onto the xy -plane. Red points are samples from the avoided component density p(⋅∣c) and blue points are samples from p(⋅∣¬c) .
η
Leak (mean) ↓
W2↓
Mode coverage ↑
0.0 (uncond)
0.253±0.016
3.6±0.6
1.00
0.1
0.193±0.014
2.7±0.4
1.00
0.2
0.150±0.013
3.5±0.7
1.00
0.5
0.068±0.008
9.7±2.4
0.99
0.7
0.041±0.007
14.1±3.8
0.97
1.0
0.018±0.005
20.5±6.3
0.94
Appendix
Table 2: NegCFG η sweep, 50 trials, mean ± std. Mode coverage =P[mink∈¬c∥x−μk∥<5] .
Table 4: Timestep-count ablation for DNG on the Gaussian mixture, mean ± std over 15 trials, at (λ0,τ)=(1,1) and M=1024 . The Oracle is recomputed at each T , and Gap is the difference between the two leak columns.
M
Oracle leak
Ours leak ↓
Oracle W2
Ours W2↓
ESS /M
Resamples
64
0.011
0.009±0.014
3.6
3.5±0.7
0.96
3.1
256
0.014
0.021±0.028
2.5
2.1±0.4
0.96
2.7
1024
0.012
0.008±0.006
1.5
1.9±0.6
0.97
2.2
Appendix
Table 5: Particle-count ablation on the Gaussian mixture with true scores, mean ± std over 15 trials. The Oracle is recomputed at each M . The row at M=1024 differs from Table 1 , which averages over 50 trials.
Class
Method
η
Rate (%) ↓
FID ↓
KL ↓
LPIPS ↑
Plane
Safe Denoiser
0.3
4.78
20.75
0.127
0.200
Safe Denoiser
0.5
2.95
45.96
0.279
0.175
Ours, λreg=10−3
0.3
3.50
24.65
0.148
0.197
Ours, λreg=1.0
0.3
4.39
23.66
0.136
0.198
Cat
Safe Denoiser
0.3
7.16
20.63
0.150
0.207
Safe Denoiser
0.5
6.10
44.53
0.282
0.183
Appendix
Table 6: PU regularization ablation at λ=1.5 , 80 seeds per cell. Our method is run at η=0.3 , while the Safe Denoiser baseline is shown at η∈{0.3,0.5} , the two values that bracket the knee of its rate–FID curve in Figure 2 and Figure 5 .
Figure 5: Avoidance rate vs. generation quality on CIFAR-10 for the cat (top), frog (middle), and ship (bottom) single-class conditions. Columns plot target-class rate against FID, KL, and average pairwise LPIPS (left to right); lower is better except for LPIPS.
Class
λ
ESS (time avg.)
ESS (min)
Ancestors /128
Resamples
Plane
0.5
126.5±0.6
59.6±22.7
105.4±19.0
1.3±1.4
Plane
1.0
125.9±0.8
43.9±25.5
85.2±27.0
2.9±2.2
Plane
2.0
125.1±0.9
37.5±24.1
66.6±28.2
4.7±3.0
Vehicles
0.5
121.9±1.8
72.5±18.0
121.1±9.2
0.4±0.5
Vehicles
1.0
116.4±2.8
33.2±18.6
94.0±14.1
2.3±1.3
Vehicles
2.0
110.4±3.1
10.1±7.1
56.2±18.4
6.4±2.3
Appendix
Table 7: ESS diagnostics on CIFAR-10 at η=0.3 , M=128 , T=250 , mean ± std over 80 runs. Ancestors counts the distinct ancestors among the final 128 particles.
Method
Plane
Vehicles
Safe Denoiser
4.78±1.99
27.60±3.98
Ours ( λ=0.5 )
3.88±2.31
25.86±5.27
Ours ( λ=1.0 )
3.28±2.66
19.87±6.09
Ours ( λ=2.0 )
2.54±3.01
20.93±18.06
Appendix
Table 8: Target-class rate [%] per 128 -sample batch on CIFAR-10 at η=0.3 , mean ± std over 80 seeds.
Figure 6: Randomly selected 8×8 image grids, from Table 9 : PU Steering at M=128 , λ=16 read as the equal-weight output after resampling, SLD at xxmax , and NegCFG at η=3 , its lowest-FID strength.
Method
Config
Rate ↓
FID ↓
KID ↓
Style-FD ↓
ESS /M
Oracle floor
2048 vs. 2048
0.238
18.52
0.0002
0.009
—
Content (no avoidance)
0.409
60.98
0.0552
0.213
—
Prompted Van Gogh
0.645
164.32
0.1982
0.661
—
SLD
max
0.334
43.08
0.0294
0.114
—
SLD
xxmax
0.317
41.82
0.0263
0.118
—
SLD
x3max
0.296
43.70
0.0284
0.180
—
Appendix
Table 9: Stable Diffusion v1.4 avoiding Van Gogh style, 2048 images per configuration, lower is better throughout. Gray rows are anchors rather than competing methods. For PU Steering, “u.” evaluates the plain images and “w.” the SMC-weighted estimate, ESS /M is the mean per-run terminal effective sample size, and λ is the terminal tempering strength. Bold marks the best FID, KID, and Style-FD among the methods outside the no-resampling ablation. Representative baseline strengths are listed; Figure 7 plots every configuration of the sweeps.
Figure 7: Stable Diffusion, avoided-style rate against FID (left) and Style-FD (right); lower left is better in both panels. Every configuration of Table 9 is plotted, including the two no-resampling points, which are the rightmost M=64 markers.