Unlearning the Unpromptable: Prompt-free Instance Unlearning in Diffusion Models
Authors: Kyungryeol Lee, Kyeonghyun Lee, Seongmin Hong, Byung Hyun Lee, Se Young Chun
Organizations: Department of Electrical and Computer Engineering (ECE), Seoul National University (SNU), Seoul, 08826, Republic of Korea · Institute of New Media and Communications (INMC), SNU, Seoul, 08826, Republic of Korea · ECE, INMC, IPAI, SNU, Seoul, 08826, Republic of Korea
Machine unlearning aims to remove specific outputs from trained models, often at the concept level, such as forgetting all occurrences of a particular celebrity or filtering content via text prompts. However, many undesired outputs, such as an individual's face or generations culturally or factually misinterpreted, cannot often be specified by text prompts. We address this underexplored setting of instance unlearning for outputs that are undesired but unpromptable, where the goal is to forget target outputs selectively while preserving the rest. To this end, we introduce an effective surrogate-based unlearning method that leverages image editing, timestep-aware weighting, and gradient surgery to guide trained diffusion models toward forgetting specific outputs. Experiments on conditional (Stable Diffusion 3) and unconditional (DDPM-CelebA) diffusion models demonstrate that our prompt-free method uniquely unlearns unpromptable outputs, such as faces and culturally inaccurate depictions, with preserved integrity, unlike prompt-based and prompt-free baselines. Our proposed method would serve as a practical hotfix for diffusion model providers to ensure privacy protection and ethical compliance.
Figures & tables
Fig. 1 : Challenge and our solution for instance unlearning in diffusion models.
Fig. 2 : Cultural and semantic misrepresentation highlight the need for instance unlearning in commercial generative models.
Fig. 3 : Comparison between (a) prompt-based and (b) instance unlearning.
Fig. 4 : Surrogate-based unlearning ( θ† ) can be better than exact unlearing ( θ~ ) in mapping preservation, i.e. , the line is closer to the original ( θ∗ ).
Fig. 5 : Surrogate data construction of (left) CelebA with TediGAN, (middle) SD3 with SDEdit, and (right) manual editing.
Fig. 6 : Overview of SSCD to validate the success of forgetting.
Fig. 7 : Qualitative results on (a) forgetting target instance on (b) preserving model integrity on generation after unlearning.
Single Instance
Multiple Instances
SSCD < 0.4
LPIPS ↓
SSIM ↑
FIDpre↓
FIDreal↓
SSCD < 0.4
LPIPS ↓
SSIM ↑
FIDpre↓
FIDreal↓
NegGrad [ 39 ]
✓(0.18)
1.00
0.34
223.5
226.1
✓(0.14)
0.96
0.07
415.4
410.8
EraseDiff [ 40 ]
✓(0.03)
0.74
0.61
80.7
87.8
✓(0.04)
0.66
0.71
45.8
51.1
SISS [ 41 ]
✓(0.21)
0.43
0.84
15.7
21.3
✓(0.22)
0.51
0.81
26.8
31.5
Ours †
✓(0.17)
0.41
0.84
15.7
24.1
✓(0.16)
0.36
0.80
13.9
27.2
Ours
✓(0.19)
0.35
0.87
9.8
18.0
✓(0.21)
0.37
0.88
17.5
23.4
TABLE I : Quantitative results on forgetting single instance and multiple instances. Ours † refers to randomly sampling the surrogate data x0s from the remember dataset Dr at each update.
Fig. 8: Comparison of instance unlearning methods on (top) forgetting target instance and (bottom) preserving related instances.
Fig. 9 : Edited images (surrogates) with varying CLIP loss weights of TediGAN.
Fig. 10 : Trade-off between forgetting (SSCD) and model integrity (SSIM, LPIPS). We can select how the surrogate is close to the original instance by adjusting CLIP loss weight during editing.
CLIP weight
SSCD
LPIPS ↓
SSIM ↑
FID pre↓
FID real↓
0.5
0.434
0.305
0.900
8.08
16.25
1.0
0.403
0.304
0.901
8.04
16.33
2.0
0.330
0.307
0.900
8.10
16.62
4.0
0.302
0.315
0.896
8.34
17.09
8.0
0.278
0.325
0.891
8.80
17.62
16.0
0.228
0.338
0.885
8.94
17.48
TABLE II : Quantitative results with varying CLIP loss weights of TediGAN.
Condition
SSCD
LPIPS ↓
SSIM ↑
FIDpre↓
FIDreal↓
λ=0.95
0.279
0.174
0.897
7.82
24.06
λ=0.99
0.289
0.173
0.894
7.18
20.31
λ=1−2⋅10−5t
0.356
0.213
0.920
6.06
15.43
λ=1−5⋅10−5t
0.355
0.213
0.920
6.06
15.43
λ=1−10−4t
0.311
0.235
0.910
6.59
15.58
TABLE III : Ablation on timestep scheduling of λ .
Condition
SSCD
LPIPS ↓
SSIM ↑
FIDpre↓
FIDreal↓
no projection
0.309
0.275
0.899
7.68
14.72
project gf
0.309
0.274
0.900
7.65
14.70
project gf , gr
0.310
0.276
0.901
8.15
16.47
TABLE IV : Ablation on gradient projection.
Surrogate method
SSCD < 0.4
LPIPS ↓
SSIM ↑
Edit (TediGAN)
✓(0.35)
0.21
0.92
Flip
✓(0.40)
0.31
0.85
Add noise ( σ=50 )
✗ (0.54)
0.23
0.91
Add noise ( σ=150 )
✓(0.32)
0.28
0.89
TABLE V : Ablation on surrogate construction methods.
Fig. 11: Qualitative comparison of surrogates. σ=50 and σ=150 denote adding Gaussian noise with standard deviations of 50 and 150 to the target image.
SSCD
LPIPS ↓
SSIM ↑
FIDpre↓
FIDreal↓
Ours (InD)
0.34
0.30
0.90
8.7
16.6
Ours (OoD)
0.32
0.31
0.90
9.0
17.1
TABLE VI : Single instance unlearning for InD and OoD data.
Existing closed-form methods for concept unlearning in text-to-image diffusion models typically derive editing directions from fixed text embeddings, which may not fully capture how concepts are expressed across latent states, timesteps, and layers. To capture this variation, we investigate cross-attention activations collected during denoising. In controlled probing experiments using the same anchor prompts, activation-derived bases achieve approximately five times the recall of text-derived bases on held-out prompts expressing the target concepts. Based on this finding, we propose Cross-Attention Subspace Erasure (CASE), a closed-form method that constructs layer-specific forget and retain subspaces from cross-attention activations. These subspaces define a retain-constrained linear operator incorporated directly into cross-attention weights, requiring no gradient-based fine-tuning or additional inference-time computation. Across ten concepts spanning four categories, CASE achieves the highest harmonic-mean score among evaluated baselines in all four categories, balancing suppression, retention, adversarial robustness, and generation quality. Further experiments demonstrate robustness to recovery attacks and a favorable suppression-retention trade-off when jointly unlearning up to 100 artistic styles. The benefits of activation-derived editing also extend to larger diffusion models, including SDXL and FLUX.
Instruction-based unlearning has proven effective for modifying the behavior of large language models at inference time, but whether this paradigm extends to other generative models remains unclear. In this work, we investigate instruction-based unlearning in diffusion-based image generation models and show, through controlled experiments across multiple concepts and prompt variants, that diffusion models systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions. By analyzing both the CLIP text encoder and cross-attention dynamics during the denoising process, we find that unlearning instructions do not induce sustained reductions in attention to the targeted concept tokens, causing the targeted concept representations to persist throughout generation. These results reveal a fundamental limitation of prompt-level instruction in diffusion models and suggest that effective unlearning requires interventions beyond inference-time language control.
Deployed text-to-image diffusion models increasingly require post-hoc concept unlearning for copyright claims, artist opt-outs, safety updates, and protected-content mitigation without full retraining. A central challenge is erase-retain imbalance, aggressive updates suppress targets but damage shared capabilities, while conservative or anchor-based updates preserve quality yet leave concepts recoverable through related, compositional, paraphrased, or adversarial prompts. Inspired by retroactive interference, we propose SurgUn, which treats forgetting as controlled competition rather than direct deletion or one-to-one reassignment. SurgUn instantiates retroactive concept interference via distractor-conditioned gradient competition: target-gradient ascent weakens target-conditioned denoising or flow-matching behavior, while descent over a semantically diverse distractor set introduces competing non-target trajectories under the same prompt context. This redistributes outputs across multiple non-target modes instead of collapsing to a single proxy. To limit collateral forgetting through shared pathways, SurgUn adds pixel-grounded weight-space localization, a lightweight diagnostic that selects attention blocks by generated-image erase-retain behavior, exploiting the asymmetry that suppression is broadly achievable whereas retention is block-selective. Across UnlearnCanvas, IP-character erasure, Holistic Unlearning, EraseBench, and Ring-A-Bell on Stable Diffusion v1.5, SDXL, and SANA-1.5, SurgUn achieves a stronger erase-retain balance than baselines. Ablations show that diverse distractors, contrastive competition, and localization are all necessary for robust suppression while preserving related and unrelated concepts.