Unlearning the Unpromptable: Prompt-free Instance Unlearning in Diffusion Models

Kyungryeol Lee; Kyeonghyun Lee; Seongmin Hong; Byung Hyun Lee; Se Young Chun

cs.LGMar 11, 2026

Unlearning the Unpromptable: Prompt-free Instance Unlearning in Diffusion Models

Authors: Kyungryeol Lee, Kyeonghyun Lee, Seongmin Hong, Byung Hyun Lee, Se Young Chun

Organizations: Department of Electrical and Computer Engineering (ECE), Seoul National University (SNU), Seoul, 08826, Republic of Korea · Institute of New Media and Communications (INMC), SNU, Seoul, 08826, Republic of Korea · ECE, INMC, IPAI, SNU, Seoul, 08826, Republic of Korea

Abstract

Machine unlearning aims to remove specific outputs from trained models, often at the concept level, such as forgetting all occurrences of a particular celebrity or filtering content via text prompts. However, many undesired outputs, such as an individual's face or generations culturally or factually misinterpreted, cannot often be specified by text prompts. We address this underexplored setting of instance unlearning for outputs that are undesired but unpromptable, where the goal is to forget target outputs selectively while preserving the rest. To this end, we introduce an effective surrogate-based unlearning method that leverages image editing, timestep-aware weighting, and gradient surgery to guide trained diffusion models toward forgetting specific outputs. Experiments on conditional (Stable Diffusion 3) and unconditional (DDPM-CelebA) diffusion models demonstrate that our prompt-free method uniquely unlearns unpromptable outputs, such as faces and culturally inaccurate depictions, with preserved integrity, unlike prompt-based and prompt-free baselines. Our proposed method would serve as a practical hotfix for diffusion model providers to ensure privacy protection and ethical compliance.

Figures & tables

Panel of Figure 1 — Fig. 1 : Challenge and our solution for instance unlearning in diffusion models.

Fig. 2 : Cultural and semantic misrepresentation highlight the need for instance unlearning in commercial generative models.

Panel of Figure 3 — Fig. 3 : Comparison between (a) prompt-based and (b) instance unlearning.

Fig. 4 : Surrogate-based unlearning (

\theta^{\dagger}

) can be better than exact unlearing (

\tilde{\theta}

) in mapping preservation, i.e. , the line is closer to the original (

\theta^{*}

Panel of Figure 5 — Fig. 5 : Surrogate data construction of (left) CelebA with TediGAN, (middle) SD3 with SDEdit, and (right) manual editing.

Fig. 6 : Overview of SSCD to validate the success of forgetting.

Panel of Figure 7 — Fig. 7 : Qualitative results on (a) forgetting target instance on (b) preserving model integrity on generation after unlearning.

	Single Instance					Multiple Instances
	SSCD $<$ 0.4	LPIPS $\downarrow$	SSIM $\uparrow$	$\text{FID}_{pre}$ $\downarrow$	$\text{FID}_{real}$ $\downarrow$	SSCD $<$ 0.4	LPIPS $\downarrow$	SSIM $\uparrow$	$\text{FID}_{pre}$ $\downarrow$	$\text{FID}_{real}$ $\downarrow$
NegGrad [ 39 ]	✓(0.18)	1.00	0.34	223.5	226.1	✓(0.14)	0.96	0.07	415.4	410.8
EraseDiff [ 40 ]	✓(0.03)	0.74	0.61	80.7	87.8	✓(0.04)	0.66	0.71	45.8	51.1
SISS [ 41 ]	✓(0.21)	0.43	0.84	15.7	21.3	✓(0.22)	0.51	0.81	26.8	31.5
Ours †	✓(0.17)	0.41	0.84	15.7	24.1	✓(0.16)	0.36	0.80	13.9	27.2
Ours	✓(0.19)	0.35	0.87	9.8	18.0	✓(0.21)	0.37	0.88	17.5	23.4

TABLE I : Quantitative results on forgetting single instance and multiple instances. Ours † refers to randomly sampling the surrogate data

x_{0}^{s}

from the remember dataset

\mathcal{D}_{r}

at each update.

Panel of Figure 9 — Fig. 8: Comparison of instance unlearning methods on (top) forgetting target instance and (bottom) preserving related instances.

Panel of Figure 10 — Fig. 9 : Edited images (surrogates) with varying CLIP loss weights of TediGAN.

Panel of Figure 11 — Fig. 10 : Trade-off between forgetting (SSCD) and model integrity (SSIM, LPIPS). We can select how the surrogate is close to the original instance by adjusting CLIP loss weight during editing.

CLIP weight	SSCD	LPIPS $\downarrow$	SSIM $\uparrow$	FID ${}_{\text{pre}}$ $\downarrow$	FID ${}_{\text{real}}$ $\downarrow$
0.5	0.434	0.305	0.900	8.08	16.25
1.0	0.403	0.304	0.901	8.04	16.33
2.0	0.330	0.307	0.900	8.10	16.62
4.0	0.302	0.315	0.896	8.34	17.09
8.0	0.278	0.325	0.891	8.80	17.62
16.0	0.228	0.338	0.885	8.94	17.48

TABLE II : Quantitative results with varying CLIP loss weights of TediGAN.

Condition	SSCD	LPIPS $\downarrow$	SSIM $\uparrow$	$\text{FID}_{pre}$ $\downarrow$	$\text{FID}_{real}$ $\downarrow$
$\lambda=\text{0.95}$	0.279	0.174	0.897	7.82	24.06
$\lambda=\text{0.99}$	0.289	0.173	0.894	7.18	20.31
$\lambda=\text{1}\!-\!\text{2}\!\cdot\!\text{10}^{-\text{5}}t$	0.356	0.213	0.920	6.06	15.43
$\lambda=\text{1}\!-\!\text{5}\!\cdot\!\text{10}^{-\text{5}}t$	0.355	0.213	0.920	6.06	15.43
$\lambda=\text{1}\!-\!\text{10}^{-\text{4}}t$	0.311	0.235	0.910	6.59	15.58

TABLE III : Ablation on timestep scheduling of

\lambda

Condition	SSCD	LPIPS $\downarrow$	SSIM $\uparrow$	$\text{FID}_{pre}$ $\downarrow$	$\text{FID}_{real}$ $\downarrow$
no projection	0.309	0.275	0.899	7.68	14.72
project $g_{f}$	0.309	0.274	0.900	7.65	14.70
project $g_{f}$ , $g_{r}$	0.310	0.276	0.901	8.15	16.47

TABLE IV : Ablation on gradient projection.

Surrogate method	SSCD $<$ 0.4	LPIPS $\downarrow$	SSIM $\uparrow$
Edit (TediGAN)	✓(0.35)	0.21	0.92
Flip	✓(0.40)	0.31	0.85
Add noise ( $\sigma=50$ )	✗ (0.54)	0.23	0.91
Add noise ( $\sigma=150$ )	✓(0.32)	0.28	0.89

TABLE V : Ablation on surrogate construction methods.

Panel of Figure 16 — Fig. 11: Qualitative comparison of surrogates. $\sigma=50$ and $\sigma=150$ denote adding Gaussian noise with standard deviations of 50 and 150 to the target image.

	SSCD	LPIPS $\downarrow$	SSIM $\uparrow$	$\text{FID}_{pre}\downarrow$	$\text{FID}_{real}\downarrow$
Ours (InD)	0.34	0.30	0.90	8.7	16.6
Ours (OoD)	0.32	0.31	0.90	9.0	17.1

TABLE VI : Single instance unlearning for InD and OoD data.

Diffusion Model Unlearning

Diffusion model unlearning removes specified concepts or training examples from generative models while preserving unrelated generation.

See all 77 papers

Concept Unlearning via Cross-Attention Activation Projection for Diffusion Models
May 25, 2026Saemi Moon, Suhyeon Jun, Seoyeon Lee +1Attention Control in Diffusion Models Diffusion Model Unlearning
CSE, POSTECH · GSAI, POSTECH
Why Instruction-Based Unlearning Fails in Diffusion Models?
Apr 2, 2026Zeliang Zhang, Rui Sun, Jiani Liu +2Diffusion Model Unlearning Machine Unlearning
University of Rochester · UCLA · UCSB
Forgetting is Competition: Rethinking Unlearning as Representation Interference in Diffusion Models
Mar 1, 2026Ashutosh Ranjan, Vivek Srivastava, Shirish Karande +1Diffusion Model Unlearning Machine Unlearning
TCS Research, India · Kalinga Institute of Industrial Technology

May 25, 2026J/K move · Enter open · S save

Concept Unlearning via Cross-Attention Activation Projection for Diffusion Models

Saemi Moon, Suhyeon Jun, Seoyeon Lee, Dongwoo Kim

CSE, POSTECH · GSAI, POSTECH

Attention Control in Diffusion Models Diffusion Model Unlearning Machine Unlearning T2I Diffusion Models Diffusion Models

Open paper AskPDF arXiv

Existing closed-form methods for concept unlearning in text-to-image diffusion models typically derive editing directions from fixed text embeddings, which may not fully capture how concepts are expressed across latent states, timesteps, and layers. To capture this variation, we investigate cross-attention activations collected during denoising. In controlled probing experiments using the same anchor prompts, activation-derived bases achieve approximately five times the recall of text-derived bases on held-out prompts expressing the target concepts. Based on this finding, we propose Cross-Attention Subspace Erasure (CASE), a closed-form method that constructs layer-specific forget and retain subspaces from cross-attention activations. These subspaces define a retain-constrained linear operator incorporated directly into cross-attention weights, requiring no gradient-based fine-tuning or additional inference-time computation. Across ten concepts spanning four categories, CASE achieves the highest harmonic-mean score among evaluated baselines in all four categories, balancing suppression, retention, adversarial robustness, and generation quality. Further experiments demonstrate robustness to recovery attacks and a favorable suppression-retention trade-off when jointly unlearning up to 100 artistic styles. The benefits of activation-derived editing also extend to larger diffusion models, including SDXL and FLUX.

Abstract

Diffusion Model Unlearning

Explore similar work

Concept Unlearning via Cross-Attention Activation Projection for Diffusion Models

Why Instruction-Based Unlearning Fails in Diffusion Models?

Forgetting is Competition: Rethinking Unlearning as Representation Interference in Diffusion Models