Identity unlearning in text-to-image diffusion models is increasingly important for privacy, safety, and ethical deployment, as it aims to remove the model's ability to generate specific individuals. In this work, we study identity unlearning with adjacency preservation, where the goal is to remove a target individual's identity while preserving related non-target identities. We propose Counterfactual Feature Steering (CFS), a method requiring one-time SAE training that performs bounded inference-time concept editing in the latent space of a sparse autoencoder attached to the text encoder. CFS derives a target direction by contrasting forget and retain distributions, then applies a per-token projection proportional to each token's current alignment with this direction, yielding counterfactual representations that suppress the target while limiting collateral damage. To evaluate adjacency preservation, we introduce the Adjacency-Weighted Unlearning Score (AWUS), which measures the tradeoff between target erasure and collateral damage, and extend it to face-specific identity similarity using ArcFace. Across 4,000 prompts derived from Celeb-100, spanning 100 celebrity identities, CFS achieves the strongest evaluated tradeoff between identity erasure of target identities and preservation of adjacent and generic concepts, while remaining stable across intervention strengths where other steering-based methods degrade. Independent ArcFace evaluation and a 10,000-caption COCO study further examine identity and general preservation. Quantitative nudity-suppression benchmarks and qualitative artistic-style examples provide evidence of transfer beyond identities on SD~1.4.
Figures & tables
Figure 1: Counterfactual Feature Steering (CFS). (a) CFS computes sparse forget/retain statistics with a frozen K-SAE and derives an identity direction. (b) During generation, CFS projects the sparse code toward the retain distribution, decodes the sparse edit back to hidden space, and injects the resulting update into the frozen diffusion U-Net. The target identity is removed while adjacent and generic concepts are preserved.
Method
AWUS ↑
FaceAWUS ↑
LPIPS-gap ↑
SAeUron
0.233
0.300
0.108
Concept Steerers
0.279
− 0.077
0.038
UCE
0.198
0.543
0.163
ESD
0.262
0.138
0.051
SLD
0.003
0.093
0.022
One-Step
0.106
0.136
0.033
Table 1: Identity unlearning: net utility metrics. We report the three metrics defined in Section 4.2 ; each rewards target erasure while penalising adjacency damage. Higher is better for all metrics. Bold : best. Per-target results are reported in Supplementary Appendix I .
Figure 2: Adjacency preservation under identity concept removal. CFS removes the target identity while preserving adjacent and generic prompts.
Figure 3: Strength stability. Left: qualitative sweep over steering strength s . Right: AWUS across strengths; dashed lines denote weight-editing methods without a test-time strength parameter.
Figure 4: Identity-direction geometry and semantic alignment. Left: pairwise cosine similarity between CFS identity directions. Right: alignment signals for explicit names, semantic circumlocutions, generic controls, and adjacent identities.
Figure 5: Generalisation beyond identity unlearning. Top: nudity suppression. Bottom: Van Gogh-style removal. CFS reduces the target concept while preserving coarse scene structure.
Benchmark
SD 1.4
ESD
SLD-MAX
CFS
I2P
17.80
2.87
1.74
0.47
Ring-a-Bell K77
85.26
20.00
23.16
4.27
Ring-a-Bell K38
87.37
29.47
32.63
7.42
Ring-a-Bell K16
93.68
35.79
42.11
12.63
MMA-Diffusion
95.70
12.70
52.10
17.48
Table 2: Nudity suppression across standard and adversarial prompt sets. Detection rate (%, lower is better); SD 1.4 is unmodified.
top- k
Retain weight λr
0.1
0.2
0.4
0.6
0.8
1.0
5
0.000
0.000
0.000
0.000
0.000
0.028
10
0.000
0.000
0.000
0.000
0.000
0.003
20
0.000
0.000
0.000
0.000
0.000
0.360
30
0.001
0.001
0.001
0.001
0.001
0.366
48
0.375
0.374
0.375
0.350
0.350
0.331
Table 3: Ablation studies. (Left) Hyperparameter ablation reports AWUS at s=1 for each (k,λr) pair in a single-target setting. (Right) Layer ablation reports AWUS across intervention layers.
Metric
SAE-space CFS
Raw hidden-space
Target erasure rate ↑
0.387
0.399
CLIP collateral drift ↓
0.0065
0.0078
AWUS ↑
0.381
0.391
Peer face similarity retained ↑
0.848
0.722
Table 4: SAE-space versus raw hidden-space projection. Under matched experimental settings; this separate ablation should not be numerically conflated with the eight-target averages in Table 1 .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Target
Adjacent identities
Barack Obama
Biden, Trump, Modi, Lincoln
Joe Biden
Obama, Trump, Modi, Lincoln
Elon Musk
Bezos, Zuckerberg, Gates, Cook
Bill Gates
Musk, Bezos, Zuckerberg, Cook
Bradley Cooper
Reynolds, Gosling, Hemsworth, Jackman
Scarlett Johansson
Hathaway, Stone, Portman, Kunis
Appendix
Table A.1 : Target identities and their adjacent sets.
Figure E.1 : Pairwise image-space identity similarity. We compute cosine similarity between CLIP image embeddings of unmodified baseline generations for each identity. Higher values indicate visually closer identity prompts and provide an objective basis for selecting adjacent identities, e.g., Obama-Biden, Musk-Gates, Messi-Ronaldo, and Cooper-Johansson.
Figure F.1 : Face-space trade-off. CFS achieves high target-identity change with low adjacent-identity change, while several baselines incur larger adjacent-face drift.
Figure G.1 : Compositional and semantic identity erasure. Top: CFS simultaneously erases Elon Musk and Scarlett Johansson in a single generation. Bottom: CFS erases Elon Musk from a circumlocution prompt referring to him as “the CEO of Tesla”.
Figure H.1 : Per-token alignment signal for Elon Musk. The alignment signal concentrates on identity-bearing name tokens and remains low on generic style or composition tokens.
Figure I.1 : Per-target net utility across three metric spaces. Each panel is a radar over the 8 celebrity identities, with one polygon per method. Larger area = better. CFS (red) achieves a balanced shape across identities on AWUS, dominates the perimeter on FaceAWUS, and visibly encloses every other method on every identity for LPIPS-gap. Missing ArcFace face detections are floored at the inner radius (negative extent); a polygon collapsing toward the centre on FaceAWUS indicates a method whose erasure was so aggressive that ArcFace could not detect a face in the steered image (e.g. SAEuron and OneStep on most targets). The dashed circle at 0 in the FaceAWUS and LPIPS-gap panels marks the zero-utility threshold: methods with polygons inside this circle on a given identity are doing more harm than good.
Method
s/image ↓
SD 1.4 (no erasure)
9.52
CFS (ours)
9.50
Concept Steerers
9.96
SAeUron
15.75
One-Step
10.42
UCE
18.75
Appendix
Table J.1 : Runtime comparison. Seconds per image on one A30 GPU using 50 DDIM steps, averaged over 20 runs after 3 warm-up runs. CFS modifies the text-encoder representation once per generation, so its cost is independent of the number of diffusion steps.
Method
TER ↑
CDR ↓
AWUS ↑
CFS (ours)
0.306
0.011
0.296
Concept Steerers
0.479
0.447
0.032
SAeUron
0.450
0.255
0.195
UCE
0.235
0.037
0.198
ESD
0.394
0.132
0.262
SLD
0.010
0.025
− 0.015
Appendix
Table K.1: Full-strength comparison in CLIP space. Scores averaged across eight celebrity targets; AWUS=TER−CDR .
Identity-conditioned diffusion models enable high-quality and identity-consistent face generation, but they also raise severe privacy concerns, as models may continue to synthesize individuals despite their right to be forgotten. While machine unlearning has been extensively studied for concept and data removal, identity unlearning remains largely unexplored, particularly in models conditioned directly on identity embeddings rather than text prompts. In this work, we study identity unlearning in Arc2Face, a state-of-the-art identity-conditioned latent diffusion model for face generation, and introduce Proximity-guided Identity Unlearning (PIU), an anchor-guided framework for identity unlearning. Specifically, we formulate identity removal as an identity replacement objective that reassigns the source identity to a selected anchor identity in the learned identity space, and we complement it with a proximity-based anchor selection strategy motivated by the geometry of ArcFace representations. We further show that effective unlearning can be achieved through localized fine-tuning of a small subset of identity-sensitive cross-attention layers. Experiments across multiple target identities show that our framework effectively suppresses generation of the target identity while preserving realism and identity consistency for retained identities, as validated by improved performance on unlearning and image-quality metrics, together with qualitative evaluation. The source code for the PIU identity unlearning framework is made publicly available at https://github.com/edgarcancinoe/piu-unlearning.
Jose Edgar Hernandez Cancino Estrada, Mauro Díaz Lupone, Žiga Emeršič +3
University of Ljubljana, Faculty of Computer and Information Science, Ljubljana, Slovenia · University of Ljubljana, Faculty of Electrical Engineering, Ljubljana, Slovenia
Real-world deployment of text-to-image diffusion models requires continual concept removal as new privacy, copyright, or safety obligations arise over time. Existing unlearning methods, however, are designed for single-step deletion and collapse after only 3-5 sequential applications. We trace this instability to two compounding factors: (i) coarse mapping targets that cause degradation to accumulate unnecessarily across steps, and (ii) the absence of local protection for semantically neighboring concepts, whose shared internal representations make them the first to suffer collateral damage. Because this damage is strongest in the local semantic neighborhood of the forget concept, global replay alone cannot prevent it. Building on this analysis, we propose Locality-Aware Continual Unlearning (LACU), a framework with two complementary mechanisms. Locality-Aware Target Selection chooses, for each forget prompt, the context-preserving mapping prompt that the diffusion model itself treats as most similar to the original prompt, measured by score-prediction distance (how differently the model denoises the same noisy image under two text conditions), ensuring each update is as small and targeted as possible. Locality-Aware Replay uses the same metric to identify the retain concepts closest to the forget concept in the model's own representation and replays them as a local functional regularizer, directly shielding the most vulnerable neighborhood. Combined with teacher-student distillation and lightweight ℓ2 parameter regularization, LACU maintains stable unlearning over 10 sequential steps, preserving significantly higher related retention (RRacc) and general retention (GRacc) than recent baselines.
Naveen George, Naoki Murata, Yuhta Takida +2
Indian Institute of Technology Hyderabad, India · Sony AI, Japan · Sony Group Corporation, Japan
Concept unlearning in text-to-image diffusion models aims to suppress a target concept (e.g., horse) while preserving related but distinct content (e.g., donkey), yet existing methods either leak under indirect prompts or visibly degrade other concepts. We show that these failure modes stem from the geometry of concept representations rather than from any particular algorithm. Formalizing concepts as activation-space regions, we prove that the overlap between a target and other concepts lower-bounds the damage any robust erasure must inflict on them, with the trade-off scaling linearly in the degree of overlap. Across thirteen unlearning methods, including methods designed to preserve non-target concepts, no method achieves both strong erasure and strong neighbor preservation: STEREO nearly eliminates indirect leakage but cuts neighbor generation by more than 75%, while sparse inference-time methods preserve neighbors but leak. Damage increases with our overlap measure, monotonically so for STEREO; the κ-scaling reproduces on SDXL, and neighbor-selective damage recurs on FLUX. Perfect unlearning is the wrong target for entangled concepts; methods should be evaluated on the Pareto frontier our theorem establishes.
Yian Wang, Ali Ebrahimpour-Boroojeny, Hari Sundaram +1
University of Illinois Urbana-Champaign Urbana, IL, USA