cs.CVNov 16, 2026

Counterfactual Feature Steering for Adjacency-Preserving Identity Unlearning

Authors: FNU Shivam, Layth Sliman, Ramana Reddy, Donald Adjeroh, Binod Bhattarai, Prashnna Gyawali

Organizations: West Virginia University, Morgantown, WV, USA · EFREI Paris, Villejuif, France · University of Aberdeen, Aberdeen, United Kingdom

Abstract

Identity unlearning in text-to-image diffusion models is increasingly important for privacy, safety, and ethical deployment, as it aims to remove the model's ability to generate specific individuals. In this work, we study identity unlearning with adjacency preservation, where the goal is to remove a target individual's identity while preserving related non-target identities. We propose Counterfactual Feature Steering (CFS), a method requiring one-time SAE training that performs bounded inference-time concept editing in the latent space of a sparse autoencoder attached to the text encoder. CFS derives a target direction by contrasting forget and retain distributions, then applies a per-token projection proportional to each token's current alignment with this direction, yielding counterfactual representations that suppress the target while limiting collateral damage. To evaluate adjacency preservation, we introduce the Adjacency-Weighted Unlearning Score (AWUS), which measures the tradeoff between target erasure and collateral damage, and extend it to face-specific identity similarity using ArcFace. Across 4,000 prompts derived from Celeb-100, spanning 100 celebrity identities, CFS achieves the strongest evaluated tradeoff between identity erasure of target identities and preservation of adjacent and generic concepts, while remaining stable across intervention strengths where other steering-based methods degrade. Independent ArcFace evaluation and a 10,000-caption COCO study further examine identity and general preservation. Quantitative nudity-suppression benchmarks and qualitative artistic-style examples provide evidence of transfer beyond identities on SD~1.4.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PIU: Proximity-guided Identity Unlearning in ID-Conditioned Diffusion Models

    May 21, 2026Jose Edgar Hernandez Cancino Estrada, Mauro Díaz Lupone, Žiga Emeršič +3Synthetic Face GenerationLatent Diffusion Models

  2. Locality-Aware Continual Unlearning for Diffusion Models

    Dec 2, 2025Naveen George, Naoki Murata, Yuhta Takida +2Diffusion Model UnlearningMachine Unlearning

  3. You Can't Have It Both Ways: Concept Entanglement Limits Diffusion Model Unlearning

    Sep 28, 2026Yian Wang, Ali Ebrahimpour-Boroojeny, Hari Sundaram +1Diffusion Model UnlearningMachine Unlearning