Similarity-Aware Machine Unlearning

Madhavan Citalamangalam Kumaran; Midhun Parakkal Unni; Vicky Kouni; Haripriya Harikumar

cs.LGJul 31, 2026

Similarity-Aware Machine Unlearning

Authors: Madhavan Citalamangalam Kumaran, Midhun Parakkal Unni, Vicky Kouni, Haripriya Harikumar

Organizations: Department of Computer Science, University of Manchester, UK · Centre for Machine Intelligence, University of Sheffield, UK · School of Computer Science, University of Sheffield, Sheffield, UK · Paris Dauphine - PSL University, Paris, France

Abstract

Machine unlearning removes the influence of user-specified training examples from a trained model, avoiding the need to retrain it from scratch. Localization-based methods improve unlearning efficiency by identifying a subset of influential model parameters. However, existing approaches select parameters based solely on forget-set importance, neglecting their role in retained dataset and often causing collateral damage to semantically similar retained examples. We address this limitation with a retain-aware localization method that considers parameter importance to both forgotten and retained data. We also introduce a retain-similar evaluation set, constructed using cosine similarity in the model embedding space, to directly measure collateral damage. Across eleven experimental settings on CIFAR-10 dataset and ResNet18 model, our method consistently reduces collateral damage while improving standard unlearning metrics, demonstrating the effectiveness of retain-aware localization for similarity-aware machine unlearning.

Explore similar work

Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning
May 29, 2026Polina Dolgova, Sebastian U. StichMachine Unlearning Forgetting
Multi-Objective Reference-Aligned Machine Unlearning
May 29, 2026Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman +1Machine Unlearning Unlearning Aims
PURGE: Projected Unlearning via Retain-Guided Erasure
Jun 2, 2026Vedant Jawandhia, Daksh Ahuja, Ghufran Alam Siddiqui +3Concept Erasure Retraining

May 29, 2026 · cs.LGJ/K move · Enter open · S save

Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning

Polina Dolgova, Sebastian U. Stich

CISPA Helmholtz Center for Information Security · Universität des Saarlandes · Saarbrücken, Germany

Machine unlearning aims to remove the influence of selected training examples without full retraining. Standard evaluations often summarize unlearning quality with aggregate metrics, such as accuracy- and forgetting-based scores, which can hide localized failures. We study this failure mode at the example level by comparing the predictions of an unlearned model to those of the model retrained after deletion. We show that this pointwise discrepancy can be highly non-uniform: for gradient-ascent and random-labeling methods, with and without retain-set fine-tuning, it grows with geometric proximity to the forget set. We call this phenomenon localized collateral forgetting. Our analysis identifies a mechanism behind the effect: surrogate targets used during unlearning can be inconsistent with the local prediction structure induced by retraining, and this inconsistency propagates through shared representations to nearby examples. Motivated by this mechanism, we propose Local Teacher Distillation, a simple mitigation strategy that replaces random targets with soft labels from a small teacher trained only on retained neighbors of the forget set. On CIFAR-100 partial-class deletion, this local teacher brings the unlearned model substantially closer to retraining, especially near the forget set, while maintaining competitive aggregate unlearning metrics.

Machine Unlearning Forgetting Singular Learning Theory Retraining Unlabeled Data Finetuning Neighbors

Open paper Ask about it PDF

Abstract

Explore similar work

Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning

Multi-Objective Reference-Aligned Machine Unlearning

PURGE: Projected Unlearning via Retain-Guided Erasure