Cross-Domain Generalization in Machine Unlearning via Label-Conditioned Energy Magnitude Regularization
Authors: Syed Ali Ahmed, Syed Bilal Ahsan, Muhammad Zaigham Zaheer
Organizations: National University of Computer and Emerging Sciences, Karachi, Pakistan · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Machine unlearning removes the influence of specific data from a trained model. However, most methods treat the forgotten concept as isolated. In this paper, we study what happens to the rest of the model when a class is forgotten, using a label-conditioned energy-based model (EBM) that assigns per-class energies, making the effect directly observable. We forget a class by raising the energy of its image-label pairs, training with a forget term, a retain anchor to the pretrained model, a global margin, and an energy regularizer that stops the energy magnitudes from growing without limit. A propagation term applies the same forget signal to retain samples, weighted by each sample's DINOv2 similarity to the forget class, so forgetting reaches images that resemble it and leaves the rest untouched. We evaluate on two benchmark datasets: 1) On a subset of DomainNet across four visual domains, we forget tiger, lion, and scissors one at a time. Forgetting a class in the sketch domain also erases it from real, clipart, and painting, with forgetting error reaching 98% and 99% for lion and scissors, and the effect carrying over to the most similar class. 2) On CIFAR-10, we turn off the propagation term and forget each of the ten classes on its own. Forgetting is complete (100%), while the other nine classes retain 98.5% of their pre-unlearning accuracy on average.
Figures & tables
Figure 1: Overview of our method. Stage 1 pretrains a label-conditioned EBM to obtain frozen reference E0 . Stage 2 trains a copy Eθ to forget a single (class, domain) cell by combining four losses (forget, retain, margin, and energy regularization) with a DINOv2 similarity-weighted propagation mechanism. Scope masking controls whether the forget signal propagates across domains or semantically related classes, selectively forgetting target knowledge while preserving unrelated concepts.
Target
Method
F/sk. ↑
F/oth. ↑
Mem pre
Mem unl↑
tiger
Retrain
19.9
3.3
96.0
96.2
NegGrad
100.0
100.0
96.0
85.3
Finetune
12.2
2.6
96.0
97.2
Ours
96.1
83.7
96.0
76.7
lion
Retrain
46.7
4.1
95.9
96.1
NegGrad
100.0
100.0
95.9
70.0
Table 1: Unlearning on DomainNet across three forget targets, each forgotten in the sketch domain. F is forgetting error ( 100acc , %, ↑ ) on the target cell (F/sk.) and on the same class in the other three domains (F/oth.). Mem is retain accuracy before and after unlearning ( ↑ ).
Class
F ↑
Mem pre
Mem unl↑
Utility ↑
Energy pres. ↑
MIA ( → 0.5)
airplane
100
69.3
67.8
97.9
0.862
0.489
automobile
100
69.3
66.4
95.7
0.872
0.496
bird
100
70.1
71.8
102.5
0.885
0.491
cat
100
71.0
72.5
102.1
0.864
0.490
deer
100
70.9
70.0
98.8
0.857
0.508
dog
100
70.6
70.4
99.8
0.813
0.484
Table 2: Isolated unlearning on CIFAR-10, all ten classes as forget targets. F is forgetting error ( 100−acc , %). Mem is reported before (pretrained) and after unlearning on the other nine classes; the pretrained model tops out near 70% on this backbone, so the near-unchanged Mem column reflects preserved utility, not degradation. Utility is the after/before ratio; retain energy preservation is the rank-order agreement of retain energies with the pretrained model ( 1.0 = identical); MIA is membership-inference AUC on forget samples ( 0.5 = indistinguishable).
Figure 2: Energy-landscape reshaping under isolated unlearning across all ten CIFAR-10 classes. Bars show the mean energy at the true label for the forget class and the retain set, before and after unlearning. For every class the forget-class energy is lifted by roughly ten units (to ≈+8.3 ), while the retain-set energy is essentially unchanged ( ≈−1.4→−1.9 ). This is the mechanism behind the energy gaps: forgetting raises only the target’s energy and leaves the retain landscape intact.
Forget target
nearest neighbour (sim.)
NN drop
r
ρ
tiger
lion (0.55)
−75%
0.76
0.44
scissors
pliers (0.78)
−46%
0.42
0.10
lion
tiger (0.56)
−9%
0.10
0.21
Table 3: Cross-class spillover (DomainNet). When a class is forgotten, degradation concentrates on its nearest visual neighbour. NN is the retain class with highest DINOv2 similarity to the forget class; “NN drop” is its accuracy drop after unlearning. r and ρ are the Pearson and Spearman correlation between a retain class’s similarity to the forget class and its accuracy drop, over all 25 retain classes.
Variant
trainable
F ↑
Mem ↑
r↑
ρ↑
Backbone capacity ( λp=3 )
Frozen (heads only)
heads only
60.6
92.1
0.52
0.22
LoRA r8, layer4
157k
66.6
89.3
0.535
0.083
LoRA r8, layer3+4
227k
62.7
87.0
0.642
0.319
Full layer4 (Ours)
8.5M
96.1
77.9
0.76
0.44
Propagation term (full capacity)
Table 4: What controls propagation (DomainNet, tiger/sketch). Backbone capacity and the propagation term each move the similarity–degradation correlation. As more of the backbone adapts, propagation ( r ) rises monotonically, and with the term off ( λp=0 ) forgetting still spreads, confirming it is partly intrinsic to the shared backbone.
Figure 3: Cross-class spillover for tiger/sketch. Each point is a retain class’s accuracy drop after unlearning, grouped by DINOv2 similarity to tiger (threshold 0.25 ). Classes above the threshold lose a median of 28% against 11% for the rest, and lion, tiger’s nearest neighbour, degrades most. Bars mark group medians.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Machine unlearning aims to remove the influence of specific training data from a model while preserving its general utility. In vision, many approximate unlearning methods pursue this goal through degradation-based heuristics, such as loss maximization or random labeling. Yet making a model worse on forget samples is not the same as making it behave as if those examples had never been seen: these signals can be poorly conditioned, destabilize optimization, and harm generalization. We argue that approximate unlearning should instead prioritize distributional indistinguishability, aligning the model's predictive behavior on forget data with that on truly unseen data. Motivated by this principle, we propose Reference-Guided Unlearning (ReGUn), a vision unlearning framework that uses disjoint held-out data to construct a principled, class-conditioned reference distribution for distillation. Rather than explicitly degrading predictions on forget examples, ReGUn guides them toward non-member behavior through held-out supervision. Across multiple architectures, natural image datasets, and forget fractions, ReGUn achieves a competitive forgetting--utility trade-off relative to standard approximate baselines while closely matching retrain-like membership inference behavior. As one instantiation of this principle, the results suggest that simple objectives designed around indistinguishability can provide an effective alternative to complex degradation-based unlearning procedures.
While numerous machine unlearning (MU) methods have recently been developed with promising results in erasing the influence of forgotten data, classes, or concepts, they are also highly vulnerable-for example, simple fine-tuning can inadvertently reintroduce erased concepts. In this paper, we address this contradiction by examining the internal representations of unlearned models, in contrast to prior work that focuses primarily on output-level behavior. Our analysis shows that many state-of-the-art MU methods appear successful mainly due to a misalignment between last-layer features and the classifier, a phenomenon we call feature-classifier misalignment. In fact, hidden features remain highly discriminative, and simple linear probing can recover near-original accuracy. Assuming neural collapse in the original model, we further demonstrate that adjusting only the classifier can achieve negligible forget accuracy while preserving retain accuracy, and we corroborate this with experiments using classifier-only fine-tuning. Motivated by these findings, we propose MU methods based on a class-mean features (CMF) classifier, which explicitly enforces alignment between features and classifiers. Experiments on standard benchmarks show that CMF-based unlearning reduces forgotten information in representations while maintaining high retain accuracy, highlighting the need for faithful representation-level evaluation of MU.
Yichen Gao, Altay Unal, Akshay Rangamani +1
Department of Computer Science & Engineering, The Ohio State University · Department of Data Science, New Jersey Institute of Technology
Machine unlearning aims to remove the influence of selected training examples without full retraining. Standard evaluations often summarize unlearning quality with aggregate metrics, such as accuracy- and forgetting-based scores, which can hide localized failures. We study this failure mode at the example level by comparing the predictions of an unlearned model to those of the model retrained after deletion. We show that this pointwise discrepancy can be highly non-uniform: for gradient-ascent and random-labeling methods, with and without retain-set fine-tuning, it grows with geometric proximity to the forget set. We call this phenomenon localized collateral forgetting. Our analysis identifies a mechanism behind the effect: surrogate targets used during unlearning can be inconsistent with the local prediction structure induced by retraining, and this inconsistency propagates through shared representations to nearby examples. Motivated by this mechanism, we propose Local Teacher Distillation, a simple mitigation strategy that replaces random targets with soft labels from a small teacher trained only on retained neighbors of the forget set. On CIFAR-100 partial-class deletion, this local teacher brings the unlearned model substantially closer to retraining, especially near the forget set, while maintaining competitive aggregate unlearning metrics.
Polina Dolgova, Sebastian U. Stich
CISPA Helmholtz Center for Information Security · Universität des Saarlandes · Saarbrücken, Germany