MeGU: Machine-Guided Unlearning with Target Feature Disentanglement
Authors: Haoyu Wang, Zhuo Huang, Xiaolong Wang, Bo Han, Zhiwei Lin, Tongliang Liu
Organizations: School of Automation, Beijing Institute of Technology · Sydney AI Centre, The University of Sydney · Department of Computer Science, Hong Kong Baptist University
The growing concern over training data privacy has elevated the "Right to be Forgotten" into a critical requirement, thereby raising the demand for effective Machine Unlearning. However, existing unlearning approaches commonly suffer from a fundamental trade-off: aggressively erasing the influence of target data often degrades model utility on retained data, while conservative strategies leave residual target information intact. In this work, the intrinsic representation properties learned during model pretraining are analyzed. It is demonstrated that semantic class concepts are entangled at the feature-pattern level, sharing associated features while preserving concept-specific discriminative components. This entanglement fundamentally limits the effectiveness of existing unlearning paradigms. Motivated by this insight, we propose Machine-Guided Unlearning (MeGU), a novel framework that guides unlearning through concept-aware re-alignment. Specifically, Multi-modal Large Language Models (MLLMs) are leveraged to explicitly determine re-alignment directions for target samples by assigning semantically meaningful perturbing labels. To improve efficiency, inter-class conceptual similarities estimated by the MLLM are encoded into a lightweight transition matrix. Furthermore, MeGU introduces a positive-negative feature noise pair to explicitly disentangle target concept influence. During finetuning, the negative noise suppresses target-specific feature patterns, while the positive noise reinforces remaining associated features and aligns them with perturbing concepts. This coordinated design enables selective disruption of target-specific representations while preserving shared semantic structures. As a result, MeGU enables controlled and selective forgetting, effectively mitigating both under-unlearning and over-unlearning.
Figures & tables
Figure 1: The entanglement among features from different concepts. Taking dinosaur and wolf as an example. They share similar features (marked as red ) while each possesses unique features ( green ). Assume that dinosaur is the class to be forgotten. The goal of our method is to disentangle its features and forget the target concept’s unique features while preserving associated features shared with the other retained concepts.
Figure 2: The proposed unlearning framework MeGU. The MLLM is employed to acquire the conceptual similarities with a small subset of the training data. Incorporated with model prediction, the perturbing labels are determined. The Fragment-Align strategy leverages a pair of feature noises trained from the frozen pretrained model to respectively restrain unique features of the target data while enhancing those associated with the retained data. The unlearning is achieved via aligning the target data towards perturbing labels while disentangling their influence.
Figure 3: The visualized transition matrix.
model
class
metric
baseline
retrain
FT
UNSIR
AMNC
SSD
MaGA
RN18
RKT
Ar
76.30
76.19
65.43
73.83
73.59
75.86
75.75
Af
82.81
0.00
0.00
41.15
0.00
0.00
0.00
MIA
96.61
8.06
10.04
3.08
28.62
0.66
0.00
MR
Ar
76.38
76.23
63.90
74.26
73.22
76.20
76.25
Af
82.03
0.00
0.00
8.07
0.00
0.00
0.00
MIA
95.65
6.01
12.22
1.68
46.06
0.25
0.00
Table 1: Class-wise unlearning on CIFAR-100.
model
class
metric
baseline
retrain
FT
UNSIR
AMNC
SSD
MaGA
RN18
veh2
Ar
82.84
82.41
73.74
81.41
82.21
83.38
82.93
Af
84.99
0.00
0.00
58.82
0.00
22.37
0.00
MIA
87.72
14.24
40.84
45.68
7.84
5.72
0.08
veg
Ar
82.59
82.24
72.56
81.23
81.66
82.64
82.53
Af
88.94
0.00
0.00
70.24
0.00
45.34
0.00
MIA
93.28
9.24
29.16
43.27
3.04
2.08
0.00
Table 2: Class-wise unlearning on CIFAR-20.
model
sublass
metric
baseline
retrain
FT
UNSIR
AMNC
SSD
MaGA
RN18
RKT
Ar
82.93
83.55
73.57
81.50
81.89
82.36
82.44
Af
81.51
1.39
15.36
60.85
0.00
5.90
4.51
MIA
84.43
22.40
12.29
43.6
9.41
5.85
4.80
sea
Ar
82.79
82.95
85.85
81.14
82.06
82.41
82.80
Af
97.66
77.34
76.14
95.49
35.76
86.02
84.38
MIA
87.10
56.67
66.05
86.60
4.00
54.65
66.68
Table 3: Sub-class unlearning on CIFAR-20.
model
metric
baseline
retrain
FT
teacher
AMNC
SSD
MaGA
RN18
Ar
95.31
92.42
87.86
90.68
90.63
91.38
93.04
Af
94.03
94.40
88.77
93.13
55.35
95.88
85.59
MIA
76.53
74.42
71.70
47.08
17.89
76.50
34.04
ViT
Ar
98.66
98.75
97.64
98.17
98.15
98.63
98.81
Af
99.71
99.41
98.24
93.52
74.88
99.39
96.27
MIA
88.66
91.67
86.83
26.80
6.40
88.22
29.18
Table 4: Random unlearning on CIFAR-10.
Figure 4: Time consumed for CIFAR-100 class-wise unlearning.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Parameters
ResNet18
ViT
exemplar n /class
10
10
batch size
32
32
noise size
256
64
noise lr
0.1
0.1
unlearn lr
0.0003
0.00005
rank τ
0.3
0.3
Appendix
Table 5: Parameter setting for unlearning.
Figure 5: The distribution of perturbing labels for class-wise unlearning designating class 2 as target on CIFAR-10 with ResNet18.
Figure 6: Visulized model prediction behavior comparison.
Figure 7: Performance on forget set under varying hyperparameters.
Figure 8: Performance on retain set under varying hyperparameters.
model
class
Metric
baseline
retrain
SSD
RND
w.o.F.N
MeGU
RN18
rocket
Ar
76.30
76.19
75.86
76.24
75.46
75.75
Af
82.81
0.00
0.00
56.51
3.32
0.00
MIA
96.61
8.06
0.66
20.4
0.00
0.00
MR
Ar
76.38
76.16
76.20
75.98
76.07
76.25
Af
82.03
0.00
0.00
57.12
4.55
0.00
MIA
95.65
5.61
0.25
10.21
0.00
0.00
Appendix
Table 6: Ablation studies on class-wise unlearning with CIFAR-100.
model
class
metric
baseline
retrain
FT
UNSIR
AMNC
SSD
MeGU
RN18
baby
Ar
76.54
75.71
64.64
71.86
73.53
76.64
76.70
Af
67.80
0.00
0.00
0.00
0.00
0.00
0.00
MIA
96.27
3.44
19.30
18.46
54.81
0.00
0.00
lamp
Ar
76.50
75.33
64.74
71.64
73.32
76.46
76.11
Af
69.10
0.00
0.00
0.00
0.00
0.00
0.00
MIA
96.20
0.63
10.74
8.68
51.89
0.00
0.00
Appendix
Table 7: Additional results of class-wise unlearning on CIFAR-100.
model
subclass
metric
baseline
retrain
FT
UNSIR
AMNC
SSD
MeGU
RN18
beetle
Ar
82.63
82.35
73.23
82.00
82.20
80.10
82.37
Af
82.64
67.71
72.05
76.13
40.97
0.78
70.40
MIA
88.43
20.26
58.64
71.00
9.40
8.81
6.26
snail
Ar
83.00
81.97
75.49
80.98
82.80
82.77
83.10
Af
69.88
37.59
16.32
55.30
9.90
6.68
50.27
MIA
84.50
9.47
16.57
47.22
10.05
4.80
4.33
Appendix
Table 8: Additional results of sub-class unlearning on CIFAR-20.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Machine unlearning aims to remove the influence of specified training data or knowledge from a trained model without requiring full retraining. This capability is particularly important for modern image classifiers and large language models (LLMs), where retraining can be computationally expensive. However, machine unlearning on complex data remains difficult because the target information is often distributed across multiple semantic elements. Existing methods mainly remove samples, modify labels, or edit model parameters to reduce the influence of the forgetting target. These approaches usually do not explicitly identify which semantic concepts connect the forgetting target to the model's prediction or generated response. As a result, it is difficult to determine which information to modify. This uncertainty may leave residual target information or unnecessarily affect knowledge that should be retained. To address this gap, we propose a concept-guided poisoning unlearning framework that explicitly identifies the concepts that contribute most strongly to the target class or knowledge and uses them to guide the model update. For image classification, our method first identifies class-relevant concepts with a Post-hoc Concept Bottleneck Model, localizes the image regions that express these concepts, and constructs replacement-based poisoned samples. For LLMs, it elicits the target knowledge through multiple questions, aggregates Integrated Gradients across the resulting responses to identify consistently important content, and masks this content to construct poisoned training targets. Experiments on multiple datasets show that the proposed framework achieves effective unlearning across different tasks while largely preserving retained model utility.
Wenhan Chang, Tianqing Zhu, Heng Xu +2
China University of Geosciences, Wuhan, China · City University of Macau, Macau SAR, China · University of Technology Sydney, Australia
Large language models deployed in sensitive applications increasingly require the ability to unlearn specific knowledge, such as user requests, copyrighted materials, or outdated information, without retraining from scratch to ensure regulatory compliance, user privacy, and safety. This task, known as machine unlearning, aims to remove the influence of targeted data (forgetting) while maintaining performance on the remaining data (retention). A common approach is to formulate this as a multi-objective problem and reduce it to a single-objective problem via scalarization, where forgetting and retention losses are combined using a weighted sum. However, this often results in unstable training dynamics and degraded model utility due to conflicting gradient directions. To address these challenges, we propose OFMU, a penalty-based bi-level optimization framework that explicitly prioritizes forgetting while preserving retention through a hierarchical structure. Our method enforces forgetting via an inner maximization step that incorporates a similarity-aware penalty to decorrelate the gradients of the forget and retention objectives, and restores utility through an outer minimization step. To ensure scalability, we develop a two-loop algorithm with provable convergence guarantees under both convex and non-convex regimes. We further provide a rigorous theoretical analysis of convergence rates and show that our approach achieves better trade-offs between forgetting efficacy and model utility compared to prior methods. Extensive experiments across vision and language benchmarks demonstrate that OFMU consistently outperforms existing unlearning methods in both forgetting efficacy and retained utility.
Machine unlearning aims to remove the influence of specific training samples while preserving the model's utility. Existing single-objective approaches, such as gradient ascent or random relabeling, often induce catastrophic forgetting due to conflicting optimization dynamics and unbounded forgetting objectives that cause the model to drift from its pre-trained knowledge. We propose Reference-Aligned UnLearning (RAUL), a multi-objective framework that jointly optimizes forgetting and retention by replacing unbounded loss maximization with a bounded KL alignment of predictions on forgotten samples toward a reference distribution representing unseen data, instantiated either as a uniform distribution or an empirical distribution from a held-out reference set, which constrains the forgetting objective and reduces gradient conflict with retention. The resulting multi-objective optimization (MOO) problem is solved via Jacobian descent, which aggregates multiple gradients into a direction that does not conflict. Our results demonstrate that RAUL achieves the closest gap compared to full retraining.
Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman +1
Department of Computer Science, Brock University, St. Catharines, ON, Canada · Yousef Haj-Ahmad Department of Engineering, Brock University, St. Catharines, ON, Canada