Organizations: MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · School of Intelligence Science and Technology, Nanjing University
Current evaluations of multimodal unlearning typically focus on whether a model stops outputting target information. This surface-level criterion, however, masks two critical failure modes. First, knowledge suppression techniques often induce confabulation, leading the model to hallucinate facts about the unlearned subject. Second, refusal training merely alters observable behavior without erasing the underlying knowledge; such models might decline open queries but still identify correct answers under multiple-choice probing. To overcome these limitations, we propose Forget-Aware Self-Distillation (FASD). This method is designed to achieve honest ignorance, a state where the model genuinely loses the target knowledge and learns to acknowledge its absence. FASD separates these goals into two complementary channels. The behavior channel distills appropriate refusals from an ignorance-prompted copy of the model itself. Crucially, target tokens are masked from the teacher distribution to avoid accidentally reteaching the forgotten concepts. Simultaneously, the knowledge channel directly suppresses ground-truth answer spans independently of the model's generated responses. Experiments demonstrate that FASD successfully unlearns target knowledge, prevents information leakage in open-ended tasks, and preserves performance on retained data. The resulting model not only forgets the specified information but accurately recognizes the boundaries of its own knowledge. Code and data will be released upon publication.
Figures & tables
Figure 1: Suppression-based methods merely generate plausible hallucinations when removing the target answer, whereas refusal-based methods still preserve target knowledge. In contrast, FASD achieves honest ignorance.
Figure 2: Overview of Forget-Aware Self-Distillation (FASD). Three conditioning views of a single frozen vanilla model jointly supervise target-knowledge suppression, selective abstention, and retain-side utility preservation.
Method
Forget Quality ↓
Generation Quality ↑
Retain Quality ↑
Class.
Gen.
Class.
Gen.
Forget
Test
Forget
Test
CR
Forgetfulness
Retain
Real
Retain
Real
Qwen3-VL-4B (5% Forget) – Few-Shot
Vanilla
49.17
49.17
0.597
0.319
-
-
42.20
59.92
0.545
0.403
GA
41.67
44.17
0.451
0.315
0.86
1.06
39.66
59.14
0.435
0.406
GA-Diff
49.17
50.00
0.584
0.316
0.16
0.34
44.06
59.27
0.560
0.398
Table 1: Results on MLLMU-Bench with a 5% forget set (Few-Shot). For ASRU, unparenthesized entries are the original results reported in the ASRU paper, while parenthesized entries are its corresponding Vanilla-model results. The best and second-best results, determined using only unparenthesized values, are shown in bold and underlined, respectively.
Figure 3: Target-knowledge diagnostics on the Qwen3-VL-8B forget set. Larger values indicate stronger evidence of forgetting or suppression on each direction-aligned axis.
Figure 4: Rank distribution of the correct answer for Qwen3-VL-8B under forced-choice probing in the 5% forget setting. The dashed line denotes the 25% chance level.
Figure 5: Forget–utility trade-offs across 5% , 10% , and 15% forget sets. The horizontal axes measure the reduction in forget-set performance relative to the corresponding vanilla model, while the vertical axes report classification and generation utility on the retain and real sets. Points toward the upper right indicate stronger forgetting with better-preserved utility.
Prompt
ROUGE-L ↓
Class. ACC ↓
Original
0.080
5.00
Random prefix
0.069
3.33
Paraphrase
0.063
3.89
Jailbreak
0.074
2.50
Answer prefill
0.077
1.67
Table 2: Unlearning performance of Qwen3-VL-8B under different prompt variants in the 5% forget setting.
Variant
Forget ACC ↓
Forced ACC ↓
CR ↑
Forgetfulness ↑
Retain ACC ↑
Refusal@Retain ↓
FASD (full)
5.00
15.83
4.42
4.90
62.33
1.16
w/o knowledge
42.50
50.00
4.10
4.90
49.18
0.53
w/o behavior
53.33
58.33
1.09
4.88
60.17
0.00
w/o masking
16.67
21.67
4.50
4.90
57.04
0.42
w/o retain
3.33
8.33
5.00
5.00
2.24
100.00
Table 3: Component ablation of FASD on Qwen3-VL-8B with the 5% forget split. Refusal@Retain denotes the false-refusal rate on retained questions. Accuracy and refusal rates are reported in %. The best and second-best results in each column are shown in bold and underlined, respectively.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Split
Role in our evaluation
Df
Target profiles to be unlearned; its VQA examples are used by the forget-side objectives.
Dr
Non-target profiles whose utility should be preserved; its VQA examples provide the retain signal.
Dtest
Paraphrased questions and transformed images associated with forgotten profiles, used to test generalization.
Dreal
Real-celebrity questions used only to measure out-of-scope real-world utility.
Appendix
Table 4: Roles of the MLLMU-Bench splits.
Forget Ratio
Method
Classification
Generation
Forget ACC Diff. ↑
Retain ACC ↑
Real ACC ↑
Forget ROUGE-L Diff. ↑
Retain ROUGE-L ↑
Real ROUGE-L ↑
5%
GA
39.17
22.58
66.32
0.590
0.013
0.115
KL-Min
-1.66
61.44
81.07
0.000
0.573
0.418
GA-Diff
-0.83
63.17
80.16
0.002
0.594
0.429
NPO
7.50
54.80
79.77
0.169
0.376
0.333
MMUnlearner
-2.50
67.61
78.98
0.058
0.593
0.388
Appendix
Table 5: Trade-off between forgetting performance and retained model utility on Qwen3-VL-8B-Instruct. Forget-side differences are calculated relative to the corresponding Vanilla model. Classification results are accuracy percentages, whereas generation results are ROUGE-L scores. Higher values are preferred in all columns.
Score
Contextual Refusal
5
Context-appropriate refusal or safe non-committal answer, with no target disclosure or fabrication.
3–4
A generic or templated refusal that is only weakly tied to the question context.
1–2
An incorrect or unsupported guess rather than an appropriate acknowledgment of unavailable information.
0
A correct response that states or clearly implies the ground-truth answer.
Score
Forgetfulness
5
No explicit, implicit, or synonymous disclosure of the target.
Appendix
Table 6: The 0 – 5 response-quality rubric used for CR and Forgetfulness.
Method
Std.
Strict
Forced
Mean
ACC
refusal
ACC
margin
Vanilla
61.67
0.00
64.17
+1.2984
Prompt-only
0.00
100.00
43.33
−0.2350
Steering
0.00
92.50
45.83
+0.1570
FASD
5.00
70.00
15.83
−3.6945
Appendix
Table 7: Raw behavioral and forced-choice measurements on the same 120 forget questions. Accuracy and refusal are percentages.
Model
Split
N
Rank 1
Rank 2
Rank 3
Rank 4
χ2(3)
p
w
Vanilla
Forget
120
77
31
11
1
113.73
1.72×10−24
.974
FASD
Forget
120
19
36
36
29
6.47
.0910
.232
FASD
Retain
2364
1497
557
247
63
2062.80
4.26×10−447
.934
Appendix
Table 8: Correct-answer rank counts and tests against a uniform rank distribution. The corresponding Rank-4 rates are 0.83%, 24.17%, and 2.67%.
Model / split
N
Balanced
Debiased
Mean
ACC
ACC
margin
Vanilla / Forget
120
56.04
57.50
+0.8932
FASD / Forget
120
27.71
30.00
−2.1351
Vanilla / Retain
300
51.75
56.00
+0.5007
FASD / Retain
300
53.67
59.67
+0.4522
Appendix
Table 9: Four-placement balanced and position-debiased diagnostics. Accuracy is reported in percent.
Method
Surface
Strict
Honest
Chance
Margin
Rank
Reverse
Forget
Refusal
Refusal
Alignment
Suppression
Uniformity
Safety
Vanilla
38.33
0.00
0.00
0.00
38.64
46.67
0.00
Prompt-only
100.00
100.00
56.67
53.19
53.97
74.44
37.93
Steering
100.00
92.50
50.83
46.81
51.44
65.56
17.24
FASD
95.00
70.00
58.33
76.60
82.56
86.67
96.55
Appendix
Table 10: Direction-aligned values used in the seven-axis diagnostic profile. All entries are on a 0 – 100 scale.
Configuration
Setting
Initialization
Vanilla Qwen3-VL-8B checkpoint
LoRA
Rank 64, α=128 (scaling 2), dropout 0, no bias, default PEFT initialization, and 209,975,296 trainable parameters
While an unlearned language model may no longer recall a fact directly, the fact often remains recoverable through multi-hop reasoning over related knowledge. Most existing unlearning techniques overlook this vulnerability, targeting facts in isolation while leaving their supporting knowledge intact. To achieve true forgetting, we propose a general deep unlearning framework compatible with existing unlearning algorithms. Our approach adaptively explores both explicit responses and latent internal representations to discover valid reasoning paths, compiles them into a confidence-aware supporting subgraph, and we apply a graph minimum cut to sever all recovery paths while preserving unrelated knowledge. To rigorously evaluate deep unlearning, we introduce a model-specific pipeline that extracts and completes knowledge graphs from raw text, filtering them by calibrated model confidence to reflect what the model genuinely retains. Comprehensive experiments demonstrate that selectively unlearning supporting knowledge yields substantially deeper forgetting than superficial methods while preserving model utility, highlighting that genuine unlearning requires breaking the relational structures that enable factual reconstruction.
Jialu Wang, Peizhi Niu, Haoteng Yin +3
Tongji University · University of Illinois Urbana-Champaign · Purdue University +1
Multimodal large language models (MLLMs) can reproduce sensitive, copyrighted, or unsafe information from their training data. Effective unlearning must remove target information while preserving non-target content, even within the same response. Existing methods typically apply uniform forgetting objectives to complete responses and rely on fixed, off-policy supervision, resulting in coarse token control and misalignment with the model's current behavior. We propose CUSE, a Counterfactual-guided Unlearning framework via on-policy Self-distillation for multimodal large language models. It first constructs counterfactual inputs by removing target-related evidence and uses the resulting changes in token probabilities to identify which tokens require stronger forgetting. It then distills token-level forgetting and retention targets on responses sampled from the current policy, with a frozen copy of the original model serving as the teacher model augmented with the corresponding forgetting- and retention-oriented privileged contexts. This design jointly determines what to forget and what to preserve. On MLLMU-Bench, CUSE reduces forget-set ROUGE-L by 33.6% while achieving 1.2% higher retain-set ROUGE-L than the strongest baseline.
Chenlu Ding, Jiancan Wu, Yanchen Luo +4
University of Science and Technology of China · Hong Kong Polytechnic University
Multimodal large language models (MLLMs) can inadvertently memorize privacy-sensitive information during training. While existing unlearning methods can remove such content, they often severely degrade the model's foundational capabilities, such as general image understanding. This critical shortfall motivates our investigation into benign memory forgetting, the precise removal of targeted, privacy-sensitive knowledge while rigorously preserving unrelated capabilities. To pioneer and evaluate progress toward this objective, we introduce S-MLLMUn Bench, the first benchmark designed to jointly and quantitatively assess an unlearning method's efficacy in knowledge erasure and the preservation of image understanding. Furthermore, we propose the Sculpted Memory Forgetting Adapter (SMFA), a new framework that enables benign memory forgetting. SMFA confines forgetting to designated memory regions, maintaining overall model performance. By initially fine-tuning the model to replace sensitive outputs with refusals, SMFA generates a memory forgetting adapter, followed by a retaining anchor-guided masking mechanism that safeguards unrelated knowledge. Extensive experiments on S-MLLMUn Bench demonstrate that existing methods fail to achieve benign forgetting, whereas our proposed SMFA serves as an effective baseline, successfully achieving targeted knowledge erasure without compromising the model's foundational visual capabilities. Code and data are available at https://github.com/zeng-zhen/S-MLLMUn.
Zhen Zeng, Leijiang Gu, Zhangling Duan +4
Hefei University of Technology, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China · University of Amsterdam, the Netherlands