Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning

Haodong Chen; Shengtao Wen; Dong Liang; Lingbing Guo; Xiang Chen

cs.AIOct 10, 2026

Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning

Authors: Haodong Chen, Shengtao Wen, Dong Liang, Lingbing Guo, Xiang Chen

Organizations: MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · School of Intelligence Science and Technology, Nanjing University

Abstract

Current evaluations of multimodal unlearning typically focus on whether a model stops outputting target information. This surface-level criterion, however, masks two critical failure modes. First, knowledge suppression techniques often induce confabulation, leading the model to hallucinate facts about the unlearned subject. Second, refusal training merely alters observable behavior without erasing the underlying knowledge; such models might decline open queries but still identify correct answers under multiple-choice probing. To overcome these limitations, we propose Forget-Aware Self-Distillation (FASD). This method is designed to achieve honest ignorance, a state where the model genuinely loses the target knowledge and learns to acknowledge its absence. FASD separates these goals into two complementary channels. The behavior channel distills appropriate refusals from an ignorance-prompted copy of the model itself. Crucially, target tokens are masked from the teacher distribution to avoid accidentally reteaching the forgotten concepts. Simultaneously, the knowledge channel directly suppresses ground-truth answer spans independently of the model's generated responses. Experiments demonstrate that FASD successfully unlearns target knowledge, prevents information leakage in open-ended tasks, and preserves performance on retained data. The resulting model not only forgets the specified information but accurately recognizes the boundaries of its own knowledge. Code and data will be released upon publication.

Figures & tables

Figure 1: Suppression-based methods merely generate plausible hallucinations when removing the target answer, whereas refusal-based methods still preserve target knowledge. In contrast, FASD achieves honest ignorance.

Figure 2: Overview of Forget-Aware Self-Distillation (FASD). Three conditioning views of a single frozen vanilla model jointly supervise target-knowledge suppression, selective abstention, and retain-side utility preservation.

Qwen3-VL-4B (5% Forget) – Few-Shot
Method	Forget Quality $\downarrow$				Generation Quality $\uparrow$		Retain Quality $\uparrow$
	Class.		Gen.				Class.		Gen.
	Forget	Test	Forget	Test	CR	Forgetfulness	Retain	Real	Retain	Real
Vanilla	49.17	49.17	0.597	0.319	-	-	42.20	59.92	0.545	0.403
GA	41.67	44.17	0.451	0.315	0.86	1.06	39.66	59.14	0.435	0.406
GA-Diff	49.17	50.00	0.584	0.316	0.16	0.34	44.06	59.27	0.560	0.398

Table 1: Results on MLLMU-Bench with a 5% forget set (Few-Shot). For ASRU, unparenthesized entries are the original results reported in the ASRU paper, while parenthesized entries are its corresponding Vanilla-model results. The best and second-best results, determined using only unparenthesized values, are shown in bold and underlined, respectively.

Figure 3: Target-knowledge diagnostics on the Qwen3-VL-8B forget set. Larger values indicate stronger evidence of forgetting or suppression on each direction-aligned axis.

Figure 4: Rank distribution of the correct answer for Qwen3-VL-8B under forced-choice probing in the

5\%

forget setting. The dashed line denotes the

25\%

chance level.

Figure 5: Forget–utility trade-offs across $5\%$ , $10\%$ , and $15\%$ forget sets. The horizontal axes measure the reduction in forget-set performance relative to the corresponding vanilla model, while the vertical axes report classification and generation utility on the retain and real sets. Points toward the upper right indicate stronger forgetting with better-preserved utility.

Prompt	ROUGE-L $\downarrow$	Class. ACC $\downarrow$
Original	0.080	5.00
Random prefix	0.069	3.33
Paraphrase	0.063	3.89
Jailbreak	0.074	2.50
Answer prefill	0.077	1.67

Table 2: Unlearning performance of Qwen3-VL-8B under different prompt variants in the 5% forget setting.

Variant	Forget ACC $\downarrow$	Forced ACC $\downarrow$	CR $\uparrow$	Forgetfulness $\uparrow$	Retain ACC $\uparrow$	Refusal@Retain $\downarrow$
FASD (full)	5.00	15.83	4.42	4.90	62.33	1.16
w/o knowledge	42.50	50.00	4.10	4.90	49.18	0.53
w/o behavior	53.33	58.33	1.09	4.88	60.17	0.00
w/o masking	16.67	21.67	4.50	4.90	57.04	0.42
w/o retain	3.33	8.33	5.00	5.00	2.24	100.00

Table 3: Component ablation of FASD on Qwen3-VL-8B with the 5% forget split. Refusal@Retain denotes the false-refusal rate on retained questions. Accuracy and refusal rates are reported in %. The best and second-best results in each column are shown in bold and underlined, respectively.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Role in our evaluation
$\mathcal{D}_{f}$	Target profiles to be unlearned; its VQA examples are used by the forget-side objectives.
$\mathcal{D}_{r}$	Non-target profiles whose utility should be preserved; its VQA examples provide the retain signal.
$\mathcal{D}_{\mathrm{test}}$	Paraphrased questions and transformed images associated with forgotten profiles, used to test generalization.
$\mathcal{D}_{\mathrm{real}}$	Real-celebrity questions used only to measure out-of-scope real-world utility.

Appendix

Table 4: Roles of the MLLMU-Bench splits.

Forget Ratio	Method	Classification			Generation
Forget Ratio	Method	Forget ACC Diff. $\uparrow$	Retain ACC $\uparrow$	Real ACC $\uparrow$	Forget ROUGE-L Diff. $\uparrow$	Retain ROUGE-L $\uparrow$	Real ROUGE-L $\uparrow$
5%	GA	39.17	22.58	66.32	0.590	0.013	0.115
	KL-Min	-1.66	61.44	81.07	0.000	0.573	0.418
	GA-Diff	-0.83	63.17	80.16	0.002	0.594	0.429
	NPO	7.50	54.80	79.77	0.169	0.376	0.333
	MMUnlearner	-2.50	67.61	78.98	0.058	0.593	0.388

Appendix

Table 5: Trade-off between forgetting performance and retained model utility on Qwen3-VL-8B-Instruct. Forget-side differences are calculated relative to the corresponding Vanilla model. Classification results are accuracy percentages, whereas generation results are ROUGE-L scores. Higher values are preferred in all columns.

Score	Contextual Refusal
5	Context-appropriate refusal or safe non-committal answer, with no target disclosure or fabrication.
3–4	A generic or templated refusal that is only weakly tied to the question context.
1–2	An incorrect or unsupported guess rather than an appropriate acknowledgment of unavailable information.
0	A correct response that states or clearly implies the ground-truth answer.
Score	Forgetfulness
5	No explicit, implicit, or synonymous disclosure of the target.

Appendix

Table 6: The

0

–

5

response-quality rubric used for CR and Forgetfulness.

Method	Std.	Strict	Forced	Mean
	ACC	refusal	ACC	margin
Vanilla	61.67	0.00	64.17	$+1.2984$
Prompt-only	0.00	100.00	43.33	$-0.2350$
Steering	0.00	92.50	45.83	$+0.1570$
FASD	5.00	70.00	15.83	$-3.6945$

Appendix

Table 7: Raw behavioral and forced-choice measurements on the same 120 forget questions. Accuracy and refusal are percentages.

Model	Split	$N$	Rank 1	Rank 2	Rank 3	Rank 4	$\boldsymbol{\chi^{2}(3)}$	$\boldsymbol{p}$	$\boldsymbol{w}$
Vanilla	Forget	120	77	31	11	1	113.73	$1.72{\times}10^{-24}$	.974
FASD	Forget	120	19	36	36	29	6.47	.0910	.232
FASD	Retain	2364	1497	557	247	63	2062.80	$4.26{\times}10^{-447}$	.934

Appendix

Table 8: Correct-answer rank counts and tests against a uniform rank distribution. The corresponding Rank-4 rates are 0.83%, 24.17%, and 2.67%.

Model / split	$N$	Balanced	Debiased	Mean
		ACC	ACC	margin
Vanilla / Forget	120	56.04	57.50	$+0.8932$
FASD / Forget	120	27.71	30.00	$-2.1351$
Vanilla / Retain	300	51.75	56.00	$+0.5007$
FASD / Retain	300	53.67	59.67	$+0.4522$

Appendix

Table 9: Four-placement balanced and position-debiased diagnostics. Accuracy is reported in percent.

Method	Surface	Strict	Honest	Chance	Margin	Rank	Reverse
	Forget	Refusal	Refusal	Alignment	Suppression	Uniformity	Safety
Vanilla	38.33	0.00	0.00	0.00	38.64	46.67	0.00
Prompt-only	100.00	100.00	56.67	53.19	53.97	74.44	37.93
Steering	100.00	92.50	50.83	46.81	51.44	65.56	17.24
FASD	95.00	70.00	58.33	76.60	82.56	86.67	96.55

Appendix

Table 10: Direction-aligned values used in the seven-axis diagnostic profile. All entries are on a

0

–

100

scale.

Configuration	Setting
Initialization	Vanilla Qwen3-VL-8B checkpoint
LoRA	Rank 64, $\alpha=128$ (scaling 2), dropout 0, no bias, default PEFT initialization, and 209,975,296 trainable parameters
Language targets	q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , and down_proj
Vision/merger targets	qkv , attn.proj , linear_fc1 , and linear_fc2
Optimizer	AdamW, learning rate $2\times 10^{-5}$ , $(\beta_{1},\beta_{2})=(0.9,0.999)$ , $\epsilon=10^{-8}$ , zero weight decay, constant schedule, no warmup, and gradient clipping at 0.1
Batching	QA micro-batch 1, gradient accumulation 1, and effective QA batch 1

Appendix

Table 11: FASD configuration for the Qwen3-VL-8B 5% forget experiment.

MLLM Unlearning

Multimodal large language model (MLLM) unlearning studies how to remove selected knowledge or behaviors while preserving other capabilities.

See all 37 papers

Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths
Sep 28, 2026Jialu Wang, Peizhi Niu, Haoteng Yin +3Factual Knowledge in Language Models Machine Unlearning
Tongji University · University of Illinois Urbana-Champaign · Purdue University +1
CUSE: Fine-Grained Multimodal Unlearning with Counterfactual Guidance and On-Policy Self-Distillation
Oct 10, 2026Chenlu Ding, Jiancan Wu, Yanchen Luo +4MLLM Unlearning Machine Unlearning
University of Science and Technology of China · Hong Kong Polytechnic University
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
Nov 25, 2025Zhen Zeng, Leijiang Gu, Zhangling Duan +4VLM Unlearning Multimodal Large Language Models
Hefei University of Technology, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China · University of Amsterdam, the Netherlands

Sep 28, 2026J/K move · Enter open · S save

Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths

Jialu Wang, Peizhi Niu, Haoteng Yin, Hans Hao-Hsun Hsu+2

Tongji University · University of Illinois Urbana-Champaign · Purdue University+1

Factual Knowledge in Language Models Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

While an unlearned language model may no longer recall a fact directly, the fact often remains recoverable through multi-hop reasoning over related knowledge. Most existing unlearning techniques overlook this vulnerability, targeting facts in isolation while leaving their supporting knowledge intact. To achieve true forgetting, we propose a general deep unlearning framework compatible with existing unlearning algorithms. Our approach adaptively explores both explicit responses and latent internal representations to discover valid reasoning paths, compiles them into a confidence-aware supporting subgraph, and we apply a graph minimum cut to sever all recovery paths while preserving unrelated knowledge. To rigorously evaluate deep unlearning, we introduce a model-specific pipeline that extracts and completes knowledge graphs from raw text, filtering them by calibrated model confidence to reflect what the model genuinely retains. Comprehensive experiments demonstrate that selectively unlearning supporting knowledge yields substantially deeper forgetting than superficial methods while preserving model utility, highlighting that genuine unlearning requires breaking the relational structures that enable factual reconstruction.

Abstract

MLLM Unlearning

Explore similar work

Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths

CUSE: Fine-Grained Multimodal Unlearning with Counterfactual Guidance and On-Policy Self-Distillation

Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning