Pre-trained vision-language models (VLMs) exhibit strong cross-domain recognition performance even without additional training. However, this robustness can also preserve undesirable domain-specific behavior, as domain-related and semantic information often remain entangled within the learned representation space, making selective domain unlearning challenging. Existing approaches typically address this problem through latent-space disentanglement and prompt- or feature-level interventions, without directly attributing and attenuating individual patch-token contributions. However, here we suggest that rather than uniformly suppressing the full representation, it may be more effective to exploit the spatial structure of vision transformers to localize and suppress patch regions that contribute disproportionately to forget-domain prediction. Patches that strongly influence forget-domain prediction may not be equally important for semantic recognition, suggesting that forgetting should be guided according to the domain contribution of different visual regions. Specifically, we propose a two-stage patch-selective framework that first estimates patch-level domain sensitivity and then selectively attenuates patches whose contribution to forget-domain prediction is stronger than their semantic utility. We evaluate our framework on Office-Home, Mini DomainNet, and DomainNet. Experimental results demonstrate improved forgetting-retention tradeoffs compared to prior methods while improving retained-domain recognition by up to 3.8%. Additional evaluations under visually overlapping and unseen-domain settings further demonstrate improved robustness under distribution shift.
Figures & tables
Figure 1: Patch-level visualization of forget-domain contribution. (a) Input image, (b) patch-level domain sensitivity heatmap where yellow: high domain contribution and purple: low contribution, and (c) top domain-predictive patches. (Best viewed in color.)
Setting
Forget
Retain
Acc. ↓
Acc. ↑
All patches
63.23
78.40
Top- K patches
37.20
77.29
Table 1: Effect of selective patch suppression on Office-Home (— Df∣=1 ) . Top-K suppresses the highest domain-scoring patches ( K expressed as % of total tokens).
Figure 3: Overview of the proposed localized patch-guided domain unlearning framework. At an intermediate layer ℓ , patch representations are evaluated using a trainable domain scorer and frozen semantic utility from class-text embeddings. Their domain and semantic scores, si and ui , define the localized domain effect ri , which determines patch-wise attenuation. The modified tokens are then propagated through the remaining frozen transformer blocks for prediction. Only gϕ is trained using Lscore=LBCE+λLmargin .
∣Dforget∣
Method
ImageNet
Office-Home
Mini DomainNet
H↑
Mem↑
For↑
H↑
Mem↑
For↑
H↑
Mem↑
For↑
∣Dforget∣=1
LP++ [ 17 ]
50.69
62.22
42.76
30.46
85.06
18.55
31.73
84.72
19.52
CLIPFit [ 23 ]
71.31
63.04
82.13
43.44
83.20
29.40
53.56
84.32
39.24
BBF [ 21 ]
45.56
35.92
65.91
31.25
80.94
19.74
32.12
81.80
20.03
ADU [ 19 ]
77.02
64.13
96.73
69.96
80.93
64.34
75.56
83.32
73.06
Global Suppression
74.66
63.24
95.41
53.29
78.40
36.37
61.07
84.14
50.41
Table 2: Comparison with Baseline CLIP Fine-Tuning Methods. Average performance over all possible retain-domain and forget-domain combinations for different numbers of forget domains ∣Df∣∈{1,2,3} . Since ImageNet contains only two domains, only the setting ∣Df∣=1 is evaluated. Performance is measured using three metrics: Mem , classification accuracy on retained domains; For , classification error on forget domains; and H , the harmonic mean of Mem and For. Higher values indicate better unlearning performance.
Table 6
Figure 4: Ablation analysis and visualization of localized patch effects. Left : patch-level visualization of semantic utility (blue), domain sensitivity (red), overlapping regions (yellow), and the resulting Localized Domain Effect (LDE) maps (green) across representative samples. Right : domain and class probe accuracy across transformer layers of the frozen model, with the selected intervention layer marked at layer l=5 .
Figure 5: Patch-domain scorer analysis. (a) Distribution of patch-domain sensitivity scores for forget-domain and retained-domain patches. (b) Effect of the margin objective on score separation between forget-domain and retained-domain patches.
Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.
Machine unlearning in Vision-Language Models (VLMs) is typically performed at the image or instance level, making it difficult to precisely remove target knowledge without affecting unrelated semantics. This issue is especially pronounced since a single image often contains multiple entangled concepts, including both target concepts to be forgotten and contextual information that should be preserved. In this paper, we propose an interpretable concept-level unlearning framework for VLMs, which constructs a compact task-specific concept vocabulary from the forgetting set using a multimodal large language model. In addition to modality alignment, visual representations are decomposed into sparse, nonnegative combinations of semantic concepts, providing an explicit interface for fine-grained knowledge manipulation. Based on this decomposition, our method formulates unlearning as concept-level optimization, where target concepts are selectively suppressed while intra-instance non-target semantics and global cross-modal knowledge are preserved. Extensive experiments across both in-domain and out-of-domain forgetting settings demonstrate that our method enables more comprehensive target forgetting, better preserves non-target knowledge within the same image, and maintains competitive model utility compared with existing VLM unlearning methods.
Shen Lin, Jing Lin, Junhao Dong +2
1Fujian Normal University · 2Nanyang Technological University · University of New South Wales +1
Vision-language models (VLMs) raise growing concerns about privacy, copyright, and bias, motivating machine unlearning to remove sensitive knowledge. However, existing methods primarily fine-tune the language decoder, leading to superficial forgetting that fails to erase underlying visual representations and often introduces object hallucination. We propose HFRU, a reinforcement unlearning framework that operates on the vision encoder for deep semantic removal. Our two-stage approach combines alignment disruption with GRPO-based optimization using a composite reward, including an abstraction reward that encourages semantically valid substitutions and mitigates hallucinations. Experiments on object recognition and face identity tasks show that HFRU achieves over 98% forgetting and retention performance, while introducing negligible object hallucination, significantly outperforming prior methods.Our code and implementation details are available at https://github.com/XMUDeepLIT/HFRU.
Vision-Language Models (VLMs) exhibit remarkable zero-shot generalization, yet they often encode unwanted or hazardous stylistic domains such as idealized textbook diagrams in medical AI or cartoon vehicles in autonomous driving. Approximate Domain Unlearning (ADU) aims to selectively erase a model's recognition of a target visual domain while preserving accuracy on the remaining domains. However, existing ADU methods operate under a flawed closed-vocabulary assumption: they evaluate unlearning solely on the specific object classes seen during the unlearning fine-tuning phase. Consequently, these methods do not unlearn the domain itself; they merely overfit to seen class-domain pairs, leaving the domain easily recognizable for unseen classes and providing a false sense of removal. We argue that true domain erasure must be class-agnostic. To address this, we formalize Open-Vocabulary Domain Unlearning (OVDU), a rigorous protocol that mandates domain forgetting must transfer to held-out classes. To solve the OVDU challenge, we propose a surgical parameter-editing framework. First, a Fisher Information mask isolates domain-sensitive weights, mathematically protecting foundational zero-shot generalization. Second, our Targeted Manifold Scattering (TMS) objective uses preference-based mining to locally scatter the forget domain's stylistic geometry. Evaluated across PACS, OfficeHome, and DomainNet, our method vastly improves open-vocabulary generalization over existing baselines. Crucially, it delivers exceptional sample efficiency, outperforming peak 8-shot baseline results with only 4 shots.