Open Vocabulary Domain Unlearning

Sumanth Udupa; Mehrtash Harandi; Yadan Luo; Mahsa Baktashmotlagh

cs.CVSep 25, 2026

Open Vocabulary Domain Unlearning

Authors: Sumanth Udupa, Mehrtash Harandi, Yadan Luo, Mahsa Baktashmotlagh

Organizations: The University of Queensland · Monash University

Abstract

Vision-Language Models (VLMs) exhibit remarkable zero-shot generalization, yet they often encode unwanted or hazardous stylistic domains such as idealized textbook diagrams in medical AI or cartoon vehicles in autonomous driving. Approximate Domain Unlearning (ADU) aims to selectively erase a model's recognition of a target visual domain while preserving accuracy on the remaining domains. However, existing ADU methods operate under a flawed closed-vocabulary assumption: they evaluate unlearning solely on the specific object classes seen during the unlearning fine-tuning phase. Consequently, these methods do not unlearn the domain itself; they merely overfit to seen class-domain pairs, leaving the domain easily recognizable for unseen classes and providing a false sense of removal. We argue that true domain erasure must be class-agnostic. To address this, we formalize Open-Vocabulary Domain Unlearning (OVDU), a rigorous protocol that mandates domain forgetting must transfer to held-out classes. To solve the OVDU challenge, we propose a surgical parameter-editing framework. First, a Fisher Information mask isolates domain-sensitive weights, mathematically protecting foundational zero-shot generalization. Second, our Targeted Manifold Scattering (TMS) objective uses preference-based mining to locally scatter the forget domain's stylistic geometry. Evaluated across PACS, OfficeHome, and DomainNet, our method vastly improves open-vocabulary generalization over existing baselines. Crucially, it delivers exceptional sample efficiency, outperforming peak 8-shot baseline results with only 4 shots.

Figures & tables

Figure 1: Overview of the Targeted Manifold Scattering (TMS) framework. Images from the retain (

\mathcal{D}_{r}

) and forget (

\mathcal{D}_{f}

) domains are processed through a Fisher-masked Image Encoder to isolate domain-specific updates. The TMS loss applies two decoupled geometric forces: (1) The Retention Component protects semantic integrity by contrasting same-class retain and forget embeddings (

y_{r}=y_{f}

). (2) The Targeted Confusion Component actively diffuses the forget representations by hard-mining cross-class retain clusters (

y_{r}\neq y_{f}

) and scattering the forget embeddings (

z_{f}

) into these unrelated regions, achieving efficient erasure without global semantic collapse.

Figure 2: Domain subspace entanglement across vision encoder layers. This figure quantifies the structural overlap between domains in pre-trained VLMs. The x-axis represents the progressive layers of the vision encoder, while the y-axis measures the mean principal angle cosine, which specifically captures the overlap between the top $k$ -singular vectors of the extracted domain subspaces (Forget and Retain). The prominent red bars denote exceptionally high structural similarity between the targeted forget domain and the retain domains across nearly all layers.

Dataset	Method	$\mathbf{\gamma=0.25}$			$\mathbf{\gamma=0.50}$			$\mathbf{\gamma=0.75}$			$\mathbf{\gamma=1.0}$
Dataset	Method	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$
OfficeHome	Baseline	59.54	65.65	50.07	51.62	67.92	56.51	36.97	71.90	67.17	31.31	77.90	73.01
	ADU	21.97	27.61	40.79	28.26	49.22	58.38	33.63	67.96	67.16	36.37	80.36	71.02
	Ours	53.33	63.24	53.71	39.34	68.33	64.27	24.90	68.68	71.75	24.61	75.45	75.42
DomainNet	Baseline	55.58	77.72	56.53	27.98	77.08	74.46	20.04	77.24	78.58	18.02	77.77	79.82
	ADU	14.64	24.45	38.01	19.68	49.25	61.06	23.41	67.27	71.63	27.86	81.48	76.53

Table 1: Unlearning performance across different class fractions (

\gamma

) with

|D_{f}|=1

. F : Forget domain accuracy (%), R : Retain domain accuracy (%), HM : Harmonic Mean (%). Best values for each fraction and dataset are in bold .

Domain	$\gamma$	Confuse Only			Retain Only			Both Losses (Ours)
Domain	$\gamma$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$
Clipart	0.50	19.68	77.25	78.75	20.63	78.04	78.70	17.93	76.82	79.35
Clipart	1.00	16.03	78.10	80.93	18.73	77.78	79.49	17.30	77.08	79.79
Painting	0.50	21.27	78.15	78.44	26.03	80.37	77.04	17.46	76.82	79.57
Painting	1.00	17.78	78.94	80.55	23.33	80.37	78.47	17.30	78.88	80.74
Real	0.50	31.27	71.48	70.08	38.57	72.86	66.66	32.38	72.70	70.06

Table 2: Ablation study on the DomainNet dataset comparing the individual components of our TMS loss formulation (Confuse-only, Retain-only, and Both). F : Forget domain accuracy (%), R : Retain domain accuracy (%), HM : Harmonic Mean (%). Best aggregate values are in bold .

Dataset	Method	$\mathbf{\gamma=0.25}$			$\mathbf{\gamma=0.50}$			$\mathbf{\gamma=0.75}$			$\mathbf{\gamma=1.0}$
Dataset	Method	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$
DomainNet	Baseline	8.92	59.87	72.25	4.54	61.06	74.48	3.26	62.13	75.67	3.33	63.45	76.61
	ADU	9.97	24.88	38.99	13.70	47.61	61.37	14.85	64.92	73.67	16.79	77.06	80.02
	Ours	7.11	60.19	73.05	5.07	62.45	75.34	4.81	66.10	78.02	4.59	67.21	78.86

Table 3: Unlearning performance on DomainNet for multiple domain forgetting (

|D_{f}|=3

) across different class fractions (

\gamma

). F : Forget domain accuracy (%), R : Retain domain accuracy (%), HM : Harmonic Mean (%). Best values for each fraction are in bold .

Forget Domain	Method	ImageNet-val (%) $\uparrow$	CIFAR-100 (%) $\uparrow$
Base Model (No Unlearning)		63.38	61.99
Clipart	Baseline	54.11	57.40
	ADU	36.73	49.37
	Ours	57.33	57.50
Painting	Baseline	51.76	50.45
	ADU	28.45	32.54

Table 4: Zero-shot generalization robustness on ImageNet-val and CIFAR-100 after unlearning specific DomainNet domains. Higher values indicate better preservation of foundational, open-vocabulary knowledge without catastrophic semantic forgetting. The original base model serves as the upper bound reference.

Figure 3: Sensitivity to the Number of Training Samples. Unlearning capacity (HM) compared across varying shots (n) per domain. While the Baseline and ADU show limited or unstable improvement in low-data regimes, our method demonstrates significantly faster convergence, achieving superior unlearning performance with just 4 shots.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: SEMU unlearning performance on PACS art_painting domain and the Domainnet clipart domain.

Figure 5: Similarity between the top forget domain and top retain domain singular vectors. The majority red amongst the domainnet domains implies that the orthogonality between domains do not exist making SEMU-based approaches very hard to get it working off the shelf.

Domain	Forget Accuracy ( $\downarrow$ )		Retain Accuracy ( $\uparrow$ )		Harmonic Mean ( $\uparrow$ )
Domain	Baseline	TMS	Baseline	TMS	Baseline	TMS
Painting	0.3016	0.2222	0.8603	0.8254	0.7710	0.8009
Real	0.8651	0.2937	0.8386	0.8069	0.2324	0.7533
Clipart	0.2714	0.1508	0.8354	0.8164	0.7784	0.8325
Sketch	0.1635	0.0841	0.8228	0.8074	0.8296	0.8582

Appendix

Table 5: Comparison of Baseline and TMS Methods on a stronger pretrained ViT-L-14 backbone (

N=8

\gamma=0.5

Domain	Forget Accuracy ( $\downarrow$ )		Retain Accuracy ( $\uparrow$ )		Harmonic Mean ( $\uparrow$ )
Domain	Baseline	TMS	Baseline	TMS	Baseline	TMS
Painting	0.2222	0.1714	0.8566	0.8381	0.8153	0.8333
Real	0.1460	0.1206	0.8063	0.8095	0.8295	0.8430
Clipart	0.2079	0.1349	0.8360	0.8190	0.8134	0.8414
Sketch	0.1222	0.0746	0.8349	0.8212	0.8558	0.8702

Appendix

Table 6: Comparison of Baseline and TMS on a stronger pretrained ViT-L-14 backbone(

N=8

\gamma=1.0

Method	Painting			Real			Clipart			Sketch
Method	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )
Baseline	0.2222	0.8566	0.8153	0.1460	0.8063	0.8295	0.2079	0.8360	0.8134	0.1222	0.8349	0.8558
ADU	0.2524	0.8815	0.8091	0.2683	0.8503	0.7866	0.2365	0.8624	0.8099	0.2365	0.8847	0.8196
TMS (Ours)	0.1714	0.8381	0.8333	0.1206	0.8095	0.8430	0.1349	0.8190	0.8414	0.0746	0.8212	0.8702

Appendix

Table 7: Comparison of Methods Across Domains (

N=8

\gamma=1.0

). Metrics are F: Forget Accuracy, R: Retain Accuracy, and HM: Harmonic Mean.

VLM Unlearning

Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.

See all 31 papers

Oct 22, 2026

Zero-Shot Domain Generalization via Domain Prompt Learning

Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {Do}main Prompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.

Shiming Chen, Shuhuang Chen, Salman Khan +1

Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia +1

Domain Generalization Multimodal Domain Generalization Prompt Learning for VLMs Test-Time Adaptation of VLMs

Ask PDF

Sep 15, 2026

Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models

Pre-trained vision-language models (VLMs) exhibit strong cross-domain recognition performance even without additional training. However, this robustness can also preserve undesirable domain-specific behavior, as domain-related and semantic information often remain entangled within the learned representation space, making selective domain unlearning challenging. Existing approaches typically address this problem through latent-space disentanglement and prompt- or feature-level interventions, without directly attributing and attenuating individual patch-token contributions. However, here we suggest that rather than uniformly suppressing the full representation, it may be more effective to exploit the spatial structure of vision transformers to localize and suppress patch regions that contribute disproportionately to forget-domain prediction. Patches that strongly influence forget-domain prediction may not be equally important for semantic recognition, suggesting that forgetting should be guided according to the domain contribution of different visual regions. Specifically, we propose a two-stage patch-selective framework that first estimates patch-level domain sensitivity and then selectively attenuates patches whose contribution to forget-domain prediction is stronger than their semantic utility. We evaluate our framework on Office-Home, Mini DomainNet, and DomainNet. Experimental results demonstrate improved forgetting-retention tradeoffs compared to prior methods while improving retained-domain recognition by up to 3.8%. Additional evaluations under visually overlapping and unseen-domain settings further demonstrate improved robustness under distribution shift.

Akanksha Singh, Vinod K. Kurmi

Indian Institute of Science Education and Research Bhopal, India

VLM Unlearning Machine Unlearning VLM Robustness

Ask PDF

Feb 21, 2026

Open-Vocabulary Domain Generalization in Urban-Scene Segmentation

Domain Generalization in Semantic Segmentation (DG-SS) aims to enable segmentation models to perform robustly in unseen environments. However, conventional DG-SS methods are restricted to a fixed set of known categories, limiting their applicability in open-world scenarios. Recent progress in Vision-Language Models (VLMs) has advanced Open-Vocabulary Semantic Segmentation (OV-SS) by enabling models to recognize a broader range of concepts. Yet, these models remain sensitive to domain shifts and struggle to maintain robustness when deployed in unseen environments, a challenge that is particularly severe in urban-driving scenarios. To bridge this gap, we introduce Open-Vocabulary Domain Generalization in Semantic Segmentation (OVDG-SS), a new setting that jointly addresses unseen domains and unseen categories. We introduce the first benchmark for OVDG-SS in autonomous driving, addressing a previously unexplored problem and covering both synthetic-to-real and real-to-real generalization across diverse unseen domains and unseen categories. In OVDG-SS, we observe that domain shifts often distort text-image correlations in pre-trained VLMs, which hinders the performance of OV-SS models. To tackle this challenge, we propose S2-Corr, a state-space-driven text-image correlation refinement mechanism that mitigates domain-induced distortions and produces more consistent text-image correlations under distribution changes. Extensive experiments on our constructed benchmark demonstrate that the proposed method achieves superior cross-domain performance and efficiency compared to existing OV-SS approaches.

Dong Zhao, Qi Zang, Nan Pu +3

Department of Information Engineering and Computer Science, University of Trento, Italy · School of Computer Science and Information Engineering, Hefei University of Technology, China

Domain Generalization Open-Vocabulary Semantic Segmentation Autonomous Driving Benchmarks

Ask PDF

Abstract

VLM Unlearning

Explore similar work

Zero-Shot Domain Generalization via Domain Prompt Learning

Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models

Open-Vocabulary Domain Generalization in Urban-Scene Segmentation