Open Vocabulary Domain Unlearning

Sumanth Udupa; Mehrtash Harandi; Yadan Luo; Mahsa Baktashmotlagh

cs.CVSep 25, 2026

Open Vocabulary Domain Unlearning

Authors: Sumanth Udupa, Mehrtash Harandi, Yadan Luo, Mahsa Baktashmotlagh

Organizations: The University of Queensland · Monash University

Abstract

Vision-Language Models (VLMs) exhibit remarkable zero-shot generalization, yet they often encode unwanted or hazardous stylistic domains such as idealized textbook diagrams in medical AI or cartoon vehicles in autonomous driving. Approximate Domain Unlearning (ADU) aims to selectively erase a model's recognition of a target visual domain while preserving accuracy on the remaining domains. However, existing ADU methods operate under a flawed closed-vocabulary assumption: they evaluate unlearning solely on the specific object classes seen during the unlearning fine-tuning phase. Consequently, these methods do not unlearn the domain itself; they merely overfit to seen class-domain pairs, leaving the domain easily recognizable for unseen classes and providing a false sense of removal. We argue that true domain erasure must be class-agnostic. To address this, we formalize Open-Vocabulary Domain Unlearning (OVDU), a rigorous protocol that mandates domain forgetting must transfer to held-out classes. To solve the OVDU challenge, we propose a surgical parameter-editing framework. First, a Fisher Information mask isolates domain-sensitive weights, mathematically protecting foundational zero-shot generalization. Second, our Targeted Manifold Scattering (TMS) objective uses preference-based mining to locally scatter the forget domain's stylistic geometry. Evaluated across PACS, OfficeHome, and DomainNet, our method vastly improves open-vocabulary generalization over existing baselines. Crucially, it delivers exceptional sample efficiency, outperforming peak 8-shot baseline results with only 4 shots.

Figures & tables

Figure 1: Overview of the Targeted Manifold Scattering (TMS) framework. Images from the retain (

\mathcal{D}_{r}

) and forget (

\mathcal{D}_{f}

) domains are processed through a Fisher-masked Image Encoder to isolate domain-specific updates. The TMS loss applies two decoupled geometric forces: (1) The Retention Component protects semantic integrity by contrasting same-class retain and forget embeddings (

y_{r}=y_{f}

). (2) The Targeted Confusion Component actively diffuses the forget representations by hard-mining cross-class retain clusters (

y_{r}\neq y_{f}

) and scattering the forget embeddings (

z_{f}

) into these unrelated regions, achieving efficient erasure without global semantic collapse.

Figure 2: Domain subspace entanglement across vision encoder layers. This figure quantifies the structural overlap between domains in pre-trained VLMs. The x-axis represents the progressive layers of the vision encoder, while the y-axis measures the mean principal angle cosine, which specifically captures the overlap between the top $k$ -singular vectors of the extracted domain subspaces (Forget and Retain). The prominent red bars denote exceptionally high structural similarity between the targeted forget domain and the retain domains across nearly all layers.

Dataset	Method	$\mathbf{\gamma=0.25}$			$\mathbf{\gamma=0.50}$			$\mathbf{\gamma=0.75}$			$\mathbf{\gamma=1.0}$
Dataset	Method	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$
OfficeHome	Baseline	59.54	65.65	50.07	51.62	67.92	56.51	36.97	71.90	67.17	31.31	77.90	73.01
	ADU	21.97	27.61	40.79	28.26	49.22	58.38	33.63	67.96	67.16	36.37	80.36	71.02
	Ours	53.33	63.24	53.71	39.34	68.33	64.27	24.90	68.68	71.75	24.61	75.45	75.42
DomainNet	Baseline	55.58	77.72	56.53	27.98	77.08	74.46	20.04	77.24	78.58	18.02	77.77	79.82
	ADU	14.64	24.45	38.01	19.68	49.25	61.06	23.41	67.27	71.63	27.86	81.48	76.53

Table 1: Unlearning performance across different class fractions (

\gamma

) with

|D_{f}|=1

. F : Forget domain accuracy (%), R : Retain domain accuracy (%), HM : Harmonic Mean (%). Best values for each fraction and dataset are in bold .

Domain	$\gamma$	Confuse Only			Retain Only			Both Losses (Ours)
Domain	$\gamma$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$
Clipart	0.50	19.68	77.25	78.75	20.63	78.04	78.70	17.93	76.82	79.35
Clipart	1.00	16.03	78.10	80.93	18.73	77.78	79.49	17.30	77.08	79.79
Painting	0.50	21.27	78.15	78.44	26.03	80.37	77.04	17.46	76.82	79.57
Painting	1.00	17.78	78.94	80.55	23.33	80.37	78.47	17.30	78.88	80.74
Real	0.50	31.27	71.48	70.08	38.57	72.86	66.66	32.38	72.70	70.06

Table 2: Ablation study on the DomainNet dataset comparing the individual components of our TMS loss formulation (Confuse-only, Retain-only, and Both). F : Forget domain accuracy (%), R : Retain domain accuracy (%), HM : Harmonic Mean (%). Best aggregate values are in bold .

Dataset	Method	$\mathbf{\gamma=0.25}$			$\mathbf{\gamma=0.50}$			$\mathbf{\gamma=0.75}$			$\mathbf{\gamma=1.0}$
Dataset	Method	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$	F $\downarrow$	R $\uparrow$	HM $\uparrow$
DomainNet	Baseline	8.92	59.87	72.25	4.54	61.06	74.48	3.26	62.13	75.67	3.33	63.45	76.61
	ADU	9.97	24.88	38.99	13.70	47.61	61.37	14.85	64.92	73.67	16.79	77.06	80.02
	Ours	7.11	60.19	73.05	5.07	62.45	75.34	4.81	66.10	78.02	4.59	67.21	78.86

Table 3: Unlearning performance on DomainNet for multiple domain forgetting (

|D_{f}|=3

) across different class fractions (

\gamma

). F : Forget domain accuracy (%), R : Retain domain accuracy (%), HM : Harmonic Mean (%). Best values for each fraction are in bold .

Forget Domain	Method	ImageNet-val (%) $\uparrow$	CIFAR-100 (%) $\uparrow$
Base Model (No Unlearning)		63.38	61.99
Clipart	Baseline	54.11	57.40
	ADU	36.73	49.37
	Ours	57.33	57.50
Painting	Baseline	51.76	50.45
	ADU	28.45	32.54

Table 4: Zero-shot generalization robustness on ImageNet-val and CIFAR-100 after unlearning specific DomainNet domains. Higher values indicate better preservation of foundational, open-vocabulary knowledge without catastrophic semantic forgetting. The original base model serves as the upper bound reference.

Figure 3: Sensitivity to the Number of Training Samples. Unlearning capacity (HM) compared across varying shots (n) per domain. While the Baseline and ADU show limited or unstable improvement in low-data regimes, our method demonstrates significantly faster convergence, achieving superior unlearning performance with just 4 shots.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: SEMU unlearning performance on PACS art_painting domain and the Domainnet clipart domain.

Figure 5: Similarity between the top forget domain and top retain domain singular vectors. The majority red amongst the domainnet domains implies that the orthogonality between domains do not exist making SEMU-based approaches very hard to get it working off the shelf.

Domain	Forget Accuracy ( $\downarrow$ )		Retain Accuracy ( $\uparrow$ )		Harmonic Mean ( $\uparrow$ )
Domain	Baseline	TMS	Baseline	TMS	Baseline	TMS
Painting	0.3016	0.2222	0.8603	0.8254	0.7710	0.8009
Real	0.8651	0.2937	0.8386	0.8069	0.2324	0.7533
Clipart	0.2714	0.1508	0.8354	0.8164	0.7784	0.8325
Sketch	0.1635	0.0841	0.8228	0.8074	0.8296	0.8582

Appendix

Table 5: Comparison of Baseline and TMS Methods on a stronger pretrained ViT-L-14 backbone (

N=8

\gamma=0.5

Domain	Forget Accuracy ( $\downarrow$ )		Retain Accuracy ( $\uparrow$ )		Harmonic Mean ( $\uparrow$ )
Domain	Baseline	TMS	Baseline	TMS	Baseline	TMS
Painting	0.2222	0.1714	0.8566	0.8381	0.8153	0.8333
Real	0.1460	0.1206	0.8063	0.8095	0.8295	0.8430
Clipart	0.2079	0.1349	0.8360	0.8190	0.8134	0.8414
Sketch	0.1222	0.0746	0.8349	0.8212	0.8558	0.8702

Appendix

Table 6: Comparison of Baseline and TMS on a stronger pretrained ViT-L-14 backbone(

N=8

\gamma=1.0

Method	Painting			Real			Clipart			Sketch
Method	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )	F ( $\downarrow$ )	R ( $\uparrow$ )	HM ( $\uparrow$ )
Baseline	0.2222	0.8566	0.8153	0.1460	0.8063	0.8295	0.2079	0.8360	0.8134	0.1222	0.8349	0.8558
ADU	0.2524	0.8815	0.8091	0.2683	0.8503	0.7866	0.2365	0.8624	0.8099	0.2365	0.8847	0.8196
TMS (Ours)	0.1714	0.8381	0.8333	0.1206	0.8095	0.8430	0.1349	0.8190	0.8414	0.0746	0.8212	0.8702

Appendix

Table 7: Comparison of Methods Across Domains (

N=8

\gamma=1.0

). Metrics are F: Forget Accuracy, R: Retain Accuracy, and HM: Harmonic Mean.

VLM Unlearning

Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.

See all 31 papers

Zero-Shot Domain Generalization via Domain Prompt Learning
Oct 22, 2026Shiming Chen, Shuhuang Chen, Salman Khan +1Domain Generalization Multimodal Domain Generalization
Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia +1
Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models
Sep 15, 2026Akanksha Singh, Vinod K. KurmiVLM Unlearning Machine Unlearning
Indian Institute of Science Education and Research Bhopal, India
Open-Vocabulary Domain Generalization in Urban-Scene Segmentation
Feb 21, 2026Dong Zhao, Qi Zang, Nan Pu +3Domain Generalization Open-Vocabulary Semantic Segmentation
Department of Information Engineering and Computer Science, University of Trento, Italy · School of Computer Science and Information Engineering, Hefei University of Technology, China

Oct 22, 2026J/K move · Enter open · S save

Zero-Shot Domain Generalization via Domain Prompt Learning

Shiming Chen, Shuhuang Chen, Salman Khan, Fahad Shahbaz Khan

Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia+1

Domain Generalization Multimodal Domain Generalization Prompt Learning for VLMs Test-Time Adaptation of VLMs

Open paper AskPDF arXiv

Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {Do}main Prompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.

Abstract

VLM Unlearning

Explore similar work

Zero-Shot Domain Generalization via Domain Prompt Learning

Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models

Open-Vocabulary Domain Generalization in Urban-Scene Segmentation