Domain-Invariant Prompt Learning for Vision-Language Models

Arsham Gholamzadeh Khoee; Yinan Yu; Robert Feldt

cs.CVMar 30, 2026

Domain-Invariant Prompt Learning for Vision-Language Models

Authors: Arsham Gholamzadeh Khoee, Yinan Yu, Robert Feldt

Organizations: Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden

Abstract

Large pre-trained vision-language models like CLIP have transformed computer vision by aligning images and text in a shared feature space, enabling robust zero-shot transfer via prompting. Soft-prompting, such as Context Optimization (CoOp), effectively adapts these models for downstream recognition tasks by learning a set of context vectors. However, CoOp lacks explicit mechanisms for handling domain shifts across unseen distributions. To address this, we propose Domain-invariant Context Optimization (DiCoOp), an extension of CoOp optimized for domain generalization. By employing an adversarial training approach, DiCoOp forces the model to learn domain-invariant prompts while preserving discriminative power for classification. Experimental results show that DiCoOp consistently surpasses CoOp in domain generalization tasks across diverse visual domains.

Figures & tables

Figure 1: Overview of Domain-invariant Context Optimization (DiCoOp). Domain First Prompting (DFP) is illustrated, where the first half of the prompt is dedicated to domain information, and the remaining half is dedicated to class information. During domain-related optimization, the class tokens remain frozen, and vice versa.

Panel of Figure 2 — Figure 2: Results of few-shot learning on the PACS dataset using the leave-one-domain-out technique. Each plot is defined by the domain name that is left out during prompt learning, and testing is performed on that same domain.

Methods	Backbone	Clipart	Painting	Sketch	Real	Avg.
CoOp	ViT-B/16	83.5	80.3	76.6	88.7	82.27
DiCoOp (SCP)	ViT-B/16	83.8	81.9	78.6	89.7	83.5
DiCoOp (DFP)	ViT-B/16	83.9	83.2	79.8	91.1	84.5
DiCoOp (CFP)	ViT-B/16	84.1	82.7	80.0	91.2	84.5

Table 1: Accuracy (%) on Mini-DomainNet for domain generalization using a leave-one-domain-out approach. Each column shows results on the domain that has been left out, comparing domain generalization performance to the baseline. Bold numbers indicate the best accuracy in each column.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Zero-Shot Domain Generalization via Domain Prompt Learning
Oct 22, 2026Shiming Chen, Shuhuang Chen, Salman Khan +1Domain Generalization Multimodal Domain Generalization
Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia +1
CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization
Sep 18, 2025Min Zhang, Yuyin Wang, Zhongxiang Dai +4VLM Adaptation Domain Generalization
East China Normal University · Xidian University · Zhejiang Key Laboratory of Artificial Intelligence of Things (AIoT) Network and Data Security +3
Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models
Apr 29, 2026Hongjun Wang, Po Hu, Kai HanVLM Adaptation Vision Foundation Model Adaptation
School of Computing and Data Science, The University of Hong Kong

Oct 22, 2026J/K move · Enter open · S save

Zero-Shot Domain Generalization via Domain Prompt Learning

Shiming Chen, Shuhuang Chen, Salman Khan, Fahad Shahbaz Khan

Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia+1

Domain Generalization Multimodal Domain Generalization Prompt Learning for VLMs Test-Time Adaptation of VLMs

Open paper AskPDF arXiv

Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {Do}main Prompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.

Abstract

VLM Adaptation

Explore similar work

Zero-Shot Domain Generalization via Domain Prompt Learning

CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization

Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models