Zero-Shot Domain Generalization via Domain Prompt Learning
Authors: Shiming Chen, Shuhuang Chen, Salman Khan, Fahad Shahbaz Khan
Organizations: Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia · Linköping University, Sweden
Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {Do}main Prompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.
Figures & tables
Figure 1: Standard Domain Generalization (DG) vs Zero-Shot Domain Generalization (ZSDG). (a)(c) Standard DG methods mainly learn the domain-invariant features or prompts from the given source domains to tackle domain shift in visual modality . They are infeasible to collect and annotate large-scale source domain data for training, resulting in limited applications of DG in realistic scenarios. (b)(d) The ZSDG method recognizes unseen classes in the target/unseen domains without data in the source domain by overcoming domain shift in text-visual modalities .
Figure 2: A schematic overview of DoP. DoP mainly consists of a text domain learning (TDL) module, a visual domain learning (VDL) module, and a classifier. Specifically, TDL first simulates various distribution shifts by learning diverse domain prompts implicitly in the text space with inter-style diversity loss and intra-style consistency loss and implicitly generates the style token S∗ that is relevant to the test image in target domains. Subsequently, VDL explicitly refines S∗ using the test visual data with self-supervised learning. Finally, DoP adopts the refined domain prompt for class prediction following the standard CLIP.
Figure 3: The process of style token selection.
Figure 4: Visual diversity across domains in four widely used DG benchmarks. Representative samples from PACS Li et al. (2017) , VLCS Fang et al. (2013) , OfficeHome Venkateswara et al. (2017) , and DomainNet Peng et al. (2019) illustrate substantial variations in rendering style, texture, background, and acquisition conditions, highlighting the difficulty of domain generalization.
Domains/Styles
Classes
Images
PACS
4
7
9,991
VLCS
4
5
10,729
OfficeHome
4
65
15,588
DomainNet
6
345
586,575
ImageNet
4
1000
98,389
Table 1: Statistics of four widely used DG datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet ) and one large-scale DG dataset (i.e., ImageNet), including the numbers of domains/styles, classes, and images.
Method
Source Data
PACS
VLCS
OfficeHome
DomainNet
Average
ResNet-50 with pre-trained weights on ImageNet
DANN Ganin et al. (2016)
✔
83.6
78.6
65.9
38.3
66.6
RSC Huang et al. (2020)
✔
85.2
77.1
65.5
38.9
66.7
SagNet Nam et al. (2021)
✔
86.3
77.8
68.1
40.3
68.1
SelfReg Kim et al. (2021)
✔
85.6
77.8
67.9
42.8
68.5
GVRT Min et al. (2022)
✔
85.1
79.0
70.1
44.1
69.6
Table 2: Domain generalization results ( % ) of the state-of-the-art DG methods on PACS, VLCS, OfficeHome, and DomainNet, including classical DG methods and VLM-based methods. Methods takes the source data for training marked with ✔ , otherwise, marked with ✗ .
Method
Source Data
ImageNet-V2
ImageNet-R
ImageNet-S
ImageNet-A
Average
CLIP Radford et al. (2021)
✗
60.8
74.0
47.8
46.1
57.2
CoOp Zhou et al. (2022b)
✔
64.2
75.2
47.9
49.7
59.3
CoCoOp Zhou et al. (2022a)
✔
64.1
76.2
48.8
50.6
59.9
TPT Shu et al. (2022)
✗
64.3
73.9
46.4
53.6
59.5
ProGrad Zhu et al. (2023)
✔
64.7
74.5
47.9
49.3
59.1
CuPL Pratt et al. (2023)
✗
63.3
77.1
48.8
50.8
60.0
Table 3: Domain generalization results ( % ) of the state-of-the-art methods on ImageNet. ViT-B/16 is used as a network backbone for fair comparison. Methods takes the source data for training marked with ✔ , otherwise, marked with ✗ .
Method
PACS
VLCS
OfficeHome
DomainNet
Average
ResNet-50 with pre-trained weights from CLIP
ZS-CLIP Radford et al. (2021)
90.6
76.0
68.6
45.6
70.2
ZS-CLIP(DP) Radford et al. (2021)
93.5
79.8
70.5
46.8
72.7
TPT Shu et al. (2022)
91.3
81.7
72.9
49.1
73.8
TPT(DP) Shu et al. (2022)
93.5
80.2
71.2
49.1
73.5
DPE Zhang et al. (2024)
91.8
82.0
73.5
47.3
73.7
Table 4: Zero-shot domain generalization results ( % ) of multiple methods on PACS, VLCS, OfficeHome, and DomainNet. “DP” denotes the methods take “ A random style of a [class] ” as text prompt.
Method
Art
Cartoon
Photo
Sketch
Average
ResNet-50 with pre-trained weights from CLIP
ZS-CLIP Radford et al. (2021)
88.9
94.4
99.3
79.8
90.6
TPT Shu et al. (2022)
92.5
93.0
99.4
80.4
91.3
PromptStyler Cho et al. (2023)
93.7
94.7
99.4
84.9
93.2
DoP (Ours)
94.7
94.9
99.4
85.8
93.7
ViT-B / 16 with pre-trained weights from CLIP
Table 5: Per-domain zero-shot domain generalization accuracy (%) on PACS using CLIP-pretrained ResNet-50 and ViT-B/16 backbones.
Method
PACS
VLCS
OfficeHome
DomainNet
Average
DoP w/o VDL
93.9
77.3
79.1
52.7
75.8
DoP w/o TDL
96.6
70.4
81.8
59.4
77.1
DoP w/o TDL(STS)
97.2
84.3
83.5
60.7
81.4
DoP (Full)
97.3
84.3
84.7
61.4
82.0
Table 6: Ablation studies for different components of our DoP on four datasets. “STS” denotes style token selection in TDL. We conduct these experiments using ViT-B/16 with pre-trained weights from CLIP.
Figure 5: We take stable diffusion (SD) to synthesize images according to the domain prompts on PACS. For each group, the first image is generated by SD according to the domain prompt with randomly initialized style tokens, the second image is the real test image in the target domain, the third image is generated by SD according to the domain prompt with the style tokens learned by our DoP. Results show that DoP effectively discovers the domain knowledge with similar styles of test images in the target domain.
Figure 6: t-SNE visualizations of text samples with random initialized style tokens and style tokens learned by TDL of our DoP. We randomly select five styles (denoted by various colors) from {Sk}k=110 and seven classes (denoted by various shapes) for easy viewing on PACS and OfficeHome datasets. Results show that DoP effectively learns diverse styles.
Figure 7: The effectiveness of the number of initialized style tokens K and confidence selection percentile ρ on PACS (a, c) and OfficeHome (b, d).
Method
Training Time (s) ↓
Testing Time (s) ↓
Per Image (ms) ↓
ZS-CLIP
–
61.2
6.1
TPT
–
4,624.1
462.8
PromptStyler
125.7
70.7
7.1
DoP (Ours)
4.8
1,703.5
170.5
Table 7: Computational efficiency comparison on the complete PACS benchmark using CLIP ViT-B/16. “–” indicates that no additional training is required.
Large pre-trained vision-language models like CLIP have transformed computer vision by aligning images and text in a shared feature space, enabling robust zero-shot transfer via prompting. Soft-prompting, such as Context Optimization (CoOp), effectively adapts these models for downstream recognition tasks by learning a set of context vectors. However, CoOp lacks explicit mechanisms for handling domain shifts across unseen distributions. To address this, we propose Domain-invariant Context Optimization (DiCoOp), an extension of CoOp optimized for domain generalization. By employing an adversarial training approach, DiCoOp forces the model to learn domain-invariant prompts while preserving discriminative power for classification. Experimental results show that DiCoOp consistently surpasses CoOp in domain generalization tasks across diverse visual domains.
Arsham Gholamzadeh Khoee, Yinan Yu, Robert Feldt
Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden
Recent advances in pre-training vision-language models (VLMs), e.g., contrastive language-image pre-training (CLIP) methods, have shown great potential in learning out-of-distribution (OOD) representations. Despite showing competitive performance, the prompt-based CLIP methods still suffer from: i) inaccurate text descriptions, which leads to degraded accuracy and robustness, and poses a challenge for zero-shot CLIP methods. ii) limited vision-language embedding alignment, which is one important factor affecting generalization performance. To tackle the above issues, this paper proposes a novel Conditional Domain prompt Learning (CoDoL) method, which utilizes readily-available domain information to form prompts and contributes to improved vision-language embedding alignment, which we identify as one factor underlying the observed OOD generalization gains. To capture both instance-specific and domain-specific information, we further propose a lightweight Domain Meta Network (DMN) to generate input-conditional tokens for images in each domain. Extensive experiments on four OOD benchmarks (PACS, VLCS, OfficeHome, and DigitDG) validate the effectiveness of our proposed CoDoL method in terms of empirically improves vision-language embedding alignment across four DG benchmarks, which we present as a contributing factor (rather than the sole cause) of the observed OOD gains.
Min Zhang, Yuyin Wang, Zhongxiang Dai +4
East China Normal University · Xidian University · Zhejiang Key Laboratory of Artificial Intelligence of Things (AIoT) Network and Data Security +3
Domain generalization aims to enhance model robustness against unseen domains with embedding distribution shifts. While large-scale vision-language models like CLIP exhibit strong generalization, their direct image-text embedding alignment suffers from inherent information asymmetry: images encode both class semantics and domain-specific styles, whereas text prompts primarily convey basic class cues. This asymmetry hinders generalization to novel domains in realistic scenarios. To address this, we propose Style-Homogenized Embedding alignment for Domain-generalization (SHED), a novel CLIP-based method that aligns style-homogenized embeddings instead of raw representations from encoders in CLIP. During training, SHED removes domain-specific style centroids from both image embeddings computed per source domains and text embeddings which are averaged across diverse prompt templates and stripped of a global centroid. For inference, considering the lack of target domain information, SHED projects diverse textual domain centroids into the visual space and aggregates predictions via membership weighting. Extensive experiments on five benchmarks show SHED achieves state-of-the-art performance, outperforming prior methods significantly (e.g., +4.0% on DomainNet vs. standard fine-tuning).
Kai Gan, Tong Wei
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China