Zero-Shot Domain Generalization via Domain Prompt Learning
Authors: Shiming Chen, Shuhuang Chen, Salman Khan, Fahad Shahbaz Khan
Organizations: Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia · Linköping University, Sweden
Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {Do}main Prompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.
Figures & tables
Figure 1: Standard Domain Generalization (DG) vs Zero-Shot Domain Generalization (ZSDG). (a)(c) Standard DG methods mainly learn the domain-invariant features or prompts from the given source domains to tackle domain shift in visual modality . They are infeasible to collect and annotate large-scale source domain data for training, resulting in limited applications of DG in realistic scenarios. (b)(d) The ZSDG method recognizes unseen classes in the target/unseen domains without data in the source domain by overcoming domain shift in text-visual modalities .
Figure 2: A schematic overview of DoP. DoP mainly consists of a text domain learning (TDL) module, a visual domain learning (VDL) module, and a classifier. Specifically, TDL first simulates various distribution shifts by learning diverse domain prompts implicitly in the text space with inter-style diversity loss and intra-style consistency loss and implicitly generates the style token S∗ that is relevant to the test image in target domains. Subsequently, VDL explicitly refines S∗ using the test visual data with self-supervised learning. Finally, DoP adopts the refined domain prompt for class prediction following the standard CLIP.
Figure 3: The process of style token selection.
Figure 4: Visual diversity across domains in four widely used DG benchmarks. Representative samples from PACS Li et al. (2017) , VLCS Fang et al. (2013) , OfficeHome Venkateswara et al. (2017) , and DomainNet Peng et al. (2019) illustrate substantial variations in rendering style, texture, background, and acquisition conditions, highlighting the difficulty of domain generalization.
Domains/Styles
Classes
Images
PACS
4
7
9,991
VLCS
4
5
10,729
OfficeHome
4
65
15,588
DomainNet
6
345
586,575
ImageNet
4
1000
98,389
Table 1: Statistics of four widely used DG datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet ) and one large-scale DG dataset (i.e., ImageNet), including the numbers of domains/styles, classes, and images.
Method
Source Data
PACS
VLCS
OfficeHome
DomainNet
Average
ResNet-50 with pre-trained weights on ImageNet
DANN Ganin et al. (2016)
✔
83.6
78.6
65.9
38.3
66.6
RSC Huang et al. (2020)
✔
85.2
77.1
65.5
38.9
66.7
SagNet Nam et al. (2021)
✔
86.3
77.8
68.1
40.3
68.1
SelfReg Kim et al. (2021)
✔
85.6
77.8
67.9
42.8
68.5
GVRT Min et al. (2022)
✔
85.1
79.0
70.1
44.1
69.6
Table 2: Domain generalization results ( % ) of the state-of-the-art DG methods on PACS, VLCS, OfficeHome, and DomainNet, including classical DG methods and VLM-based methods. Methods takes the source data for training marked with ✔ , otherwise, marked with ✗ .
Method
Source Data
ImageNet-V2
ImageNet-R
ImageNet-S
ImageNet-A
Average
CLIP Radford et al. (2021)
✗
60.8
74.0
47.8
46.1
57.2
CoOp Zhou et al. (2022b)
✔
64.2
75.2
47.9
49.7
59.3
CoCoOp Zhou et al. (2022a)
✔
64.1
76.2
48.8
50.6
59.9
TPT Shu et al. (2022)
✗
64.3
73.9
46.4
53.6
59.5
ProGrad Zhu et al. (2023)
✔
64.7
74.5
47.9
49.3
59.1
CuPL Pratt et al. (2023)
✗
63.3
77.1
48.8
50.8
60.0
Table 3: Domain generalization results ( % ) of the state-of-the-art methods on ImageNet. ViT-B/16 is used as a network backbone for fair comparison. Methods takes the source data for training marked with ✔ , otherwise, marked with ✗ .
Method
PACS
VLCS
OfficeHome
DomainNet
Average
ResNet-50 with pre-trained weights from CLIP
ZS-CLIP Radford et al. (2021)
90.6
76.0
68.6
45.6
70.2
ZS-CLIP(DP) Radford et al. (2021)
93.5
79.8
70.5
46.8
72.7
TPT Shu et al. (2022)
91.3
81.7
72.9
49.1
73.8
TPT(DP) Shu et al. (2022)
93.5
80.2
71.2
49.1
73.5
DPE Zhang et al. (2024)
91.8
82.0
73.5
47.3
73.7
Table 4: Zero-shot domain generalization results ( % ) of multiple methods on PACS, VLCS, OfficeHome, and DomainNet. “DP” denotes the methods take “ A random style of a [class] ” as text prompt.
Method
Art
Cartoon
Photo
Sketch
Average
ResNet-50 with pre-trained weights from CLIP
ZS-CLIP Radford et al. (2021)
88.9
94.4
99.3
79.8
90.6
TPT Shu et al. (2022)
92.5
93.0
99.4
80.4
91.3
PromptStyler Cho et al. (2023)
93.7
94.7
99.4
84.9
93.2
DoP (Ours)
94.7
94.9
99.4
85.8
93.7
ViT-B / 16 with pre-trained weights from CLIP
Table 5: Per-domain zero-shot domain generalization accuracy (%) on PACS using CLIP-pretrained ResNet-50 and ViT-B/16 backbones.
Method
PACS
VLCS
OfficeHome
DomainNet
Average
DoP w/o VDL
93.9
77.3
79.1
52.7
75.8
DoP w/o TDL
96.6
70.4
81.8
59.4
77.1
DoP w/o TDL(STS)
97.2
84.3
83.5
60.7
81.4
DoP (Full)
97.3
84.3
84.7
61.4
82.0
Table 6: Ablation studies for different components of our DoP on four datasets. “STS” denotes style token selection in TDL. We conduct these experiments using ViT-B/16 with pre-trained weights from CLIP.
Figure 5: We take stable diffusion (SD) to synthesize images according to the domain prompts on PACS. For each group, the first image is generated by SD according to the domain prompt with randomly initialized style tokens, the second image is the real test image in the target domain, the third image is generated by SD according to the domain prompt with the style tokens learned by our DoP. Results show that DoP effectively discovers the domain knowledge with similar styles of test images in the target domain.
Figure 6: t-SNE visualizations of text samples with random initialized style tokens and style tokens learned by TDL of our DoP. We randomly select five styles (denoted by various colors) from {Sk}k=110 and seven classes (denoted by various shapes) for easy viewing on PACS and OfficeHome datasets. Results show that DoP effectively learns diverse styles.
Figure 7: The effectiveness of the number of initialized style tokens K and confidence selection percentile ρ on PACS (a, c) and OfficeHome (b, d).
Method
Training Time (s) ↓
Testing Time (s) ↓
Per Image (ms) ↓
ZS-CLIP
–
61.2
6.1
TPT
–
4,624.1
462.8
PromptStyler
125.7
70.7
7.1
DoP (Ours)
4.8
1,703.5
170.5
Table 7: Computational efficiency comparison on the complete PACS benchmark using CLIP ViT-B/16. “–” indicates that no additional training is required.
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China