Single domain generalization (SDG) aims to learn a model from one labeled source domain that generalizes to unseen target domains. A common strategy is to enrich the source distribution with augmented or generated samples, and recent text-to-image (T2I) diffusion models provide a strong generative prior for this purpose. However, diversity alone is insufficient for robust generalization, because useful generated samples should also capture variations that the current classifier finds difficult. Motivated by distributionally robust optimization (DRO), we define a semantic ambiguity set in the class-conditional generative space of a pretrained T2I model and search it for samples with high classification loss under the current classifier. To this end, we introduce PAPT++, a risk-aware adversarial generation-training framework for SDG. PAPT++ first learns diverse semantic reference images for each class through image-text alignment and intra-class diversity regularization. These references then serve as denoising targets during classifier-guided diffusion synthesis, reducing semantic drift while guiding generation toward challenging variations. The generated samples are combined with the source data to update the classifier, and the updated classifier guides the next synthesis round in return. In this way, PAPT++ progressively exposes the classifier to challenging yet semantically consistent variations. Extensive experiments on standard SDG benchmarks demonstrate the superiority of the proposed PAPT++ method and the effectiveness of its main components.
Figures & tables
Fig. 1: Comparison of data augmentation strategies for SDG. (A) Conventional augmentation expands the source distribution within predefined transformation spaces. (B) Diffusion-based augmentation provides broader semantic variations but is not explicitly guided by the current classifier. (C) PAPT++ searches the class-conditional T2I generative space for classifier-challenging samples while preserving class semantics.
Fig. 2: The framework of our proposed PAPT++. PAPT++ consists of two modules: (A) Class-Level Semantic Reference Learning (CSRL) and (B) Classifier-Guided Adversarial Diffusion Synthesis (CADS). CSRL learns diverse class-level semantic reference images by encouraging image-text alignment and intra-class diversity . CADS uses these reference images as denoising targets to regularize the diffusion parameters during classifier-guided adversarial synthesis, encouraging the generation of high-risk samples with reduced semantic drift. The generated challenge images are combined with source-domain data to update the downstream model, and the updated model provides feedback for the next synthesis round.
Fig. 3: Visualization of images generated with and without diversity regularization. (A) Without diversity regularization, the generator tends to produce eyeglass images with similar front-view layouts, similar frame shapes, and colorful blurred backgrounds. (B) With diversity regularization, the generated references show richer intra-class variations, including different viewpoints, frame shapes, lens colors, backgrounds, and eyeglass types.
Fig. 4: Optimization dynamics of the risk term Lrisk(r) in Eq. 24 and the semantic-reference denoising loss Lden(r) in Eq. 28 . The risk term guides the generator toward samples with high classification loss, while the denoising loss discourages semantic drift and non-semantic artifacts. Their balance produces high-risk yet semantically consistent samples for classifier training.
Algorithm 1 Training pipeline of our proposed PAPT++
Fig. 5: Comparison between the one-step denoising optimization used in PAPT and our progressive denoising optimization strategy. The proposed strategy gradually increases the number of denoising steps for back-propagation, providing longer trajectory-level guidance and improving image quality.
Method
Venue
level1
level2
level3
level4
level5
Avg.
ResNet-18
DANN [ 40 ]
IJCAI’16
75.40 ± 0.40
72.60 ± 0.30
69.70 ± 0.20
65.60 ± 0.00
59.60 ± 0.20
68.60
CORAL [ 41 ]
ICCV’16
76.00 ± 0.40
72.90 ± 0.20
69.90 ± 0.00
65.80 ± 0.10
59.60 ± 0.10
68.80
MMD [ 42 ]
ICCV’18
76.40 ± 0.40
72.90 ± 0.20
69.90 ± 0.00
65.80 ± 0.10
59.60 ± 0.40
68.80
Mixup [ 43 ]
-
76.30 ± 0.30
73.20 ± 0.20
70.20 ± 0.20
66.10 ± 0.10
60.10 ± 0.10
69.20
GroupDRO [ 20 ]
ICLR’20
76.00 ± 0.10
72.90 ± 0.10
69.80 ± 0.20
65.50 ± 0.30
59.50 ± 0.50
68.70
TABLE I: SDG results (%) on CIFAR-10-C with the backbone of ResNet-18. Each level (name in column) is viewed as a target domain, and a higher level denotes more severe corruption. The best results are in red and the second highest results are in blue .
Method
Venue
A
C
P
S
Avg.
ResNet-18
Augmix [ 58 ]
ICLR’20
66.54
70.16
38.30
52.48
56.87
RSC [ 44 ]
ECCV’20
73.40
75.90
41.60
56.20
61.80
L2D [ 10 ]
ICCV’21
76.91
77.88
52.29
53.66
65.18
RSC+ASR [ 44 ]
CVPR’21
76.70
79.30
54.60
61.60
68.10
pAdaIn [ 59 ]
CVPR’21
64.96
65.24
33.66
32.04
49.98
TABLE II: SDG results (%) on PACS with the backbone of ResNet-18. One domain is used as the source domain and the others are used as the target domain. The best results are in red and the second highest results are in blue .
Method
Venue
A
C
P
S
Avg.
ResNet-50
DANN [ 40 ]
IJCAI’16
79.00
76.50
48.70
57.90
65.50
CORAL [ 41 ]
ICCV’16
76.30
79.20
45.90
57.00
64.60
MMD [ 42 ]
ICCV’18
75.40
80.10
45.20
58.20
64.70
Mixup [ 43 ]
-
77.40
80.00
47.30
58.20
65.70
GroupDRO [ 20 ]
ICLR’20
79.00
79.00
42.00
60.80
65.20
TABLE III: SDG results (%) on PACS with the backbone of ResNet-50. One domain (name in column) is used as the source domain and the others are used as the target domains. The best results are in red and the second highest results are in blue .
Method
Venue
V
L
C
S
Avg.
ResNet-18
Augmix [ 58 ]
ICLR’20
75.25
59.52
45.90
57.43
59.53
ERM [ 47 ]
ICLR’21
76.72
58.86
44.95
57.71
59.56
pAdaIn [ 59 ]
CVPR’21
76.03
65.21
43.17
57.94
60.59
Mixstyle [ 9 ]
ICLR’21
75.73
61.29
44.66
56.57
59.56
EFDMix [ 60 ]
CVPR’22
72.35
61.41
52.34
63.28
62.33
TABLE IV: SDG results (%) on VLCS with the backbone of ResNet-18. One domain is used as the source domain and the others are used as the target domains. The best results are in red and the second highest results are in blue .
Method
Venue
A
C
P
R
Avg.
ResNet-50
DANN [ 40 ]
IJCAI’16
55.20
49.30
48.40
58.40
52.80
CORAL [ 41 ]
ICCV’16
55.60
52.80
50.30
59.40
54.50
MMD [ 42 ]
ICCV’18
55.10
52.00
50.30
59.30
54.20
Mixup [ 43 ]
-
55.50
54.10
49.40
59.40
54.60
GroupDRO [ 20 ]
ICLR’20
55.10
52.00
50.30
59.30
54.20
TABLE V: SDG results (%) on OfficeHome with the backbone of ResNet-50. One domain is used as the source domain and the others are used as the target domains. The best results are in red and the second highest results are in blue .
Method
Venue
A
C
P
R
Avg.
ResNet-50
DANN [ 40 ]
IJCAI’16
59.90
53.00
73.60
76.90
65.90
CORAL [ 41 ]
ICCV’16
64.10
54.50
76.20
77.80
68.20
GroupDRO [ 20 ]
ICLR’20
61.30
53.30
75.40
76.00
66.50
RSC [ 44 ]
ECCV’20
60.70
51.40
74.80
75.10
65.50
VREx [ 46 ]
ICML’21
60.70
53.00
75.30
76.60
66.40
TABLE VI: Multi-source DG results (%) on OfficeHome with the backbone of ResNet-50. One domain is used as the target domain and the others are used as the source domains. The best results are in red and the second highest results are in blue .
Method
Venue
V
L
C
S
Avg.
ResNet-50
DANN [ 40 ]
IJCAI’16
51.10
65.10
99.00
73.10
78.60
CORAL [ 41 ]
ICCV’16
77.50
66.10
98.30
73.40
78.80
MLDG [ 91 ]
AAAI’18
75.30
65.20
97.40
71.00
77.20
GroupDRO [ 20 ]
ICLR’20
76.70
63.40
97.30
69.50
76.70
RSC [ 44 ]
ECCV’20
75.60
62.50
97.90
72.30
77.10
TABLE VII: Multi-source DG results (%) on VLCS with the backbone of ResNet-50. One domain is used as the target domain and the others are used as the source domains. The best results are in red and the second highest results are in blue .
Method
Venue
L100
L38
L43
L46
Avg.
ResNet-50
DANN [ 40 ]
IJCAI’16
51.10
40.60
57.40
37.70
46.70
CORAL [ 41 ]
ICCV’16
51.60
42.20
57.00
39.80
47.70
GroupDRO [ 20 ]
ICLR’20
41.20
38.60
56.70
36.40
43.20
RSC [ 44 ]
ECCV’20
50.20
39.20
56.30
40.80
46.60
VREx [ 46 ]
ICML’21
48.20
41.70
56.80
38.70
46.40
TABLE VIII: Multi-source DG results (%) on TerraInc with the backbone of ResNet-50. One domain is used as the target domain and the others are used as the source domains. The best results are in red and the second highest results are in blue .
Idx
Method
PACS
VLCS
A
C
P
S
Avg.
Δ
V
L
C
S
Avg.
Δ
1
Source Only
58.60
66.40
34.00
27.50
46.60
↑26.83
71.81
61.06
52.60
62.32
61.95
↑12.26
2
PAPT
69.84
78.57
62.12
80.34
72.72
↑0.71
73.16
74.69
69.66
75.96
73.37
↑0.84
3
w/o CSRL
72.42
79.39
60.40
70.63
70.71
↑2.72
73.15
75.02
62.87
75.41
71.61
↑2.60
4
w/o CADS
67.44
74.17
56.30
72.04
67.49
↑5.94
73.57
73.93
67.95
70.21
71.42
↑2.79
5
w/o Diversity ( λdiv=0 )
70.23
76.83
49.60
76.01
68.17
↑5.26
73.23
76.99
62.35
75.53
72.03
↑2.18
TABLE IX: Ablation results (%) on PACS and VLCS with the ResNet-18 backbone.
Fig. 6: Effect of the denoising backpropagation schedule in CSRL. Progressive (PAPT++) and fixed one-step optimization (PAPT) are compared on PACS and VLCS using HPSv2 and ImageReward.
Fig. 7: Effect of the number of accumulated CADS rounds R on VLCS.
Idx
Method
Guide CE ↑
Transfer CE ↑
CLIP Cons. (%) ↑
ImageReward ↑
LPIPS ↑
PACS Avg. (%) ↑
1
w/o CADS
0.76
3.76
90.3
0.42
0.67
67.49
2
w/o Risk Guidance ( λrisk=0 )
0.78
6.05
93.1
0.78
0.59
72.67
3
w/o Reference Denoising ( Lden )
0.93
7.30
61.8
-1.47
0.60
70.65
4
PAPT++
0.79
6.70
87.5
-0.58
0.62
73.43
TABLE X: Analysis of generated samples and downstream performance for different CADS variants on PACS.
Method
Venue
Diffusion
Diffusion Params
GPU Usage (GB,BS=1)
Trainable Params
Extra Image Storage
VLCS
TerraInc
OfficeHome
Avg.
Max.
CDGA [ 93 ]
–
SD-v1.4
1.07B
5.0
6.0
23.50M
2.25 MiB
78.90
–
68.20
DomainFusion [ 94 ]
ECCV’24
SD-v1.4
1.07B
4.7
5.5
23.50M
0.29 MiB
79.20
51.10
72.40
Terra [ 95 ]
NeurIPS’24
SDXL
3.50B
20.0
24.0
70.50M
0.75 MiB
78.25
–
69.63
FDS-ERM [ 96 ]
WACV’25
SD-v1.5
1.07B
13.5
16.0
883.5M
3.73 MiB
79.80
–
71.10
TRIDENT [ 97 ]
–
SD-v2.1-unCLIP
1.93B
8.0
10.0
34.00M
3.38 MiB
77.80
–
70.00
TABLE XI: Comparison of computational and storage overheads and multi-source DG performance (%) among diffusion-based methods. The best results are in red and the second highest results are in blue .
Fig. 8: Hyper-parameter analysis of λdiv , λrisk , Ma , and Mg on PACS and VLCS.
Idx
Method
V
L
C
S
Avg.
1
CFG ( g=1 )
71.34
69.78
62.98
73.79
69.47
2
CFG ( g=3 )
72.50
70.77
65.66
72.85
70.44
3
CFG ( g=5 )
71.07
69.49
63.90
71.45
68.98
4
CFG++ ( λ=0.4 )
70.77
70.93
64.36
72.43
69.62
5
CFG++ ( λ=0.6 )
72.24
73.43
66.50
72.87
71.26
6
CFG++ ( λ=0.8 )
71.91
71.32
62.78
73.01
69.76
TABLE XII: Impact of CFG/CFG++ on VLCS (%) with ResNet-18 under a high risk-guidance weight ( λrisk=0.5 ). The best results are in red , and the second-highest results are in blue .
Fig. 9: Class-wise and average ImageReward scores over CADS synthesis rounds on PACS under different CFG scales g and CFG++ coefficients λ . Each class panel shows the score trajectory for one class, whereas “Avg” shows the mean across the seven classes. Higher scores indicate stronger agreement with human preferences for prompt-conditioned image generation.
Fig. 10: Class-wise and average ImageReward scores over CADS synthesis rounds on VLCS under different CFG scales g and CFG++ coefficients λ . Each class panel shows the score trajectory for one class, whereas “Avg” shows the mean across the five classes. Higher scores indicate stronger agreement with human preferences for prompt-conditioned image generation.
Idx
Method
A
C
P
S
Avg.
1
CFG ( g=1 )
67.12
77.77
60.00
78.04
70.73
2
CFG ( g=3 )
68.70
77.42
60.19
79.30
71.40
3
CFG ( g=5 )
71.43
77.95
65.55
74.68
72.40
4
CFG++ ( λ=0.4 )
70.64
82.08
62.56
77.32
73.15
5
CFG++ ( λ=0.6 )
73.02
81.54
67.79
78.88
75.31
6
CFG++ ( λ=0.8 )
73.61
78.02
64.56
77.15
73.34
Appendix
TABLE XIII: Impact of CFG/CFG++ on PACS (%) with ResNet-18 under a high risk-guidance weight ( λrisk=0.5 ). The best results are in red , and the second-highest results are in blue .
Symbol
Description
Basic setting and diffusion notation
C,k
Number of classes and class index, respectively, with k=1,…,C .
T,t,r
Total number of diffusion steps, diffusion timestep, and synthesis-round index, respectively. The subscript T in target-domain quantities such as PTk and πTk denotes the target domain rather than a diffusion timestep.
Ma,Mg
Numbers of semantic reference images and generated high-risk samples per class, respectively.
Z,FE,FD
Latent space and the frozen VAE encoder and decoder, respectively.
z0,zt,ϵ
Clean latent, noisy latent at diffusion timestep t , and Gaussian noise sampled from N(0,I) , respectively.
Appendix
TABLE XIV: Principal notation used in the theoretical analysis.
Fig. 11: Qualitative visualization of challenging samples conditioned on the dog class during PAPT++ generation–training rounds 1–4 using CFG++ with λ=0.6 . Each row corresponds to one round, and the mean ImageReward (IR) of the displayed samples is reported above the row.
Fig. 12: Qualitative visualization of challenging samples conditioned on the dog class during PAPT++ generation–training rounds 5–8 using CFG++ with λ=0.6 . Each row corresponds to one round, and the mean ImageReward (IR) of the displayed samples is reported above the row.
Single domain generalization (SDG) aims to learn a robust model, which could perform well on many unseen domains while there is only one single domain available for training. One of the promising directions for achieving single-domain generalization is to generate out-of-domain (OOD) training data through data augmentation or image generation. Given the rapid advancements in AI-generated content (AIGC), this paper is the first to propose leveraging powerful pre-trained text-to-image (T2I) foundation models to create the training data. However, manually designing textual prompts to generate images for all possible domains is often impractical, and some domain characteristics may be too abstract to describe with words. To address these challenges, we propose a novel Progressive Adversarial Prompt Tuning (PAPT) framework for pre-trained diffusion models. Instead of relying on static textual domains, our approach learns two sets of abstract prompts as conditions for the diffusion model: one that captures domain-invariant category information and another that models domain-specific styles. This adversarial learning mechanism enables the T2I model to generate images in various domain styles while preserving key categorical features. Extensive experiments demonstrate the effectiveness of the proposed method, achieving superior performances to state-of-the-art single-domain generalization approaches.
Zhipeng Xu, De Cheng, Xinyang Jiang +3
Xidian University · Microsoft Research Asia · Chongqing University of Posts and Telecommunications
Generalizing from a single labeled source domain to unseen target domains, without access to any target data during training, remains a fundamental challenge in robust machine learning. We address this underexplored setting, known as Single Domain Generalization (SDG), by proposing BiSDG, a bi-level optimization framework that explicitly decouples task learning from domain modeling. BiSDG simulates distribution shifts through surrogate domains constructed via label-preserving transformations of the source data. To capture domain-specific context, we propose a domain prompt encoder that generates lightweight modulation signals to produce augmenting features via feature-wise linear modulation. The learning process is formulated as a bi-level optimization problem: the inner objective optimizes task performance under fixed prompts, while the outer objective maximizes generalization across the surrogate domains by updating the domain prompt encoder. We further develop a practical gradient approximation scheme that enables efficient bi-level training without second-order derivatives. Extensive experiments on various SGD benchmarks demonstrate that BiSDG consistently outperforms prior methods, setting new state-of-the-art performance in the SDG setting.
Marzi Heidari, Hanping Zhang, Hao Yan +1
School of Computer Science, Carleton University, Ottawa, Canada · Canada CIFAR AI Chair, Amii, Canada
Incorporating diffusion-generated synthetic data into adversarial training (AT) has been shown to substantially improve the training of robust image classifiers. In this work, we extend the role of diffusion models beyond merely generating synthetic data, examining whether their internal representations, which encode meaningful features of the data, can provide additional benefits for robust classifier training. Through systematic experiments, we show that diffusion models offer representations that are both diverse and partially robust, and that explicitly incorporating diffusion representations as an auxiliary learning signal during AT consistently improves robustness across settings. Furthermore, our representation analysis indicates that incorporating diffusion models into AT encourages more disentangled features, while diffusion representations and diffusion-generated synthetic data play complementary roles in shaping representations. Experiments on CIFAR-10, CIFAR-100, and ImageNet validate these findings, demonstrating the effectiveness of jointly leveraging diffusion representations and synthetic data within AT.