PAPT++: Risk-Aware Adversarial Tuning and Generation for Single Domain Generalization

Zhipeng Xu; De Cheng; Xinyang Jiang; Lingfeng He; Huaijie Wang; Dongsheng Li; Nannan Wang; Xinbo Gao

cs.CVSep 4, 2026

PAPT++: Risk-Aware Adversarial Tuning and Generation for Single Domain Generalization

Authors: Zhipeng Xu, De Cheng, Xinyang Jiang, Lingfeng He, Huaijie Wang, Dongsheng Li, Nannan Wang, Xinbo Gao

Organizations: Xidian University, Xi’an, China · Microsoft Research Asia, Shanghai, China

Abstract

Single domain generalization (SDG) aims to learn a model from one labeled source domain that generalizes to unseen target domains. A common strategy is to enrich the source distribution with augmented or generated samples, and recent text-to-image (T2I) diffusion models provide a strong generative prior for this purpose. However, diversity alone is insufficient for robust generalization, because useful generated samples should also capture variations that the current classifier finds difficult. Motivated by distributionally robust optimization (DRO), we define a semantic ambiguity set in the class-conditional generative space of a pretrained T2I model and search it for samples with high classification loss under the current classifier. To this end, we introduce PAPT++, a risk-aware adversarial generation-training framework for SDG. PAPT++ first learns diverse semantic reference images for each class through image-text alignment and intra-class diversity regularization. These references then serve as denoising targets during classifier-guided diffusion synthesis, reducing semantic drift while guiding generation toward challenging variations. The generated samples are combined with the source data to update the classifier, and the updated classifier guides the next synthesis round in return. In this way, PAPT++ progressively exposes the classifier to challenging yet semantically consistent variations. Extensive experiments on standard SDG benchmarks demonstrate the superiority of the proposed PAPT++ method and the effectiveness of its main components.

Figures & tables

Fig. 1: Comparison of data augmentation strategies for SDG. (A) Conventional augmentation expands the source distribution within predefined transformation spaces. (B) Diffusion-based augmentation provides broader semantic variations but is not explicitly guided by the current classifier. (C) PAPT++ searches the class-conditional T2I generative space for classifier-challenging samples while preserving class semantics.

Fig. 2: The framework of our proposed PAPT++. PAPT++ consists of two modules: (A) Class-Level Semantic Reference Learning (CSRL) and (B) Classifier-Guided Adversarial Diffusion Synthesis (CADS). CSRL learns diverse class-level semantic reference images by encouraging image-text alignment and intra-class diversity . CADS uses these reference images as denoising targets to regularize the diffusion parameters during classifier-guided adversarial synthesis, encouraging the generation of high-risk samples with reduced semantic drift. The generated challenge images are combined with source-domain data to update the downstream model, and the updated model provides feedback for the next synthesis round.

Fig. 3: Visualization of images generated with and without diversity regularization. (A) Without diversity regularization, the generator tends to produce eyeglass images with similar front-view layouts, similar frame shapes, and colorful blurred backgrounds. (B) With diversity regularization, the generated references show richer intra-class variations, including different viewpoints, frame shapes, lens colors, backgrounds, and eyeglass types.

Fig. 4: Optimization dynamics of the risk term $\mathcal{L}_{\mathrm{risk}}^{(r)}$ in Eq. 24 and the semantic-reference denoising loss $\mathcal{L}_{\mathrm{den}}^{(r)}$ in Eq. 28 . The risk term guides the generator toward samples with high classification loss, while the denoising loss discourages semantic drift and non-semantic artifacts. Their balance produces high-risk yet semantically consistent samples for classifier training.

Panel of Figure 5 — Algorithm 1 Training pipeline of our proposed PAPT++

Fig. 5: Comparison between the one-step denoising optimization used in PAPT and our progressive denoising optimization strategy. The proposed strategy gradually increases the number of denoising steps for back-propagation, providing longer trajectory-level guidance and improving image quality.

ResNet-18
Method	Venue	level1	level2	level3	level4	level5	Avg.
DANN [ 40 ]	IJCAI’16	75.40 $\pm$ 0.40	72.60 $\pm$ 0.30	69.70 $\pm$ 0.20	65.60 $\pm$ 0.00	59.60 $\pm$ 0.20	68.60
CORAL [ 41 ]	ICCV’16	76.00 $\pm$ 0.40	72.90 $\pm$ 0.20	69.90 $\pm$ 0.00	65.80 $\pm$ 0.10	59.60 $\pm$ 0.10	68.80
MMD [ 42 ]	ICCV’18	76.40 $\pm$ 0.40	72.90 $\pm$ 0.20	69.90 $\pm$ 0.00	65.80 $\pm$ 0.10	59.60 $\pm$ 0.40	68.80
Mixup [ 43 ]	-	76.30 $\pm$ 0.30	73.20 $\pm$ 0.20	70.20 $\pm$ 0.20	66.10 $\pm$ 0.10	60.10 $\pm$ 0.10	69.20
GroupDRO [ 20 ]	ICLR’20	76.00 $\pm$ 0.10	72.90 $\pm$ 0.10	69.80 $\pm$ 0.20	65.50 $\pm$ 0.30	59.50 $\pm$ 0.50	68.70

TABLE I: SDG results (%) on CIFAR-10-C with the backbone of ResNet-18. Each level (name in column) is viewed as a target domain, and a higher level denotes more severe corruption. The best results are in red and the second highest results are in blue .

ResNet-18
Method	Venue	A	C	P	S	Avg.
Augmix [ 58 ]	ICLR’20	66.54	70.16	38.30	52.48	56.87
RSC [ 44 ]	ECCV’20	73.40	75.90	41.60	56.20	61.80
L2D [ 10 ]	ICCV’21	76.91	77.88	52.29	53.66	65.18
RSC+ASR [ 44 ]	CVPR’21	76.70	79.30	54.60	61.60	68.10
pAdaIn [ 59 ]	CVPR’21	64.96	65.24	33.66	32.04	49.98

TABLE II: SDG results (%) on PACS with the backbone of ResNet-18. One domain is used as the source domain and the others are used as the target domain. The best results are in red and the second highest results are in blue .

ResNet-50
Method	Venue	A	C	P	S	Avg.
DANN [ 40 ]	IJCAI’16	79.00	76.50	48.70	57.90	65.50
CORAL [ 41 ]	ICCV’16	76.30	79.20	45.90	57.00	64.60
MMD [ 42 ]	ICCV’18	75.40	80.10	45.20	58.20	64.70
Mixup [ 43 ]	-	77.40	80.00	47.30	58.20	65.70
GroupDRO [ 20 ]	ICLR’20	79.00	79.00	42.00	60.80	65.20

TABLE III: SDG results (%) on PACS with the backbone of ResNet-50. One domain (name in column) is used as the source domain and the others are used as the target domains. The best results are in red and the second highest results are in blue .

ResNet-18
Method	Venue	V	L	C	S	Avg.
Augmix [ 58 ]	ICLR’20	75.25	59.52	45.90	57.43	59.53
ERM [ 47 ]	ICLR’21	76.72	58.86	44.95	57.71	59.56
pAdaIn [ 59 ]	CVPR’21	76.03	65.21	43.17	57.94	60.59
Mixstyle [ 9 ]	ICLR’21	75.73	61.29	44.66	56.57	59.56
EFDMix [ 60 ]	CVPR’22	72.35	61.41	52.34	63.28	62.33

TABLE IV: SDG results (%) on VLCS with the backbone of ResNet-18. One domain is used as the source domain and the others are used as the target domains. The best results are in red and the second highest results are in blue .

ResNet-50
Method	Venue	A	C	P	R	Avg.
DANN [ 40 ]	IJCAI’16	55.20	49.30	48.40	58.40	52.80
CORAL [ 41 ]	ICCV’16	55.60	52.80	50.30	59.40	54.50
MMD [ 42 ]	ICCV’18	55.10	52.00	50.30	59.30	54.20
Mixup [ 43 ]	-	55.50	54.10	49.40	59.40	54.60
GroupDRO [ 20 ]	ICLR’20	55.10	52.00	50.30	59.30	54.20

TABLE V: SDG results (%) on OfficeHome with the backbone of ResNet-50. One domain is used as the source domain and the others are used as the target domains. The best results are in red and the second highest results are in blue .

ResNet-50
Method	Venue	A	C	P	R	Avg.
DANN [ 40 ]	IJCAI’16	59.90	53.00	73.60	76.90	65.90
CORAL [ 41 ]	ICCV’16	64.10	54.50	76.20	77.80	68.20
GroupDRO [ 20 ]	ICLR’20	61.30	53.30	75.40	76.00	66.50
RSC [ 44 ]	ECCV’20	60.70	51.40	74.80	75.10	65.50
VREx [ 46 ]	ICML’21	60.70	53.00	75.30	76.60	66.40

TABLE VI: Multi-source DG results (%) on OfficeHome with the backbone of ResNet-50. One domain is used as the target domain and the others are used as the source domains. The best results are in red and the second highest results are in blue .

ResNet-50
Method	Venue	V	L	C	S	Avg.
DANN [ 40 ]	IJCAI’16	51.10	65.10	99.00	73.10	78.60
CORAL [ 41 ]	ICCV’16	77.50	66.10	98.30	73.40	78.80
MLDG [ 91 ]	AAAI’18	75.30	65.20	97.40	71.00	77.20
GroupDRO [ 20 ]	ICLR’20	76.70	63.40	97.30	69.50	76.70
RSC [ 44 ]	ECCV’20	75.60	62.50	97.90	72.30	77.10

TABLE VII: Multi-source DG results (%) on VLCS with the backbone of ResNet-50. One domain is used as the target domain and the others are used as the source domains. The best results are in red and the second highest results are in blue .

ResNet-50
Method	Venue	L100	L38	L43	L46	Avg.
DANN [ 40 ]	IJCAI’16	51.10	40.60	57.40	37.70	46.70
CORAL [ 41 ]	ICCV’16	51.60	42.20	57.00	39.80	47.70
GroupDRO [ 20 ]	ICLR’20	41.20	38.60	56.70	36.40	43.20
RSC [ 44 ]	ECCV’20	50.20	39.20	56.30	40.80	46.60
VREx [ 46 ]	ICML’21	48.20	41.70	56.80	38.70	46.40

TABLE VIII: Multi-source DG results (%) on TerraInc with the backbone of ResNet-50. One domain is used as the target domain and the others are used as the source domains. The best results are in red and the second highest results are in blue .

Idx	Method	PACS						VLCS
Idx	Method	A	C	P	S	Avg.	$\Delta$	V	L	C	S	Avg.	$\Delta$
1	Source Only	58.60	66.40	34.00	27.50	46.60	$\uparrow 26.83$	71.81	61.06	52.60	62.32	61.95	$\uparrow 12.26$
2	PAPT	69.84	78.57	62.12	80.34	72.72	$\uparrow 0.71$	73.16	74.69	69.66	75.96	73.37	$\uparrow 0.84$
3	w/o CSRL	72.42	79.39	60.40	70.63	70.71	$\uparrow 2.72$	73.15	75.02	62.87	75.41	71.61	$\uparrow 2.60$
4	w/o CADS	67.44	74.17	56.30	72.04	67.49	$\uparrow 5.94$	73.57	73.93	67.95	70.21	71.42	$\uparrow 2.79$
5	w/o Diversity ( $\lambda_{\mathrm{div}}=0$ )	70.23	76.83	49.60	76.01	68.17	$\uparrow 5.26$	73.23	76.99	62.35	75.53	72.03	$\uparrow 2.18$

TABLE IX: Ablation results (%) on PACS and VLCS with the ResNet-18 backbone.

Fig. 6: Effect of the denoising backpropagation schedule in CSRL. Progressive (PAPT++) and fixed one-step optimization (PAPT) are compared on PACS and VLCS using HPSv2 and ImageReward.

Fig. 7: Effect of the number of accumulated CADS rounds $R$ on VLCS.

Idx	Method	Guide CE $\uparrow$	Transfer CE $\uparrow$	CLIP Cons. (%) $\uparrow$	ImageReward $\uparrow$	LPIPS $\uparrow$	PACS Avg. (%) $\uparrow$
1	w/o CADS	0.76	3.76	90.3	0.42	0.67	67.49
2	w/o Risk Guidance ( $\lambda_{\mathrm{risk}}=0$ )	0.78	6.05	93.1	0.78	0.59	72.67
3	w/o Reference Denoising ( $\mathcal{L}_{\mathrm{den}}$ )	0.93	7.30	61.8	-1.47	0.60	70.65
4	PAPT++	0.79	6.70	87.5	-0.58	0.62	73.43

TABLE X: Analysis of generated samples and downstream performance for different CADS variants on PACS.

Method	Venue	Diffusion	Diffusion Params	GPU Usage (GB,BS=1)		Trainable Params	Extra Image Storage	VLCS	TerraInc	OfficeHome
Method	Venue	Diffusion	Diffusion Params	Avg.	Max.	Trainable Params	Extra Image Storage	VLCS	TerraInc	OfficeHome
CDGA [ 93 ]	–	SD-v1.4	1.07B	5.0	6.0	23.50M	2.25 MiB	78.90	–	68.20
DomainFusion [ 94 ]	ECCV’24	SD-v1.4	1.07B	4.7	5.5	23.50M	0.29 MiB	79.20	51.10	72.40
Terra [ 95 ]	NeurIPS’24	SDXL	3.50B	20.0	24.0	70.50M	0.75 MiB	78.25	–	69.63
FDS-ERM [ 96 ]	WACV’25	SD-v1.5	1.07B	13.5	16.0	883.5M	3.73 MiB	79.80	–	71.10
TRIDENT [ 97 ]	–	SD-v2.1-unCLIP	1.93B	8.0	10.0	34.00M	3.38 MiB	77.80	–	70.00

TABLE XI: Comparison of computational and storage overheads and multi-source DG performance (%) among diffusion-based methods. The best results are in red and the second highest results are in blue .

Fig. 8: Hyper-parameter analysis of

\lambda_{\mathrm{div}}

\lambda_{\mathrm{risk}}

M_{a}

, and

M_{g}

on PACS and VLCS.

Idx	Method	V	L	C	S	Avg.
1	CFG ( $g=1$ )	71.34	69.78	62.98	73.79	69.47
2	CFG ( $g=3$ )	72.50	70.77	65.66	72.85	70.44
3	CFG ( $g=5$ )	71.07	69.49	63.90	71.45	68.98
4	CFG++ ( $\lambda=0.4$ )	70.77	70.93	64.36	72.43	69.62
5	CFG++ ( $\lambda=0.6$ )	72.24	73.43	66.50	72.87	71.26
6	CFG++ ( $\lambda=0.8$ )	71.91	71.32	62.78	73.01	69.76

TABLE XII: Impact of CFG/CFG++ on VLCS (%) with ResNet-18 under a high risk-guidance weight (

\lambda_{\mathrm{risk}}=0.5

). The best results are in red , and the second-highest results are in blue .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 9: Class-wise and average ImageReward scores over CADS synthesis rounds on PACS under different CFG scales $g$ and CFG++ coefficients $\lambda$ . Each class panel shows the score trajectory for one class, whereas “Avg” shows the mean across the seven classes. Higher scores indicate stronger agreement with human preferences for prompt-conditioned image generation.

Fig. 10: Class-wise and average ImageReward scores over CADS synthesis rounds on VLCS under different CFG scales $g$ and CFG++ coefficients $\lambda$ . Each class panel shows the score trajectory for one class, whereas “Avg” shows the mean across the five classes. Higher scores indicate stronger agreement with human preferences for prompt-conditioned image generation.

Idx	Method	A	C	P	S	Avg.
1	CFG ( $g=1$ )	67.12	77.77	60.00	78.04	70.73
2	CFG ( $g=3$ )	68.70	77.42	60.19	79.30	71.40
3	CFG ( $g=5$ )	71.43	77.95	65.55	74.68	72.40
4	CFG++ ( $\lambda=0.4$ )	70.64	82.08	62.56	77.32	73.15
5	CFG++ ( $\lambda=0.6$ )	73.02	81.54	67.79	78.88	75.31
6	CFG++ ( $\lambda=0.8$ )	73.61	78.02	64.56	77.15	73.34

Appendix

TABLE XIII: Impact of CFG/CFG++ on PACS (%) with ResNet-18 under a high risk-guidance weight (

\lambda_{\mathrm{risk}}=0.5

). The best results are in red , and the second-highest results are in blue .

Symbol	Description
Basic setting and diffusion notation
$C,\;k$	Number of classes and class index, respectively, with $k=1,\ldots,C$ .
$T,\;t,\;r$	Total number of diffusion steps, diffusion timestep, and synthesis-round index, respectively. The subscript $T$ in target-domain quantities such as $P_{T}^{k}$ and $\pi_{T}^{k}$ denotes the target domain rather than a diffusion timestep.
$M_{a},\;M_{g}$	Numbers of semantic reference images and generated high-risk samples per class, respectively.
$\mathcal{Z},\;\mathcal{F}_{E},\;\mathcal{F}_{D}$	Latent space and the frozen VAE encoder and decoder, respectively.
$\mathbf{z}_{0},\;\mathbf{z}_{t},\;\boldsymbol{\epsilon}$	Clean latent, noisy latent at diffusion timestep $t$ , and Gaussian noise sampled from $\mathcal{N}(\mathbf{0},\mathbf{I})$ , respectively.

Appendix

TABLE XIV: Principal notation used in the theoretical analysis.

Fig. 11: Qualitative visualization of challenging samples conditioned on the dog class during PAPT++ generation–training rounds 1–4 using CFG++ with $\lambda=0.6$ . Each row corresponds to one round, and the mean ImageReward (IR) of the displayed samples is reported above the row.

Fig. 12: Qualitative visualization of challenging samples conditioned on the dog class during PAPT++ generation–training rounds 5–8 using CFG++ with $\lambda=0.6$ . Each row corresponds to one round, and the mean ImageReward (IR) of the displayed samples is reported above the row.

Explore similar work

Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization
Jun 19, 2026Zhipeng Xu, De Cheng, Xinyang Jiang +3Synthetic Data Augmentation Adversarial Prompt Generation
Xidian University · Microsoft Research Asia · Chongqing University of Posts and Telecommunications
Bi-Level Optimization for Single Domain Generalization
Apr 7, 2026Marzi Heidari, Hanping Zhang, Hao Yan +1Cross-Domain Generalization
School of Computer Science, Carleton University, Ottawa, Canada · Canada CIFAR AI Chair, Amii, Canada
Expanding the Role of Diffusion Models for Robust Classifier Training
Feb 23, 2026Pin-Han Huang, Shang-Tse Chen, Hsuan-Tien LinAdversarial Robustness Adversarial Training
National Taiwan University

Jun 19, 2026J/K move · Enter open · S save

Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization

Zhipeng Xu, De Cheng, Xinyang Jiang, Nannan Wang+2

Xidian University · Microsoft Research Asia · Chongqing University of Posts and Telecommunications

Synthetic Data Augmentation Adversarial Prompt Generation T2I Diffusion Models

Open paper AskPDF arXiv

Single domain generalization (SDG) aims to learn a robust model, which could perform well on many unseen domains while there is only one single domain available for training. One of the promising directions for achieving single-domain generalization is to generate out-of-domain (OOD) training data through data augmentation or image generation. Given the rapid advancements in AI-generated content (AIGC), this paper is the first to propose leveraging powerful pre-trained text-to-image (T2I) foundation models to create the training data. However, manually designing textual prompts to generate images for all possible domains is often impractical, and some domain characteristics may be too abstract to describe with words. To address these challenges, we propose a novel Progressive Adversarial Prompt Tuning (PAPT) framework for pre-trained diffusion models. Instead of relying on static textual domains, our approach learns two sets of abstract prompts as conditions for the diffusion model: one that captures domain-invariant category information and another that models domain-specific styles. This adversarial learning mechanism enables the T2I model to generate images in various domain styles while preserving key categorical features. Extensive experiments demonstrate the effectiveness of the proposed method, achieving superior performances to state-of-the-art single-domain generalization approaches.

Abstract

Explore similar work

Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization

Bi-Level Optimization for Single Domain Generalization

Expanding the Role of Diffusion Models for Robust Classifier Training