VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalization

Minghui Chen; Cheng Wen; Feng Zheng; Fengxiang He; Ling Shao

cs.CVApr 25, 2022

VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalization

Authors: Minghui Chen, Cheng Wen, Feng Zheng, Fengxiang He, Ling Shao

Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology · The University of Sydney · JD Explore Academy, JD.com Inc · National Center for Artiﬁcial Intelligence, Saudi Data and Artiﬁcial Intelligence Authority, Riyadh, Saudi Arabia

Abstract

Invariance to diverse types of image corruption, such as noise, blurring, or colour shifts, is essential to establish robust models in computer vision. Data augmentation has been the major approach in improving the robustness against common corruptions. However, the samples produced by popular augmentation strategies deviate significantly from the underlying data manifold. As a result, performance is skewed toward certain types of corruption. To address this issue, we propose a multi-source vicinal transfer augmentation (VITA) method for generating diverse on-manifold samples. The proposed VITA consists of two complementary parts: tangent transfer and integration of multi-source vicinal samples. The tangent transfer creates initial augmented samples for improving corruption robustness. The integration employs a generative model to characterize the underlying manifold built by vicinal samples, facilitating the generation of on-manifold samples. Our proposed VITA significantly outperforms the current state-of-the-art augmentation methods, demonstrated in extensive experiments on corruption benchmarks. Code: https://github.com/MinghuiChen43/VITA.

Figures & tables

Panel of Figure 1 — Figure 1: Diverse on-manifold samples boost performance. (a) shows an underfitting phenomenon caused by the lack of sufficiently diverse vicinal samples. (b) takes a strong adversarial attack as an example to show the performance degradation caused by off-manifold samples. Classifiers tend to overfit these off-manifold samples and fail to construct a proper underlying manifold. (c) demonstrates the benefit of tangent transfer. While maintaining sample variety, it will not generate severe out-of-manifold samples. For a better understanding of samples that depart from the manifold, see our appendix for a three-dimensional schematic diagram.

Figure 2: Our proposed VITA includes tangent transfer and multi-source integration. We adopt vicinal difference ( $\Delta X$ ) as an approximation of the manifold tangent and use $X+\lambda\cdot\Delta X$ as an initial augmented sample. Our generative model includes a $pix2pix$ model, which is designed to generate diverse on-manifold samples. The goal of multi-source integration is to learn, based on dataset $\mathcal{D}=\{X^{v},Y\}$ ( $X^{v}=\{X^{\text{aug}},X^{adv}\}$ ), an embedding that imitates the generation process of vicinal samples $P(x^{v}|x,\delta x)$ , where $x$ is an original samples and $\delta x\in\Delta X(\Delta X=X^{v}-X$ ). Note that $\delta x$ is a transferred vicinal difference. In our robust multi-source training, models are trained with multi-source samples and samples from a well-trained translator.

Figure 3: Effectiveness of directly transferring differences. We compare the corruption robustness of models trained with different augmentation methods, including Gaussian noise (Gaussian), adversarial perturbations (adv), transferred perturbations (trans adv), weak augmentation (aug), transferred differences from weak augmentation (trans aug). We evaluate corruption robustness (error rate, the lower, the better) on CIFAR-10-C with an AllConvNet architecture. Here, clean indicates the performance on clean (uncorrupted) images, noise/blur/weather/digital is robustness towards the corresponding corruption types and mCE is mean corruption error for all types of corruption. As seen, transferring vicinal differences improves robustness significantly.

		Standard	Cutout	Mixup	CutMix	AutoAug	Adv Train	AugMix	ME-ADA	VITA
CIFAR-10-C	AllConvNet	30.8	32.9	24.6	31.3	29.2	28.1	15.0	21.8	10.6
	DenseNet	30.7	32.1	24.6	33.5	26.6	27.6	12.7	23.1	9.7
	WideResNet	26.9	26.8	22.3	27.1	23.9	26.2	11.2	16.7	9.5
	ResNeXt	27.5	28.9	22.6	29.5	24.2	27.0	10.9	16.6	8.9
Mean		29.0	30.2	23.5	30.3	26.0	27.2	12.5	19.5	9.7
CIFAR-100-C	AllConvNet	56.4	56.8	53.4	56.0	55.1	56.0	42.7	48.8	36.3

Table 1: Mean corruption error (mCE) on CIFAR-10-C and CIFAR-100-C. Average classification error as percentages. Across several architectures, our method obtains obvious enhancement in corruption robustness. Specifically, we reduced the error rate of corrupted data to 8.9 on ResNeXt.

		Noise			Blur				Weather				Digital
Network	Clean	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG	mCE
Standard	23.9	79	80	82	82	90	84	80	86	81	75	65	79	91	77	80	80.6
Patch Uniform	24.5	67	68	70	74	83	81	77	80	74	75	62	77	84	71	71	74.3
AutoAug	22.8	69	68	72	77	83	80	81	79	75	64	56	70	88	57	71	72.7
MaxBlur pool	23.0	73	74	76	74	86	78	77	77	72	63	56	68	86	71	71	73.4
SIN	27.2	69	70	70	77	84	76	82	74	75	69	65	69	80	64	77	73.3

Table 2: Clean error, mean corruption error (mCE) and all types of corruption error rate values for various methods on ImageNet-C. We compare against other data augmentation methods for improving the corruption robustness. Our proposed VITA hugely improves corruption robustness and achieves balanced performance toward different corrupted images.

Figure 4: Ablation study on vicinal differences in VITA, demonstrating the necessity of transferred vicinal differences as input to a translator. We evaluate corruption robustness (error rate, the lower, the better) on CIFAR-10-C with an AllConvNet architecture. Here, w/o diff is the translator trained and inferred without transferred differences, and w/ noise is the translator trained and inferred with the addition of Gaussian noise. Training process is the same setting as for default robust training process (50% gen + 25 % adv + 25% aug). As seen, a translator trained with Gaussian noise or without vicinal differences ( i.e. merely original data) performs worse than VITA.

Defense	Natural	FGSM	PGD-20	C $\&$ W ∞	PGD-100	AutoAttack
TRADES ( $\beta=1.0$ )	88.64	56.38	49.14	-	-	-
FAT for TRADES	89.94	61.00	49.70	49.35	48.35	47.22
VITA for Adv. Training	89.35	68.02	52.33	50.21	50.04	48.38
TRADES ( $\beta=6.0$ )	84.92	61.06	56.61	54.47	55.47	53.08
FAT for TRADES	86.60	61.97	55.98	54.29	55.34	53.27
VITA for Adv. Training	85.75	67.99	57.63	55.32	56.87	54.35

Table 3: Evaluations (test accuracy) of deep models (WRN-34-10) on the CIFAR-10 dataset. Results of TRADES (

\beta=1.0

and

6.0

) are reported in ( Zhang et al. 2019 ) . Results of FAT for TRADES are reported in ( Zhang et al. 2020 ) . Our proposed framework of VITA and multi-source integrated training can also improve the adversarial robustness of the model.

Figure 5: Ablation study on the data source in multi-source robust training reveals the importance of training with samples from VITA. We evaluate corruption robustness (error rate, the lower the better) on CIFAR-10-C with an AllConvNet architecture. Here, w/o aug/adv/gen indicate that specific types of samples lack in our multi-source robust training. For example, we train with 50% of samples with shuffled perturbations and 50% samples generated by VITA during training under the w/o aug setting. Obviously, when robust training does not include samples generated by VITA, corruption robustness decreases significantly.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Corruption Type	AlexNet Corruption Error
Gaussian Noise	88.6%
Shot Noise	89.4%
Impulse Noise	92.3%
Defocus Blur	82.0%
Glass Blur	82.6%
Motion Blur	78.6%

Appendix

Table 4: AlexNet errors in ImageNet-C

Figure 6: Examples of the 15 corruptions with severity level of 3 in the ImageNet-C corruption benchmark.

Panel of Figure 7 — Figure 7: Three-dimensional schematic diagram to show that diverse on-manifold samples boost performance. (a) shows an underfitting phenomenon caused by the lack of sufficiently diverse vicinal samples. (b) takes a strong adversarial attack as an example to show the performance degradation caused by off-manifold samples. Classifiers tend to overfit these off-manifold samples and fail to construct a proper underlying manifold. (c) demonstrates the benefit of tangent transfer. While maintaining sample variety, it will not generate severe out-of-manifold samples.

Panel of Figure 8 — Figure 8: Shared difference on data manifold. Left images are from the edge2handbags dataset. Transferring the difference from the backpack, a colored purse or satchel can be crafted using our proposed generative model (for more detailed generated images see our supplementary). Right images are from the CIFAR-10 dataset. We utilize the vicinal difference between the original airplane and the augmented one to guide the generation of vicinal samples of other images. As shown on the right, a ship and a car can also be transformed as the airplane.

Figure 9: The excellent and balanced performance of VITA on Corruption datasets. It can be seen that even if a diverse augmentation strategy AugMix is adopted, the model will still have extremely uneven performance on different types of corrupted images. In contrast, VITA not only has significantly higher overall corruption robustness performance than AugMix, but also performs evenly towards different types of corruption. In particular, when VITA is used in an advanced network architecture such as ResNeXt, the performance gap between the best corruption type and the worst corruption type is less than 10%, unlike AugMix, which has a gap of about 20%.

Figure 10: Training with corruption Gaussian noise , shot noise , impulse noise , defocus blur and glass blur .

Figure 11: Training with corruption motion blur , zoom blur , snow , frost and fog .

Figure 12: Training with corruption brightness , contrast , elastic transform , pixelate and JPEG compression .

Figure 13: Ablation on VITA components. On the whole, the initial settings of VITA can get a good robust performance, and our generation framework is not very sensitive to the parameter settings of the components of the framework.

Panel of Figure 14 — Figure 14: Images from CIFAR-10 dataset. Here, (a) is the original image, (b) is the target image and (c) is the generated image.

Panel of Figure 15 — Figure 15: Images from edges2handbags dataset. Here, (a) is the input (original) image, (b) is the target (ground truth) image, (c) is one generated image from VITA and (d) is another generated image from VITA

Network Architectures	Method	Clean	FGSM	PGD-20	C&W ( $l_{\infty}$ )
WRN-58-10	FAT for TRADES ( $\epsilon_{\text{train}}=8/255$ )	87.09	68.70	57.17	55.43
	VITA for Adv. Training ( $\epsilon_{\text{train}}=8/255$ )	88.42	73.05	60.55	55.87
	FAT for TRADES ( $\epsilon_{\text{train}}=16/255$ )	85.28	68.08	58.39	55.89
	VITA for Adv. Training ( $\epsilon_{\text{train}}=16/255$ )	86.85	72.76	60.83	56.77

Appendix

Table 5: Evaluations of adversarial robustness (test accuracy) of Wide ResNet on the CIFAR-10 dataset. It can be seen that our method not only improves the robustness but also improves the accuracy of clean samples significantly. Here,

\epsilon_{\text{train}}

is the maximum value of adversarial perturbation during training (

\epsilon_{\text{test}}=8/255

same standard adversarial robustness evaluation). In standard adversarial training, performance on clean samples hurt with larger

\epsilon_{\text{train}}

. In contrast, a model trained with VITA via multi-source integrated training can have a better performance on clean samples.

Corruption Type	All Source	Only Aug	Only Adv	Only Gen	w/o Aug	w/o Adv	w/o Gen
Clean	6.5	6.4	9.5	9.9	6.5	5.6	6.9
Noise	14.0	20.9	41.9	21.1	14.8	18.2	18.1
Gaussian Noise	13.0	24.4	42.9	23.0	15.2	21.3	18.6
Shot Noise	10.6	18.1	35.7	18.5	12.3	15.5	14.1
Impulse Noise	18.3	20.1	47.0	21.9	16.9	17.9	21.5
Blur	9.2	12.2	18.3	15.1	10.2	10.3	11.8

Appendix

Table 6: Contributions of different data sources to improve the corruption robustness (AllConvNet on CIFAR-10-C). Without samples generated by our method, the corruption robustness of the model is poor, which reflects the irreplaceability of our generated samples to significant improvement on the corruption robustness. Besides, only using any one or two sources of data for training does not have the effect of using all sources of data. This shows the necessity of multi-source integrated training.

Explore similar work

InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation
Jul 21, 2026Khawar Islam, Arif Mahmood, Xin Jin +1Data Augmentation Inpainting
The University of Melbourne, Melbourne, Australia · The Information Technology University, Lahore, Pakistan · Westlake University, China
Suppress and Diversify: Refining Robust Pathways for Corruption Robustness
Aug 7, 2026Jiangang Yang, Wenhui Shi, Xiaoran Xu +4Robustness Verification Adversarial Corruption
Institute of Microelectronics, Chinese Academy of Sciences, Beijing, China · University of Chinese Science, Beijing, China
How (and How Not) to Use Data Augmentation in VLA Post-Training
Oct 5, 2026Bram Grooten, Joaquin Vanschoren
TU Eindhoven

Jul 21, 2026, cs.CVJ/K move · Enter open · S save

InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation

Khawar Islam, Arif Mahmood, Xin Jin, Naveed Akhtar

The University of Melbourne, Melbourne, Australia · The Information Technology University, Lahore, Pakistan · Westlake University, China

Data Augmentation Inpainting Convolutional Neural Networks

Open paper AskPDF arXiv

In image and video technologies, data augmentation is widely used to improve the generalization of deep visual models, and mixup-based strategies that interpolate between samples have become the dominant approach. However, computing informative mixing regions adds substantial overhead, and blending content across different images frequently disrupts the semantic integrity of the resulting sample. We propose \our{}, a data augmentation method that constructs challenging yet label-consistent training samples entirely within a single visual sample. \our{} first extracts multi-scale salient patches from the sample using a lightweight saliency detector, refines each patch with an instruction-guided generative model, and blends the edited patch back into the non-salient regions of the same sample; because the generative edits are computed once and cached offline, this step adds negligible training cost. To further diversify the learned representation, \our{} injects self-similar fractal structure into the same salient regions at an adaptive ratio, so each training sample carries both fractal and non-fractal structure. We derive a second-order approximation of the resulting vicinal risk, showing that the method simultaneously enforces invariance to the generative edit and suppresses curvature along the perturbed salient directions, and we verify both predictions empirically. We evaluate on small to large backbones for instance Convolutional Neural Networks (CNNs), Vision Transformers (ViTs) and Vision-Language Foundational Models (VLMs) across seven benchmarks covering coarse- and fine-grained classification, robustness to corruption and occlusion, calibration, and transfer and self-supervised learning, InstructMixup outperforms nine competing augmentation methods, surpassing the strongest baseline across all benchmarks.

Abstract

Explore similar work

InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation

Suppress and Diversify: Refining Robust Pathways for Corruption Robustness

How (and How Not) to Use Data Augmentation in VLA Post-Training