Learning Robustness at Test-Time from a Non-Robust Teacher

Stefano Bianchettin; Giulio Rossolini; Giorgio Buttazzo

cs.CVApr 13, 2026

Learning Robustness at Test-Time from a Non-Robust Teacher

Authors: Stefano Bianchettin, Giulio Rossolini, Giorgio Buttazzo

Organizations: Department of Excellence in Robotics and AI, Sant’Anna School of Advanced Study, Via Moruzzi, 1, Pisa, 56124, Tuscany, Italy

Abstract

Nowadays, pretrained models are increasingly used as general-purpose backbones and adapted at test-time to downstream environments where target data are scarce and unlabeled. While this paradigm has proven effective for improving clean accuracy on the target domain, adversarial robustness has received far less attention, especially when the original pretrained model is not explicitly designed to be robust. This raises a practical question: can a pretrained, non-robust model be adapted at test-time to improve adversarial robustness on a target distribution? To face this question, this work studies how adversarial training strategies behave when integrated into adaptation schemes for the unsupervised test-time setting, where only a small set of unlabeled target samples is available. It first analyzes how classical adversarial training formulations can be extended to this scenario, showing that straightforward distillation-based adaptations remain unstable and highly sensitive to hyperparameter tuning, particularly when the teacher itself is non-robust. To address these limitations, the work proposes a label-free framework that uses the predictions of a non-robust teacher model as a semantic anchor for both the clean and adversarial objectives during adaptation. We further provide theoretical insights showing that our formulation yields a more stable alternative to the self-consistency-based regularization commonly used in classical adversarial training. Experiments evaluate the proposed approach on CIFAR-10 and ImageNet under induced photometric transformations. The results support the theoretical insights by showing that the proposed approach achieves improved optimization stability, lower sensitivity to parameter choices, and a better robustness-accuracy trade-off than existing baselines in this post-deployment test-time setting.

Figures & tables

Figure 1: Overview of the proposed Teacher-guided Robust Adaptation ( TgRA ) framework, detailed in Section 5 : the student model is fine-tuned at test-time under distribution shift, using a non-robust teacher as a reference for both the accuracy and robustness objectives, unlike TRADES-like approaches.

$\beta$	TgRA (Ours)				TRADES-U
$\beta$	Clean	PGD	PGN	Square	Clean	PGD	PGN	Square
Target Domain Severity = 0 (No shift, i.e., robust fine-tuning)
6	88.22	46.39	54.77	53.18	79.06	45.50	49.69	46.94
8	87.95	46.87	55.40	53.29	76.12	44.93	48.31	45.48
10	88.11	47.30	55.42	53.63	73.49	43.67	46.72	43.67
12	87.89	47.47	55.63	53.55	66.65	39.32	41.13	38.52

Table 1: Clean and robust accuracy on the CIFAR-10 test set for unsupervised robust test-time adaptation on CIFAR-10 training set, evaluated across different values of

\beta

for TgRA and TRADES-U under varying target-domain severity levels (for both training and testing).

Table 2: Clean and robust accuracy on the ResNet-50 ImageNet validation set across different values of

\beta

for TgRA and TRADES-U under varying target-domain severity levels.

$\beta$	TgRA (Ours)				TRADES-U
	Clean		Robust		Clean		Robust
	T1	T5	T1	T5	T1	T5	T1	T5
Target Domain Severity = 0
6	71.92	90.29	39.34	73.99	67.03	87.10	40.08	69.70
8	71.70	90.06	39.62	74.11	66.01	86.30	40.37	69.33
10	71.42	89.85	39.80	74.14	63.76	84.81	38.80	67.36

Table 3: Clean and robust accuracy on the ViT-B ImageNet validation set across different values of

\beta

for TgRA and TRADES-U under varying target-domain severity levels. For attacks

\varepsilon=2

Panel of Figure 2 — Figure 2: Clean and robust test accuracy under PGD-20 attacks during test-time finetuning on CIFAR-10 ( $\varepsilon=8/255$ ) and ImageNet-Val ( $\varepsilon=2/255$ ), evaluated across different levels of distribution-shift severity.

$\beta$	TgRA (Ours)		TRADES-U
$\beta$	Clean	Robust	Clean	Robust
Target Domain Severity = 0
6	73.46 ( $-14.76$ )	28.54 ( $-17.85$ )	66.42 ( $-12.64$ )	13.70 ( $-31.80$ )
8	73.38 ( $-14.57$ )	30.28 ( $-16.59$ )	58.92 ( $-17.20$ )	14.46 ( $-30.47$ )
10	73.78 ( $-14.33$ )	31.40 ( $-15.90$ )	51.30 ( $-22.19$ )	15.02 ( $-28.65$ )
12	74.20 ( $-13.69$ )	32.24 ( $-15.23$ )	47.72 ( $-18.93$ )	16.10 ( $-23.22$ )

Table 4: Clean and robust Accuracy on the split 50/50 CIFAR-10 test set for test-time unsupervised robust adaptation, evaluated across different values of

\beta

for TgRA and TRADES-U under varying target-domain severity levels. For clean and PGD robustness, the value in parentheses denotes the gap with respect to the corresponding result obtained in Table 1 .

Table 5: Comparison between TgRA and supervised adversarial finetuning approaches, evaluated in terms of clean and PGD-20 robust accuracy under varying target-domain severity levels (for both training and testing).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Clean and robust accuracy (Top-1) on ImageNet validation under increasing photometric corruption severity. Robust accuracy is evaluated using PGD-20 with different $\epsilon$ .

Panel of Figure 5 — Figure 5: Examples of the target-domain corruptions for CIFAR-10 (top) and ImageNet (bottom).

Severity	Noise $\sigma$	Blur kernel	Jitter $(b,c,s,h)$
1	0.03	3	(0.1, 0.1, 0.1, 0)
2	0.06	5	(0.2, 0.2, 0.2, 0)

Appendix

Table 6: Compound corruption settings used to define the target-domain shifts.

Split	$\varepsilon$	Clean		Robust
Split	$\varepsilon$	Top-1	Top-5	Top-1	Top-5
50/10	$2/255$	65.42	85.90	35.34	64.40
70/10		66.56	86.64	37.30	66.32
80/10		66.20	86.74	38.22	66.68
90/10		68.02	87.66	34.56	62.78
50/10	$4/255$	57.86	79.84	16.84	39.82

Appendix

Table 7: Analysis of the effect of test-time adaptation dataset size on ResNet50 and the ImageNet validation-set split, where the first value denotes the finetuning set size and the second the test set size, with no data corruption in either the finetuning or test data.

Explore similar work

Sample-wise Targeted Adversarial Attacks on Test-time Adaptation
May 22, 2026Phuc Duc Nguyen, Quang Duc NguyenTest-Time Adaptation Adversarial Attacks
College of Computing and Data Science Nanyang Technological University Singapore
Reliability-Guided Adaptive Ensembling for Robust Test-Time Adaptation
Jun 21, 2026Adam Koziak, Yuhong GuoTest-Time Adaptation Ensemble Learning
School of Computer Science, Carleton University, Ottawa, Canada · Canada CIFAR AI Chair, Amii, Canada
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Jul 10, 2026Xingyu Zhu, Huanshen Wu, Shuo Wang +4VLM Robustness VLM Adaptation
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology

May 22, 2026J/K move · Enter open · S save

Sample-wise Targeted Adversarial Attacks on Test-time Adaptation

Phuc Duc Nguyen, Quang Duc Nguyen

College of Computing and Data Science Nanyang Technological University Singapore

Test-Time Adaptation Adversarial Attacks Adversarial Robustness

Open paper AskPDF arXiv

Test-time adaptation (TTA) mitigates distribution shifts by adapting models to unlabeled test inputs, but also exposes them to adversarial manipulation. Existing class-wise targeted attacks remain suboptimal for stealthy exploitation in this setting: since TTA operates on batches, forcing a subset of samples toward a target label unintentionally pulls similar benign samples along, resulting in a conspicuously high frequency of the target label that is easy to detect. To capture a more realistic threat, we introduce a sample-wise targeted attack. Unlike prior approaches, the attacker aims to misclassify only inputs carrying an attacker-chosen trigger, while preserving a benign-like prediction distribution to evade detection. To achieve this, we propose a meta-learning-based attack with a novel priority-aware gradient alignment strategy that explicitly prioritizes attack success. The strategy formulates the gradient update as an ellipsoidal trust-region problem, mitigating gradient conflict between the attack and stealth objectives, while providing theoretical guarantees for effective optimization of the attack objective in the presence of gradient misalignment. Extensive experiments on CIFAR-10-C, CIFAR-100-C, and ImageNet-C across TTA protocols demonstrate that our method achieves high targeted success rates while maintaining prediction behavior close to benign adaptation under multiple label-free stealth metrics, making it difficult to detect in unlabeled TTA deployment scenarios. Furthermore, we demonstrate that our attack shows strong robustness against existing defenses.

Abstract

Explore similar work

Sample-wise Targeted Adversarial Attacks on Test-time Adaptation

Reliability-Guided Adaptive Ensembling for Robust Test-Time Adaptation

Robustifying Vision-Language Models via Test-Time Prompt Adaptation