Test-Time Defense Against Adversarial Attacks via Stochastic Resonance of Latent Ensembles
Organizations: LSU Vision Lab · UCLA Vision Lab · Yale Vision Lab
Abstract
We propose a test-time defense mechanism against adversarial attacks: imperceptible image perturbations that significantly alter the predictions of a model. Unlike existing methods that rely on feature filtering or smoothing, which can lead to information loss, we propose to "combat noise with noise" by leveraging stochastic resonance to enhance robustness while minimizing information loss. Our approach introduces small translational perturbations to the input image, aligns the transformed feature embeddings, and aggregates them before mapping back to the original reference image. This can be expressed in a closed-form formula, which can be deployed on diverse existing network architectures without introducing additional network modules or fine-tuning for specific attack types. The resulting method is entirely training-free, architecture-agnostic, and attack-agnostic. Empirical results show state-of-the-art robustness on image classification and, for the first time, establish a generic test-time defense for dense prediction tasks, including stereo matching and optical flow, highlighting the method's versatility and practicality. Specifically, relative to clean (unperturbed) performance, our method recovers up to 68.1% of the accuracy loss on image classification, 71.9% on stereo matching, and 29.2% on optical flow under various types of adversarial attacks.
Figures & tables
| Method | Architecture | Natural | FGSM | PGD-20 | PGD-100 | C&W | Ensemble | AutoAttack |
| AT Madry et al. (2017) | ResNet-18 | 85.02 | 56.21 | 48.22 | 46.37 | 47.38 | 45.90 | 44.11 |
| + FD Xie et al. (2019) | 85.14 | 56.81 | 48.54 | 46.70 | 47.72 | 45.82 | 44.57 | |
| + CAS Bai et al. (2021) | 85.78 | 55.57 | 50.42 | 49.91 | 53.47 | 46.46 | 44.23 | |
| + CIFS Yan et al. (2021) | 79.87 | 56.53 | 49.80 | 48.17 | 49.89 | 47.26 | 43.94 | |
| + FSR Kim et al. (2023) | 81.46 | 58.07 | 52.47 | 51.02 | 49.44 | 48.34 | 46.41 | |
| + TTE Pérez et al. (2021) | 85.25 | 59.20 | 53.00 | 51.65 | 52.45 | 50.60 | - |
| Att. Strength ( ) | 8/255 | 4/255 | 2/255 | 1/255 | ||||
|---|---|---|---|---|---|---|---|---|
| Metric | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 |
| No Defense | 4.51 | 19.55 | 12.25 | 42.15 | 29.79 | 65.09 | 48.09 | 78.13 |
| Ours ( ) | 11.66 | 46.27 | 27.85 | 65.80 | 45.94 | 77.63 | 57.48 | 83.43 |
| Ours ( ) | 18.78 | 58.58 | 36.42 | 72.86 | 51.58 | 80.85 | 60.41 | 84.77 |
| Ours ( ) | 25.77 | 65.88 | 42.52 | 76.75 | 54.94 | 82.64 | 62.08 | 85.50 |
| Attack Strength ( ) | 0.02 | 0.01 | 0.005 | 0.002 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Metric | MAE | RMSE | D1-err | MAE | RMSE | D1-err | MAE | RMSE | D1-err | MAE | RMSE | D1-err | |
| FGSM | No Defense | 14.83 | 24.10 | 97.33 | 8.49 | 14.53 | 90.49 | 5.05 | 7.70 | 74.71 | 3.01 | 3.49 | 38.12 |
| Latent Smoothing Addepalli et al. (2021) | 13.42 | 22.09 | 96.61 | 8.12 | 13.69 | 89.25 | 4.89 | 7.05 | 73.01 | 2.89 | 3.32 | 36.25 | |
| Randomized Smoothing Cohen et al. (2019) | 14.20 | 22.41 | 97.26 | 7.38 | 11.21 | 89.44 | 4.26 | 5.19 | 70.49 | 2.78 | 3.18 | 30.76 | |
| Ours ( ) | 10.12 | 15.81 | 95.80 | 6.46 | 9.92 | 86.30 | 4.39 | 5.93 | 68.55 | 2.78 | 3.18 | 31.99 | |
| Ours ( ) | 9.22 | 13.88 | 94.61 | 6.13 | 8.87 | 84.49 | 4.19 | 5.43 | 66.74 | 2.73 | 3.12 | 31.18 | |
| mIoU | Boundary mIoU | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Attack | No SR | No SR | |||||||
| CosPGD | 18.40 | 25.53 | 31.14 | 32.75 | 15.14 | 21.57 | 26.71 | 28.21 | |
| 17.16 | 22.42 | 26.29 | 27.38 | 13.74 | 18.61 | 22.26 | 23.28 | ||
| 15.39 | 19.15 | 21.71 | 22.41 | 12.18 | 15.57 | 17.96 | 18.62 | ||
| SegPGD | 1.02 | 5.64 | 18.24 | 23.46 | 1.08 | 6.06 | 17.84 | 22.11 | |
| 0.47 | 1.50 | 6.43 | 9.06 | 0.49 | 1.66 | 6.79 | 9.32 | ||
| Backbone | Method | RMSE | RMSE-log | AbsRel | SqRel |
|---|---|---|---|---|---|
| ViT-B/14 | Clean | 0.396 | 0.135 | 0.100 | 0.061 |
| No Defense | 0.645 | 0.191 | 0.168 | 0.383 | |
| Ours | 0.564 | 0.181 | 0.155 | 0.163 | |
| Error Reduced (%) | 32.5 | 17.9 | 19.1 | 68.3 | |
| ViT-L/14 | Clean | 0.373 | 0.127 | 0.093 | 0.054 |
| No Defense | 0.588 | 0.189 | 0.163 | 0.236 |
| Defense | Clean | Baseline | |||
|---|---|---|---|---|---|
| No Defense | 0.302 | -0.025 | 0.064 | 0.145 | 0.179 |
| AT | 0.302 | 0.136 | 0.195 | 0.231 | 0.249 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Layer | Mean Correlation |
|---|---|
| conv_proj | 0.0897 |
| block_00 | 0.2107 |
| block_01 | 0.2518 |
| block_02 | 0.3311 |
| block_03 | 0.3712 |
| block_04 | 0.3781 |
| Translation Radius | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| Distance (Adv. vs. Clean) | 12.30 | 5.29 | 3.50 | 2.64 | 2.12 |
| Metric | ||||
|---|---|---|---|---|
| # Forward Passes | 1 | 9 | 25 | 49 |
| Wall-clock Time (s) | 0.095 | 0.114 | 0.133 | 0.158 |
| SR Forward Time (excluding interpolation) | 0.007 | 0.010 | 0.019 | 0.028 |
| Top-1 Acc @ | 8.51 | 18.40 | 20.01 | 21.17 |
| Top-1 Acc @ | 9.45 | 19.54 | 21.02 | 22.04 |
| Top-1 Acc @ | 11.10 | 21.02 | 22.44 | 23.44 |
| Metric | ||||
|---|---|---|---|---|
| # Forward Passes (can be parallelized) | 1 | 9 | 25 | 49 |
| Memory (CIFAR) | 114.50 MB | 141.50 MB | 182.93 MB | 260.50 MB |
| Memory (ImageNet) | 157.80 MB | 225.19 MB | 384.03 MB | 650.47 MB |
| Dataset / Metric | No Defense | SR ( ) | SR ( ) |
|---|---|---|---|
| ImageNet Top-1 | 74.90 | 75.09 | 75.16 |
| ImageNet Top-5 | 92.21 | 92.30 | 92.42 |
| CIFAR-10 Accuracy | 93.26 | 93.50 | 93.64 |
| Att. Strength ( ) | 8/255 | 4/255 | 2/255 | 1/255 | ||||
|---|---|---|---|---|---|---|---|---|
| Metric | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 |
| No Defense | 8.51 | 26.19 | 9.45 | 27.92 | 11.10 | 30.63 | 15.21 | 36.76 |
| Initial Conv. | 16.91 | 34.66 | 17.36 | 35.28 | 18.12 | 36.30 | 19.59 | 38.25 |
| Res. Block 1 | 22.44 | 44.95 | 23.39 | 46.03 | 24.84 | 47.64 | 27.57 | 50.80 |
| Res. Block 2 | 24.88 | 48.54 | 25.90 | 49.66 | 27.23 | 51.19 | 30.11 | 54.15 |
| Res. Block 3 | 21.76 | 44.28 | 22.73 | 45.47 | 24.14 | 47.14 | 27.18 | 50.20 |
| Att. Strength ( ) | 8/255 | 4/255 | 2/255 | 1/255 | ||||
|---|---|---|---|---|---|---|---|---|
| Metric | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 |
| No Defense | 8.51 | 26.19 | 9.45 | 27.92 | 11.10 | 30.63 | 15.21 | 36.76 |
| Ours ( ) | 18.40 | 40.36 | 19.54 | 42.65 | 21.02 | 43.86 | 24.66 | 47.81 |
| w/ Rotation | 20.01 | 41.32 | 20.84 | 42.39 | 22.10 | 43.89 | 24.72 | 46.89 |
| Ours ( ) | 20.01 | 42.22 | 21.02 | 43.45 | 22.44 | 45.32 | 25.80 | 48.97 |
| w/ Rotation | 18.86 | 38.67 | 19.55 | 39.50 | 20.36 | 40.48 | 22.01 | 42.50 |
| Layer | Corr. (Trans.) | Corr. (Rot.) | CosSim (Trans.) | CosSim (Rot.) |
|---|---|---|---|---|
| Block 1 | 0.7540 | 0.6487 | 0.8357 | 0.7739 |
| Block 2 | 0.5933 | 0.4569 | 0.6756 | 0.5766 |
| Block 3 | 0.6752 | 0.5095 | 0.7771 | 0.6756 |
| Block 4 | 0.8808 | 0.7719 | 0.8954 | 0.8032 |
| Setting | Clean | FGSM | PGD | ||||
|---|---|---|---|---|---|---|---|
| No Defense | 89.13 | 31.41 | 24.01 | 21.76 | 0.00 | 0.01 | 0.03 |
| Ours ( ) | 89.99 | 71.68 | 71.51 | 71.55 | 58.19 | 57.03 | 49.87 |
| Ours ( ) | 89.94 | 73.40 | 73.30 | 73.44 | 69.08 | 68.14 | 61.41 |
| Ours ( ) | 89.92 | 74.05 | 73.99 | 74.08 | 73.32 | 72.30 | 65.38 |
| Setting | Accuracy (%) |
|---|---|
| Attacked | 1.72 |
| Feature-space defense [ 70 ] | 8.13 |
| Feature-space defense [ 70 ] + Ours | 8.56 |
| Setting | Accuracy (%) |
|---|---|
| Attacked | 2.14 |
| Feature-space defense [ 70 ] | 11.09 |
| Ours only | 12.36 |
Adversarial Robustness
Adversarial robustness studies how well AI systems withstand deliberately crafted inputs, training-data corruption, and other attacks.
See all 508 papers