Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
Organizations: Sungkyunkwan University · Amazon
Abstract
Test-time adaptation (TTA) enhances the zero-shot robustness under distribution shifts by leveraging unlabeled test data during inference. Despite notable advances, several challenges still limit its broader applicability. First, most methods rely on backpropagation or iterative optimization, which limits scalability and hinders real-time deployment. Second, they lack explicit modeling of class-conditional feature distributions. This modeling is crucial for producing reliable decision boundaries and calibrated predictions, but it remains underexplored due to the lack of both source data and supervision at test time. In this paper, we propose ADAPT, an Advanced Distribution-Aware and backPropagation-free Test-time adaptation method. We reframe TTA as a Gaussian probabilistic inference task by modeling class-conditional likelihoods using gradually updated class means and a shared covariance matrix. This enables closed-form, training-free inference. To correct potential likelihood bias, we introduce lightweight regularization guided by CLIP priors and a historical knowledge bank. ADAPT requires no source data, no gradient updates, and no full access to target data, supporting both online and transductive settings. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts with superior scalability and robustness.
Figures & tables
| Method | BP-Free | Distribution-Aware | Task Setting | |
| Online | Transductive | |||
| Prompt Tuning Manli et al. (2022) ; Zhang et al. (2024b) ; Ma et al. (2023) ; Samadh et al. (2023) ; Xiao et al. (2025) | ✗ | ✗ | ✓ | ✗ |
| Adapter Tuning Gao et al. (2024) ; Zhu et al. (2023) ; Yu et al. (2023) ; Sui et al. (2025) ; Zhang et al. (2024a) | ✗ | ✗ | ✓ | ✗ |
| Similarity Score Karmanov et al. (2024) ; Zhang et al. (2024c) ; Zhang et al. (2024d) ; Zhang et al. (2022) ; Udandarao et al. (2023) | ✓ | ✗ | ✓ | ✗ |
| Transductive Learning Kalantidis et al. (2024) ; Li et al. (2025b) ; Zanella et al. (2024a) ; Zhu et al. (2024a) ; Wang et al. (2024) | ✓ | ✓ | ✗ | ✓ |
| ADAPT (Ours) | ✓ | ✓ | ✓ | ✓ |
| Method | BP-free | ImageNet | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Avg. | Avg. | |
| Online | CLIP Radford et al. (2021) | - | 66.74 | 47.79 | 60.89 | 73.99 | 46.12 | 57.20 | 59.11 |
| Tip-Adapter Zhang et al. (2022) | ✗ | 70.75 | 51.04 | 63.41 | 77.76 | 48.88 | 60.27 | 62.37 | |
| TPT Manli et al. (2022) | ✗ | 68.98 | 54.77 | 63.45 | 77.06 | 47.97 | 60.81 | 62.45 | |
| DiffTPT Feng et al. (2023) | ✗ | 70.30 | 55.68 | 65.10 | 75.00 | 46.80 | 60.65 | 62.58 | |
| C-TPT Yoon et al. (2024) | ✗ | 68.50 | 51.60 | 62.70 | 76.00 | 47.90 | 59.55 | 61.34 | |
| DMN Zhang et al. (2024d) | ✗ | 72.25 | 58.28 | 65.17 | 78.55 | 53.20 | 63.80 | 65.49 |
| Method | Blur | Weather | Digital | Noise | Avg. | ||||||||||||
| Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pix. | JPEG | Gauss. | Shot | Impu. | |||
| Online | CLIP Radford et al. (2021) | 24.25 | 15.71 | 24.46 | 22.60 | 33.08 | 31.06 | 37.61 | 55.62 | 17.11 | 13.43 | 33.04 | 33.70 | 13.25 | 14.16 | 13.48 | 25.50 |
| TPT Manli et al. (2022) | 27.56 | 15.48 | 26.16 | 26.94 | 36.74 | 34.28 | 39.38 | 60.22 | 16.96 | 15.64 | 40.74 | 37.90 | 10.64 | 11.94 | 10.92 | 27.43 | |
| DiffTPT Feng et al. (2023) | 25.63 | 16.96 | 26.74 | 25.40 | 35.99 | 34.57 | 39.83 | 59.01 | 17.32 | 17.16 | 38.43 | 35.47 | 12.97 | 13.60 | 13.21 | 27.49 | |
| TDA Karmanov et al. (2024) | 26.53 | 17.91 | 27.35 | 25.90 | 36.50 | 34.84 | 40.53 | 58.57 | 20.16 | 16.62 | 35.65 | 36.69 | 15.42 | 16.46 | 16.03 | 28.34 | |
| DMN Zhang et al. (2024d) | 26.06 | 17.19 | 26.61 | 25.23 | 34.81 | 33.48 | 38.93 | 58.70 | 19.38 | 15.40 | 35.32 | 36.49 | 14.33 | 15.33 | 14.69 | 27.46 | |
| Method | BP-free | Aircraft | Caltech | Cars | DTD | EuroSAT | Flower | Food101 | Pets | Sun397 | UCF101 | Avg. | |
| Online | CLIP Radford et al. (2021) | - | 23.70 | 92.98 | 65.24 | 44.44 | 41.42 | 67.28 | 83.80 | 87.98 | 62.55 | 65.08 | 63.45 |
| TPT Manli et al. (2022) | ✗ | 24.78 | 94.16 | 66.87 | 47.75 | 42.44 | 68.98 | 84.67 | 87.79 | 65.50 | 68.04 | 65.10 | |
| DiffTPT Feng et al. (2023) | ✗ | 25.60 | 92.49 | 67.01 | 47.00 | 43.13 | 70.10 | 87.23 | 88.22 | 65.74 | 68.22 | 65.47 | |
| C-TPT Yoon et al. (2024) | ✗ | 24.00 | 93.60 | 65.80 | 46.00 | 43.20 | 79.80 | 83.70 | 88.20 | 64.80 | 65.70 | 64.48 | |
| DMN Zhang et al. (2024d) | ✗ | 30.03 | 95.38 | 67.96 | 55.85 | 59.43 | 74.49 | 85.08 | 92.04 | 70.18 | 72.51 | 70.30 | |
| TPS Li et al. (2025b) | ✗ | 26.27 | 94.56 | 67.00 | 53.80 | 42.11 | 71.69 | 84.78 | 87.82 | 68.25 | 71.18 | 66.75 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Low-dim | Freq of p>0.05 (%) | p-value Avg. | |
| Henze–Zirkler | 2 | 100 | 0.39 |
| 4 | 99.90 | 0.32 | |
| 6 | 99.00 | 0.27 | |
| 8 | 96.30 | 0.22 | |
| 10 | 92.90 | 0.19 | |
| Shapiro-Wilk | 2 | 100 | 0.31 |
| Method | Time | Acc (%) | Gain (%) | Mem.(GB) |
| CLIP | 8m | 66.74 | - | 0.79 |
| ADAPT (Online) | 1h 11m | 70.91 | +4.17 | 0.93 |
| w/ Separate Cov. | 1h 44m | 53.80 | -12.94 | 2.89 |
| ADAPT (Trans.) | 0.73m | 71.56 | +4.82 | 3.37 |
| w/ Separate Cov. | 1.40m | 66.16 | -0.58 | 4.62 |
| Key Distinctions | ADAPT (Ours) | T3A [ 19 ] | DPCore [ 67 ] | DynaPrompt [ 55 ] |
| Core Paradigm | Distribution-Aware | Prototype-Based | Prompt-Tuning | Prompt-Tuning |
| Knowledge Bank Elements | Confident Test Samples | – | Learnable Prompts, Statistics | Learnable Prompts |
| Training/Tuning Free | ✓ | ✓ | ||
| Update Basis | Sample-by-Sample | Batch-Based | Batch-Based | Sample-Wise Tuning |
| Key Dependencies | – | Batch of Data | Batch of Data | Augmentation, Hard Thresholds |
| Applicability (Trans.) | ✓ | ✓ | ✓ | ✓ |
| Task 1: Natural Distribution Shift | |||||||
| Method | ImageNet | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Avg. | Avg. |
| ADAPT | |||||||
| Method | BP-free | ImageNet | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Avg. | Avg. | |
| Online | CLIP [ 40 ] | - | 58.16 | 21.83 | 51.41 | 56.15 | 33.37 | 40.69 | 44.18 |
| TPT [ 34 ] | ✗ | 60.74 | 26.67 | 54.70 | 59.11 | 35.09 | 43.89 | 47.26 | |
| DiffTPT [ 9 ] | ✗ | 60.80 | 31.06 | 55.80 | 58.80 | 37.10 | 45.69 | 48.71 | |
| C-TPT [ 56 ] | ✗ | 60.20 | 23.40 | 54.70 | 58.00 | 35.10 | 42.80 | 46.28 | |
| DMN [ 66 ] | ✗ | 63.87 | 28.57 | 56.12 | 61.44 | 39.84 | 46.49 | 49.97 | |
| TPS [ 47 ] | ✗ | 62.27 | 29.80 | 60.04 | 55.49 | 35.74 | 45.27 | 48.67 |
| Method | Blur | Weather | Digital | Noise | Avg. | ||||||||||||
| Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pix. | JPEG | Gauss. | Shot | Impu. | |||
| Online | CLIP [ 40 ] | 9.54 | 3.40 | 7.46 | 12.62 | 12.29 | 15.72 | 22.08 | 41.69 | 6.24 | 4.67 | 11.01 | 14.24 | 2.43 | 3.07 | 2.52 | 11.27 |
| TPT [ 34 ] | 8.02 | 2.74 | 5.34 | 10.97 | 10.59 | 12.92 | 16.17 | 35.67 | 4.45 | 3.73 | 11.56 | 16.68 | 1.43 | 1.94 | 1.42 | 9.58 | |
| DiffTPT [ 9 ] | 10.50 | 3.90 | 8.62 | 12.74 | 11.31 | 15.03 | 19.64 | 37.73 | 4.98 | 4.74 | 13.58 | 16.56 | 2.69 | 3.59 | 2.08 | 11.18 | |
| TDA [ 22 ] | 9.84 | 4.4 | 7.38 | 13.74 | 13.74 | 17.16 | 23.76 | 44.16 | 7.00 | 5.79 | 11.24 | 15.26 | 2.54 | 3.26 | 2.72 | 12.13 | |
| ADAPT | 10.54 | 4.44 | 8.57 | 14.34 | 13.85 | 17.84 | 24.56 | 45.67 | 7.76 | 5.85 | 11.96 | 15.86 | 2.91 | 3.77 | 2.92 | 12.72 | |
| Method | BP-free | Aircraft | Caltech | Cars | DTD | EuroSAT | Flower | Food101 | Pets | Sun397 | UCF101 | Avg. | |
| Online | CLIP [ 40 ] | - | 15.66 | 85.88 | 55.70 | 40.37 | 23.69 | 61.75 | 73.97 | 83.57 | 58.80 | 58.84 | 55.82 |
| TPT [ 34 ] | ✗ | 17.58 | 87.02 | 58.46 | 40.84 | 28.33 | 62.69 | 74.88 | 84.49 | 61.46 | 60.82 | 57.66 | |
| DiffTPT [ 9 ] | ✗ | 17.60 | 86.89 | 60.71 | 40.72 | 41.04 | 63.53 | 79.21 | 83.40 | 62.72 | 62.67 | 59.85 | |
| C-TPT [ 56 ] | ✗ | 17.00 | 86.90 | 56.50 | 42.20 | 27.80 | 65.20 | 74.70 | 84.10 | 61.00 | 59.70 | 57.51 | |
| DMN [ 66 ] | ✗ | 22.77 | 90.14 | 60.02 | 50.41 | 48.72 | 67.93 | 76.70 | 86.78 | 64.39 | 65.34 | 63.32 | |
| TPS [ 30 ] | ✗ | 18.30 | 89.80 | 59.40 | 48.40 | 24.30 | 68.20 | 76.20 | 84.40 | 62.70 | 64.30 | 59.60 |
| Update | Update | Task 1 | Task 2 | Task 3 | |
| ✗ | ✗ | ✗ | 59.11 | 25.50 | 63.45 |
| ✗ | ✗ | ✓ | 49.64 | 9.58 | 60.02 |
| ✗ | ✓ | ✗ | 61.54 | 25.42 | 67.03 |
| ✗ | ✓ | ✓ | 49.65 | 9.58 | 60.04 |
| ✓ | ✗ | ✗ | 64.89 | 25.08 | 67.06 |
| ✓ | ✗ | ✓ | 64.05 | 19.49 | 67.95 |
| Task | Method | Vanilla | Ensemble | CLIP Template | GPT | GPT & Ensemble | GPT & CLIP Template |
| Task 1 | TDA | ||||||
| ADAPT (Online) | |||||||
| ADAPT (Trans.) | |||||||
| Task 2 | TDA | ||||||
| ADAPT (Online) | |||||||
| ADAPT (Trans.) |
| Method | ImageNet | Aircraft | Caltech101 | Cars | DTD | EuroSAT | Flower102 | Food101 | Pets | Sun397 | UCF101 | Avg. |
| AWT | 71.32 | 29.22 | 95.54 | 69.93 | 55.56 | 58.61 | 75.07 | 85.54 | 92.53 | 70.58 | 72.51 | 70.58 |
| AWT ∗ | 69.03 | 27.48 | 94.77 | 65.94 | 53.96 | 60.41 | 75.11 | 84.76 | 91.88 | 68.18 | 69.81 | 69.21 |
| ADAPT (Online) | 70.91 | 28.95 | 94.48 | 68.19 | 55.20 | 68.19 | 75.56 | 83.81 | 92.01 | 70.57 | 70.66 | 70.78 |
| ADAPT (Trans.) | 71.56 | 30.81 | 95.46 | 71.32 | 56.86 | 65.93 | 80.11 | 85.15 | 92.59 | 72.25 | 73.86 | 72.35 |
| Severity Level | 1 | 2 | 3 | 4 | 5 |
| CLIP | 59.68 | 52.97 | 46.79 | 37.51 | 25.50 |
| TDA | 60.42 | 54.15 | 48.35 | 36.84 | 28.34 |
| ADAPT (Online) | 61.37 | 54.70 | 48.61 | 39.28 | 28.56 |
| ADAPT (Trans.) | 62.04 | 55.51 | 49.68 | 40.76 | 30.29 |
| N | 2 | 4 | 6 | 8 | 10 | |
| Online | Task 1 | 59.61 | 64.19 | 65.50 | 66.10 | 66.33 |
| Task 2 | 22.95 | 26.00 | 27.26 | 27.93 | 28.22 | |
| Task 3 | 63.87 | 67.46 | 69.04 | 69.89 | 70.31 | |
| Trans. | Task 1 | 67.02 | 67.11 | 67.09 | 67.10 | 67.04 |
| Task 2 | 30.26 | 30.29 | 30.29 | 30.23 | 30.23 | |
| Task 3 | 72.15 | 72.21 | 72.43 | 72.24 | 72.25 |
| Method | ImageNet | ImageNet-A |
| ADAPT (Online) | 70.91 | 63.32 |
| w/ 0.1 pruning | 69.16 | 50.69 |
| w/ 0.3 pruning | 69.03 | 50.60 |
| w/ 0.5 pruning | 69.02 | 50.56 |
| w/ 0.7 pruning | 68.97 | 50.52 |
| Method | Head class | Med class | Tail class | Avg. |
| CLIP | 68.44 | 69.69 | 68.42 | 69.33 |
| CLIP w/ Noise | ||||
| ADAPT (Online) | 69.22 | 70.31 | 69.08 | 69.83 |
| ADAPT w/ Noise |
Test-Time Adaptation
Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.
See all 342 papers