Test-time adaptation (TTA) aims to improve model robustness under distribution shifts by adapting to unlabeled test data, but most existing methods rely on backpropagation (BP), which is computationally costly and incompatible with non-differentiable models such as quantized models, limiting practical deployment on numerous edge devices. Recent BP-free approaches alleviate overhead but remain either architecture-specific or limited in optimization capacity to handle high-dimensional models. We propose ZOTTA, a fully BP-free TTA framework that performs efficient adaptation using only forward passes via Zeroth-Order Optimization (ZOO). While ZOO is theoretically appealing, naive application leads to slow convergence under high-dimensional parameter spaces and unstable optimization due to the lack of labels. ZOTTA overcomes these challenges through 1) Distribution-Robust Layer Selection, which automatically identifies and freezes layers that already extract distribution-invariant features, updating only domain-sensitive layers to reduce the optimization dimensionality and accelerate convergence; 2) Spatial Feature Aggregation Alignment, which stabilizes ZOO by aligning globally aggregated spatial features between source and target to reduce gradient variance. Together, these components enable architecture-agnostic and stable BP-free adaptation. Extensive experiments on ImageNet-C/R/Sketch/A show that ZOTTA outperforms or matches BP-based methods, e.g., it reduces memory usage by 84% and improves accuracy by 3.9% over SAR on ImageNet-C. Code:https://github.com/Zhang-Ronghao/zotta.
Figures & tables
Method
BP-Based
BP-Free & Learning-Free
BP-Free & Learning-Based
Ours
(e.g., SAR [ 37 ] )
(e.g., T3A [ 21 ] )
(e.g., FOA [ 35 ] )
Rely on BP?
Yes
No
No
No
Learn from an objective?
Yes
No
Yes
Yes
Learnable params
Norm layer
N/A
Prompt
Norm layer
Model compatibility
ViTs, CNNs, etc.
ViTs, CNNs, etc.
ViTs
ViTs & CNNs, etc.
Acc. & Mem. (ViT)
62.7% (5,166 MB)
56.9% (957 MB)
62.4% (830 MB)
66.6% (825 MB)
TABLE I: Comparison of our method with previous backpropagation (BP)-based and BP-free test-time adaptation methods. Runtime accuracy and memory are the average results of the full-precision ViT-Base and ResNet50-GN models on ImageNet-C (corruption level 5) with batch size 64. FOA results are reported under the same number of forward passes as ours.
Figure 2
Fig. 3: An overview of the proposed ZOTTA framework, which enables BP-free TTA through two main components: 1) Distribution-Robust Layer Selection : Before TTA, we identify distribution-invariant layers and freeze them, updating only distribution-sensitive layers to reduce the optimization dimensionality. 2) Zeroth-Order Gradient Estimation via Spatial Feature Aggregation Alignment : For each TTA step, we inject Gaussian perturbations into the selected parameters, aggregate spatial/token features into global descriptors, and align their statistics with the source domain. This alignment objective provides a more stable zeroth-order gradient obtained from the loss difference of two-sided perturbed forward passes.
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
56.8
56.8
57.5
46.9
35.6
53.1
44.8
62.2
62.5
65.7
77.7
32.6
46.0
67.0
67.6
55.5
BP-based
TENT
60.3
61.6
61.8
59.2
56.5
63.5
59.2
54.3
64.5
2.3
79.1
67.4
61.5
72.5
70.6
59.6
CoTTA
63.6
63.8
64.1
55.5
51.1
63.6
55.5
70.0
69.4
71.5
78.5
9.7
64.5
73.4
71.2
61.7
SAR
59.2
60.5
60.7
57.5
55.6
61.8
57.6
65.9
63.5
69.1
78.7
45.7
62.4
71.9
70.3
62.7
EATA
61.2
62.3
62.7
60.0
59.2
64.7
61.7
69.0
66.6
71.8
79.7
66.8
65.0
74.2
72.3
66.5
TABLE II: Comparisons with state-of-the-art methods on ImageNet-C with ViT-Base w.r.t Accuracy(%).
Type
Method
R
Sketch
A
Avg.
/
NoAdapt
59.5
44.9
0.1
34.8
BP-based
TENT
63.9
49.1
52.9
55.3
CoTTA
63.5
50.0
52.2
55.2
SAR
63.3
48.7
52.5
54.8
EATA
63.3
50.9
53.4
55.9
DeYO
66.1
52.2
54.1
57.5
TABLE III: Comparisons with state-of-the-arts methods on ImageNet-R, ImageNet-Sketch and ImageNet-A with ViT-Base w.r.t Acc.(%).
Type
Method
#FP
#BP
ImageNet-C
Runtime(s)
Memory(MB)
/
NoAdapt
1
0
55.5
54
819
BP-based
TENT
1
1
59.6
123
5165
CoTTA
3or35
1
61.7
620
16,516
SAR
[1,2]
[0,2]
62.7
243
5,166
EATA
1
1
66.5
128
5,175
EATA(GC)
1
1
66.5
226
2,701
TABLE IV: Resource usage on ViT-Base. #FP/#BP indicate forward/backward passes per sample. We report average accuracy on ImageNet-C (level 5). Wall-clock time (s) and memory (MB) are measured on 50,000 images using an RTX4090 GPU. EATA(GC) employs gradient checkpointing to reduce memory. EATA* updates only the norm layers of the last two ViT blocks. K is the population size; k is the number of gradient estimations—both influence #FP.
Type
Method
C
R
Sketch
A
Avg.
/
NoAdapt
31.3
40.2
28.1
13.9
28.4
BP-based
TENT
31.4
40.3
28.2
13.9
28.5
CoTTA
32.7
40.9
26.3
14.4
28.6
SAR
46.8
51.4
36.2
20.2
38.6
EATA
47.1
47.7
37.5
15.9
37.0
DeYO
34.9
41.8
28.3
14.3
29.8
TABLE V: Comparisons with state-of-the-art methods on ImageNet-C/R/Sketch/A with ResNet50-GN w.r.t Accuracy(%).
Figure 8Figure 9
Fig. 8: Batch size sensitivity analysis on ImageNet-C using ViT-Base.
Fig. 9: Batch size sensitivity analysis on ImageNet-C using ViT-Base.
Fig. 10: Effect of SFAA loss weight λ2 on ImageNet-C using ViT-Base.
Noise
Blur
Weather
Digital
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
NoAdapt
55.3
55.1
56.0
45.9
34.5
51.9
42.1
60.8
60.7
63.3
77.2
22.3
44.2
65.9
66.7
53.5
T3A
55.6
55.7
55.7
45.8
34.4
51.1
41.2
59.5
61.9
66.8
76.4
45.5
43.4
65.6
67.5
55.1
FOA
58.3
59.0
60.6
52.7
36.0
54.0
46.5
63.0
62.2
71.0
77.6
55.4
45.7
68.9
68.2
58.6
Ours
60.1
61.8
62.0
56.3
56.4
61.8
57.6
69.2
66.2
70.8
80.0
58.8
65.8
73.3
72.5
64.8
TABLE VII: Results on quantized ViT-Base models on ImageNet-C with 8-bit floating-point precision w.r.t Accuracy(%).
Type
Method
MT-VQA
G-VQA
Total Acc.(%)
/
NoAdapt
51.7
22.6
34.7
BP-based
TENT
56.7
27.4
39.6
BP-free
FOA
53.3
23.8
36.1
Ours
56.7
26.2
38.9
TABLE VIII: Results on MathVista using the Qwen2.5VL-3B-Instruct model. “MT-VQA” denotes math-targeted VQA, and “G-VQA” denotes general VQA.
Type
Method
Mixed distribution shift
Batch size=1
Label shift
/
NoAdapt
55.5
55.5
55.5
BP-based
TENT
60.4
59.3
59.4
SAR
60.7
59.1
63.8
EATA
64.6
66.5
62.5
DeYO
63.9
67.7
64.2
BP-free
FOA
60.8
64.6
60.4
TABLE IX: Comparisons with state-of-the-arts methods on ImageNet-C under three type of wild TTA settings with ViT-Base w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
56.8
56.8
57.5
46.9
35.6
53.1
44.8
62.2
62.5
65.7
77.7
32.6
46.0
67.0
67.6
55.5
BP-based
TENT
57.6
59.8
60.9
51.8
49.6
59.8
53.4
64.0
62.7
68.0
78.6
66.6
54.5
70.0
69.8
61.8
SAR
59.2
61.2
61.7
54.2
55.3
58.7
55.7
61.0
61.9
64.6
76.8
58.4
58.4
68.5
68.8
61.6
EATA
61.2
64.8
65.4
59.1
60.3
64.7
63.2
68.3
68.9
72.5
79.9
60.8
67.5
73.3
72.8
66.8
DeYO
59.8
63.5
64.6
56.0
57.7
62.1
56.4
66.5
66.2
71.7
79.3
65.6
63.8
72.1
70.9
65.1
TABLE X: Comparisons with state-of-the-arts methods on ImageNet-C under continual setting with ViT-Base w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
23.2
23.8
23.3
19.1
10.9
21.0
24.5
38.5
47.5
38.6
68.4
32.3
17.9
28.2
53.0
31.3
BP-based
TENT
22.2
24.4
24.1
20.0
13.3
23.6
25.4
40.5
42.7
38.1
67.5
38.5
17.0
36.9
52.0
32.4
SAR
39.8
48.3
47.8
9.1
19.6
13.9
41.0
48.3
50.7
56.3
70.1
50.6
10.1
11.0
57.7
38.3
EATA
40.0
43.1
41.0
26.1
31.2
35.0
42.0
45.8
47.8
53.9
66.6
47.5
43.4
49.7
55.5
44.6
DeYO
25.3
33.8
36.8
19.5
16.9
22.6
24.1
36.4
39.4
40.1
62.7
37.7
16.8
38.0
48.4
33.2
TABLE XI: Comparisons with state-of-the-arts methods on ImageNet-C under continual setting with ResNet50-GN w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
44.5
47.0
34.0
76.6
46.0
72.5
80.8
76.6
75.2
60.4
86.1
53.8
66.3
53.2
66.6
62.6
BP-based
TENT
46.0
52.2
37.4
76.8
47.6
73.9
81.8
76.5
77.5
63.8
85.7
61.2
66.8
57.8
68.0
64.9
SAR
44.6
50.8
35.1
75.3
43.5
72.1
80.4
73.9
75.0
61.0
84.9
53.6
64.9
53.8
66.8
62.4
EATA
62.8
65.9
61.9
80.6
65.1
78.1
84.1
79.0
80.0
71.4
87.3
76.5
73.4
76.9
69.6
74.2
DeYO
63.2
66.2
64.0
81.7
67.0
79.7
84.5
80.6
81.1
74.3
88.0
79.1
73.8
77.8
71.1
75.5
TABLE XII: Comparisons with state-of-the-arts methods on CIFAR100-C with ViT-Base w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
17.8
19.9
7.2
68.0
20.4
63.0
73.2
67.8
60.3
58.3
79.1
67.7
49.6
32.2
50.4
49.0
BP-based
TENT
14.3
16.6
5.7
69.5
19.1
64.1
73.7
68.6
61.3
60.1
79.3
69.4
50.7
33.1
50.7
49.1
SAR
16.2
18.2
7.1
69.0
19.5
63.8
73.7
68.1
60.9
59.8
79.2
68.9
50.6
33.0
50.5
49.2
EATA
41.1
42.2
22.3
72.3
41.8
69.1
77.1
69.8
68.0
66.8
79.9
74.7
56.8
67.9
53.5
60.2
DeYO
11.9
12.5
3.6
73.0
9.1
69.5
77.3
70.5
66.9
65.9
80.3
75.8
55.3
62.5
52.1
52.4
TABLE XIII: Comparisons with state-of-the-arts methods on CIFAR100-C with ResNet50-GN w.r.t Acc.(%).
Fig. 11: Per-layer clustering purity of ResNet50-GN.
Fig. 12: Per-layer clustering purity of ViT-Large.
Adapt Layers
Acc.
Adapt Layers
Acc.
Adapt Layers
Acc.
Adapt Layers
Acc.
0
57.9
0,1
59.7
0,1,2
60.4
0,1,2,3
62.2
1
60.0
1,2
61.6
1,2,3
65.4
1,2,3,4
65.6
2
64.5
2,3
66.1
2,3,4
66.4
2,3,4,5
66.7
3
64.8
3,4
66.2
3,4,5
66.6
0,1,2,3,4
62.9
4
64.9
4,5
65.9
4,5,6
65.7
1,2,3,4,5
66.1
5
63.8
5,6
64.7
5,6,7
64.8
All
63.7
TABLE XIV: Results of updating different layers on ImageNet-C with ViT-Base w.r.t Acc.(%). Underlined entries denote results surpassing full-layer update.
Fig. 13: Per-layer clustering results of CLS token for model before and after adaptation on ImageNet-C gaussian noise using ViT-Base.