Test-time adaptation (TTA) aims to improve model robustness under distribution shifts by adapting to unlabeled test data, but most existing methods rely on backpropagation (BP), which is computationally costly and incompatible with non-differentiable models such as quantized models, limiting practical deployment on numerous edge devices. Recent BP-free approaches alleviate overhead but remain either architecture-specific or limited in optimization capacity to handle high-dimensional models. We propose ZOTTA, a fully BP-free TTA framework that performs efficient adaptation using only forward passes via Zeroth-Order Optimization (ZOO). While ZOO is theoretically appealing, naive application leads to slow convergence under high-dimensional parameter spaces and unstable optimization due to the lack of labels. ZOTTA overcomes these challenges through 1) Distribution-Robust Layer Selection, which automatically identifies and freezes layers that already extract distribution-invariant features, updating only domain-sensitive layers to reduce the optimization dimensionality and accelerate convergence; 2) Spatial Feature Aggregation Alignment, which stabilizes ZOO by aligning globally aggregated spatial features between source and target to reduce gradient variance. Together, these components enable architecture-agnostic and stable BP-free adaptation. Extensive experiments on ImageNet-C/R/Sketch/A show that ZOTTA outperforms or matches BP-based methods, e.g., it reduces memory usage by 84% and improves accuracy by 3.9% over SAR on ImageNet-C. Code:https://github.com/Zhang-Ronghao/zotta.
Figures & tables
Method
BP-Based
BP-Free & Learning-Free
BP-Free & Learning-Based
Ours
(e.g., SAR [ 37 ] )
(e.g., T3A [ 21 ] )
(e.g., FOA [ 35 ] )
Rely on BP?
Yes
No
No
No
Learn from an objective?
Yes
No
Yes
Yes
Learnable params
Norm layer
N/A
Prompt
Norm layer
Model compatibility
ViTs, CNNs, etc.
ViTs, CNNs, etc.
ViTs
ViTs & CNNs, etc.
Acc. & Mem. (ViT)
62.7% (5,166 MB)
56.9% (957 MB)
62.4% (830 MB)
66.6% (825 MB)
TABLE I: Comparison of our method with previous backpropagation (BP)-based and BP-free test-time adaptation methods. Runtime accuracy and memory are the average results of the full-precision ViT-Base and ResNet50-GN models on ImageNet-C (corruption level 5) with batch size 64. FOA results are reported under the same number of forward passes as ours.
Figure 2
Fig. 3: An overview of the proposed ZOTTA framework, which enables BP-free TTA through two main components: 1) Distribution-Robust Layer Selection : Before TTA, we identify distribution-invariant layers and freeze them, updating only distribution-sensitive layers to reduce the optimization dimensionality. 2) Zeroth-Order Gradient Estimation via Spatial Feature Aggregation Alignment : For each TTA step, we inject Gaussian perturbations into the selected parameters, aggregate spatial/token features into global descriptors, and align their statistics with the source domain. This alignment objective provides a more stable zeroth-order gradient obtained from the loss difference of two-sided perturbed forward passes.
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
56.8
56.8
57.5
46.9
35.6
53.1
44.8
62.2
62.5
65.7
77.7
32.6
46.0
67.0
67.6
55.5
BP-based
TENT
60.3
61.6
61.8
59.2
56.5
63.5
59.2
54.3
64.5
2.3
79.1
67.4
61.5
72.5
70.6
59.6
CoTTA
63.6
63.8
64.1
55.5
51.1
63.6
55.5
70.0
69.4
71.5
78.5
9.7
64.5
73.4
71.2
61.7
SAR
59.2
60.5
60.7
57.5
55.6
61.8
57.6
65.9
63.5
69.1
78.7
45.7
62.4
71.9
70.3
62.7
EATA
61.2
62.3
62.7
60.0
59.2
64.7
61.7
69.0
66.6
71.8
79.7
66.8
65.0
74.2
72.3
66.5
TABLE II: Comparisons with state-of-the-art methods on ImageNet-C with ViT-Base w.r.t Accuracy(%).
Type
Method
R
Sketch
A
Avg.
/
NoAdapt
59.5
44.9
0.1
34.8
BP-based
TENT
63.9
49.1
52.9
55.3
CoTTA
63.5
50.0
52.2
55.2
SAR
63.3
48.7
52.5
54.8
EATA
63.3
50.9
53.4
55.9
DeYO
66.1
52.2
54.1
57.5
TABLE III: Comparisons with state-of-the-arts methods on ImageNet-R, ImageNet-Sketch and ImageNet-A with ViT-Base w.r.t Acc.(%).
Type
Method
#FP
#BP
ImageNet-C
Runtime(s)
Memory(MB)
/
NoAdapt
1
0
55.5
54
819
BP-based
TENT
1
1
59.6
123
5165
CoTTA
3or35
1
61.7
620
16,516
SAR
[1,2]
[0,2]
62.7
243
5,166
EATA
1
1
66.5
128
5,175
EATA(GC)
1
1
66.5
226
2,701
TABLE IV: Resource usage on ViT-Base. #FP/#BP indicate forward/backward passes per sample. We report average accuracy on ImageNet-C (level 5). Wall-clock time (s) and memory (MB) are measured on 50,000 images using an RTX4090 GPU. EATA(GC) employs gradient checkpointing to reduce memory. EATA* updates only the norm layers of the last two ViT blocks. K is the population size; k is the number of gradient estimations—both influence #FP.
Type
Method
C
R
Sketch
A
Avg.
/
NoAdapt
31.3
40.2
28.1
13.9
28.4
BP-based
TENT
31.4
40.3
28.2
13.9
28.5
CoTTA
32.7
40.9
26.3
14.4
28.6
SAR
46.8
51.4
36.2
20.2
38.6
EATA
47.1
47.7
37.5
15.9
37.0
DeYO
34.9
41.8
28.3
14.3
29.8
TABLE V: Comparisons with state-of-the-art methods on ImageNet-C/R/Sketch/A with ResNet50-GN w.r.t Accuracy(%).
Figure 8Figure 9
Fig. 8: Batch size sensitivity analysis on ImageNet-C using ViT-Base.
Fig. 9: Batch size sensitivity analysis on ImageNet-C using ViT-Base.
Fig. 10: Effect of SFAA loss weight λ2 on ImageNet-C using ViT-Base.
Noise
Blur
Weather
Digital
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
NoAdapt
55.3
55.1
56.0
45.9
34.5
51.9
42.1
60.8
60.7
63.3
77.2
22.3
44.2
65.9
66.7
53.5
T3A
55.6
55.7
55.7
45.8
34.4
51.1
41.2
59.5
61.9
66.8
76.4
45.5
43.4
65.6
67.5
55.1
FOA
58.3
59.0
60.6
52.7
36.0
54.0
46.5
63.0
62.2
71.0
77.6
55.4
45.7
68.9
68.2
58.6
Ours
60.1
61.8
62.0
56.3
56.4
61.8
57.6
69.2
66.2
70.8
80.0
58.8
65.8
73.3
72.5
64.8
TABLE VII: Results on quantized ViT-Base models on ImageNet-C with 8-bit floating-point precision w.r.t Accuracy(%).
Type
Method
MT-VQA
G-VQA
Total Acc.(%)
/
NoAdapt
51.7
22.6
34.7
BP-based
TENT
56.7
27.4
39.6
BP-free
FOA
53.3
23.8
36.1
Ours
56.7
26.2
38.9
TABLE VIII: Results on MathVista using the Qwen2.5VL-3B-Instruct model. “MT-VQA” denotes math-targeted VQA, and “G-VQA” denotes general VQA.
Type
Method
Mixed distribution shift
Batch size=1
Label shift
/
NoAdapt
55.5
55.5
55.5
BP-based
TENT
60.4
59.3
59.4
SAR
60.7
59.1
63.8
EATA
64.6
66.5
62.5
DeYO
63.9
67.7
64.2
BP-free
FOA
60.8
64.6
60.4
TABLE IX: Comparisons with state-of-the-arts methods on ImageNet-C under three type of wild TTA settings with ViT-Base w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
56.8
56.8
57.5
46.9
35.6
53.1
44.8
62.2
62.5
65.7
77.7
32.6
46.0
67.0
67.6
55.5
BP-based
TENT
57.6
59.8
60.9
51.8
49.6
59.8
53.4
64.0
62.7
68.0
78.6
66.6
54.5
70.0
69.8
61.8
SAR
59.2
61.2
61.7
54.2
55.3
58.7
55.7
61.0
61.9
64.6
76.8
58.4
58.4
68.5
68.8
61.6
EATA
61.2
64.8
65.4
59.1
60.3
64.7
63.2
68.3
68.9
72.5
79.9
60.8
67.5
73.3
72.8
66.8
DeYO
59.8
63.5
64.6
56.0
57.7
62.1
56.4
66.5
66.2
71.7
79.3
65.6
63.8
72.1
70.9
65.1
TABLE X: Comparisons with state-of-the-arts methods on ImageNet-C under continual setting with ViT-Base w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
23.2
23.8
23.3
19.1
10.9
21.0
24.5
38.5
47.5
38.6
68.4
32.3
17.9
28.2
53.0
31.3
BP-based
TENT
22.2
24.4
24.1
20.0
13.3
23.6
25.4
40.5
42.7
38.1
67.5
38.5
17.0
36.9
52.0
32.4
SAR
39.8
48.3
47.8
9.1
19.6
13.9
41.0
48.3
50.7
56.3
70.1
50.6
10.1
11.0
57.7
38.3
EATA
40.0
43.1
41.0
26.1
31.2
35.0
42.0
45.8
47.8
53.9
66.6
47.5
43.4
49.7
55.5
44.6
DeYO
25.3
33.8
36.8
19.5
16.9
22.6
24.1
36.4
39.4
40.1
62.7
37.7
16.8
38.0
48.4
33.2
TABLE XI: Comparisons with state-of-the-arts methods on ImageNet-C under continual setting with ResNet50-GN w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
44.5
47.0
34.0
76.6
46.0
72.5
80.8
76.6
75.2
60.4
86.1
53.8
66.3
53.2
66.6
62.6
BP-based
TENT
46.0
52.2
37.4
76.8
47.6
73.9
81.8
76.5
77.5
63.8
85.7
61.2
66.8
57.8
68.0
64.9
SAR
44.6
50.8
35.1
75.3
43.5
72.1
80.4
73.9
75.0
61.0
84.9
53.6
64.9
53.8
66.8
62.4
EATA
62.8
65.9
61.9
80.6
65.1
78.1
84.1
79.0
80.0
71.4
87.3
76.5
73.4
76.9
69.6
74.2
DeYO
63.2
66.2
64.0
81.7
67.0
79.7
84.5
80.6
81.1
74.3
88.0
79.1
73.8
77.8
71.1
75.5
TABLE XII: Comparisons with state-of-the-arts methods on CIFAR100-C with ViT-Base w.r.t Acc.(%).
Noise
Blur
Weather
Digital
Type
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
/
NoAdapt
17.8
19.9
7.2
68.0
20.4
63.0
73.2
67.8
60.3
58.3
79.1
67.7
49.6
32.2
50.4
49.0
BP-based
TENT
14.3
16.6
5.7
69.5
19.1
64.1
73.7
68.6
61.3
60.1
79.3
69.4
50.7
33.1
50.7
49.1
SAR
16.2
18.2
7.1
69.0
19.5
63.8
73.7
68.1
60.9
59.8
79.2
68.9
50.6
33.0
50.5
49.2
EATA
41.1
42.2
22.3
72.3
41.8
69.1
77.1
69.8
68.0
66.8
79.9
74.7
56.8
67.9
53.5
60.2
DeYO
11.9
12.5
3.6
73.0
9.1
69.5
77.3
70.5
66.9
65.9
80.3
75.8
55.3
62.5
52.1
52.4
TABLE XIII: Comparisons with state-of-the-arts methods on CIFAR100-C with ResNet50-GN w.r.t Acc.(%).
Fig. 11: Per-layer clustering purity of ResNet50-GN.
Fig. 12: Per-layer clustering purity of ViT-Large.
Adapt Layers
Acc.
Adapt Layers
Acc.
Adapt Layers
Acc.
Adapt Layers
Acc.
0
57.9
0,1
59.7
0,1,2
60.4
0,1,2,3
62.2
1
60.0
1,2
61.6
1,2,3
65.4
1,2,3,4
65.6
2
64.5
2,3
66.1
2,3,4
66.4
2,3,4,5
66.7
3
64.8
3,4
66.2
3,4,5
66.6
0,1,2,3,4
62.9
4
64.9
4,5
65.9
4,5,6
65.7
1,2,3,4,5
66.1
5
63.8
5,6
64.7
5,6,7
64.8
All
63.7
TABLE XIV: Results of updating different layers on ImageNet-C with ViT-Base w.r.t Acc.(%). Underlined entries denote results surpassing full-layer update.
Fig. 13: Per-layer clustering results of CLS token for model before and after adaptation on ImageNet-C gaussian noise using ViT-Base.
Test-Time Adaptation (TTA) is essential for enabling deep learning models to handle real-world data distribution shifts. However, current approaches face significant limitations: backpropagation-based methods are not suitable for low-end deployment devices, due to their high computation and memory requirements, as well as their tendency to modify model weights during adaptation; while traditional backpropagation-free techniques exhibit constrained adaptation capabilities. In this work, we propose Forward-Only Zeroth-Order Optimization (FOZO), a novel and practical backpropagation-free paradigm for TTA. FOZO leverages a memory-efficient zeroth-order prompt optimization, which is led by objectives optimizing both intermediate feature statistics and prediction entropy. To ensure efficient and stable adaptation over the out-of-distribution data stream, we introduce a dynamically decaying perturbation scale during zeroth-order gradient estimation and theoretically prove its convergence under the TTA data stream assumption. Extensive continual adaptation experiments on ImageNet-C, ImageNet-R, and ImageNet-Sketch demonstrate FOZO's superior performance, achieving 59.52% Top-1 accuracy on ImageNet-C (5K, level 5) and outperforming main gradient-based methods and SOTA forward-only FOA (58.13%). Furthermore, FOZO exhibits strong generalization on quantized (INT8) models. These findings demonstrate that FOZO is a highly competitive solution for TTA deployment in resource-limited scenarios.
Test-time adaptation (TTA) aims to enhance the cross-domain performance of pre-trained models by adapting to unlabeled test data. While most existing TTA methods rely on backpropagation (BP) for finetuning, BP-free methods such as zeroth-order (ZO) methods are more desired in practical on-device scenarios. ZO methods rely only on forward computation, which can largely reduce the complexity and memory overhead of on-device deployment. However, ZO methods suffer from much higher variance compared with first-order methods in estimating the gradient. To address this, we propose an improved ZO method to substantially boost the performance of ZO optimization based TTA. First, we provide an observation to reveal the persistent low-rank Hessian structure of the loss during the adaptation process. Based on this insight, we then propose a loss-landscape curvature-aware zeroth-order (CAZO) method, which leverages a sliding-average estimation of the diagonal Hessian to construct a covariance matrix for anisotropic perturbation sampling. CAZO operates by freezing pretrained weights and optimizing minimal adapter parameters via forward-only passes based gradient estimation, which can substantially reduce the memory overhead compared to BP-based methods. Extensive experiments demonstrate that CAZO significantly outperforms existing TTA methods, achieving state-of-the-art performance while maintaining an excellent balance between accuracy and memory efficiency. Code is available at https://github.com/Hollyming/CAZO.
Test-time model evolution offers a promising way for deployed models to improve from unlabeled test-time experience, yet most existing methods depend on backpropagation (BP), which incurs substantial memory overhead and makes them difficult to deploy on edge devices, quantized models, specialized accelerators, or black-box models. In this work, we study test-time model evolution under a strict two-forward budget, a setting that pushes adaptation toward highly efficient real-world deployment. We reveal three key obstacles in zeroth-order test-time optimization: susceptibility to shortcut solutions, uncontrolled weight drift, and ineffective update direction estimation. To overcome them, we propose EVA-0, a minimal zeroth-order adaptation framework that: 1) keeps the loss scale-invariant to prevent shortcut solutions; 2) devises an anchor-guided optimization strategy to alleviate weight drift; 3) uses sample-wise symmetric two-sided perturbation for update direction estimation and inference. EVA-0 requires no BP and performs both inference and adaptation within only two forward passes per sample. Results on ImageNet-C & ViT-Base show that EVA-0 outperforms both BP-based DeYO and BP-free FOA, while achieving a 14x speed-up over FOA. Code will be released.
Guohao Chen, Shuaicheng Niu, Geng Li +4
Nanyang Technological University · Tulane University