FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptation

Xingyu Wang; Tao Wang

cs.CVMar 5, 2026

FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptation

Authors: Xingyu Wang, Tao Wang

Organizations: College of Computer Science, Sichuan University

Abstract

Test-Time Adaptation (TTA) is essential for enabling deep learning models to handle real-world data distribution shifts. However, current approaches face significant limitations: backpropagation-based methods are not suitable for low-end deployment devices, due to their high computation and memory requirements, as well as their tendency to modify model weights during adaptation; while traditional backpropagation-free techniques exhibit constrained adaptation capabilities. In this work, we propose Forward-Only Zeroth-Order Optimization (FOZO), a novel and practical backpropagation-free paradigm for TTA. FOZO leverages a memory-efficient zeroth-order prompt optimization, which is led by objectives optimizing both intermediate feature statistics and prediction entropy. To ensure efficient and stable adaptation over the out-of-distribution data stream, we introduce a dynamically decaying perturbation scale during zeroth-order gradient estimation and theoretically prove its convergence under the TTA data stream assumption. Extensive continual adaptation experiments on ImageNet-C, ImageNet-R, and ImageNet-Sketch demonstrate FOZO's superior performance, achieving 59.52% Top-1 accuracy on ImageNet-C (5K, level 5) and outperforming main gradient-based methods and SOTA forward-only FOA (58.13%). Furthermore, FOZO exhibits strong generalization on quantized (INT8) models. These findings demonstrate that FOZO is a highly competitive solution for TTA deployment in resource-limited scenarios.

Figures & tables

Figure 1: Convergence Curves of Forward-Only Test-Time Adaptation Algorithms. Average ACC@1 (%) of various Test-Time Adaptation (TTA) methods on ImageNet-C (level 5) versus adaptation time (s). Per original settings, FOA and ZOA use 28 forward propagation (FP), while our FOZO method employs 26 FP. FOZO (Dynamic) consistently surpasses FOZO (Base) in accuracy, demonstrating the effectiveness of our dynamic perturbation strategy. Furthermore, FOZO achieves superior performance and faster convergence than FOA and ZOA. Notably, FOZO reaches 65% ACC@1 in only 66% of the runtime required by FOA and ZOA, respectively.

Figure 2: Overview of our proposed FOZO. (a) Forward-Only Zeroth-Order Optimization. This diagram illustrates how FOZO adaptively updates learnable visual prompts (

\mathbf{P}

) at test time using a zeroth-order gradient estimation. For each test batch, the model performs two forward passes by perturbing the prompt

\mathbf{P}

in positive (

\mathbf{P}+\epsilon_{t}\mathbf{Z}

) and negative (

\mathbf{P}-\epsilon_{t}\mathbf{Z}

) directions. Here,

\mathbf{Z}

is a perturbation vector generated from a random seed

s

, and

\epsilon_{t}

is a dynamically adjusted perturbation step size. (b) Deep-Shallow Aligning Loss Function. We compute the mean (

\mu^{T}

) and standard deviation (

\sigma^{T}

) of the [CLS] token activations by grouping shallow (

e_{1}^{0},\ldots,e_{N/2}^{0}

) and deep (

e_{N/2+1}^{0},\ldots,e_{N}^{0}

) layers of the model, and align them with pre-computed source domain statistics (

\mu^{S},\sigma^{S}

	FP	Gauss	Shot	Impul	Defcs	Gls	Mtn	Zm	Snw	Frst	Fg	Brt	Cnt	Els.	Px.	JPG	Avg	Time	Memory	#Params
NoAdapt	1	57.1	57.0	57.7	46.8	35.9	52.8	45.9	62.4	62.6	65.8	78.0	32.1	45.3	67.0	67.4	55.57	94	819	0
LAME	1	56.7	56.4	57.4	46.2	35.2	52.1	45.1	58.6	61.8	62.9	77.7	24.6	43.9	66.8	66.9	54.16	97	819	0
T3A	1	56.8	57.7	59.1	40.8	36.9	53.2	46.1	62.2	59.6	60.4	78.2	12.1	45.5	68.2	69.5	53.76	311	823	0
FOA	2	57.2	57.5	58.1	48.6	37.7	54.3	47.6	64.2	65.2	66.8	78.0	51.3	48.6	67.7	69.1	58.13	224	831	2304
ZOA	2	58.0	58.0	59.5	49.8	39.5	55.3	49.0	64.6	64.3	66.8	78.0	48.9	50.2	67.8	68.7	58.56	198	859	26145
FOZO	2	57.8	57.8	58.9	50.2	39.2	55.3	49.2	65.3	66.8	67.5	78.6	56.9	51.0	68.3	70.1	59.52	179	831	2304

Table 1: Continual adaptation results : Comparison with forward-only methods on ImageNet-C (5k, level 5). FP denote the number of forward propagations per single sample. For a fair comparison with the same number of forward propagations, and referencing ZOA’s reproduced results, FOA’s population size K is set to 2, FOZO’s number of SPSA is set to 1, and ZOA’s steps is set to 1. Time (s) is measured for processing ImageNet-C (5k, level 5) on an NVIDIA RTX 4090 GPU. Memory usage (MiB) is the peak GPU memory allocated by tensors, measured on a single adaptation batch. #Params is the number of parameters updated during TTA.

	Gauss	Shot	Impul	Defcs	Gls	Mtn	Zm	Snw	Frst	Fg	Brt	Cnt	Els.	Px.	JPG	Avg	Time	Memory	#Params
NoAdapt	57.1	57.0	57.7	46.8	35.9	52.8	45.9	62.4	62.6	65.8	78.0	32.1	45.3	67.0	67.4	55.57	94	819	0
TENT	57.4	60.1	60.8	42.0	41.0	53.7	49.2	62.7	59.9	64.1	78.4	60.0	48.9	68.4	68.3	58.32	208	5495	39936
DEYO	58.1	60.5	61.0	46.5	49.7	57.8	51.9	64.1	63.0	64.3	78.5	61.4	57.5	68.3	68.9	60.76	282	5499	39936
SAR	58.3	60.9	61.3	48.2	48.7	58.4	50.7	62.6	61.9	64.0	78.4	59.9	54.6	68.0	69.4	60.36	393	5495	29184
EATA	57.0	61.6	61.5	52.0	48.4	59.5	52.9	63.9	61.8	66.0	78.7	62.8	54.7	69.3	69.8	61.35	218	5496	39936
FOA (FP=28)	57.4	60.8	62.9	53.1	45.1	58.3	53.2	66.5	67.2	67.6	80.1	57.6	51.3	68.5	70.2	61.32	2391	831	2304

Table 2: Continual adaptation results : Comparison with backpropagation methods on ImageNet-C (5k, level 5). When FP=28, ZOA’s steps is set to 14, and FOA’s population size K is 28. For FOZO, the number of SPSA

n

is set to 13 when FP=26, and 14 when FP=28.

	Gauss	Shot	Impul	Defcs	Gls	Mtn	Zm	Snw	Frst	Fg	Brt	Cnt	Els.	Px.	JPG	Avg
NoAdapt	56.2	55.9	56.5	45.8	34.6	52.1	43.5	61.4	62.1	65.9	77.3	30.4	43.5	65.5	66.5	54.48
LAME	55.8	55.6	56.2	45.5	33.6	51.7	42.7	58.4	61.1	64.5	77.1	16.4	42.0	64.9	66.3	52.79
T3A	56.0	56.4	57.8	42.3	36.1	53.0	43.7	62.3	58.2	56.4	77.3	11.1	43.9	66.9	68.6	52.67
FOA	56.5	56.5	57.3	47.7	36.2	53.1	45.1	63.2	64.6	69.9	77.4	48.6	46.0	66.2	67.6	57.07
ZOA	57.1	56.8	57.7	47.8	37.6	54.1	46.7	63.4	62.9	65.3	77.4	46.2	46.1	67.0	67.4	56.91
FOZO	57.5	56.6	57.7	49.3	38.3	54.1	47.7	64.1	64.7	68.1	77.1	51.4	48.5	66.6	68.2	58.00

Table 3: Comparison with the forward-only methods on quantized ViT models (8bit PTQ4ViT).

	FP	R	Sketch	Memory	#Params
NoAdapt	1	59.5	44.9	819	0
LAME	1	58.9	44.4	819	0
T3A	1	55.1	48.3	823	0
FOA	2	63.2	49.7	831	2304
ZOA	2	63.6	49.8	859	26145
FOZO	2	64.1	50.5	831	2304

Table 4: Performance comparison on ImageNet-R and ImageNet-Sketch datasets.

-	Acc. (Top-1)	$\Delta$
NoAdapt	55.1	-
Base FOZO (ZO + Entropy)	57.3	+2.2
+ Deep-shallow Feature Alignment	60.1	+2.8
+ Dynamic Perturbation	62.7	+2.6

Table 5: Ablation study of FOZO components. The results are obtained with ImageNet-C (5k, level5).

Figure 3: Ablation Study of FOZO Hyperparameters on ImageNet-C (Gaussian Noise, level 5).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Time budget	0	100	200	300	400	500	600	700	800	900	1000	1100	1200	1300	1400
FOZO (base)	59.5	59.6	60.0	60.6	60.9	61.3	61.7	62.0	62.3	62.5	62.8	63.0	63.2	63.3	63.4
FOA	58.0	59.0	60.5	61.6	62.4	63.1	63.6	64.1	64.4	64.7	64.9	65.1	65.4	65.5	65.7
ZOA	58.7	59.7	61.2	62.5	63.4	63.8	64.1	64.3	64.6	64.8	65.0	65.1	65.2	65.3	65.4
FOZO	59.5	59.8	61.7	63.0	63.7	64.3	64.8	65.2	65.5	65.7	66.0	66.2	66.3	66.4	66.6

Appendix

Table C.1: Detailed data on the time budget and accuracy comparison of Forward-Only Test-Time Adaptation Algorithms.

	$\alpha$ =0.1	$\alpha$ =0.2	$\alpha$ =0.3	$\alpha$ =0.4	$\alpha$ =0.5	$\alpha$ =0.6	$\alpha$ =0.7	$\alpha$ =0.8	$\alpha$ =0.9
ACC@1	59.13	59.15	59.04	59.34	59.33	59.45	59.47	59.50	59.52

Appendix

Table D.1: Effects of the decay factor

\alpha

in Eqn. 12. We report the continual adaptation results on ImageNet-C (5k,level 5) with 2 forward passes.

	$\beta$ =0.1	$\beta$ =0.2	$\beta$ =0.3	$\beta$ =0.4	$\beta$ =0.5	$\beta$ =0.6	$\beta$ =0.7	$\beta$ =0.8	$\beta$ =0.9
ACC@1	59.02	59.05	59.14	59.23	59.27	59.26	59.38	59.50	59.52

Appendix

Table D.2: Effects of the moving average factor

\beta

in Eqn. 12. We report the continual adaptation results on ImageNet-C (5k,level 5) with 2 forward passes.

	$\lambda$ =0.1	$\lambda$ =0.2	$\lambda$ =0.3	$\lambda$ =0.4	$\lambda$ =0.5	$\lambda$ =0.6	$\lambda$ =0.7	$\lambda$ =0.8	$\lambda$ =0.9
ACC@1	59.31	59.30	59.35	59.52	59.50	59.34	59.30	59.27	59.16

Appendix

Table D.3: Effects of trade-off parameter

\lambda

in Eqn. 15. We report the continual adaptation results on ImageNet-C (5k,level 5) with 2 forward passes.

Figure E.1 : Mixed shift: Performance comparison on ImageNet-C (5K, level 5).

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 307 papers

ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization
Mar 15, 2026Ronghao Zhang, Shuaicheng Niu, Qi Deng +3Test-Time Adaptation Zeroth-Order Optimization
South China University of Technology, Guangzhou, 510000, China. · Nanyang Technological University, 639798, Singapore. · Xi’an Jiaotong University, Xi’an, 710049, China. +1
Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
Aug 12, 2026Junming Zhang, Shuyu Yin, Peilin Liu +2On-Device Training Test-Time Adaptation
Shanghai Jiao Tong University, China
EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample
May 15, 2026Guohao Chen, Shuaicheng Niu, Geng Li +4Efficient Neural Network Inference Test-Time Adaptation
Nanyang Technological University · Tulane University

Mar 15, 2026J/K move · Enter open · S save

ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization

Ronghao Zhang, Shuaicheng Niu, Qi Deng, Yanjie Dong+2

South China University of Technology, Guangzhou, 510000, China. · Nanyang Technological University, 639798, Singapore. · Xi’an Jiaotong University, Xi’an, 710049, China.+1

Test-Time Adaptation Zeroth-Order Optimization Image Classification

Open paper AskPDF arXiv

Test-time adaptation (TTA) aims to improve model robustness under distribution shifts by adapting to unlabeled test data, but most existing methods rely on backpropagation (BP), which is computationally costly and incompatible with non-differentiable models such as quantized models, limiting practical deployment on numerous edge devices. Recent BP-free approaches alleviate overhead but remain either architecture-specific or limited in optimization capacity to handle high-dimensional models. We propose ZOTTA, a fully BP-free TTA framework that performs efficient adaptation using only forward passes via Zeroth-Order Optimization (ZOO). While ZOO is theoretically appealing, naive application leads to slow convergence under high-dimensional parameter spaces and unstable optimization due to the lack of labels. ZOTTA overcomes these challenges through 1) Distribution-Robust Layer Selection, which automatically identifies and freezes layers that already extract distribution-invariant features, updating only domain-sensitive layers to reduce the optimization dimensionality and accelerate convergence; 2) Spatial Feature Aggregation Alignment, which stabilizes ZOO by aligning globally aggregated spatial features between source and target to reduce gradient variance. Together, these components enable architecture-agnostic and stable BP-free adaptation. Extensive experiments on ImageNet-C/R/Sketch/A show that ZOTTA outperforms or matches BP-based methods, e.g., it reduces memory usage by 84% and improves accuracy by 3.9% over SAR on ImageNet-C. Code:https://github.com/Zhang-Ronghao/zotta.

Abstract

Test-Time Adaptation

Explore similar work

ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization

Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation

EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample