Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation

Damian Sójka; Sebastian Cygert; Marc Masana

cs.LGMar 30, 2026

Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation

Authors: Damian Sójka, Sebastian Cygert, Marc Masana

Organizations: Poznan University of Technology · Akces NCBR · NASK - National Research Institute · Gdańsk University of Technology ·

Abstract

We introduce PACE, a backpropagation-free continual test-time adaptation system that directly optimizes the affine parameters of normalization layers. Existing derivative-free approaches struggle to balance runtime efficiency with learning capacity, as they either restrict updates to input prompts or require continuous, resource-intensive adaptation regardless of domain stability. To address these limitations, PACE leverages the Covariance Matrix Adaptation Evolution Strategy with the Fastfood projection to optimize high-dimensional affine parameters within a low-dimensional subspace, leading to superior adaptive performance. Furthermore, we enhance the runtime efficiency by incorporating an adaptation stopping criterion and a domain-specialized vector bank to eliminate redundant computation. Our framework achieves state-of-the-art accuracy across multiple benchmarks under continual distribution shifts, reducing runtime by over 50% compared to existing backpropagation-free methods.

Figures & tables

Figure 1: Accuracy versus runtime trade-off on the ImageNet-C benchmark using a ViT-B model, across various adaptation stopping thresholds $\epsilon$ (star marks the default setting $\epsilon\!=\!0.045$ ). The horizontal dotted line represents the NoAdapt baseline accuracy. Existing BP-free methods typically face a trade-off: they either achieve high accuracy at the expense of computational efficiency or reduce runtime by sacrificing precision. Our approach not only outperforms current baselines but also introduces a tunable mechanism to balance the accuracy-runtime trade-off, preventing the inefficient use of resources for diminishing returns.

Figure 2: Observed performance gap . A comparison of updating the affine parameters of normalization layers ( Norm .) versus three input prompts ( Prompt ) for a ViT-B model during test-time adaptation on ImageNet-C . We adapt these using ground-truth labels with SGD optimizer with varying learning rate. Updating the normalization layers allows the model to more effectively ‘correct’ the covariate shift at each network depth for all reported learning rate values.

Figure 3: Intrinsic dimensionality of continual TTA gradients . The affine parameters of normalization layers from a ViT-B model are optimized via SGD with the loss function from FOA [ 25 ] . The concatenated gradients from all ImageNet-C domains reveal that only

566

components explain

90\text{\,}\mathrm{\%}

of the variance. This highlights the low-dimensional nature of the adaptation space. Note that only the most significant components are shown for clarity. The analysis is based on

11\,729

gradient batches.

Figure 4: Marginal accuracy gain per unit time across adaptation budgets on ImageNet-C with ViT-B model. For each consecutive pair of adaptation step budgets, we compute the increase in mean accuracy across domains and divide it by the additional estimated wall-clock time required for that interval using our Subspace Adaptation with CMA-ES algorithm. The mean of CMA-ES distribution

\bm{m}

is used as the evaluated model update. Higher bars indicate more efficient use of adaptation time, while decreasing bars indicate diminishing returns from further adaptation.

Figure 5: Diagram of PACE . 1) Subspace Adaptation : we adapt the model by adding a high-dimensional random projection of a small, learnable vector to the model’s normalization layer weights. We use the CMA-ES strategy to iteratively evolve a population of these vectors, selecting the one that minimizes the loss on current test samples. 2) Adaptation Stopping : for efficiency, we stop the adaptation when the mean of the distribution optimized by CMA-ES is lower than a threshold. Along with the Domain-Specialized Vector Bank , they make an effective and efficient TTA system.

		ViT-B			DeiT-B
Method	Backprop.	IN-C	IN-R	DN-126	IN-C	IN-R	DN-126	Avg.
NoAdapt	✗	55.5	59.5	53.1	51.6	52.8	60.2	55.5
TENT	✔	61.7	63.9	53.8	55.5	56.1	60.4	58.6
CoTTA	✔	58.4	63.5	62.0	55.4	53.0	60.4	58.8
SAR	✔	61.5	63.3	53.8	59.4	57.4	60.7	59.4
LAME	✗	54.1	59.0	51.6	50.9	52.5	58.9	54.5

Table 1: Accuracy on ImageNet-C (IN-C), ImageNet-R (IN-R) and DomainNet-126 (DN-126) with ViT-B and DeiT-B. Bold indicates best result, underlined second best.

		Noise			Blur				Weather				Digital
Model	Method	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Avg.
8-bit	NoAdapt	55.8	55.8	56.5	46.7	34.7	52.1	42.5	60.8	61.4	66.7	76.9	24.6	44.7	65.8	66.7	54.1
	T3A	55.6	55.6	55.9	45.9	34.7	51.8	42.0	59.7	62.0	65.4	76.4	48.5	43.0	65.3	67.7	55.6
	FOA	60.7	61.4	61.3	57.2	51.5	59.4	51.3	68.0	67.3	72.4	80.3	63.2	57.0	72.0	69.8	63.5
	ZOA	57.9	60.6	61.4	51.2	44.6	56.6	51.8	64.6	61.8	63.6	78.1	54.5	54.4	68.7	68.3	59.7
	ZOA ( $\text{fp}\!\!=\!\!28$ )	60.4	62.6	63.3	56.8	53.6	62.1	58.5	67.2	67.0	70.4	79.4	60.8	63.7	72.2	72.2	64.7

Table 2: Corruption Accuracy (%) on ImageNet-C with Quantized ViT-B models.

Method	Backprop.	#FP	#BP	Avg. Acc. (%)	Runtime (hours)	Memory (MB)	Adapt. Batches (%)
NoAdapt	✗	1	0	55.5	0.01	819	0
TENT	✔	1	1	61.7	0.03	5,165	100
SAR	✔	[1, 2]	[0, 2]	61.5	1.1	5,166	100
CoTTA	✔	3 or 35	1	58.4	1.5	16,836	100
T3A	✗	1	0	56.9	0.7	957	100
FOA	✗	28	0	65.0	11.6	832	100

Table 3: Computation complexity comparison on ImageNet-C with ViT-B . Forward and backward passes ( #FP / #BP ) are counted for processing a single sample. The wall-clock time (hours) and memory usage (MB) are measured for processing the whole ImageNet-C on a single RTX 4090 GPU . Adapted batches are indicated by the percentage on which adaptation was performed.

Method	adapt. stopping	vector bank	subspace adapt.	Avg. Acc.	Runtime (hours)
NoAdapt				55.5	0.01
Baseline (FOA)				65.0	11.6
PACE v1			✔	66.1	10.3
PACE v2		✔	✔	67.2	10.3
PACE v3	✔		✔	66.3	5.4
PACE	✔	✔	✔	67.0	5.4

Table 4: Ablation study on each component of PACE on ImageNet-C with ViT-B .

Method	round 1	round 2	round 3	round 4	round 5
NoAdapt	55.5	55.5	55.5	55.5	55.5
T3A	55.4	55.9	55.2	55.0	54.6
FOA	65.0	65.6	66.1	66.2	66.4
ZOA	61.5	63.0	63.2	63.9	64.0
ZOA ( $\text{fp}\!=\!28$ )	66.3	67.4	67.3	67.8	68.0
PACE (ours)	67.0	67.9	67.8	68.1	67.8

Table 5: Comparisons with state-of-the-art methods on ImageNet-C with ViT-B in long-term continual adaptation. We report average accuracy at each round of continual adaptation. Bold indicates best result, underlined second.

Figure 6: Smoothed per-batch accuracy for the repeated Glass Blur domain from ImageNet-C with ViT-B model. The throughput was measured on a Jetson Xavier NX . The gray vertical lines indicate the start of each repetition. The throughput from ZOA (

\text{fp}\!=\!28

) is covered by FOA.

	Fastfood (Def.)	Dense	PCA
PACE (ours)	67.0	67.0	64.8

Table 6: Accuracy (%) of our method on ImageNet-C, evaluated across different subspace projection methods using a ViT-B model.

	300	768	2304	3000
PACE (ours)	63.8	65.4	67.0	66.5

Table A.1 : Accuracy (%) of PACE on ImageNet-C with a ViT-B model.

Panel of Figure 14 — Figure A.1 : 1D curvature profiles for Fastfood ( Left panels), Dense ( Middle panels), and PCA ( Right panels) subspace projection methods evaluated on a ViT-B model across different ImageNet-C domains using the representative batches.

Figure A.2 : Distribution of 1D curvature estimates for Fastfood , Dense , and PCA subspace projection methods across all of the ImageNet-C domains and chosen axes on a ViT-B model. Horizontal markers indicate the mean and median.

Figure A.3 : Smoothed per-batch accuracy for the ImageNet-C benchmark with ViT-B model. The throughput was measured on Jetson Xavier NX. The gray horizontal lines indicate the start of the domain.

		Noise			Blur				Weather				Digital
Method	BP	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Avg.
NoAdapt	✗	56.8	56.8	57.5	46.9	35.6	53.1	44.8	62.2	62.5	65.7	77.7	32.6	46.0	67.0	67.6	55.5
TENT	✔	57.6	59.8	60.9	51.2	49.4	59.6	53.2	64.0	62.7	67.8	78.6	66.5	54.5	70.0	69.7	61.7
CoTTA	✔	57.4	58.4	59.7	47.5	38.3	54.9	47.3	62.4	63.4	69.9	77.8	54.3	47.8	68.0	68.6	58.4
SAR	✔	59.1	61.1	61.6	54.2	55.1	58.6	55.7	60.3	61.5	64.3	76.6	58.2	58.1	68.6	68.9	61.5
LAME	✗	56.5	56.5	57.2	46.4	34.7	52.7	44.2	58.4	61.6	63.1	77.4	24.7	44.6	66.6	67.2	54.1

Table A.2 : Comparisons with SOTA methods on ImageNet-C with ViT-B regarding Accuracy (%) . BP is short for backward propagation.

Subspace Optimization

See all 12 papers

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions
Jul 9, 2026Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang +6Source-Free Domain Adaptation Distribution Shift Robustness
The University of Texas at Dallas, USA · LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS), McGill - ÉTS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec · Tulane University, USA +2
FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptation
Mar 5, 2026Xingyu Wang, Tao WangTest-Time Adaptation Continual Test-Time Adaptation
College of Computer Science, Sichuan University
Source-Free Controlled Adaptation of Teachers for Continual Test-Time Adaptation
Jul 26, 2026Anurag Roy, Riddhiman Moulick, Vinay Kumar Verma +2Source-Free Domain Adaptation Domain Adaptation
IIT Kharagpur · IBM Research · Amazon +1

Jul 9, 2026J/K move · Enter open · S save

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang, Shuaicheng Niu+5

The University of Texas at Dallas, USA · LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS), McGill - ÉTS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec · Tulane University, USA+2

Source-Free Domain Adaptation Distribution Shift Robustness Test-Time Adaptation Continual Test-Time Adaptation

Open paper AskPDF arXiv

Deep neural nets achieve remarkable performance when training and test data share the same distribution, but this assumption frequently breaks in real-world deployment, where data undergoes continual distributional shifts. Continual Test-Time Adaptation (CTTA) addresses this challenge by adapting pretrained models to non-stationary target distributions on-the-fly, without access to source data or labeled targets, while mitigating two critical failure modes: catastrophic forgetting of source knowledge and error accumulation from noisy pseudo-labels over extended time horizons. In this comprehensive survey, we formally define the CTTA problem, analyze the diverse continual domain shift patterns that characterize different evaluation protocols, and propose a hierarchical taxonomy that categorizes existing methods into three families: optimization-based strategies (entropy minimization, pseudo-labeling, parameter restoration), parameter-efficient methods (normalization layer adaptation, adaptive parameter selection), and architecture-based approaches (teacher-student frameworks, adapters, visual prompting, masked modeling). We systematically review representative methods within each category and present comparative benchmarks and experimental results across standard evaluation settings. Finally, we discuss the limitations of current approaches and highlight emerging research directions, including the adaptation of foundation models and black-box systems, thereby providing a roadmap for future research in robust continual test-time adaptation.

Abstract

Subspace Optimization

Explore similar work

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptation

Source-Free Controlled Adaptation of Teachers for Continual Test-Time Adaptation